Detecting duplicate agent plans with Jaccard distance

To check whether the plans an agent proposes are really different or duplicates under new titles, encode each plan's operation type, roles, relationship edges, and stages as a set of markers, compute the Jaccard distance between every pair, and judge the batch by its closest pair. A plan's title and summary stay out of the sets, so rewriting them does not change its distance to the other plans. This article explains that check as we implemented it in research code, with a worked example that uses made-up teaching numbers. The tests show the code does what it is meant to do. They do not show that readers choose better because of it.

Zhang Chenxi (Nature), who builds agent systems for manufacturers and distributors. 18 October 2026. Drafted with AI assistance.

The problem: three titles, one plan

An agent that proposes several collaboration plans for one request is supposed to give people different options. Consider three plans: "arrange a skill-sharing session," "organize an experience exchange," and "hold a knowledge workshop." A response can give each its own tone, audience, goal, and expected benefit. If all three have the same person teaching the same group the same content in the same time slot, the person actually has one option.

Our collaboration-discovery research uses a structural diversity check after plan generation to catch this. The check asks whether the plans in a batch are different arrangements or the same arrangement described three ways.

What the check compares

Jaccard distance is an existing method for comparing two sets, and we did not invent it. The standard definition and a reference implementation are in the SciPy documentation for the Jaccard distance. Our choices are about what goes into each set and how a batch is judged.

The research code reads four things from each plan: the operation type, the role each participant plays, the directed and typed relationship edges between roles, and the stage numbers together with each stage's action. Our generator is asked to return each plan as a JSON record that carries these fields, plus a separate title and a one-paragraph summary, so the structure does not have to be parsed out of prose. The function structural_tokens turns these into one set of structural markers per plan. The title and summary are not read. A second function, jaccard_distance, compares two such sets, and evaluate_batch applies the batch rule described below.

Each marker is a string in one of four shapes:

operator:<operation type>
role:<participant>:<role>
edge:<from role>:<relationship type>:<to role>
stage:<stage number>:<action>

A plan's set is all of these strings, and two plans share a marker only when the whole string matches.

Because the title and summary never reach the marker set, two plans that differ only in those fields have a distance of 0. The role names and action labels inside the structure are still words, which causes the naming problem described below.

A worked example: repairing versus teaching

The numbers and markers in this example are made up for teaching, not data from a real event. We recomputed them with the actual function.

A community wants to help residents repair old household items. Plan A is "have a repairer fix the items directly." Plan B is "have a repairer teach residents to fix them on the spot." The two texts mention the same repairer, residents, and repairing, so a text-similarity check, such as cosine similarity between embeddings of the two plan texts, could rate them as near-duplicates. The relationships differ: in A the repairer delivers a repaired item, and in B the repairer delivers teaching while residents do the work.

Write Plan A as four structural markers. Plan B keeps the two role markers and changes the other two.

Each set has four markers and the sets share two. Their union has six markers. The Jaccard similarity is 2/6, and the distance is 1 − 2/6, about 0.667.

Shared:           repairer is provider, resident is receiver
Only in Plan A:   direct service, delivers repair
Only in Plan B:   training, delivers teaching

distance = markers that differ / all distinct markers = 4/6

Now rename Plan A from "Old-item repair day" to "Community renewal campaign" and write the summary with more enthusiasm, leaving the four structural markers as they are. The set is unchanged, so the distance to its old self is 0.

Duplication that a reader would otherwise have to sense from the text becomes a list of structural differences that anyone can inspect.

Judging a batch by its closest pair

When the system hands over a batch, the algorithm compares every pair. It looks at the smallest distance in the batch and ignores the average. Three teaching plans show why: two of them are structurally identical, and the third differs a lot from both. An average over the three pairs lets the third plan's large distances dilute the duplicate. The minimum keeps the 0 between the identical pair, so the duplicate stays visible.

For this round, a batch passes only if it has at least three candidates, its closest pair is at least 0.4 apart, and no two plans have identical structural signatures (the same full set of markers). The configuration file that holds these values carries a flag saying it was frozen before generation, and evaluate_batch refuses to run without that flag. The flag does not prove the timing by itself. Here, version control does: the commit that added the file is about two hours older than the first generation call. This threshold is a specific setting for this round. It is not a natural boundary of creativity. Raising it can turn a passing batch into a failing one without any change in the plans. Whether a higher bar produces more valuable plans cannot be derived from the mathematics.

Comparing every pair is straightforward for a batch of a few dozen plans. For n plans there are n(n−1)/2 pairs, so 10 plans have 45 pairs and 100 plans have 4,950. With a very large candidate pool, grouping first and then comparing within groups could cost less. For small batches, adding an index only to make the method look more advanced is unnecessary.

Where the representation breaks

The most important limits come from the markers themselves.

Suppose one plan says "repairer" and another says "maintenance worker" for the same role. If nothing upstream unifies the names, the sets treat them as different. In the other direction, two plans may have identical roles and identical exchange types while one takes a week and the other three months. The structural markers in this version do not include duration, so an important difference disappears. Quantity, price, and many other concrete conditions are also outside this distance.

One of our research runs shows the first problem is not a corner case. In its 15 batches, the model wrote its own role names, relationship types, and stage actions as free text. In one batch, the same community sensor network is community_measurement_source in the first plan, community_sensor_network in the second, and measurement_origin in the third, and every first step is a 15- or 16-word instruction. No marker was shared between any two plans within a batch, so every batch scored the maximum distance of 1.0 and passed. If you use this check, limit roles, relationship types, and stage actions to a fixed vocabulary, the way our generator prompt lists five allowed operation types, and reject or map any label outside the list before computing distances. Our research code does not do that rejection yet. Otherwise, free-text labels can push every pair to 1.

A distance of 0 therefore means "identical on these structural fields," and a distance of 1 means "no encoded markers overlap." Neither is a final judgment about the real plans. A distance of 1 does not mean the two plans are completely different, and a larger distance does not make one plan more creative.

A generator that learns the threshold could also change role names or operation labels to raise the distance while the actual arrangement stays the same. This is the same problem as varying titles to fake diversity, moved into the structural fields. The remedy is to align the markers with the concrete, executable plan, and where needed to check role synonyms, real responsibilities, and changes in resources. Raising the distance threshold further does not repair a flawed representation.

What the tests show

In the existing test setup, three plans with different wording and identical structure score a distance of 0 and fail the batch, and plans that differ in operations and role relationships pass the distance bar. That shows the code implements the intended structural comparison. It does not show that readers made better choices as a result.

What the check does offer is a more specific way to ask for different plans. Who does the work, who learns, what is exchanged, how the order of steps changes: once those differences can be stated, people can discuss the cost and meaning of each plan and stop guessing among three titles that all sound reasonable.

Where the check fits in a generation loop

In our research code the check sits inside the generation loop. If a batch fails this check or the separate grounding check (which confirms that every capability a plan cites matches a recorded source), the next prompt carries the full distance report and asks for different role and mechanism structures; after three failed attempts the run stops with an error. That feedback is also where the relabeling risk comes from: the generator sees its distances and can learn to rename instead of rearrange. In the run described above, every first attempt scored 1.0 before any feedback: 14 of the 15 batches passed on the first try, and the one that needed a second try had failed the separate grounding check, not the distance check. So the 1.0 scores came from free-text labels, not from the generator gaming the report.

Whether it runs inside an automated loop like ours or once by hand, the comparison comes after plan generation: let the agent propose several possibilities first, then lay out the actual structure and see which plans add a real choice. The distance is a limited but clear measure, and the final decision still rests on a person's goals, conditions, and the work they are willing to take on.

A manual check before you accept a batch

Before you accept a batch of plans, list for each one who does what, what is exchanged, and in what order. If two lists match, you have one plan with two titles.

The structural check in this article comes out of ToWow's research on agents that coordinate work between people. If you work on agent planning and want to compare notes, write to hi@towow.ai.

An earlier Chinese version of this article is at towow.net/articles/towow-jaccard-plan-diversity.

If you want a system like this built around your own workflow, see AI agent systems for manufacturers and distributors or write to hi@towow.ai.