We Labeled Six Repos with a Thousand Sub-Agents

When I started on the DataParade scanner, the question I was trying to answer was simple: what does it need to discover, and how do we know it did? That turned into four layers of things to find in a codebase: mentions of personal data, the data items those mentions belong to, the components that handle them, and the flows between components. We score each layer against ground truth, and we deliberately do not roll the four up into one number. A scanner that is great at finding mentions and terrible at grouping them is a specific kind of broken, and an overall score would hide which kind.

The problem with scoring against ground truth is that you need ground truth. There is no public corpus that says, for a real web application, here are the 340 places an email address appears and here are the 11 data items they belong to. We needed one for six repositories, and we needed it to be good enough to steer by.

The short version: we built the corpus with a swarm of coding-agent sub-agents. Small models did the first pass, a mid-sized model did a second pass, and the largest model arbitrated where they disagreed. It ran overnight on an ordinary subscription account and moved the weekly quota by about 1%. The labels have errors. They also immediately found errors in the labels we had made by hand.

The job

For each repository, the label set has to say, for every mention the scanner could plausibly find: is this personal data, which data item does it belong to, and which other mentions belong with it? That last one is the expensive part. We score grouping pairwise, so the corpus needs to be right about pairs, and the number of pairs grows with the square of the mentions.

Six repositories, chosen for variety: an e-commerce platform, a CMS, a headless commerce backend, a customer support application, and two smaller ones we use for quick checks.

Three sizes of model, one judge

The swarm was a coding agent orchestrating sub-agents, which is a thing you can do now from a laptop. The shape:

1. First pass, small model. Hundreds of sub-agents, each handed one file or one candidate group, asked to label it. Cheap enough that the number of agents is not a constraint. You can spin up a full Sorcerer's Apprentice situation and nothing terrible happens.
2. Second pass, mid-sized model. Same inputs, independent labels. Costs more, not much.
3. Arbitration, largest model. Only where the first two disagree. It sees both labels, the code, and the reasoning each pass recorded, and it decides.

Over a weekend that came to more than a thousand sub-agents. The largest model was the final judge on a small fraction of items, which is what kept it affordable.

The labels found our mistakes first

Before the swarm, we had hand-labeled fixtures for the four layers. The first useful thing the swarm did was disagree with them. In a handful of cases the hand label was wrong: a mention we had marked as one data item was two, or a test fixture we had counted as personal data was a placeholder. We fixed the fixtures. The corpus got better before the scanner did.

That is the right order. If your evaluation is wrong, every improvement you make to the thing it evaluates is suspect.

What the compass says today

With the corpus in place, the deterministic scanner alone finds about a third of the expected data item groups across the two repositories we use for daily work. A week before it was under a quarter. Adding a decision model on top takes that materially higher (that is its own article). The number I watch is not the recall figure, it is the direction it moved since yesterday and which repository moved it.

The pairwise scorer is published from the scanner package so that the CLI and our evaluation tooling import the same one. There is no local copy anywhere. If the scorer changes, every historical run is rescored, not reinterpreted.

The labels will never be perfect

Three models voting is not ground truth. It is three opinions and a tiebreak. There are data items in those six repositories that all three passes got wrong in the same way, and we will not find those until a person reads the code or the scanner disagrees loudly enough to make us look. Our own evaluations will never reach 100% against this corpus, and if they did I would assume the scanner had learned the corpus rather than the task.

That is fine. The corpus does not have to be true. It has to point the same direction as the truth, consistently, so that when a change moves the number we know which way we went.

What we haven't tested

• Agreement rates. We have not published how often the first two passes agreed, or how often the arbiter sided with each. That is the number that would tell a reader how much to trust the corpus, and we should compute it.
• Human spot checks. A sample of a few hundred pairs, read by a person, would give us an error rate for the corpus itself. We have not done it yet.
• A design partner's repository. The corpus is public code. The point of all this is to take one partner's project and comb through it until the scanner finds what it needs to find there. That is the next run.

Lessons

1. Build the ruler before the thing you are measuring. The first week with a real corpus taught us more about the scanner than the month before it.
2. Disagreement is the signal. Three passes that agree are cheap confirmation. The items where they split are where the hard cases live, and that is where the expensive model earns its cost.
3. Don't average the layers. Four numbers that each mean something beat one number that means nothing.

The fixtures, the scorer and the swarm scripts are in the scanner repository's eval directory.