# A Decision Model That Walks the Parse Tree

> Sorting data item mentions into data items is an agent task: look at a candidate group, read around it, decide. A frontier coding model does it well and costs too much to run on every scan. We built the same loop around Jev, gave it parse-tree tools instead of a file viewer, and it landed above the…

By Ryan Porter · Published 2026-10-08

Canonical: https://dataparade.io/posts/a-decision-model-that-walks-the-parse-tree/

Once the deterministic layer of the DataParade scanner has grouped what the parse tree can prove, there is a pile of leftovers: mentions of personal data that might belong to an existing data item, might be a new one, or might be noise. Someone has to look at each one and decide. In our first version that someone was a frontier coding model running an agent loop with a file read tool. The harness showed it one candidate group at a time, it poked around the codebase, and it sorted the group into buckets.

It worked. Using the mid-sized frontier model, the data item groups on our test repositories came out clean enough that I stopped and said "wow" in Slack, which is not something I do for most results. The small frontier model worked too, not as well. And then the problem: a mid-sized frontier model and a free tier are not compatible. We want people to run this scanner on every pull request, and some of them will never pay us.

The short version: we rebuilt the same agent loop around Jev, a decision model, and gave it tools that expose the parse tree rather than raw file text. On the grouping task it now beats the small frontier model and trails the mid-sized one. On one repository it beats the mid-sized one. And because Jev prices a request by its input, not by the number of questions you ask, we collapsed three calls into one and started asking a lot more questions.

## Why a decision model and not a smaller LLM

Jev does not generate text. You hand it some context and a set of typed questions, and it returns an answer to each with a confidence. For this task that is the right shape. The decision at each step is bounded: does this mention belong with that group, yes or no? Is this declaration a new data item? Which of these three candidates is the best match? None of that needs prose.

It also means the loop is cheap in a way that matters for how you design it. The first prototype ran three Jev calls per step with three different questions over the same inputs. Then I remembered that the number of questions in a Jev request does not change the price of a prediction. One question or twenty-five, same cost. So three requests became one request, and that one request now asks many more questions, which gives the harness many more signals about what the model thinks it is looking at.

## Tools that look like an LSP

The frontier-model version used a file read tool, because frontier models are trained on raw source and are good at reading it. Jev is not a code model. So instead of showing it files, the harness exposes the parse tree: the declaration for this identifier, the type of that receiver, the call sites of this function, the parameters it binds. It is something like a language server protocol, with Jev as the client. The model walks the tree one step at a time and the harness records what it decided at each step and why.

The harness still has a file view tool. Jev mostly does not reach for it.

## Results on the grouping task

We score grouping pairwise against a labeled corpus (there is a separate article on how that corpus was built). On the two test repositories we use most, as of early October:

| Enhancement layer | Grouping result | Cost |
|:---|:---|:---|
| **Parse tree only, no model** | About a third of expected groups found | Free |
| **Small frontier model, file tools** | Better than parse tree only | Low |
| **Jev, parse-tree tools** | Above small frontier model, below mid-sized; above mid-sized on one repo | Very low |
| **Mid-sized frontier model, file tools** | Best overall | Too high for every PR |

Two caveats. The parse-tree-only number is a floor that keeps moving up, since every time we find a grouping rule the parser can apply deterministically, we move it down into that layer and the models have less to do. And the mid-sized frontier model's lead is on recall more than precision. The Jev pipeline leads on precision for both repositories and trails on recall for email items, which is where the cross-file import gap in the deterministic layer bites hardest.

## Routing by confidence

Since Jev returns a confidence with each answer, there is an obvious next step: accept the confident decisions, send the rest to a bigger model. We built that too, as an LLM enhancement layer that takes the parse-tree information plus Jev's trail of decisions and tries to improve both recall and precision. It exists. Whether it works is not yet proven, and I am saying that in the same breath so nobody reads this and thinks it is done.

The experiment we are running now sends the least-confident 10% of data items to the LLM layer, then 20%, then 30%, and measures what changes. The thing to find out is where the curve flattens, because that is the number that sets the cloud price. On the same test repositories, a newer, cheaper frontier model did better on this step than the coding model we started with, at about 18 times less on input and 28 times less on output.

One detail that only matters if you build this: Jev's confidence is not a probability until you calibrate it against your own labels. We wrote about that for sentiment at Anthus. The labeled grouping corpus gives us the same calibration set for this task, which is a nice side effect of building it.

## What we haven't tested

* **Other decision models.** Kev and Laya run in the same harness and we have not compared them on this task yet.
* **A local decision model.** The whole point of the open-core idea is that you could run the decision layer with your own key or a model on your own machine. We have not tried the second half.
* **Stability.** Asking Jev the same question twice in one request does not always give the same answer, and averaging an ensemble of copies makes the answer more repeatable without making it more accurate. We have not measured how much the grouping output changes between two runs on identical code, and for a tool that is supposed to diff scans, that matters.

## Lessons

1. **Shape the task to the model.** A frontier model reads files. A decision model answers bounded questions. Give each the tool it was built for and the small one gets a lot closer than you expect.
2. **Free questions change your design.** When extra questions cost nothing, ask for every signal you can think of and let the harness sort them.
3. **Prove the expensive path first.** We knew the loop could work because the mid-sized frontier model had already done it. Swapping in Jev was an optimization with a known target, not a hope.

The harness and the eval runs are in the scanner repository.