PR review is not a probability

Note: this follows on from part one and part two. It is shorter than both, because the thing it describes did not work. I am posting it anyway, for reasons that are most of the point.

Part two ended in a specific place. Last Light’s PR review pipeline had stopped being bad at finding defects and started being bad at mentioning them. Eighteen of twenty five real defects generated somewhere inside the machine; four or five actually posted on the pull request. I called that the saying half, and I said it was where the next work was.

There is an obvious suspect, and I have been circling it for a month. Every finding the pipeline produces carries a confidence field, and the attention boundary - the code that decides whether you see something inline, in the review body, or not at all - is gated on that number. It has six per-family thresholds and a floor. It is a genuinely nice piece of machinery.

It has never done anything, because the number does not move. Median confidence across findings sits between 0.95 and 1.00. A model asked to rate its own certainty says very to nearly everything, and a threshold applied to a constant is decoration. I have a boundary I cannot tune because I have no axis to tune it on.

So when TypeSafe crossed my desk, I paid attention for exactly one reason.

A different kind of number

TypeSafe’s model, Jev, is what they call a System One model. It does not generate text. You hand it some state and a set of typed questions - yes/no, pick-one, score-against-a-rubric - and it hands back probabilities. That is the entire surface. You cannot ask it to write a review comment; you can only ask it things that have shaped answers.

The pitch that mattered to me is that those probabilities are calibrated, and the economics are absurd: $42 per billion input tokens, output free, most calls under a second. Three orders of magnitude below the chat models I have been paying for.

Put those together and you get a very specific hypothesis. My adjudicator’s confidence is decorative because a text model rating itself is not really doing measurement. A model trained to emit a calibrated probability and nothing else might give me the axis the boundary has been waiting for. If that worked, the payoff is not a cheaper reviewer. It is a reviewer whose saying half can finally be tuned.

That is the thing I wanted to be true. This post is about finding out that it is not.

The cheapest possible way to be wrong

Here is where four months of building an evals harness paid for itself in a way I did not plan.

Back in August I wrote a measurement script called aacr-adjudicate. It scores an adjudicator - hand it a review comment somebody else wrote, can it tell a correct one from an incorrect one - against AACR-Bench, 2,145 review comments from real pull requests, every one verified by a human expert as correct or not. I wrote it deliberately before building the adjudicator it was meant to grade, so that the adjudicator would have an external target rather than being marked by the instrument I designed alongside it.

That script has a tiny interface. An arm takes a row, returns keep-or-drop and optionally a confidence, and that is all:

interface Arm {
  name: string;
  usesModel: boolean;
  describe(): string;
  decide(row: Row): Promise<Decision>;
  estimateTokens?(rows: Row[]): { input: number; output: number };
}

It ships with two deterministic floors and one model arm. The floors are the important part. keep-all posts every comment it is handed, which is what Last Light does today, because there is no filtering adjudicator in production. drop-all is the opposite wall. Any real arm has to land between them and beat keep-all on F1, or it is trading recall for nothing.

It also has a threshold sweep that re-scores every row at eleven cut-offs from 0.0 to 1.0. I built that to check a confidence axis is monotone - t=0.0 must reproduce keep-all and the top must approach drop-all, so the two floors bracket the curve by construction. In the months since I wrote it I have never had a confidence worth sweeping.

So the whole experiment was: add a fourth arm, about 150 lines, touching zero production code. No new phase type, no workflow change, nothing that could reach a real pull request. Build the question, point it at 2,145 rows somebody else labelled, read the sweep.

Total cost of finding out, including the two runs I got wrong: under ten cents, and about an hour.

The first answer was mine, not the model’s

I asked Jev four yes/no questions per comment, in a single request - they evaluate in parallel, so the extra three are nearly free. The one that drove the decision was this:

The review comment identifies a real defect in the code it points at.

I ran fifty rows. The headline was bad, but the category breakdown was interesting enough that I stopped reading the headline:

Jev was throwing away nine out of ten valid maintainability comments. Which is not a calibration failure. It is me writing a bad question. AACR’s label means this comment is correct, and a reviewer saying “this function is doing three things and should be split” is correct. I had asked about defects, and the model had answered about defects, precisely, and then I had graded it against a label that means something broader.

This is documented behaviour, and I should have read it first. TypeSafe publish a page of known jagged edges for the model, and the top entry is that it “answers the question you wrote, not the one you meant”. Duly noted.

So I added a second question that actually matches the label - a maintainer of this project would agree this comment is correct and worth acting on, with maintainability spelled out as qualifying - and made the decision axis switchable so one run could measure both.

The full result

Then I ran all 2,145 rows on both axes. Thirty four seconds a pass, about four cents each.

And the sweep, which is the bit that actually settles it. Here is the wider axis across all eleven thresholds:

The axis is monotone. t=0.0 reproduces keep-all exactly and the top reproduces drop-all, so it brackets, so it is a real axis and not noise. After a month of a confidence field pinned at 0.97 that was almost a relief to see.

And it is useless. Precision crawls from 70.2% to 76.3% - six points - while retention falls off a cliff from 100% to 10%. No threshold on either axis beats keeping everything. The best row in the whole table is the one where you do nothing.

The single number that says it cleanest is the one I computed afterwards. As a classifier for the thing AACR actually labels, P(correct) scores an AUC of 0.534. The other axis manages 0.544. A coin lands on 0.500.

The human-authored rows - 548 of the 2,145, and the honest half, since three quarters of that corpus is machine-written comments that experts then verified - are worse on both axes than the AI-authored ones. If anything the pooled figure is flattering it.

That is a clean negative. The lever I have been planning for a month, that I was fairly confident about, does not exist.

The number I did not expect

Except.

I went back to the category breakdown at full scale, on the narrow “is this a defect” question, and it is not a weak classifier at all:

As a detector for “is this comment about a defect or is it about tidiness”, P(correct) scores an AUC of 0.897.

So the same probabilities that are a coin flip against one question are a sharp instrument against another. The model was never miscalibrated. It answered the question I wrote with high fidelity, and I spent two runs grading it against a different one. The failure is entirely in the join between my question and the benchmark’s label, and only the breakdown tables made that visible - the headline F1 would have let me write “the model is poorly calibrated on code review”, which is false, and which I would have believed.

There is one more slice that stops me getting carried away. Restrict to the 1,022 Code Defect rows only, where the category confusion cannot apply, and ask whether P(correct) separates the correct comments from the incorrect ones. AUC 0.569. Still basically nothing.

Which I think is the actual finding, and it is not really about Jev. Every arm in this harness sees the comment text and its file and line number. None of them see the code. Deciding whether a review comment is right without reading what it points at may just be close to impossible - and the fact that no adjudicator anybody has run through this script has ever beaten keep-all (Haiku 0.803, GLM 0.745, now Jev at 0.789) is more consistent with the task being the problem than with three unrelated models all being bad at it.

I have been treating “beat keep-all” as a bar my adjudicator kept failing to clear. It is looking more like a wall.

What I am not going to claim

The tempting move here is obvious, and I want to name it rather than make it.

Last Light’s boundary does not need to know whether a comment is correct. It needs to know whether a comment is worth interrupting somebody with. And a classifier that keeps 87% of defects while dropping 81% of tidiness notes, for four hundredths of a cent, is aimed almost exactly at that.

That is a hypothesis this probe produced. It is not a result, and I have not measured it, and this project has a long history of exactly that kind of reasoning being wrong in a way that costs two weeks. Separating defects from nits is a much easier problem than the one my adjudicator is failing at - it files real, confirmed defects as internal because of the grammar they are written in, and no amount of category sorting fixes that. So it goes on the list, behind the things that have evidence.

What the probe does settle: Jev as a blind filter on review comments is dead. I am not building it, and the ten cents is the whole cost of knowing.

What it leaves alone is narrower than I expected. Jev with real evidence in front of it - the diff, the hypothesis, the probe transcript - is a different question that a blind benchmark structurally cannot answer. And the boring use is untouched: Last Light’s re-review triage currently spins up a whole sandboxed Haiku session, with a workspace and a database row, to emit one line that says REVIEW_DEPTH: full or REVIEW_DEPTH: light. That is a pick-one question wearing a very expensive coat. Swapping it does not need this result to be true, because its bar is “match a one-line decision”, not “beat keep-all”.

The bit that generalises

I have written three posts now about making an agent workflow measurable, and this is the first one where the measuring was the entire deliverable. Nothing shipped. The post is a hole where a feature was going to go.

The thing I would take from it is not about probabilities. It is that the cost of being wrong is a property of your harness, not of your judgement. I did not avoid this mistake. I had the hypothesis, I liked it, I had already sketched where the new phase type would slot into the workflow engine and which handler it would mirror. On a worse day, or in a repository without that script sitting in it, I would have built the phase, wired the key through, run it against my own eight pull requests, got a number inside the noise band, and argued with myself about it for a fortnight.

What stopped that was a 150-line arm dropped into an interface that already existed, pointed at a corpus somebody else labelled, with two deterministic floors already pinned so I could not mistake a plausible number for a good one. That script was written a month before the feature it was built to grade, for a reason I only half understood at the time, and its whole value turned out to be killing something.

If you take one thing from this: write the measuring stick before the thing it measures, and make sure it has a floor in it. keep-all is not a strawman arm I added for completeness. It is the reason I know a 0.789 is a failure rather than a decent first attempt. Without that row in the table, 0.789 with a monotone axis is a very encouraging result, and I would be three days into building on it right now.

And keep your negative results cheap enough that you are willing to have them. An afternoon and ten cents is a price at which I will happily test ideas I believe in. Two weeks and a pipeline change is a price at which I start wanting to be right, and wanting to be right is how you end up with a number you tuned instead of measured.

The saying half is still where the work is. It just is not going to be fixed by a better number.

This follows PR review is not a prompt and part two. Earlier posts in this thread: The harness is the product, What is harness engineering, and Meet agentic-pi. All of the numbers here came out of the Last Light evals harness; the AACR-Bench corpus is Alibaba-Aone/aacr-bench, Apache-2.0.