PR review is not a prompt

Note: this ends with the work held rather than shipped, and I have kept the wrong turns in, because they were more useful than the destination. Feedback very welcome.

Last Light has had a PR review workflow since almost the beginning. It is one of the first things you build once you have a harness that can watch a repository: a webhook fires on pull_request, you hand a coding agent the diff and a skill that tells it what a good review looks like, it writes some findings to a JSON file, and a deterministic phase posts them back to GitHub as a formal review.

Here it is on a real pull request. It is fluent, it is specific, it uses the right vocabulary, and it approved with no inline comments at all. What you cannot tell by looking at it is whether it read the code or wrote a confident summary of the description.

This did not feel like a hard problem. Anthropic ship a code review skill. Matt Pocock has written about wiring one up. Half the agentic engineering community has a code-review/SKILL.md in a repo somewhere, and they all look roughly the same: a rubric, some severity levels, an instruction to look for the usual suspects, and a schema for the output. Mine was no different. It ran, it produced sensible looking reviews, and for a few months I was quietly pleased with it.

The reason I am writing this post is that it was not working, I did not know it was not working, and the thing that told me was not a bug report. It was a measurement.

The uncomfortable bit: it looks fine until you count

Here is the shape of the problem with reviewing anything with an LLM. A bad review and a good review look identical from the outside. Both are well written. Both are confident. Both use the right vocabulary. If the bot says “this looks good to me, nice separation of concerns” then you have no idea whether it read the code properly or whether it pattern-matched on the diff and produced a plausible noise.

And Last Light was saying that a lot. Over one representative period: 94 pr-review runs across 43 pull requests, 71% of them APPROVE, and of 59 approvals, 58 carried zero inline findings.

You can read that two ways. Either the code going through was genuinely clean, or the reviewer was not finding anything. I had been reading it the first way for months.

Building a golden dataset

The only way to settle it is to compare the bot against a human on the same code, and to do that you need gold answers: real review comments, written by a named human, on a known commit SHA.

I built one from skillspro, an internal Nearform repository with a healthy review culture. The criteria were deliberately strict. For each case I needed a PR where a human reviewer had reviewed the exact head SHA that Last Light had already approved. Not a later revision, not an earlier one. If the human reviewed a different tree, the comparison is meaningless, and this caught me out immediately: two of the four defects I originally wanted to use turned out not to exist at the commit the bot approved.

One thing to state plainly now rather than in a footnote, because every number below depends on it: a “match” is decided by a model. A judge, at temperature zero, reads what the bot posted and what the human wrote and decides which findings correspond. That is standard for this kind of benchmark and it is also a weakness I will come back to, because it turned out to be wrong in a way that flattered us for weeks.

That left me with eight cases containing 25 gold findings. In honesty it is really four pull requests, since review rounds on the same PR are correlated, with a three case blind split held out so I could tell overfitting from improvement. It is a small dataset. It is nowhere near enough to make a general claim, and I will come back to that. But it is enough to answer the only question I actually had, which was “is this thing working at all”.

The harness for it lives in lastlight-evals, which runs Last Light’s real workflows against a mocked GitHub so a full arm costs model spend and nothing else.

One of twenty five

The baseline arm cost $5.65 and produced this:

It posted nothing at all on five of the seven recall cases.

The part that bothered me most was not the number. It was what the transcripts showed. This was not a lazy agent. It ran 54 to 68 turns per review. It opened files, followed imports, wrote a genuinely good cross-file trace of what the change touched, and then concluded “no findings” and threw all of it away.

It was not failing from lack of effort. It was failing at something else, and I did not yet know what.

Three theories, all wrong

What follows is about $38 of model spend and three rounds of work, each of which killed a theory I was confident about. I am writing them out because the falsifications turned out to be worth more than anything that worked.

Theory one: it is hallucinating, so it needs verification. This is the obvious one, and it is what most of the literature was pointing at. I built a version that generated findings, ran them through a machine-checked evidence gate where every claim had to be quote-backed, and then handed the survivors to a fresh-context adjudicator.

Mechanically it worked perfectly. Every disposition became quote-backed and machine checked. Posted findings went from 2 to 8. And recall went from 1 of 25 to 2 of 25, while F1 halved, the precision canary went from 1.00 to 0.00, and it cost 2.4 times as much. I deleted the machinery.

Theory two: it is a prompt problem. Three rounds of instruction work. A 22 item checklist got acknowledged in a single sentence and skipped entirely. A 17 row ledger got honestly discharged, row by row, and still produced zero findings. The model was not disobeying. It was doing exactly what I asked, at the level of abstraction I asked it, and that level was too high to find anything.

Theory three: it needs a bigger model. This one is worth dwelling on because it is the reflex answer for everything right now. It is wrong here, and there is published evidence: Haiku 4.5 beats Sonnet 4.6 on review recall, 41.2% against 22.1%. Martian’s leaderboard shows a roughly 28 point gap attributable to scaffolding at a fixed model class. The scaffolding is the variable. The model is not.

Three dead theories later, the actual diagnosis is embarrassingly simple:

The model’s question set does not contain the human’s questions. Discovery is the ceiling, not verification.

You cannot prompt your way to a question about a file you never knew was connected to the diff. All three of my theories were about improving the answers. None of them were about improving the questions.

So how good is the state of the art anyway?

At this point I went looking for what the ceiling actually is, half expecting to find that everyone else had solved this and I had missed a memo.

I had not.

• CR-Bench (arXiv:2603.11078) puts GPT-5.2 single-shot review agents at 27.0% recall and 3.6% precision.
• c-CRAB (arXiv:2603.23448) grades a review by synthesising a test from each human comment and checking whether it goes from failing to passing after the fix, which removes the judge problem entirely. Claude Code scores 32.1%, Devin Review 24.8%, PR-Agent 23.1%.

Two things fall out of that. The first is that my 0.040 is genuinely bad and I should not feel good about it. The second is more interesting: frontier precision on this task is 3 to 5%. Every serious reviewer in the field over-generates massively and eats the noise. My baseline posted 2 findings across 8 pull requests, which I had been reading as admirable restraint. It is the exact opposite of what the field does.

There is a lovely detail in the CR-Bench numbers that made me rethink the whole approach. Adding a second pass whose only instruction is “find what the first pass missed” moves GPT-5.2 from 27.0% to 32.8% recall. It also drops signal-to-noise from 5.11 to 1.95. You buy recall with noise. That is the trade, and it is not optional.

The products are good. They are also not skills.

Here is the thing that reframed it. The commercial products in this space are meaningfully better than a bare prompt, and when you read how they work, not one of them is a skill.

CodeRabbit clones the repo into an isolated microVM, compiles the project, runs twenty to fifty linters and analysers in parallel, builds a fresh call graph per review, and then has a separate judge model score every finding against that gathered context and drop what it cannot ground. Greptile maintains a semantic graph of the whole repository before a pull request even arrives, and its TREX work spins a sandbox to actually execute the behaviour in question and attaches the logs as evidence. Cursor Bugbot, the only one publishing before-and-after numbers, went from 0.2 to 0.5 resolved bugs per pull request by moving from a fixed pipeline to an agent pulling context at runtime, across 40 experiments in which, in their words, many intuitive changes surprisingly regressed the metrics.

Compiled projects, sandboxes, call graphs, parallel analysers, separate judges, executable oracles. Not one of them is a rubric in a markdown file.

I had been comparing my skill to other people’s skills, when the thing I should have been comparing it to was other people’s pipelines.

What I built instead

The ASCII sketch I had been carrying around is now nine phases with an actual specification, so here it is properly.

One piece of that is worth stating because it is invisible in a diagram. The posting step does not depend on the adjudicator, it merely runs after it. That sounds like a distinction without a difference and it is not: if posting depended on adjudication, then an adjudicator that crashed would take the review down with it, the run would be recorded as failed, and a cron would re-dispatch the entire nine-phase pipeline every thirty minutes forever. That exact shape has cost me real money before. So the dependency is on the plain reviewer alone, and the ordering is a matter of declaration.

One fact, from a real pull request on cal.com. The field doing the most work is inDiff. A symbol that changed and is only used inside the diff is usually fine; a symbol that changed and is used at four sites the author never opened is where the defects live, and no amount of reading the diff will tell you which one you have.

And here is one of the survey passes discharging its questions. Every stated obligation comes back either quoted to a line or explicitly marked absent, and anything it could not settle becomes a hypothesis for the phases downstream:

Each survey pass gets exactly one of these five families. A question only exists if the deterministic layer found something specific to hang it on, which is what stops the whole thing from being a checklist:

The first four are minted from static analysis, so they are only ever about code that really moved and really has consumers. spec is the odd one out: its questions come from what a human said they were going to do, which is the axis a standards review structurally cannot check.

There are four ways to close one of these questions, and only one of them is a clean bill of health: quote the line that answers it, quote the line and name the gap it still leaves, say it can only be settled by running something, or say that every candidate was read and no such line exists.

That last one is the point of the whole exercise. It is not a failure to answer the question. It is the finding.

A sixth family, tests, was cut on the way to shipping. It needed a coverage report that only the probe phase can produce, the probe phase is off by default, and so it had produced zero artifacts in every measured run. It was paying a fifth of the fan-out to write “not measured”.

Two rules in that diagram I would have got wrong without the measurements.

The deterministic layer generates hypotheses. It does not filter them. This is theory one, inverted. BitsAI-CR (arXiv:2501.15134) independently reproduced the result I had paid for: a review filter raised precision from 54.5 to 67.1 and cut recall from 45.5 to 39.8.

The both-ends rule in seed is not fastidiousness. IRIS’s ablation measured it: naming both ends is roughly 2x recall, while a half mechanism scored -3, which is actively worse than asking no question at all.

And one number bounds the whole approach. Take the human’s findings, pull out the identifiers they mention, and ask whether the deterministic layer even names the thing the finding is about. For TypeScript that is 46.2%; for everything else it is 2.7%. Naming is necessary and nowhere near sufficient, but it is the difference between “the information is not there” and “the information is there and nothing is using it”.

The seam between what you can test and what you cannot

The pipeline has a hard line through the middle of it, and it turned out to be the most useful property of the design.

Everything up to and including seed is deterministic. Same commit in, same facts out, same questions out. I can say that with more confidence than I usually can about software, because when I ran the identical configuration three times, the generated questions came back byte-identical on all eight cases - same identifiers, same families, same counts. That half behaves like ordinary code. I can unit-test it, diff it, pin it with fixtures and reason about it.

Everything after it is not. Handed that byte-identical brief, the models produced 18 hypotheses on one run and 43 on the next for the same pull request. 10 against 23 on another. And across three identical runs of the whole pipeline, matched defects went 8, then 2, then 5.

Two things follow from that, and they are why I would build it this shape again.

Push work across the seam wherever you can. Not because deterministic code is cleverer - it plainly is not, it cannot review anything - but because it is the only half that holds still. Every question the static layer mints is a question the model does not have to think of, and one I can regression-test forever. The model’s job shrinks to the part only it can do: looking at the code and forming a judgement.

On the other side of the seam, repetition is the only instrument you have. There is no unit test for “does this prompt find defects”. A single run of a non-deterministic system is an anecdote. I have a number I like and a number I do not - 0.320 and 0.080 - from the same code, on the same fixtures, on the same evening, and nothing but running it again distinguished them.

That is what an eval actually buys you, and it is not a score. It is the ability to tell a change from a draw. I did not appreciate the difference until I had a result I wanted to believe.

Where it lands

Eight pull requests, 25 real defects. The shipped reviewer, then the pipeline, then the pipeline after one more change I will come to:

The baseline column is the one to sit with. The shipped reviewer posted one finding across eight pull requests containing 25 real defects, and matched none of them.

That is worse than the 0.040 I opened with, and the two are not comparable: the model changed, the harness changed, and the fixtures were repaired in between after I found that my own test setup had been handing the agent its previous review of the same pull request. Take 0.040 as the number that started this and this one as the number I now measure against. Neither of them is good.

So the pipeline finds real defects the prompt never did, and it has never lost a finding the baseline had already caught. That is the thesis, and it holds.

The change that finally moved recall

For a long time it did not. The middle column above was the state a few days ago: precision up, volume down, and recall statistically flat - the set of defects found rotated rather than grew, ten swapping in and ten swapping out from one unchanged configuration. I could defend the precision half and nothing else.

What changed is not a model or a prompt. It came from looking at the defects nobody had ever found and noticing something dull about them: every rule I had for generating a question required a reference from outside the diff. A value defined and used entirely within newly added code has no such reference, so no question was ever asked about it. Four of the never-found defects sat in files no question touched at all.

So seed learned two new ways to mint a question - one for symbols whose every use is inside the diff, one for route and hook registration order - and the result is the third column:

• defects generated: 12 of 25 to 21 of 25. Paired per defect, that is ten gained against one lost, at p = 0.006. It is the only change in this entire project that is significant on its own terms.
• defects actually posted: 12 to 17 of 25, nine gained against four lost, at p = 0.133. Real, not significant.
• mean recall 0.320 to 0.400, which is the first change to clear its own run-to-run noise band. The tool that scores these calls it KEEP, and it has said REVERT or INDISTINGUISHABLE to everything before it.
• and it costs the same money: $16.90 against $17.11.

And the bill for it

Precision fell from 0.220 to 0.161 and posted volume went up 40%, from 41 findings to 56.

That is not a surprise, it is the trade the field says it is. You buy recall with noise, and I bought some. It is also the exact opposite of the previous change, which bought precision and no recall - which tells you something uncomfortable about how much of this is one dial.

Two things are still true and neither is fixed by this.

Discovery moved; saying did not. Twenty one defects generated against seventeen posted, and the significance sits on the first number, not the second. The pipeline is still finding things and then failing to tell anyone about roughly a fifth of them.

The per-run expectation barely moved. Across repeats, 17 of 25 were found at least once and only 3 of 25 every time - down from 4. A user does not get the union; they get one run, and which run they get is still close to a draw.

And one earlier improvement is partly circular: the question catalogues were derived from defects the pipeline had historically missed, including some in the held-out split. Real improvement on those cases, but not evidence it generalises - which is why the next section exists.

Does any of it generalise? Ten pull requests say maybe

Everything above is eight pull requests from one private repository, tuned against its own gold. That is not evidence of anything general, and I have been saying so throughout.

So here is the same pipeline, unchanged, on ten cal.com pull requests from Martian’s public review benchmark - 31 defects, open-source code, gold written by other people, none of it used to build or tune any part of this. The models are wired the way production would wire them rather than forced onto one model for comparability.

Paired defect by defect, the pipeline gained seven and lost none, at a one-sided p of 0.008. On the generated surface it is eight gained against one lost, p = 0.020.

That is the strongest evidence in this post, and it is worth being precise about why. It is not the size of the gain. It is that nothing was lost - the claim I had been making on private data, that the pipeline never gives up a finding the old reviewer caught, held on a repository nobody tuned it against. And both repeats returned exactly 0.581. The variance that dominates everything above did not turn up here at all.

Now the caution, because there is plenty.

The baseline is far better on this set than on mine - 15 of 31 against 0 of 25. These are different difficulty regimes measuring different things, and I am not going to pool them or quote a combined figure. Whatever makes cal.com’s defects findable, my eight cases do not have it.

The baseline is a single run, so its own noise is unbounded, and some of that seven-point gap could be it. The scorer refuses to call this anything better than indistinguishable for exactly that reason, and it is right to. Repeating the baseline is the next thing I owe this number.

And precision halved again, 0.441 to 0.257, on double the posted volume. Same trade as before, on a different corpus, which at least makes it look like a property of the approach rather than of one repository.

Ten pull requests is still ten pull requests. But it is the first result here that was not measured on the material it was built from, and it went the right way.

Twelve minutes, which is the cost I under-rated

Three to seven times the money depending on the repository, which the tables above put at two or three dollars a pull request. I find I can live with that, and much less easily with the other one: a review that took two and a half minutes now takes twelve, and on the larger cal.com pull requests, sixteen.

That changes what the tool is. At two and a half minutes it comments while you are still looking at the page, and it is part of the conversation about the change. At twelve you have moved on, and it becomes something you find when you come back - much closer to CI than to a reviewer. Neither is wrong, but they are different products, and I arrived at the second one by accident rather than by choosing it.

Twelve is also the good number. Before the five survey passes were made to run concurrently instead of one after another it was twenty nine minutes, which is not a code review at all, it is a nightly job.

Where the money goes is not where I assumed either. The survey model is deliberately the cheap one, and then the fan-out multiplies its price by five. Meanwhile the original reviewer is still in there doing a complete independent review of its own - 137 seconds and $0.30 of every single run - while forty-odd hypotheses sit unread beside it. That is straightforwardly waste. It is known, and it is not fixed, and it is the first thing I would cut.

So: is it worth it? Not yet, and the reason is reliability rather than price. Seventeen of twenty five found at least once is a genuinely useful reviewer. Three of twenty five found every time is not, and the second number is the one a person actually experiences. I would not ask anyone to pay seven times over for a reviewer whose output changes that much between identical runs.

The encouraging part is that the change which finally moved recall was free. More questions out of the same evidence costs nothing; the money is in the model passes, and they did not grow. If the rest of the recall comes from asking better questions rather than from buying more inference, this gets better and cheaper together, which is not the direction I expected to be arguing in.

Where it goes next

The seeding change is the shape of the remaining work, and it is not where I would have guessed four months ago. It was not a better model, a longer prompt or another phase. It was noticing that a whole class of defect could never be asked about, and adding two rules to a deterministic file. Ten defects appeared that no amount of prompting had ever surfaced.

So the next thing is more of that: find the defects nobody has ever found, work out what question would have caught them, and see whether that question can be minted from evidence rather than guessed at. The variance is the other half - seventeen found at least once against three found every time is the gap that decides whether any of this is usable, and I do not yet know whether it closes by asking better questions or by asking them more than once.

Then more of what the cal.com run started. Ten public pull requests going the right way is the first thing here that is not measured on the material it was built from, and the cheapest thing I can do next is repeat that baseline so the comparison has two ends rather than one. After that, more repositories, and gold I have never read.

If you are building something in this shape, the thing I would press on you is smaller than a pipeline. Decide which parts of your system are allowed to be non-deterministic, make that boundary explicit, and put everything you can on the deterministic side of it - not because it is smarter, but because it is the only side you can hold still while you change the other one. Then measure the other side by running it more than once, because a single run of a language model is a story, not a number.

I would genuinely like to compare notes if you are down the same road, especially if you have found a way to make the discovery half stable. I have not.

Previous posts in this thread: The harness is the product, What is harness engineering, and Meet agentic-pi.