Skip to content
Rooz Jalali
Go back

Generation got cheap, review did not

A team I know doubled the number of pull requests they open in a week. Their merge rate went up by about a fifth. The rest is sitting in a queue, and the queue is getting longer every week.

This is the central fact of the last two years and I think a lot of teams have not sat down and worked through what it implies. Producing a plausible change got dramatically faster. Deciding whether a change is correct got no faster at all.

Table of contents

Open Table of contents

Why only one half sped up

Writing code was always partly a typing problem. Not entirely, but enough that removing the typing makes a visible difference. You know roughly what you want, and the distance between knowing and having is mechanical work that a model is good at.

Review is not a typing problem. To approve a change you have to build a model of what the code does, compare it against what it should do, and look for the gap. That work happens in a human head at a human speed, and nothing about the last two years has made human heads faster.

So you have a pipeline where one stage got several times quicker and the next stage did not move. Anyone who has looked at a production line knows what happens. The throughput of the whole thing is set by the slowest stage, and all you have achieved upstream is a bigger pile of work in front of it.

The pile has a cost beyond the waiting. Changes sitting in review go stale, conflict with each other, and have to be rebased. The author forgets the context and has to reload it to answer a question. A change that waits four days costs more than the same change reviewed in an hour, and the cost is paid twice, once by the reviewer and once by the author who has to come back.

What teams do about it, and why it does not work

The common responses are all variations on ignoring the constraint.

Some teams add more capacity to generation. More licences, more agents, more people encouraged to use them. This makes the queue longer.

Some measure the wrong thing and declare victory. Pull requests opened, lines changed, commits per engineer. All of those went up. None of them is delivery.

Some quietly stop reviewing properly. This is the most common and the least discussed. Nobody announces it. The approvals just get faster, and the comments get shorter, and after a while a review is somebody scanning a diff for anything that looks odd and clicking approve. The ritual continues and the control is gone.

That last one is worth sitting with because it does not feel like a decision. A reviewer facing nine hundred lines of competent looking code, with six more changes behind it, is not going to read it line by line. They are going to skim, trust, and move on. The incentive structure produces that outcome without anybody choosing it.

The things that actually help

I have seen four that make a difference, and none is clever.

Cap the size of a change. Not as a guideline, as a limit the tooling enforces. The relationship between diff size and review quality falls off a cliff somewhere around a few hundred lines, and the cliff is well documented. Generation makes large diffs effortless to produce, which means the discipline that used to come from the effort of writing them has to come from somewhere else now.

Make the author do the comprehension work. The person opening the change should have already read it properly, understood it and be able to explain it. Whether a model wrote it is irrelevant to that obligation. A description that says what changed, why, what was considered and rejected, and what the risky part is, converts an hour of reviewer archaeology into ten minutes of checking. Plenty of teams have let this slip precisely because the change was quick to produce.

Review behaviour, not text. A diff tells you what characters changed. A test tells you what the system now does. When the volume is high, the most efficient review is to read the tests first, decide whether they describe the right behaviour, then skim the implementation for anything surprising. That inverts the usual order and it works much better at volume.

Spend the review attention unevenly. Not every change deserves the same scrutiny, and pretending otherwise is how everything ends up getting the same shallow pass. A change to a configuration file in a sandbox and a change to the permission checking code are not the same risk. Deciding in advance which parts of the codebase get real review, and which get a glance, at least makes the triage deliberate instead of accidental.

The uncomfortable implication

If review is the constraint, then the way to go faster is to spend more on review, not less. More senior time, not less. Better tooling for reviewers, which barely exists, rather than more tooling for authors, which is where all the investment has gone.

That is a hard sell, because it looks like slowing down. The team that doubles its output of pull requests can point at a number. The team that assigns two experienced engineers to review full time cannot point at anything except a queue that stopped growing.

I do not have a neat answer to the sell. What I would say is that the alternative is not faster delivery. The alternative is the same delivery rate with a lot more unreviewed code in it, and you find out about that later, in a way that does not get attributed back to the decision.

There is a version of this I have seen work, which is to stop treating review as an interruption to real work and start treating it as the work. Put it in the sprint. Give it the hours. Let people say they spent Tuesday reviewing and have that be a complete answer.

A note on reviewing with a model

The obvious response is to use a model for review as well, and close the gap that way.

It helps for a narrow class of things. Mechanical consistency, obvious omissions, a forgotten error path, the test that does not assert anything. Run it as a first pass and it will remove some noise before a person looks.

It does not help with what review is actually for, which is deciding whether this change is a good idea in a system the reviewer understands and the model does not. The model has no stake in the codebase existing in two years. It has not been paged at three in the morning by the code it is about to approve. The judgement that review exists to apply is exactly the part that does not transfer.

So it widens the pipe a bit at the narrow end. It does not move the constraint. I will come back to the skill of reading generated code in a separate piece, because it turns out to be a different skill from reviewing a colleague’s work and almost nobody is taught it.


Share this post:

Previous Post
Cedar, and the decision point my AI flow was missing
Next Post
Reading code nobody wrote