Every productivity figure I have seen for these tools measures the same thing: time from starting a task to finishing it. That number has genuinely improved and I am not going to argue with it.
It is also only one side of a ledger. Accepting a suggestion that turns out to be wrong has a cost, and that cost shows up weeks later, in a different team’s incident, attributed to something else. Nobody subtracts it, because by the time it arrives nobody can connect it to the decision that caused it.
I want to try to put rough numbers on the other side.
Table of contents
Open Table of contents
The shape of the problem
For any suggestion you accept, the expected cost is roughly the chance it is wrong, multiplied by what it costs to find and fix later.
Both factors moved in the wrong direction at once, which is the problem.
The chance of being wrong is not high in absolute terms. Most suggestions are fine. But the volume went up a lot, and a small rate applied to a much larger number produces more absolute errors than a higher rate applied to a smaller one. If you used to write forty lines an hour by hand with a one in fifty chance of a defect per meaningful decision, and you now accept four hundred lines with a one in a hundred chance, you have more defects, not fewer, despite being individually more accurate.
The cost of finding it later moved too, and this is the part I had not thought about properly. Defects get more expensive the longer they survive, and what determines survival is whether anybody looks closely. Generated code is read less carefully than hand written code, for all the reasons in the two previous pieces. So the defects that do occur have a better chance of making it to production, where fixing them is an order of magnitude more expensive than fixing them in review.
Put those together and you can be individually more accurate, produce more defects in total, and have each one cost more. Nothing about that is visible in a cycle time measurement.
The class that matters
Not all wrong suggestions cost the same, and the distribution is unhelpful.
A suggestion that does not compile costs almost nothing. You find out in seconds. Same for one that fails an obvious test, or that is clearly nonsense when you read it. These are the failures people think of when they think about the tools being wrong, and they are the cheap ones.
The expensive class is the suggestion that is plausible, passes review, passes the tests, and is wrong in a way that only shows up under a condition nobody tested. An off by one at a boundary. A race that needs two requests in the same millisecond. An error path that swallows something it should propagate. A permission check that is correct for the case in front of you and wrong for the case in the next ticket.
These survive precisely because they look right. The property that makes the tools useful, producing code that looks like code a competent person would write, is the same property that gets the bad ones through. You cannot have one without the other, and filtering on appearance will not separate them.
Why the measurement goes wrong
The asymmetry is in when and where the two numbers land.
Time saved is immediate, local, and attributable. It happened to a named engineer on a named task this week. It is easy to collect and it makes a good slide.
Cost incurred is delayed by weeks or months, lands on whoever is on call rather than whoever accepted the suggestion, and arrives labelled as a production incident rather than as a consequence of a tooling decision. No reporting system connects those two events. I have never seen a retrospective conclude that the root cause was a suggestion accepted in March without being read.
So the organisation measures the benefit accurately and the cost not at all, and then acts surprised when the productivity gain does not appear in delivery or quality figures. It is in the figures. It is sitting on the other side, uncounted.
Matching the tool to the blast radius
What I have settled on is not a rule about when to use the tools. It is a rule about how much scrutiny to apply, based on what it costs to be wrong in that particular place.
Some code is cheap to be wrong in. A script that runs once. A prototype that will be thrown away. A test fixture. Anything where the failure is immediate, visible and affects one person. Generate freely, skim, move on. The expected cost is genuinely near zero and treating it with ceremony is waste.
Some code is expensive to be wrong in. Anything shared across services. Data migrations, which are often irreversible. Permission and authentication logic. Anything in a path that handles money or personal data. Retry and backoff behaviour, where being wrong turns a small failure into an outage. In these places the saving from a fast first draft is trivial compared with the cost of a plausible error surviving, and the draft should be read as carefully as if a stranger had sent it, because one did.
Most code is in between and your judgement is the tool.
What I would push against is a single policy applied to the whole codebase, in either direction. Banning the tools wastes most of the benefit. Using them identically everywhere concentrates the risk exactly where you can least afford it, because the expensive places are also, usually, the places where the code is intricate enough that a plausible wrong answer is hardest to spot.
What I would actually track
If somebody asked me to produce an honest figure rather than a flattering one, I would want to know how long defects are surviving, not how fast changes are produced.
Pick a sample of incidents and ask where the defect was introduced and how many reviews it passed. Compare it against the same figure from two years ago. If defects are surviving longer and passing more reviews, that is the cost arriving, and it will keep arriving regardless of how good the cycle time number looks.
That is a slower and less satisfying measurement than counting merged pull requests. It has the advantage of measuring something that matters.