I threw away an implementation last month and had a new one in about ninety seconds. The tests took a day to write and I would have been upset to lose them.
That ratio has reversed in a way I keep noticing. For most of my career the code was the asset and the tests were overhead that you wrote because you should. The code encoded the thinking. The tests checked it. If you lost the tests you were annoyed; if you lost the code you were finished.
Now the implementation is close to disposable and the tests are the part carrying the information.
Table of contents
Open Table of contents
What the tests are actually holding
A test suite is the only place in most codebases where somebody has written down what the system is supposed to do, in a form that is checked.
Documentation drifts and nobody notices. Comments describe an older version. The ticket that explained why is in a tool you no longer use. The person who knew has left. The tests are the one artefact that fails loudly when reality moves away from the intent, which makes them the only durable record of what anyone decided.
That mattered before. It matters more now, because the question “can we rebuild this” has a different answer. Given a clear enough statement of required behaviour, you can regenerate a service. You cannot regenerate the knowledge of what it was supposed to do. The intent was always the expensive part and it is now the only expensive part.
Which reframes what a test is for. Not a safety net under the implementation. The specification, which happens to be executable.
The circularity problem
There is an obvious way to get this wrong and I have watched several people do it, including me.
You generate the implementation. Then you ask for tests. You get a suite that passes, with good coverage, and it is close to worthless, because it was derived from the implementation rather than from the requirement. It tests that the code does what the code does. Every misunderstanding baked into the implementation is faithfully reproduced in the suite that is supposed to catch misunderstandings.
Coverage looks excellent, incidentally. That is what makes it dangerous.
The fix is not complicated but it does require discipline. The specification has to come from somewhere other than the implementation. In practice that means one of:
Write the tests first, from the requirement, before any implementation exists. The old discipline, now with a much better reason behind it than it used to have.
Write them from the requirement document, by a person, and accept that this is slow.
Derive them from observed behaviour of the real system, for anything you are replacing, which gives you a specification grounded in what actually happens rather than what anyone believes happens.
Have a different person, or at least a different session with no sight of the code, produce them from the requirement alone.
The common thread is that the checker and the checked need independent sources. That has always been true in principle and was easy to ignore when writing tests was tedious enough that nobody generated them from the code anyway.
Techniques that got more valuable
Property based testing. Instead of a list of examples, you state a property that should hold for all inputs and let the framework hunt for a counterexample. The reason this suits the current situation is that properties are claims about intent, expressed at a level the implementation cannot trivially satisfy by accident. A model can produce code that passes your twelve examples. Producing code that satisfies “the output is always sorted and the same length as the input, for any input” requires actually doing the work.
Characterisation tests. When you are replacing something old, capture what it does now, including the behaviour nobody intended, and assert that the new version matches. This is how you discover the four callers depending on a quirk. It has become more useful because replacing an old implementation is now something you might attempt on a Tuesday rather than planning for a quarter.
Contract tests at the edges. Where your component meets another, assert the shape of the exchange from both sides. These catch the invented method and the assumed field, which is a failure mode that got noticeably more common.
Tests that read as documentation. If the suite is the specification, a person has to be able to read it and learn what the system does. A test named test_handler_3 fails at that even when it passes. This used to be a style preference. It is now the difference between having a specification and having a pile of assertions.
What this does to coverage as a number
Badly, is the short answer.
Coverage measures which lines executed during the suite. It never measured whether the behaviour was right, and the gap between those two has widened, because generating code that exercises every line is easy and generating code that checks the right properties is not.
I would not throw the metric away. Low coverage still tells you something real. High coverage now tells you considerably less than it used to, and a team optimising for it will get exactly what it asks for.
What I would rather look at is whether the suite catches a deliberate break. Change a comparison operator, flip a boundary, remove an error path, and see whether anything fails. That question has a useful answer in a way that a percentage does not, and it is the same instinct as planting records to measure a scanner you cannot otherwise evaluate.
The honest difficulty
Writing a good specification is harder than writing code, and always was. It requires knowing what you want, precisely, before you have something in front of you to react to.
A lot of software development has quietly depended on not doing that. You build an approximation, show it to someone, and they tell you what is wrong with it. The implementation was the medium through which the requirement got discovered. That is not a failure of process, it is how people actually work.
I do not think that changes. What changes is which artefact you keep afterwards. Build the approximation, learn what you actually wanted, then write that down as tests and let the approximation go. The thinking was the output all along. We just used to store it in the wrong place.