Here is a category of tool you probably own several of. A sensitive data scanner. A vulnerability scanner. An anomaly detector. A secrets scanner in your build pipeline. A spam or fraud classifier. Each one produces a list of things it found, and each one is silent about the things it did not.
That silence is the important part. If a service goes down you find out. If a detector misses something, nothing happens at all, which is a much worse property than it sounds, and it is why so many organisations are confident about controls that barely work.
There is a straightforward fix, it is old, and hardly anyone does it.
Table of contents
Open Table of contents
Why the numbers you have are the wrong numbers
Every console gives you the same shape of information. Findings this month. Severity breakdown. Trend over time. Coverage, meaning the proportion of your estate the tool looked at.
None of that tells you whether the tool works. It tells you what it did. A detector that finds nothing produces a beautiful dashboard.
The number you want is recall: of all the things that were genuinely there, what fraction did it report? That number is absent from every product I have used, and its absence is not an oversight. The vendor cannot compute it either, because computing it requires knowing the truth, and if anybody knew the truth they would not need the detector.
Precision, the proportion of findings that were real, is at least approachable. Someone reviews a sample of findings and marks them. That happens, and it is why false positives get attention. Recall is the one that matters for anything with a compliance or safety consequence, and it is the one nobody has.
Plant the records
You cannot enumerate the truth across your whole estate. You can inject a small piece of truth that you control completely.
Generate synthetic records that resemble your real sensitive data. Not generic examples, yours: your field names, your file formats, your identifier shapes, your languages. Place a known quantity of them in known locations. Then look at what the tool reports, and count.
If you plant two hundred and it finds sixty, your recall on that population is thirty percent, and you now know something about your control that no vendor will tell you. Repeat it after every configuration change, every product update, every new bucket naming convention, and you have a regression test for a thing you previously took on faith.
That is the whole method. The interesting part is doing it in a way that produces a number worth trusting.
Doing it properly
Make the plants indistinguishable in form and obviously fake in content. They must have the same structure as real data so the detector treats them the same way. They must be unmistakably synthetic to any human, so that finding one later never triggers a real incident. Reserved test ranges help here: card schemes publish test numbers, and most identifier systems have documented invalid ranges you can use.
Cover the hard cases deliberately, not just the easy ones. The temptation is to plant clean, well labelled examples, which measures the case you already know works. Plant the awkward variants: no column headers, opaque field names, nested archives, a scanned image rather than text, a value split across two fields, a language other than English. The gap between your clean score and your awkward score is the most useful output of the whole exercise.
Record placements outside the system being tested. Keep the manifest of what you planted and where somewhere the detector’s own reporting cannot influence. A spreadsheet is fine. If the manifest lives in the tool you are auditing, you will eventually measure the tool against itself.
Plant across the range of storage, not one representative spot. The bucket somebody set up properly is not where your exposure is. Include the badly configured, the legacy, the region nobody visits, the archive tier.
Decide the pass mark before you look. Write down what recall you would accept, and what you would do at fifty percent, before running it. Otherwise the result becomes a negotiation with yourself, and every number gets rationalised.
Label everything and set a removal date. Tag the plants so they can be found and deleted by a mechanism independent of the detector. Synthetic test data left in production for years becomes real confusion for the next person.
What to do with a bad number
You will get a bad number. Thirty to sixty percent is a normal first result for contextual detection, and the reaction matters more than the figure.
The wrong response is to switch products. The replacement will score similarly on the awkward cases, because the awkward cases are hard for structural reasons rather than product ones.
The useful responses are mostly upstream. If detection depends on context, supply context: field names that describe their contents, headers on exports, consistent formats. That improves every tool you will ever buy. If the tool supports rules of your own, write them for your internal identifier formats, which no vendor catalogue will ever include and which are frequently the most sensitive things you hold.
And then adjust what you claim. A control measured at forty percent is still worth running. It is not worth describing to an auditor, a board or a customer as though it were complete. The number’s real value is that it converts a vague comfort into a stated limitation, and a stated limitation can be managed.
Where else this applies
Once you have the habit, the same move works anywhere the failure is silent.
Vulnerability scanning: deploy a host with known old packages and see what appears. Secrets detection: commit a synthetic credential in the several formats you actually use, in a comment, in a config file, in a notebook, and see which are caught. Alerting and correlation: inject a synthetic incident and watch whether it groups correctly and reaches a human. Backups, which are the original example of this idea, since a backup you have never restored is a hypothesis. Access reviews: create an account that should fail the review and see whether it does.
In each case you are converting an article of faith into a measurement, and the cost is an afternoon.
The general principle
Controls whose failures are invisible drift toward being decorative, because nothing in the normal course of events corrects them. Nobody is lying. The evidence that would prompt a fix never arrives.
The way out is to manufacture the evidence yourself, on a schedule, and to treat any control you have never tested as unverified rather than working. That is not cynicism about tooling. It is the same reason you run fire drills in a building that has never burned down.
Assume every detector is worse than its dashboard suggests, and find out by how much.