An AI agent says it ran your test suite – great. But did it really run it? The only way you can tell is if the results of those tests were recorded.
People make false claims about their own work all the time, and they do it on purpose. Somebody signs off on a document they skimmed. Somebody ticks the box because the deadline was yesterday. Somebody sends the status report up green because amber will trigger a conversation they would rather not have.
Because it is deliberate, everything we use to catch it works. There is a motive to look for. There is a story that stops hanging together when you ask a second question. There is a person who knows what they did, and who hesitates before answering it. And there are consequences, which is why most people, most of the time, don't.
An agent's false claims are sincere. It is not lying, and there is nothing to interrogate – no motive, no cover story, nobody in the building who knows. It produces the sentence its training says fits the situation, and it produces the same sentence whether the work happened or not. The model that ran the suite and the model that only says it ran the suite write identical words, because the words were never derived from what the agent actually did.
So every human tell is noise here. Fluency, hesitation, an account that does not quite add up: the instruments a manager uses without noticing they are using them read nothing at all on an agent.
Which leaves one move. You are not asking the agent to be more honest – it already is. You are arranging for the work to leave something behind, so that there is an artefact to read instead of a sentence about it.
What this register does
The truth register names the 15 claims an agent makes about its own work, what each one costs when it turns out to be false, and the strongest control available against it. The register carries eight evidenced claims and seven candidates; the hub's headline counts count evidenced rows only. Nine can be converted into a committed artefact a machine checks on every commit. Six come down to judgement that no tool settles. None can be prevented outright.
The register produces one working rule: commit the artefact, then trust the artefact rather than the claim.
This is a technical page listing every claim, what it costs, what closes it, and what has actually gone wrong here. It sits under the hub.
Many of the controls on this page are standard software engineering – CI, code review, committed test output – and experienced engineers already run some of them.
What the register adds is the failure behind each control: which failures are only detected, and which ones nothing closes. This shows where your remaining gaps are.
Why these failures occur quietly
The claims in this register have a second property, following from the first.
Bugs announce themselves eventually. Something breaks, someone reports it, it gets fixed, and the cost is limited by the noise it makes.
False claims do not announce themselves. They sit underneath decisions until whatever was built on top of them collapses. By then the claim is months old and nobody connects the two.
The severity grading makes that concrete. Every claim is scored S1 to S4 on what it costs and how quietly it fails, and S1 – the cheap one, where somebody spots the problem the same day and one round trip fixes it – has no rows on this register at all. Not one of the 15 claims fails loudly enough to be noticed on the day it happens. That is what makes this a truth problem rather than a bug list, and it is the reason the register has to be assembled in advance rather than only accumulated from incidents.
Most people learn this the same way. Not from an agent that behaves suspiciously, but from one that behaves perfectly and turns out to have been wrong the whole time.
How this register works
The truth register lists 15 claims and their failure states, measured against a base install – an agent, a repository, a CI pipeline, and nothing else pointed at this problem. It then names the strongest guard you can add against each claim. Whether any of them is actually switched on is a separate question, and one you answer against your own configuration. The whole catalogue is published on GitHub – clone it or download the zip.
The claims were derived by analysing the types of claims an agent makes in one harness and one production codebase. We worked through the claims, row by row, and asked what would make each one false, what that failure would cost, and what could catch it.
We followed this approach deliberately. A register built only from recognised incidents would fail to identify the worst-case scenarios: the highest-severity claims are the ones that never announce themselves, so selecting for "we have seen this go wrong" filters out the dangerous half and leaves a list of loud, cheap failures.
Every register in this framework is built the same way, but this is the one where the difference shows, because a truth claim that goes wrong leaves nothing behind to log.
Every sentence on this register is one we have watched an agent write. We have not always watched it turn out to be false. Eight of the 15 carry a dated incident from our own work. The other seven carry the claim and the reasoning about what would make it false, with no logged instance of it happening.
Those seven are the register's weak point. An empty evidence line means one of two things. Either the control was already switched on before we went looking, or the failure leaves nothing behind to log.
Kind 2 – "did it come out clean" – is the first case: the cheapest controls on the register, installed first, one dated failure across five claims. Kinds 4 and 5 are the second, where a judgement claim that goes wrong produces no incident for anybody to log. The evidence column records which failures left a trace. It is not a ranking of risk.
There is a version of this you can check yourself. If you run agents daily and cannot remember the last time one faked a result, the likely reason is not that faked results are rare. It is that you have already closed the cheap group, where they would have been visible.
The goal of the register is to identify the claims your agents make constantly, and to help you install the setting, tool or process that would prove each one. Everything that cannot be covered that way stays under your judgement. The list of the remaining claims that you are responsible for is at the end of this page.
The five kinds of claim
Claims in this register can be grouped by what the agent is actually asserting. This is more useful for spotting them in your own work, because it maps to moments in a working day.
Every claim on the truth register answers one of five questions.
- Did it happen? Claims that an action was performed at all. The action either happened or it did not; nothing here is about quality. Example: "I verified it." These show up in closing summaries and pull request descriptions.
- Did it come out clean? Claims about the state of an artefact after the work – a pass, a currency, a conformance. Example: "The generated file is current." These show up whenever the agent reports a status rather than an action.
- Was the checking any good? Claims about the adequacy of the checking itself. The suite ran. The question is whether running it meant anything. Example: "The tests are good." These show up in exactly the sessions that feel most reassuring.
- Is the design right? Claims of judgement, where correctness is a property of a decision rather than of an execution. Example: "The data model is right." These show up at the start of work and are paid for months later.
- Is this safe to ship? Claims about release and about risk acceptance. Example: "The residual risk is acceptable." These show up at the moment of handover, which is the moment nobody is checking.
The two taxonomies are independent. The important finding comes out of a cross-cut between the two.
- Kinds 1 and 2 – did it happen, did it come out clean – are almost entirely closeable.
- Kinds 4 and 5 are almost entirely not.
- Kind 3, the claims about whether the checking was any good, is the only group that straddles both classes and closes in neither.
Nothing that checks the checking is closed anywhere, and you only see that by cross-cutting the two taxonomies. Sort by control alone and it disappears.
The three classes: what you can do about each claim
Every claim on the register is assigned to one of three classes. The class describes the strongest control available against that claim, not the seriousness of the claim itself.
What decides the class is whether the work behind the claim, when it is genuinely done, leaves behind something a machine can compare against what the agent is claiming. "I verified it" and "the residual risk is acceptable" are the same kind of sentence: flat, plain, written at the end of a session by an agent that means it. One is Class B (detected) and the other is Class C (survives), and the only difference between them is that a verification produces an output you can commit and diff, and a risk acceptance produces nothing.
- Class A – prevented, by withholding. The gold standard. The best control is to make a false claim impossible: the action is refused, and there is nothing to check afterwards because nothing happened. On this register it is empty. No claim an agent makes about its own work can currently be prevented by construction. Other registers in the framework do have Class A rows; this one does not.
- Class B – detected, by executing then diffing (nine claims). Commit the output rather than the claim, and let a machine compare the two on every commit. This is where most of the register lives, and the machinery is ordinary. No part of it is novel, expensive or research-grade. The cost is that detection is after the fact. The failure is written, it is caught, and then somebody repairs it. You are reading the red, not the code.
- Class C – survives: nothing retires these claims, so they reach you. Judgement only (six claims). "This finding is real." "The data model is right." "The residual risk is acceptable." Nothing converts these into machinery. They are not a backlog to automate later. They are the part of the work that was never delegable. Independent re-derivation reduces the risk, but it cannot eliminate it. The register's job here is to identify exactly which six claims need a person.
So the score for this register is zero prevented, nine detected, six surviving. Not one claim can be made impossible. Everything is either caught afterwards or lands on somebody's desk.
A note: the three classes don't sort neatly into good news and bad news. TL-04, "I verified it", is the most severe band on the chart and also the most completely closeable. Commit the verification output, let CI compare it against the claim, and the agent still writes the sentence but the sentence stops mattering.
TL-09, "the tests are good", has a mature mechanism that most teams have never installed. TL-13, "the RLS design is sound", never closes at all.
Does judgement require a human? Why can't an agent do it?
The obvious question about Class C is why a second agent cannot simply check it. It can, but only on one condition.
A second agent settles anything that has an oracle – a test log, a diff, a committed file. It cannot settle "did we think of every way this policy leaks?", because that question has no oracle to check against.
A second agent from the same model family also brings correlated blind spots. That is why the register's wording is "blind spots that aren't correlated with the author's" rather than "a second opinion".
A second agent counts as independent when it is a different vendor and a different model, in a seat the first agent does not control. That condition has its own row – TL-03, "the audit was independent" – and it is the row everything else leans on. Two sources agreeing is probability, not certainty. One source agreeing with itself is neither.
Vocabulary: this register reads its three classes as prevented, detected and survives. Five of the six registers use these words, and the class letter means the same thing everywhere – how complete the remedy is. [A note on vocabulary] on the hub covers the one exception and the three things still unreconciled across the framework.
