Once people step outside using Claude and ChatGPT in their browsers and start using them on their files, they quickly see the limitations. Yes, you can get work done in a single morning that would take you a week. But you also need to constantly check whether the agent actually did the work that it said it did, saved it in the right place and didn't wreck anything in the process.
So the day splits in two. Half of it is the fun part – flying through your work. The other half is reading back what the agent did and deciding whether to believe it. That second half is where you lose a lot of the productivity benefits generated by the first half. And it also makes you question everything your agent does.
There are various controls you can add to your AI agent which limit its mistakes. These controls are bundled into a concept called a harness, which helps you drive your agent like a harness on a horse. Harnesses fall into the governance category, which scales all the way up to enterprise products such as ServiceNow AI Control Tower and Rimini Govern for AI.
New harnesses are released constantly. The hard part is knowing which ones actually work.
Over the past month I measured the ways that agents failed me and turned it into a free assurance framework called False Floors. It consists of several assets:
False Floors is designed as a free resource for anyone who wants to increase the quality of their experience with AI agents. If you want to understand the philosophy behind it – how agents need a code of conduct at work like we have for humans – you can read the essay, The Return of the Fool. (It's the first in a four-part series.)
This is all quite abstract so I recommend scanning the whole framework at a high level by flipping through these seven diagrams.
There has already been some excellent research in this space listing controls for agents (Engineering Reliable Coding Agents, Agentic Risk and Capability Framework).
Better harnesses and governance models arrive weekly. What nobody is doing is measuring what a harness actually covers, and what it leaves open. Hold a governance product against False Floors and you can see which failures it misses.
Why run this assessment? Because if you have the right controls in place, you will reduce the time spent checking if your agent did what you asked it to. Or simply, you will trust your agent more.
This is critical to handing more of your work to an agent. At a larger scale, it's critical to hiring agents to perform roles carried out by humans.
Over the next few weeks I'll be releasing reports showing which failures each harness actually catches, and which ones get through.
If you want to know the origin story, read on.
When I started using agents properly – for hours a day, every day – I quickly became frustrated at silly mistakes. The agent forgetting what I had told it, or decisions I had already made, or changing something that I didn't ask to change.
I started writing rules to stop it from making mistakes, and then wanted to know how many rules I had to make. Would I reach an end point where the agent was fully built, or was this a Sisyphean task?
That question turned into a research project.
I started keeping a register: one row per failure, written down in the agent's own operating terms, against a personal software project.
I recorded more than a log of "the agent made a mistake". I wanted to know what type of mistake – was I running into the same thing multiple times, or different things? Diagnosing the failures required detailed names like "a prohibition was read as a preference and it went ahead anyway, with a justification".
Each row records its catch point (the moment inside the agent's turn where the failure can still be stopped), what mechanism catches it there, and whether that mechanism prevents the failure, merely detects it, or leaves it to survive.
The rows sorted themselves into groups, and the groups had names that I recognised as something you would hear in a workplace staffed by humans. Did the agent do what it was told? Did it know what it needed to know? What information could it access?
There are six registers published, with a seventh in the wings. They split by the mechanism that fails.
The name is False Floors because a false floor is a control that looks like it's holding your weight but isn't. It is false by construction rather than by fraud. Nobody built the hollow board to deceive you. You find it by standing on it.
Once every failure is a named row with a stated catch point, you can write a scripted version of that failure, run it against any tool that claims to prevent it, and see whether anything actually stops it. Do that across the whole catalogue and you have a test suite for agent setups.
As mentioned above, a harness is a product wrapped around an agent to direct or check its work: templates it has to fill in, steps it has to follow, scripts that check something before it moves on.
Claude and ChatGPT have their own set of base controls, although they don't really call this a harness. You're more likely to know of GitHub Spec Kit, OpenSpec and BMAD-METHOD as popular harnesses for coding agents. Your own rules file, hooks and pre-commit checks are an informal one. If you run agents at all you have a harness, whether or not you'd call it that.
The False Floors test suite runs a planted failure through a harness and reads what the harness does, not what it says. Each scenario carries a twin: a near-identical action that is legitimate and must be allowed through. A check that blocks everything guards nothing, so both halves have to pass before a control counts as closing anything.
Before any of it is allowed to grade a product, every scenario has to fail on a bare repository with no harness at all, and pass against a control setup built to catch it. A test that has never been seen catching anything is not evidence.
The output is a coverage map with named gaps. It is not a score or a ranking. A ranking invites a vendor to optimise for the ranking, and it hides the only thing you actually wanted to know, which is whether the control you are relying on actually works.
The Big Four do something similar in their paid engagements. KPMG announced in June 2026 that it would sell its Trusted AI agent governance to clients as a service; Deloitte launched a connected agentic network under its Trustworthy AI framework the same month.
I am doing it in public and releasing the framework for free. A procurement team can read the registers themselves rather than take a supplier's word for a proprietary internal harness. Other firms get a shared vocabulary they can be measured against, the way OWASP became the common language for web security even though every serious shop had its own internal review.
And the validation discipline is published for you to review, including the parts that didn't work.
Reliability, not performance. Almost every published comparison of coding agents measures how well they do the work. This measures whether you can trust the result: whether the agent did what it was told, stopped when it should have stopped, and told you the truth about what it did.
Two layers of it are running now. One tests harnesses, the products that sit around the agent. The other tests the models themselves as their own vendor's command line runs them, which is the unit a team actually installs: Claude Code driving Opus 5 and Sonnet 5, and Codex driving pinned GPT versions. The tests both run the same planted failures, the same reading, and each family is judged by a model from the other family.
The gaps between these setups are wider than I expected and they are not where I expected them, on ten kinds of failure across 600 runs. The Claude and ChatGPT reports will be free. Add your email address below to find out when they are available.
Read the registers and work out which failure modes your current setup covers, where there are gaps, and what you can do to cover them. The end result is fewer frustrations with your agent, and more trust that it can do the work you're asking it to.
If you are writing software with agents, or running Claude Code or Codex over the files on your desktop, that list is worth an hour of your time.
The catalogue and the method are on GitHub now, free: MIT for the code, CC BY 4.0 for the content, and attribution is the only condition. The tools I build around it in my own work go out under the same licences, each one as it becomes usable by someone other than me.
Two caveats. This is a moving target: the platforms keep shipping guards against these failures, and I keep finding new failures. And the catalogue is bounded by my personal experience. It holds every way I have seen an agent fail, which is not the same as every way an agent can fail.
If you have hit something the catalogue doesn't hold, open an incident and tell me. One free-text box, describe what happened in your own words.
Every incident gets a public verdict on the thread, including the verdict that nothing in the catalogue owned it, in which case a row gets added and you get credited on it.
Interviews, registers and essays. Free.
Make your agents work.
