Somewhere in your company there is an AI-built system that works. The demo ran clean. The tests pass. Everyone who saw it nodded.
It has never touched a customer.
That gap shows up in almost every 10 to 500 person company I talk to. Stuff got built, sometimes fast and sometimes impressively, and then it sat.
Nobody could say out loud why they trusted it. The person who built it moved on, or the agent that built it produced a green checkmark and a README.
The leader holding the risk looked at both and quietly decided not to find out.
The instinct at that moment is to add capability. A better model. More tests. A stricter prompt. I spent a full build cycle learning why none of those close the gap, and what does.
Why does an AI-built system pass every test and still fail the first human who uses it?
Because the same party writes the code and the evidence.
When an AI agent builds a feature, it also writes the tests that judge the feature. Those tests encode what the builder imagined could go wrong.
They cannot encode what the builder never thought of. The failures that end up mattering live almost entirely in that second category.
A courtroom would never accept this arrangement. Testimony from the defendant gets weighed as exactly what it is.
In software we look at 800 green checkmarks from the defendant and call it assurance.
Here is what that looks like in practice. Last week I built signal-desk, an open pipeline that turns inbound buying signals into outreach drafts.
It refuses to pass anything it cannot prove. Eight stages, every decision written to a tamper-evident ledger, a human approval bound to the exact text being approved.
By the second milestone it had 520 passing tests.
An independent reviewer then walked the workflow the way a first-time user would. Run the pipeline, approve a draft, run it again, replay the audit trail.
The final step failed. The replay command never loaded the approval decisions the run command did.
The one sequence every real user would perform ended in an error that blamed the user.
Five hundred and twenty tests had passed over that bug. Each stage was tested alone, and each stage was correct. The failure lived in the seam.
Test coverage is a map of what you imagined. A stranger walking your demo is a map of what you promised.
The mistakes that taught me this, so you can skip them
The generous part of this piece is the list of things my own system got wrong with every test green.
Each one was caught before merge by the review layer described below. Each one is visible in the public record if you want to check my claims.
The same bug shipped twice, after the lesson was written down. Two milestones after the replay bug, a new entry point omitted the same decision store, in a fresh spot.
The earlier lesson was quoted verbatim in the builder's instructions. The reviewer caught the recurrence by walking the workflow again.
I take one thing from this above everything else. Telling the builder about a failure class does very little. A gate that checks for the class catches it every time.
Written review compounds. Institutional memory does not.
A credential sat in the shareable artifact, and the leak test was blind to it. The pipeline produces a run directory you can hand to a colleague as an audit record.
The signing secret was inside it, in plaintext. There was a test greping every written byte for leaked credentials. It knew about one of the two secrets.
830 tests passed while the artifact leaked. The suite encoded the marketing sentence, hand this to anyone, and never the trade underneath it.
A fix passed all its own tests and preserved the bug one level up. In a companion tool, a wrong environment variable was silently ignored.
The fix made the inner function strict, added tests for the function, and went green. The composed module still fell back to a default binary, so the bug survived intact.
The only test that could see it had to spawn the real CLI as a subprocess. The broken behavior only assembled at that level.
Two good properties composed into a defect. Run IDs were content hashes, for determinism. The queue sorted by run ID to find the latest run.
Both choices were individually right. Together they showed a human a stale approval queue about half the time.
The documented API shape had already drifted, and only a real call could say so. Every provider test in the build ran against fakes built from the docs.
The first live call the code ever made came back with a 400. The provider had deprecated a parameter the docs-shaped request still carried.
One line, fixed and independently reviewed within the hour. The smoke test that found it exists because a fake can only test the shape you imagined.
If you have an AI agent building anything in your shop, some version of every one of these exists in it right now.
The agent is careless in none of these stories. The classes above are simply invisible from inside the build.
What does an adversarial review layer actually look like?
The layer that caught everything above is a second, independent agent with a written contract and the power to refuse the merge.
The contract has five passes, and the order matters.
1. Blast radius first. For every behavior the change touches, find every other place that shares the behavior and check those too.
A finding is closed when its class is closed, never when the single instance is patched.
2. Claim versus code. Read every sentence the README and the docs promise, and test whether the code still keeps each one.
A doc that promises an early refusal the code performs late is a finding, even though nothing is broken.
3. Refute, do not confirm. Construct hostile inputs and try to make the system fail. Forge the audit ledger. Feed it a payload with a credential embedded.
Approve a draft, then edit the draft, and check whether the stale approval still applies.
The reviewer's job is to lose faith. Every probe the system survives is trust you can name.
4. Severity, calibrated in writing. Every finding graded against a stated line for what blocks a release.
A reviewer that blocks on nitpicks is as useless as one that waves everything through. The only defense is making the line explicit before grading starts.
5. The missing question. What would a senior ask that the author never did?
On my build the answers were, does the shareable artifact hold a credential, and can the approval loop close in live mode. Both answers were no. Both had green tests.
The verdict converts to action by one rule. Only an explicit pass merges.
A hedge, a conditional, a review that never reaches a verdict, all of those park the change for a human.
The reviewer also walks the system as a first-time user against the documented flow. That walk is the pass that caught the replay failure a whole suite missed.
One honest note on independence. My builder and reviewer are both AI agents, from the same model family.
The independence is procedural. Different context, different instructions, an adversarial brief, and no shared memory of the build.
That was enough to catch two blockers, a credential leak, and a repeat of a known class in one day. Whether it stays enough at team scale is a question I have not answered yet.
How do you install this on your own system?
Everything above is public and runnable. You do not need my pipeline to use the method, and both repos run without any API keys.
To see the method working, clone the worked example.
git clone https://github.com/derrtaderr/signal-desk
cd signal-desk && npm test
node bin/signal-desk.mjs runFourteen synthetic signals go in. Twelve are refused, each with a named reason in a sealed ledger you can replay byte for byte.
The pull requests on that repo carry the full review records, including the blockers described above. You can read exactly what each pass caught and what it cost to fix.
To apply it to the system sitting in your shop, take the contract. The reviewer contract, the five passes and the verdict rule, lives at github.com/derrtaderr/ship-check.
It is a template you can hand to an agent or a human. The short version of the install is three steps.
-
Write down the promise. One page describing what a first-time user should be able to do with your system, step by step, including every error and its recovery. If nobody can write this page, that is the first finding.
-
Hand the contract to a reviewer that did not build the thing. A second agent session with no shared context works. So does an engineer from another team.
-
Convert the verdict mechanically. Explicit pass merges, everything else parks for a human. The moment a hedged review can ship, the layer is decoration.
The first run of this on your own system will be uncomfortable, because it will find things. That discomfort is the entire value.
Every finding it produces was already in the system, invisible, waiting for a customer to find it instead.
What this does not prove yet
I want to be precise about the boundary of the evidence, because the boundary is where the interesting work is.
This ran on one codebase, over one build cycle, with agents on both sides of the table. Nobody outside my machine has run the pipeline yet.
The live path has made exactly one real provider call, the deliberately small one that caught the drift above. Everything else about live mode is proven against fakes.
The refusal ratio in the demo comes from fixtures I authored.
Whether the review discipline holds when the builder is a human team with feelings about their code, or a codebase ten years old instead of one day, I do not know.
The mechanism that caught a repeated failure class twice in one day is the part I trust. The scaling claims are the part I am still earning.
FAQ
Is this just code review?
It is code review with three properties most review lacks. The reviewer is adversarial by contract rather than collegial by habit.
The reviewer walks the promised user journey instead of reading the diff. And the verdict converts to a merge decision mechanically, so a hedge cannot ship.
Do I need two AI agents to do this?
No. The contract works with any reviewer who did not build the thing and has permission to say no.
Agents make it cheap enough to run on every change. That cost drop is what turned it from a ceremony into a gate in my build.
My AI system has no tests at all. Where do I start?
Start with the promise page from the install steps, then the stranger walk. On an untested system the walk itself is the highest-value audit available, and it costs an afternoon.
Doesn't refusing this much slow everything down?
The build described here shipped four milestones in one day through this gate, including the fix waves. Refusal is fast.
The slow thing is discovering in production what a reviewer would have found in an hour.