I'm building URIP, a risk-intelligence platform, with a team of AI agents doing most of the implementation. Agents write the code, agents write the tests, and agents often verify their own work before I see any of it. Four times now, that whole chain reported green on something that wasn't true. Each one is logged, with a root cause and, in three of the four cases, a fix. None of them were caught by the check that was supposed to catch them — they were caught by someone asking a narrower, more suspicious question than "did the test pass."
Failure 1 — Green, but never committed
How do I verify data was actually committed to the database?
Read it back from a different database session or connection than the one that wrote it — never from the same session, and only after the write has actually committed. A same-session read can see its own uncommitted change and report it as proof, when a completely separate query a moment later would find nothing there at all.
That rule was documented and repeated in the next test brief. An independent review still found six of fifteen tests with the same anti-pattern: five never committed before reading, and one committed but still read through the writer session. It was the third recurrence. The review concluded that documentation was not enough. P33-0708 remains triaged and now calls for an enforced fixture.
Failure 2 — A mutation test that had never actually tested the mutation
What is mutation testing and why does it matter for AI-written code?
Mutation testing deliberately breaks or deletes small pieces of already-passing logic, one at a time, and checks whether any test notices. It matters for AI-written code because a suite that's green against the current implementation can still be green against a broken one — a passing test proves the code ran, not that anything checked what it did.
A risky-user scoring test asserted 76.0, flagged red, for eight duplicate rows sharing one dedup key. The suite passed. Once the dedup constraint was declared in the ORM, that state became unreachable; the only possible score was 9.5, green. The original test described a state the database could no longer hold.
Failure 3 — The mechanism worked; production never called it
How do I check a feature is actually wired into production?
Grep for every call site of the function or task in question — not just its own module and its own tests, but the schedulers, routers, and background workers that would actually invoke it on an ongoing basis. A feature that runs correctly once, by hand, and is never called again afterward isn't wired in; it's a proof of concept that looks finished.
The enrichment was correct but frozen after that run: later devices and ownership changes became invisible. The fix added a scheduled task and daily beat entry, with a twelve-test suite including a separate-session committed-state read. P33-0711 is resolved.
Failure 4 — A large metric that did not mean what it said
Device correlation across telemetry sources depends on joining records by hostname. Measured on production data, one connector reported 2,276 distinct hostnames, another 461, a third 46, a fourth 17 — a lot of raw material, on paper, for a feature meant to connect activity seen by one tool to the same device seen by another. Checking how many of those hostnames actually matched between any two sources told a different story: zero overlap between three of the six possible source pairs, seven between a fourth. The volume was real. The join surface it was supposed to feed almost didn't exist, because each vendor names the same physical machine in its own private universe, and nothing had ever checked whether those names actually lined up.
The count of hostnames was a true number. It just wasn't measuring what the feature needed it to measure. P33-0695 is still open as I write this.
What changed
Why do all my tests pass when the software is still wrong?
Usually because the test is checking the wrong thing — it reads through the same session that wrote the data, reruns a manual step instead of trying to break the logic, or counts something without checking whether that count means what it's assumed to mean. A pass only rules out the specific failure the test was built to catch, not every failure that could exist.
Every one of these four passed something before it was found wrong — a UAT, a test suite, a manual production run, a row count. What none of them survived was a check aimed at the specific way that kind of claim goes false: a fresh session instead of the one that wrote the data, a mutation instead of a rerun, a caller grep instead of a rerun of the same manual step, a cardinality check instead of a row count. This project logs every defect as a row rather than a note, and closing one requires a person or agent other than whoever fixed it to verify it — of the ones closed so far, all of them were checked by someone other than the fixer, none were self-verified. That structure didn't stop any of these four from happening. It's the reason all four were caught at all.
Don't ask whether the agent completed the task. Ask what evidence would make the task's claim false — then go get that evidence from somewhere the task itself doesn't control.
