'Done' Is a Claim, Not a Fact

An agent can report "done" with total confidence and no necessary relationship to whether the work is finished. That claim needs a structural check, not a reminder.

I run a set of AI agents that build and maintain software projects, and every defect, bug, or new requirement they find becomes a row in a SQLite ledger, not a bullet in a document. That distinction turned out to matter because of one closure record: P39-0005.

What shipped green

P39-0005 was a defect in the installer for another one of my agents. A commit had added two new placeholder tokens — one inside a launchd plist file, one inside a deploy script — without adding a substitution site for either. The rendering function that was supposed to fill in real values only knew about two older, different placeholders; it had never been taught about the new ones.

The bug shipped green, because the existing test asserted that one specific, already-known placeholder was gone from the rendered output. It never checked for the two new ones. A passing test suite and a broken installer coexisted for as long as nobody looked for the right thing.

What closing actually requires

Why shouldn't the agent that wrote the fix be the one to verify it?

Because the agent that just finished writing the fix already believes the fix is correct — that belief formed while writing it, so there's no independent frame left to check it against. A verifier who is also the author isn't verifying anything; they're re-reading their own conclusion and agreeing with it.

The fix was written by a Sonnet subagent. Before that report becomes a closed row, the tracker requires a different verifier and an evidence row of kind TEST or REPRO. Both rules are database constraints.

The record

Here is what the row for P39-0005 actually holds, read from the live ledger:

resolved_by          p39-plist-fix-sonnet-subagent
verified_by          claude-opus-5-orchestrator
closed_by            claude-opus-5-orchestrator
root_cause  commit ddccdae added two placeholders
  (__DEPLOY_DIR__ in the plist ProgramArguments,
  __FACTORY_ROOT__ in deploy/run_daemon.sh) with no
  substitution site for either; _render_plist only
  handled the two older placeholders. It shipped
  green because the test asserted just one named
  placeholder was absent.
evidence (TEST)  RED/GREEN probe: git checkout HEAD --
  src/pehredaar/cli.py -> pytest
  tests/test_launchd_integration.py = 2 failed
  (leaked placeholders); cli.py restored -> 3 passed

The verifier is a different identity than the fixer. verification_method records what was run: a diff stat, a real pytest execution, and a RED/GREEN probe.

Why the RED/GREEN probe is the point

What evidence should be required before closing an issue?

Not a sentence claiming testing happened, but a record of what was actually run — ideally a RED/GREEN probe: revert the fix, confirm the suite fails in exactly the way the bug predicts, then restore the fix and confirm it passes. A single passing run after a fix only tells you today's code works; a RED/GREEN probe tells you the test would have caught the original bug.

A single passing run only says today's code works. The RED/GREEN probe is falsifiable: it records the predicted failure before the fix and the passing result after it, showing that the test would have caught the original bug.

The aggregate, and the gap I won't hide

P39-0005 isn't an outlier I picked because it's flattering.

That ratio isn't a norm the agents choose; it's what the schema permits. An agent that wants to self-close cannot make the insert succeed.

The honest part: 13 of those 557 closures carry a verified_by with no resolved_by recorded at all — a data-completeness gap, not a self-verification one, but a gap. It's tracked as its own row, P0-0002, opened during work adjacent to P39-0005 itself. Its root cause is specific: the check that blocks an unresolved issue from closing was added at the close step, but the equivalent guard was never added at the verify step — so a row can reach a verified state attesting to work with no recorded fixer, the same failure the rule exists to prevent, just with the fixer field left empty instead of matching the verifier. That defect is still open. I'm naming it here rather than fixing it first, because a ledger that only reports the numbers making it look complete isn't a ledger, it's marketing.

What to copy

How do I stop an AI agent claiming work is done when it isn't?

Don't rely on the agent remembering to check its own work — make a false "done" claim structurally impossible to record. A closure should require a different verifying identity than the one that did the fix, plus an actual evidence row from a real test run rather than a sentence saying it was tested, enforced by the database rejecting any write that skips either.

"Done" is what someone says before the work is checked. Whether it's true is a separate fact, and the only way I've found to keep those two things from collapsing into each other is to make the system refuse to record the second without evidence for it.