When AI “Cleans Up” the Past
One of the most attractive promises of an AI coding agent is that it can see inconsistency faster than a human team can. Give it a repository and it will find duplicated branches, strange status values, old fallbacks, awkward interfaces, and conditions that no longer appear to fit the architecture. It can make the system look cleaner in a single pass.
That ability is useful. It is also where a subtle class of failure begins.
An old code path is not merely an old implementation. Sometimes it is a container for historical knowledge: data already stored in production, an external system that still depends on a compatibility contract, a degraded mode that keeps an important workflow alive, or an edge case discovered by an incident nobody documented properly. Source code does not always tell you which of these explanations is true.
Why legacy behaviour is difficult for AI
AI agents are very good at local reasoning. They can inspect a function, compare it with a new design document, and identify a mismatch. They are less reliable at reconstructing the undocumented reasons a mismatch exists. Those reasons may be distributed across migrations, fixtures, call sites, operational habits, old API clients, and the shape of the data itself.
The optimisation pressure also points toward internal consistency. If a new architecture defines a clean set of states, the agent naturally wants every caller to use only those states. If an old branch looks redundant, removing it feels like progress. If a database value is ambiguous, excluding it feels safer than interpreting it.
Sometimes that is exactly right. Sometimes it turns a truthfulness improvement into a capability outage.
A fictional failure in event correlation
Consider an event-processing platform that correlates records over time. The system has thousands of historical events with usable timestamps, but older versions did not record where those timestamps came from. The data contains time, but not time provenance.
A new design introduces an explicit field:
native— supplied directly by the producer;reconstructed— derived from stored evidence;fallback— supplied by a deliberately degraded path;unknown— usable provenance cannot be established.
The new design makes an important correction: the system must not present an unknown timestamp as vendor-native evidence. That protects users from a false claim.
An AI coding agent then makes an apparently logical change. It updates the correlation eligibility rule so that only native and reconstructed events can participate. The implementation is tidy. The new tests pass. The migration has already labelled all historical rows unknown.
After release, the scheduler still runs. The correlation job completes. There is no exception, timeout, or failed health check. Its output is simply zero because the job quietly excluded the historical input that made the feature useful.
This is more dangerous than a crash. A crash is noisy, localisable, and usually tied to a visible alert. A successful zero can be mistaken for a legitimate result. It can travel through dashboards, exports, reports, and downstream decisions while retaining the appearance of normal operation.
“Do not claim this data is native” and “do not use this data” are two completely different decisions.
The first is a statement about truthfulness. The second is a statement about capability. They may be related, but one does not logically imply the other.
The safer question is not “is this old?”
A more careful design might preserve the historical events, mark them as degraded or unknown, and allow them into correlation where the semantics permit. The output could carry a clear confidence or provenance warning so that downstream users cannot mistake historical evidence for fully proven native data.
Newly created unknown values could be counted separately. A historical unknown may be an unavoidable migration fact; a new unknown may indicate a producer defect. Treating them as one population hides an important operational distinction.
Stored evidence could also support gradual reconstruction. If an old row can be upgraded to reconstructed from facts already present in the record, do that. If proof is missing, leave it unknown. Do not improve the numbers by inventing certainty.
This is not a universal recipe. Some unknown data must be excluded for safety, regulatory, or semantic reasons. The point is that exclusion needs a demonstrated reason. Fixing a claim does not automatically authorise deleting a capability.
Understand before changing
Before changing existing behaviour, an autonomous agent should answer five questions:
- What does the current behaviour do today?
- Why might it have been designed this way?
- What useful capability, compatibility, safety property, or customer value does it preserve?
- What exact defect are we trying to fix?
- Can we fix only that defect while preserving the useful behaviour?
The answers should come from evidence, not intuition. The agent should inspect code, tests, migrations, production-shaped fixtures, historical data, comments, API contracts, call sites, logs, downstream consumers, and previous issue history where available. A sentence in an architecture document is evidence of intended design; it is not proof that the running system has no other contract.
Preserving behaviour does not mean preserving bad implementation. A slow fallback can be replaced with a faster one. A confusing interface can be wrapped and deprecated. A dangerous compatibility path can be disabled after its consumers are identified and migrated. The distinction is between preserving the capability and preserving the exact old code.
Controls for autonomous local development
AI agents can still work autonomously. They need controls that make behaviour changes inspectable before they become invisible changes to system history.
Record the behaviour change
Every meaningful change should have a small audit record: old behaviour, its likely purpose, the defect, the proposed change, behaviour being preserved, behaviour deliberately removed, and the evidence used. This is not bureaucracy for its own sake. It is a forcing function against the sentence “the old path looked wrong.”
Cover every changed file
Every changed file should map to an audited behaviour change. One behaviour may touch many files, but the audit should check each affected read and write path individually: APIs, exports, dashboards, background jobs, counts, administrative views, and scoring or context reads. A broad label is acceptable only after its concrete paths have been examined.
Test the old capability as well as the new rule
Focused tests ask whether the new feature is correct. Regression tests must also ask what useful old behaviour still has to work. For data migrations, run those tests against representative historical shapes, not only freshly generated fixtures.
Test the tests
Mutation testing is valuable because a safety test can be present without proving anything. Deliberately remove the intended protection and confirm that the test fails. Be precise about the mutation: an incorrect mutation can leave the protection intact and create a false sense of coverage.
Use independent reviewers
Where practical, use two independent AI reviewers. Do not give the second reviewer the first reviewer’s verdict before it forms its own conclusion. Ask both: Did this change fix the intended defect without accidentally removing a valid legacy behaviour or capability? Disagreement should trigger investigation, not majority voting.
Verify artifacts, not declarations
An agent saying “done” is not evidence. Verify the actual file, diff, commit, test output, database state, and runtime state. A passing process is not proof that the right records were examined or that the output is meaningful.
Watch for the broken zero
Systems that process historical data need more than success and failure metrics. Record inputs examined, records eligible, records excluded, exclusion reasons, outputs created, degraded-data counts, and new unexpected states.
These counters make a silent compatibility break visible. They distinguish a legitimate zero — the system evaluated real input and found nothing — from a broken zero wearing an honest zero’s clothes: the system examined nothing because a new filter accidentally removed the population, then reported success anyway.
Fail visibly wherever the semantics allow it. If the system cannot safely produce an answer, a loud degraded state is usually more useful than a clean but misleading result.
Keep disproven findings
There is one more part of the control loop that matters over time. If an AI reviewer raises a concern that later proves incorrect, do not erase the finding. Mark it withdrawn or invalidated and record why. The point is not to preserve every suspicion as truth. It is to preserve the reasoning trail so future agents and engineers can learn which evidence changed the conclusion.
That record creates organisational memory around failures in reasoning, not only failures in code. It also makes future review faster: an apparently strange branch may already have a documented explanation, or a previously accepted explanation may have been disproven.
AI is powerful; autonomy needs a wider definition of correctness
This is not an argument that AI is uniquely dangerous, or that legacy systems should never change. Humans make the same mistake when they see untidy code and assume the untidiness has no value. AI changes the scale and speed. A plausible cleanup can touch more paths before anyone has time to notice what disappeared.
Code correctness is local: a function follows its new specification, a test asserts the expected result, and the implementation is internally coherent. System behaviour correctness is broader: historical data remains meaningful, external consumers keep working, degraded modes stay honest, and outputs continue to represent the capability users depend on.
Green focused tests do not prove system compatibility. A specification can be perfectly implemented while omitting an important historical contract. The answer is not to slow every change to a halt. It is to make behaviour preservation a first-class acceptance condition alongside code quality, security, and performance.
Old behaviour is evidence to understand, not technical debt to automatically delete. Sometimes the right conclusion is to remove it. Sometimes the right conclusion is to name it, constrain it, instrument it, and preserve it until the surrounding system is ready.
The safest autonomous developer is not the one that changes code fastest. It is the one that can explain what the old system was protecting before it changes it.
