Studies of AI-written code almost all look at what got committed, which is the wrong place to find mistakes, because the obvious ones never make it that far. A researcher at Trinity College Dublin kept everything instead, a working 21,000-line Python tool built by Claude from scratch, without a human-authored line of code or test anywhere in its history, plus every session transcript behind it. Then he went back and fact-checked what the agent had said while building it.
The code errors landed roughly where a working engineer would guess. About one in seven code-generation events introduced an error, each one caught by the test suite before anything was committed, which is exactly why commit-history studies of AI code quality can't see them. Most were broken tests or fixtures rather than broken logic. The author makes a fair point about that, which is that the same agent that made the mistakes also wrote the suite that caught them.
Every substantive statement the agent made to its user was then pulled apart and checked against the codebase as it stood at the moment it was said. Plain factual claims (a test count, whether a function exists) were nearly always right. Proposed fixes were wrong about one time in five. Explanations of existing code, and diagnoses of why something broke, sat in between.
"Overall roughly 1 in 4-5 responses contain at least one inaccuracy."
Read in that order, accuracy falls exactly where the stakes rise. The agent is steadiest on things a developer could check in seconds, shakiest on the judgement calls that get acted on without checking. One appendix example lands harder than any rate. Asked why a file could not be found, the agent blamed a boundary between its sandbox and the user's shell, then ran the same command from that very environment and hit the same error.
"The AI disproves its own diagnosis without appearing to notice."
The asymmetry matters more than either rate. Code written by an agent gets gated by tests, CI and review. The prose it writes about that code gets gated by nothing, and the prose is what teams act on, like the root cause in the incident channel, the "safe to merge" or the refactor floated in a planning thread. The sensible working rule on this evidence is to treat a diagnosis as a hypothesis rather than a result, and ask for the command and its output instead of the conclusion. It also sets a fair test for any vendor quoting a reliability figure, because reporting a fact and diagnosing a cause are different jobs, and one number covering both is hiding the gap.
There are two things the study cannot settle beyond the limitation its author leads with, which is that this is one project, with one developer at the keyboard. There is no human comparison for the pre-commit error rate either, and he says why.
"The human analogue would be errors caught and fixed within a developer's own write-test-fix iterations before committing, a rate existing measurement cannot see for the same reason."
The second is sharper, and the paper doesn't raise it. The rate counts only errors the test suite caught, and that suite came from the same agent. Anything both got wrong in the same direction is missing from the count, making the figure a floor rather than an estimate.
Expect the headline to be over-read as "the agent is wrong a quarter of the time". It is not that. Most flagged replies held a single bad statement inside an otherwise sound answer, and the worst-scoring category counted proposals the user simply declined as misses; strip those and it looks much like the rest. The defensible version is narrower. On this evidence, an agent's account of its own work is the least verified thing it produces, and the thing teams read most.
Sources
- Leith, D. J., "Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase" — https://arxiv.org/abs/2609.29744