Coding benchmarks start every agent on a clean repository. Real work rarely does. A developer opens a new session on something an earlier session called done, or one agent in a pipeline builds on what another one submitted. Researchers at the University of Hong Kong, Tencent and elsewhere took that situation and tested it. They let cheaper agents attempt real tasks, kept only the attempts that were submitted as complete but still failed the hidden tests, and then gave five frontier agents the same request with that leftover work sitting in the repository. Nobody told them an attempt had happened or that it had failed.
Every agent did worse, by 20 to 53 percentage points, on the same tasks with the same tests. The rankings flipped too: the two agents that led from a clean start came third and fourth once they had to inherit work. The leftovers looked convincing. On the code-repository track, the typical inherited state already passed 98% of the hidden tests.
What the agents did with the leftovers explains the drop. On the code track, every run looked at the inherited changes, and not one discarded them and started again, though nothing stopped it. Runs that failed kept almost all of the earlier code, and most of them finished with exactly the same tests failing as when they began. Those are tests the same agent passes from a clean start. Files the earlier attempt never touched were often opened and then left alone.
"The tests become hard only once a failing version of the work is present."
The authors call this acceptance bias: the agent treats the earlier work as the definition of the job instead of checking it against the request. That reading fits the evidence, but the study can't show why it happens. It also cannot say whether a one-line handoff note ("this attempt failed") would close the gap, because withholding that signal was the whole design. Both stay open.
The more practical read is that "continue from here" is a separate skill from "solve this", and clean-start benchmarks don't measure it. A vendor's benchmark score describes the clean-start case. If a team's agents routinely resume sessions, pick up abandoned branches or sit downstream of other agents, that score probably overstates what they'll get. The cheap fix is procedural: when work changes hands, say plainly what is known to be broken, and have reviewers check the result against the original request rather than against the diff the last agent left. On this evidence, a green test run on inherited work tells you less than it does on fresh work, since the inherited state was already mostly green.
The paper's own limitation is narrow: every predecessor here was an agent, so whether people's half-finished work triggers the same deference is untested. The place this finding will get over-read is the word "agents". Human developers anchor on existing work too, and the authors cite that research themselves. This piece's view: treating it as an AI-only flaw that a better model will fix would be a mistake.
"Clean-start scores therefore overstate how well an agent can take over work in progress, and Starting-State Robustness deserves to be evaluated alongside them."
Sources
- Li, J., Liu, W., Xu, J. et al., "SWE-Residual: Evaluating Starting-State Robustness of Coding Agents" — https://arxiv.org/abs/2610.25899