A team took real open-source repositories from two standard agent benchmarks and rewrote them into code that does exactly the same thing, with if/else branches swapped and the condition negated, local variables renamed to synonyms and unreachable methods bolted onto classes. They touched about 7% of source lines, left the test files alone, and checked against full test suites that nothing behaved differently. Then they pointed four frontier models at the original and the rewrite, each one wrapped in two different agent scaffolds (the tool-and-prompt harness that sits between a model and a codebase).
The agents resolved fewer issues on the rewritten code in most of the combinations tested, but the average drop was small, under 7 percentage points at its very worst. Read alone, that's a reassuring number. Two things sitting underneath it are not.
The first is which ranking survived. Ordering the models by how many issues they solved gave nearly the same league table before and after the rewrite; a leaderboard would have registered almost nothing. Ordering them by how much they lost gave a different answer every time the scaffold or the codebase changed. One model was the sturdiest of the four under the simpler harness and the most brittle under the richer one. The strongest performer overall took the largest hit in the study on one benchmark and was the least affected on the other.
"We call this a jagged robustness frontier: robustness is a joint property of the model, the scaffold, and the workload, not of the model alone."
The second is the bill. On one benchmark, cost rose in every single configuration after the rewrite, by as much as 23%, counting only the runs where the agent got the right answer. Step counts moved far less, so the agents weren't mostly taking more turns; they were burning more on each one. The paper's trajectory analysis shows what that looks like up close. It found decoy variables carrying the same keyword as the bug report, polluting the agent's search results; an agent widening its patch to tidy up code that was never broken; another that blamed a failing test on a split string literal and submitted anyway.
"Measured by resolve rates, most configurations are impacted modestly by the perturbations, but, measured by what the agent spends, the impact is more drastic."
For a team, spend is the leading indicator and pass rate is the lagging one. If agent runs get metered at all, meter cost per resolved task per repository. That's the number that moved here, and it moves before anything shows up in a success metric. It also reframes what a vendor's robustness claim is worth, since capability appears to travel between harnesses, reliability doesn't, and reliability is the half being bought.
What the study can't establish is which of the rewrites did the damage. Each variant drew a few transformations at random from the catalogue, and the results are never broken out by type, so there's no support here for the tempting idea that tidier code makes agents work better. The authors don't claim it either. The instances were also pre-screened to ones the strongest model already solves, making this the easy end of the work, and the losses concentrated hard, with a handful of individual issues carrying most of the largest average drop.
This will get over-read as proof that coding agents are fragile. On these numbers they mostly aren't. What the study undercuts is the metric the industry ranks them on, which turns out to measure the one property that survives a rewrite.
Sources
- Mahmud, H. N., Gupta, S., Chaudhary, I. et al., "A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformations" — https://arxiv.org/abs/2608.18389