Coding agents now open pull requests that claim to make software faster. Someone has to decide whether to merge them, and a passing test can't settle it, because a test shows the code still works, not that it got quicker. Academic researchers went looking at how that decision actually gets made, across more than 1,200 performance fixes that commercial agents opened in public repositories. Then they did the part nobody had done before. They rebuilt a sample of the fixes and ran them.
Of thirty merged fixes the researchers rebuilt and re-executed, nine delivered no meaningful gain or made the code slower. The tests that shipped with them were no help either, since in most cases they passed just as happily against the unfixed code. Nearly half the fixes changed behaviour on inputs no test in the pull request ever reached. A compiler emitted a string where a number belonged. A permissions check let a child role's failure override a parent's grant.
"Nine of the 30 merged fixes show no significant gain or regress at the primary workload, and nearly half change behavior on inputs their tests never reach, so a merge records that a maintainer accepted the fix and not that the fix does what its description says."
The other half of the study is about the decision itself. The researchers compared merged against rejected fixes on everything they could code, including the kind of inefficiency, the fix the agent applied, the size of the change, and whether it carried a measurement or a test. None of it separated the two outcomes. What predicted a merge was the repository's prior relationship with the agent. Fixes landing in projects that had already merged most of the agent pull requests they'd seen were accepted around 84% of the time, against 33% in projects that had mostly turned them down.
"The outcome tracks the repository's history with the agent rather than the coded content of the fix, and a merge does not show that the fix delivers what it claims."
The authors conclude that maintainers should re-run a claimed speed-up before deciding, and that researchers using merge rates as a quality signal need a within-repository control. Their data supports the pattern but not the mechanism, because the study is observational, and can't say whether a maintainer who did check would have decided differently.
What follows for a team is not in the paper. Merge rate, acceptance rate and pull requests landed are the numbers most engineering organisations use to judge whether an agent rollout is working. On this evidence, they track how accustomed a team has become to the agent at least as much as the quality of what it produced. A team that had stopped looking closely would produce a rising acceptance rate too, and it would look identical to success. Any vendor quoting one should be asked what else moved alongside it.
For performance work there's a cheap control, and the study points at it without quite prescribing it. Require that the test accompanying a speed-up fails on the old code. That turns a claim into something CI checks on every run, and it is the condition most of these fixes didn't meet. Most carried no test of the speed at all.
Where this will get over-read is the nine-in-thirty. Those thirty were chosen for what could be rebuilt and driven deterministically, which favoured small, self-contained changes, the ones whose gains vanish most easily inside untouched work. The authors say plainly it may overstate the rate across the wider population. The study also can't tell anyone whether adding a performance test makes a fix more likely to deliver; it looked, found too few cases either way, and declined to draw the conclusion.
The finding that travels is narrower and harder than "agent performance fixes don't work". A merge records a decision, not a verification. Where the usual tests can't check the claim at all, a team treating those two as the same thing doesn't know what it has shipped.
Sources
- Qi, Z., Li, H. et al., "Merged, Not Measured: An Empirical Study of Performance Issues Fixed by Coding Agents" — https://arxiv.org/abs/2609.37985