Upgrading a dependency is the work nobody wants and every team owes. Dependabot and Renovate open the pull request, but they stop at the version number. If the new release moved a signature, widened a type or changed what a call returns, somebody still has to repair the code around it. That repair is exactly the job coding agents are sold for.
Researchers from Monash, Peking University and Microsoft built a benchmark to find out. They collected merged upgrade pull requests from real open-source projects (203 of them, across JavaScript, Java, Go, Rust and Python), keeping only cases where a human had been forced to write code beyond the version bump. Two things stay hidden from the agent, the fix the original developer wrote, and the tests that developer changed to prove it worked. The strongest agent setup repaired just over half of the tasks. Every other configuration did worse.
The failures are where this gets useful. In the failed runs the researchers examined closely, roughly three in five showed the same pattern. The project's own tests and builds passed, and only the hidden post-upgrade tests caught the mistake. In most failures the agent was not stuck or crashing; it had every signal it could see telling it the job was done.
"The result suggests that many failures are not simple execution crashes: agents often satisfy the old visible test surface but miss the dependency-upgrade behavior encoded by the hidden test patch."
The reason is structural, and it applies to human upgrades too. A repository's test suite was written against the version being replaced, so it cannot assert behaviour that only exists in the new one. The most common labelled cause of failure was an upgrade propagated partway. The agent found the right place, changed the import or the constructor, and left a wrapper, a return type or a test fixture still speaking the old contract. In one case a Go library widened its identifiers from a narrow integer type to a wider one; the agent updated the client wrapper and left a helper still handing back the old type. It compiled, it ran, and it was wrong.
Which makes green CI on a dependency-bump pull request close to worthless as evidence. On an upgrade PR, the question is not whether the tests pass but which test in the diff would have failed against the old version. If the answer is none, nothing in that pull request has checked the upgrade at all. Teams already treat bump PRs as low-risk paperwork, and automation makes that instinct worse by producing more of them, faster, each with a green tick.
The model was not the only thing that mattered. Holding the model fixed and changing only the runner around it moved the result by 33 tasks, a swing on the same scale as switching models.
"Current harnesses often treat upgrades like ordinary test-driven repair, rather than carrying the upgraded library's API or semantic requirement through all repository layers exercised by the hidden tests."
On this evidence, a disappointing result from an in-house upgrade agent may say more about the runner than the model, and swapping the runner is much the cheaper experiment.
One obvious remedy goes untested. The researchers counted how often each agent read the upgraded library's release notes or migration guide, and found it varied enormously between setups. But they are explicit that they only recorded the behaviour, never testing whether it helped. So the intuitive fix, making the agent read the changelog first, has no evidence here either way. The sample is narrow too, since every task came from a project already running an upgrade bot, which is not most codebases.
The practical move is small. Require that any agent-authored upgrade arrives with at least one test that fails on the old version, and treat an upgrade PR with no such test as unreviewed rather than approved.
Sources
- Luo, Z., He, R., Gao, P. et al., "Update from Hell: Can Coding Agents Survive Hidden Breakage in Dependency Upgrades?" — https://arxiv.org/abs/2608.30300