Every engineering organisation has one, whether the service stuck on a framework nobody maintains or the C library that should have been Rust years ago. Migrations like these are expensive and almost entirely manual, exactly the shape of work a coding agent is meant to absorb.
Researchers tested that properly. They took twenty real open-source projects, SQLite, zlib and libsodium among them, and asked coding agents to rebuild each one on a different language, framework, host platform or build system, with hours of unsupervised working time and no network beyond the model. Then they graded three ways. Did the migration happen, does the software still behave identically, and can a panel of independent coding agents find a difference nobody thought to test for?
That first check exposes a hole in how this work usually gets graded. A migration starts from a repository whose tests already pass, so handing it back untouched scores full marks. The researchers call this Blindness, and it isn't hypothetical. Thirty submissions came back with the repository barely changed and passed every behavioural check, spread across nearly every model tested.
"Fixed tests measure behavioural correctness, but they cannot establish whether the migration actually occurred."
With all three gates applied, only 5% of attempts produced a migration that was complete, still behaved identically, and survived the verifiers. Most of the twenty projects were never finished by anyone. A small group preserved behaviour by not really migrating; a much larger group migrated and broke something. Doing the work and not breaking anything are separate skills, missed in opposite directions.
The near-misses matter more. Among runs that did complete the migration, most got the behaviour nearly all correct and still failed. Several models converged on the same solitary defect, in which a rewritten web app dropped the original's hash-based URLs, breaking every bookmark and shared link to the site.
"Not one of these checks can be called nitpicking: each corresponds to a regression that would cause trouble in production, and the original repository passes all of them."
Even a clean sweep was no protection. An independent agent found a behavioural difference in two thirds of the submissions that passed everything, typically within about 17 minutes.
For a leader weighing whether to hand over a migration, the 5% is the least useful figure here. What matters is that the test suite most teams would reach for as the acceptance gate can't, even in principle, tell them whether the work was done. On this evidence the deliverable in an agent-led migration is the acceptance harness, built before anything else by someone whose job is catching a no-op.
One failure never surfaces as a failing test at all. On a C-to-Rust task, submissions copied the original's control flow line for line and moved its manual memory management straight across. Every behavioural test passed. Memory safety, the entire reason for that migration, was never delivered. "Why are we migrating" has to become an acceptance criterion in its own right, because no behavioural test can express it. The same goes for vendor demos. One that ends on a green test run is showing the single thing that would look identical had nothing happened.
That 5% comes from a deliberately brutal setup, and will be over-read if that gets dropped. It used fully autonomous runs, one attempt each, with a single failing check anywhere scoring zero. The study cannot say whether the same agents, supervised module by module by an engineer who knows the codebase, would do better, and that is the arrangement most teams would actually use. The authors are careful about their own ceiling too, noting that acceptance here means survival rather than proof.
"An accepted submission is therefore not a migration proven correct but a migration that survived the strongest adversaries we could field—and as models improve, this same fixed set of 20 tasks will be scored more strictly."
For anyone piloting this, make "the old stack is gone" a separate sign-off from "the tests pass". CI already handles the second. Nobody currently owns the first.
Sources
- Hong, D., Chi, Y., Li, W. et al., "SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?" — https://arxiv.org/abs/2608.23564