Almost every coding benchmark grades one job and stops there. That isn't how teams use agents. A developer asks for a component, then asks for it to be extended, upgraded and made faster, and each request lands on code the agent wrote the week before. Multi-step SWE is built to test that pattern. Each of its tasks chains five to fifteen dependent requests on a single codebase, every one a multi-hour job: carry a Lua interpreter written in Java from version 5.1 through 5.5, walk a Rails app from 4.0 to 8.1, speed up BoringSSL one family of cryptographic routines at a time. Each step starts from whatever the agent left behind, and a task only counts as solved if every step passes.

Whole chains were rarely finished. The strongest of nine frontier models solved 18.5% of its attempts. It passed more than half of the individual steps, which is closer to the number a single-task benchmark would have reported. More than half of the tasks were never solved by any model, and on the version-upgrade tasks, where earlier behaviour has to change release by release, every model failed the overwhelming majority of its attempts.

Failures also bunched together. A step that followed a passed step was passed about 70% of the time, but after a failed step that fell to about 14%, and for every model the steepest drop came early in the chain. It looks like compounding, with one bad step leaving a broken foundation for the next. The study cannot yet say how much of the drop is compounding and how much is simply harder later steps. The authors list a baseline that would separate the two, starting each step from a correct reference solution, as future work.

Failing agents rarely ran out of time. For most models, a typical failed step used about a quarter to a third of its budget and left more than an hour on the clock.

"In 52% of failed steps, the agent reports the step as complete and names no failing check or missing requirement."

Those reports were labelled by Claude Opus 5.5, the same model that topped the leaderboard, though an independent second labelling of a subset agreed with the first in nearly every case. The agents' own reports were weak evidence of success. Among steps where the agent declared the job finished and named no gaps, more than half still failed the hidden checks. The authors read the timing data as agents being limited by when they judge the work complete rather than by the clock, and they say plainly that their data cannot settle the obvious follow-up.

"Whether a larger budget would help cannot be determined from these runs, because failing agents seldom use the budget they already have."

What separated passing runs from failing ones, in the case studies, was what the agent checked its work against. Failing runs stopped once the public sample tests or their own tests went green. Passing runs found an independent reference, such as an official conformance suite or a comparison with the original application's behaviour. In one migration, an agent upgraded a dependency nobody asked about, which quietly changed how two web requests were routed; a passing run made the same upgrade, compared behaviour against the original, noticed the difference and reverted it.

"Tests that an agent writes encode its own reading of the specification, so they confirm a misreading instead of exposing it."

For an engineering leader, this suggests the agent's completion message should be treated as a request for review, not a status update. On this evidence, a practical next step is to give the agent an independent check it did not write, such as a recorded baseline of production behaviour, a reference implementation or a conformance suite. "All tests pass" means little if the agent wrote the tests. It also argues for gating each step of a long piece of agent work, because a defect accepted on Monday becomes the foundation for Tuesday's request.

The finding is easy to over-read in both directions. These tasks are unusually large, from interpreters and network protocols to cryptography, and most are in C or C++, so a team's routine feature work may chain more forgivingly. But vendors who quote single-task benchmark scores are reporting something closer to the per-step pass rate than to the chance of a month of delegated work holding together, and they should be asked to explain the gap between those two numbers.

Sources

  1. Zeng, X., Liu, Z., Lin, Z., "Multi-step SWE: Can Coding Agents Keep Building on Their Own Code Over Long Horizons?" — https://arxiv.org/abs/2611.07166