Most coding benchmarks hand an agent one task, on code a human wrote, and grade it once. Teams are starting to use agents differently. The pitch is delegation, where an engineer hands over a run of work and comes back later. In that setup, every request lands on whatever the agent left behind last time, checked or not.
Two Amazon researchers built a benchmark around exactly that. They took stretches of commit history from more than a hundred open-source Python projects and turned each into a chain of 20 requests that, done in order, rebuild the developers' own changes. Each request went to the agent in a fresh conversation, on the repository as the previous request left it. Hidden tests graded every step, and the agent was never told how it had done.
Every one of the ten agents completed individual requests more often than whole chains. The agents passed roughly two-thirds to four-fifths of individual requests as they came, but on average only 51% of chains ended with everything working. The best model, Claude Opus 5.5, finished about two in three.
The study compared verified agent code with verified human code. A new request did nearly as well on either foundation.
"In summary, when the code beneath a request is verified, models leave code nearly as buildable as a human’s."
What did the damage was the mistake nobody caught. A request built on top of an unnoticed failure was much more likely to fail itself, and the odds of failing roughly doubled with each additional unnoticed failure underneath. The original mistake was rarely fixed later.
Almost every submission came after a test run. They were testing the wrong thing. More than half of the missed requests were submitted after a passing local test run, and a green local run barely changed the chance that the work was wrong.
"The agents test what they wrote, not what was asked; a passing run of their own tests is what makes them believe a request complete."
A second agent reviewing the work did not rescue it either. Shown finished implementations, some right and some wrong, the lenient models waved through most of the wrong ones, while the strict models rejected about half of the correct ones. None told good work from bad much better than chance.
External feedback improved completion. Giving the agent a "verification failed" message and one repair round after failed requests lifted chain completion from about 51% to 59% in the paired first attempts. The repair rounds also increased regressions and required more model calls. In separate continuation experiments, requests built on repaired code failed no more often than those built on verified code. Saying which test failed did more. The paper puts it well:
"Naming the failing tests once achieves what insisting achieves in about three rounds: insisting repeats the agent’s belief about what it built; naming the test replaces it with a fact."
The authors are candid about the limits. The benchmark is Python only, many of the projects are small, and the request descriptions were written and checked by models, not yet by people. Correctness here means passing the developers' tests, not readability or fitness for production.
On this evidence, an agent's "done" carries almost no information on its own. An agent's report that its tests passed describes the tests it chose to run. For a team, the question to ask is what independent check sits between one request and the next. A team running agents unattended for a week of tickets is, in effect, running this benchmark without the hidden grader.
That also makes the agent-reviews-agent setups now common in vendor pitches worth questioning. The reviewer models in this study were free to write and run their own tests, and nearly always did, yet they still could not reliably tell right from wrong. Several were also noticeably softer on their own wrong work than on other models'. The study did not test human reviewers or a reviewer given the hidden requirements, so it cannot say how much either would recover. On this evidence, a second opinion from the same kind of model is not the check it is sold as.
The practical step is to check the result. Before the next request builds on a change, run the change or try the feature, and when it fails, say what failed rather than just "try again". Timing counts too. A repaired mistake stopped causing trouble, while one found after more work had piled on top of it had already spread.
The main benchmark resets the conversation after each request. Additional continuous-conversation experiments use different protocols and do not isolate how memory affects an agent’s ability to catch its own mistakes.
Sources
- Cuadron, A., Gupta, A., "Built on Broken Assumptions (BOBA): Why Coding Agents Cannot Be Left Alone" — https://arxiv.org/abs/2611.02207