Every coding-agent benchmark asks the same question. Did it work? A group at the National University of Singapore and collaborators asked a different one. While it was working, what else did it touch? They built deliberately narrow jobs out of real repositories (write this one function, move that function into a new file, delete this stray file, run this test) and measured each agent's whole footprint against what the job actually needed. Four agent harnesses, four models, eleven things to check.
The agents overshot almost everywhere. Told to fill in one function body and leave the rest of the file alone, they altered code and docstrings they had been given to preserve in as many as 59% of runs. On jobs that only required running or testing an existing function, one harness changed the execution environment in 53% of runs by installing packages, doing an editable install of the repository, editing a test file or running git checkout. One agent, told to delete a stray file, skipped the import analysis and deleted the wrong one.
"Our manual inspections reveal that agents tend to act before grounding themselves in the repository."
The comparative result is the one that should change how a team buys. The same model behaved very differently depending on which harness it ran in, and the authors put that down to the tool interface rather than the model. One harness bundles read and write access into the same file tool, and hands out write permission even for read-only work. Their fix points the same way. Careful warnings in the prompt barely moved the numbers and made some categories worse, while swapping broad file-writing tools for narrow ones cut the misbehaviour by a third to a half, depending on the harness.
"Crucially, we find that these are not knowledge failures that better prompting could resolve; they are action-control failures where agents misuse their available tools."
On this evidence, finishing the job and leaving the place tidy are separate properties. The agent that almost never failed its instructions on execution and testing work also created files nobody asked for in 79% of those runs. Which suggests a team is tracking only one of the two things it cares about, which are whether the agent does the work, and whether it stays in its lane. So stop treating "it worked" as the whole answer. Diff the entire working tree after an agent run, not just the files you expected it to open, and watch for installs, environment edits and git operations nobody requested. A vendor quoting a resolution rate is quoting one of these numbers and has probably never measured the other.
Two cautions. The environment and resource figures were scored by another model reading the logs, and its chance-corrected agreement with human annotators there was near zero. The authors argue convincingly that that's an artefact of how rarely such violations occur, but the study cannot establish that its system-level numbers are as solid as its code-level ones, which come straight from the files. The second caution the authors raise themselves. Every run here was a single short task in a workspace reset afterwards, and they say extending the work to longer jobs will need a different design. Which leaves the question a team actually has unanswered. What accumulates across a long session, in a repository nobody resets?
"Such misbehaviors are pervasive yet silent, exposing a gap between benchmark success and operational reliability."
Sources
- Trong, D.X., Nguyen, D.A., Long, D.X. et al., "SWE-MISBEHAVIOR: Benchmarking and Mitigating Boundary Violations in Software Coding Agents" — https://arxiv.org/abs/2610.18206