A benchmark from New York University and Hong Kong Polytechnic makes one agent habit visible by pairing up tasks. Each exists in two versions, identical but for a single fact. In one version something in the repository or the instruction settles how a risk ought to be treated, such as the type system already excluding bad input, continuous integration already running the suite, or the user having said they will write the tests themselves. In the other that fact is gone. Everything else is held still, so any difference in behaviour traces back to it.
What it catches is the agent protecting the code against something that cannot go wrong. Across eight model-and-harness combinations, agents did that in between 11% and 59% of runs where the evidence had already ruled the protection out, adding redundant guard clauses, duplicate validation, backups of files version control already tracks, and requests to reconfirm a decision the user had plainly made. The evidence was always a statement of fact, never a prohibition. No task said "do not add checks."
The usual instruments are blind to it. More than nine in ten of the runs carrying unnecessary work still passed their tests. Developers notice anyway. Twenty experienced engineers rated runs without knowing which was which, and said they would take the output as it stood 38% of the time for runs with the extra work, against 78% for the rest.
"The results indicate that unnecessary treatment imposes real costs that functional metrics fail to capture."
The sharpest finding is about bounds. Where an instruction named a magnitude (this one test file, this many retries, a report this long), agents mostly stopped there. Remove the bound and they sprawled. In one logged run with no bound stated, an agent asked only whether the tests passed reached for the repository-wide target and came back with dozens of unrelated errors.
"Without a stated bound, the agent chooses the broadest standard test target, reaches unrelated components, and returns a less useful result at greater cost."
Capability bought no restraint. Every configuration finished almost every task, and the ordering by task success did not match the ordering by unnecessary work. Within one vendor's lineup, the model that completed the most tasks also did the most unnecessary work, by a wide margin. The authors read this as judgment about risk being a capability of its own, and suggest models may have overgeneralised the habit of checking their own work. Plausible, and the design can't test it. The benchmark records what agents did, not why. Nor can it separate model from scaffolding, since each model ran only in its native harness. The table isn't a model ranking.
What that leaves a team is narrow and cheap. The evidence that worked was never a rule about behaviour; it was a fact sitting where the agent would run into it. On this evidence, bounding the task beats policing it. Name the files, name the scope, say when a decision is final and when someone else owns the next step, rather than adding another "don't over-engineer" line to a rules file. Review shifts too, and the question about an agent's diff is not only whether the change is wrong, but whether half of it answers something nobody asked. And a vendor quoting only task completion is quoting the one number this paper shows conceals the problem.
The rates are where this will get over-read. Each agent here gets one clean decisive fact, where a real repository scatters the equivalent across a contributing guide, a CI config and a stale Slack thread; the authors flag that gap themselves. Read these numbers as a controlled instrument, not a forecast.
Sources
- Luo, H., Zhang, X., Xu, Z. et al., "ParanoiaEval: Benchmarking Unnecessary Defensive Work in Agentic Coding" — https://arxiv.org/abs/2610.08662