The appealing thing about a coding agent that writes its own tests is that it looks like a closed loop. The agent proposes a fix, writes a check, runs it, and stops when the check goes green. The patch arrives with evidence attached. A team at Microsoft Research and the University of Wisconsin–Madison measured what that evidence is actually worth.
They held one repair agent fixed and varied nothing but where its tests came from, across a standard set of real GitHub issues. With no tests at all it resolved 61.2% of the issues; handed tests written by a model of its own strength, it resolved 57.3%. The tests ran. They failed on the broken code and passed once the agent had patched it. The agent still finished the job worse than if it had never written them.
"The patch and the test agree, but they agree on the same incomplete interpretation of the issue."
The mechanism is not exotic. An agent that misreads an issue (fixes the ordinary case, misses the empty-list case) writes a test for the ordinary case as well, because one misreading produces both. The check then confirms the misreading and tells the agent it is finished. Judged on the single job a bug-fix test exists to do, fail on the broken code and pass on the official fix, fewer than a quarter of that same-strength model's tests managed both. Tests from a much stronger model cleared that bar most of the time, and helped rather than hurt.
The authors' answer is to split the roles. One agent writes the test, a harness freezes it, and a second agent may only change source code. They are unusually careful about what this buys.
"Here, independence refers to the generation context and write permissions, not statistical independence of Test and Repair errors: the two agents can still share a mistaken interpretation of the issue."
That marks the edge of what the work establishes. It shows that procedural separation stops an agent from editing away the check that is failing it. It can't say how often two agents cut from the same model quietly agree on the wrong requirement.
The consequence for a team is narrower than a blanket "agent-written tests are bad." Better tests plainly helped; what decided the outcome was who wrote them. A green test written by the agent that wrote the patch demonstrates consistency, not correctness, and consistency is what a reviewer already assumes. So treat an agent-authored test as part of the change under review rather than as the review of it, and discount any tool reporting "verified" on evidence it generated itself. The cheapest available check is the researchers' own qualifying step, which is to run the new test against the code as it stood before the fix. If it passes there too, it's decoration. That question can be asked of work a team has already merged.
Sources
- Tao, L., Peng, B., Wang, H. et al., "ExecCritic: Learn to Test, Test to Improve for Coding Agents" — https://arxiv.org/abs/2609.09133