Getting an AI to write tests looks like the one job where the machine can check itself, since a test either runs or it doesn't, and it either catches a bug or it doesn't. That premise sits behind a whole class of systems that generate tests, run them, and improve each round.
Closing the loop needs an answer key, and the usual shortcut is to push the generated input through one program already known to be correct and treat whatever comes out as the truth. Six researchers, mostly at the Chinese Academy of Sciences, took that shortcut apart. They had two models write tests for several hundred programming tasks, then re-ran the selected tests against two more independently written correct programs and discarded anything the three didn't agree on.
The model's own judgement went first. Asked to supply both a test input and the expected output, the weaker model produced an answer all three correct programs agreed with about 28% of the time. The stronger one managed roughly half.
The shortcut fails because one reference program will happily accept an input the specification never allowed, produce something for it, and hand back a confident answer. A test built on that answer disagrees with a pile of broken submissions, and takes the credit for catching them.
"Optimization can then reward tests that disagree with many wrong submissions for the wrong reason."
Measured against one program, refining the tests worked, scoring about twelve percentage points better than a single attempt. Audited against all three, nearly all of that gain vanished. Given the same budget of candidate tests, simply generating tests three separate times beat the refine-and-mutate loop in every condition tested. A blinded review by two software engineering researchers judged around 94% of the discarded tests to be junk, inputs outside what the problem ever allowed, which is the check on whether the audit itself was sound.
The authors then built the control most self-improvement papers skip, a placebo with the same three-round loop, the same history and the same tally of bugs caught, but with the mapping between each test and the bugs it actually found randomly scrambled. The placebo reproduced the real loop's results almost exactly. Their reading is that the gain came from the shape of the interaction rather than the content of the feedback, and they say plainly that their design can't settle whether precise feedback carries some smaller value underneath.
"This does not show that execution feedback is universally useless."
A loop can look like it's learning when it's only getting better at finding its grader's blind spots, and nothing inside the loop can tell the difference. The same move is available to any agent scored by a checker it can influence, whether that's a coverage number, a lint pass, a security scan or another model's sign-off.
The defence is cheap. One extra independent implementation caught most of the inflation, so this is an offline check on a tool, not a production cost. When a vendor shows a self-improving curve, ask who wrote the answer key and whether anyone graded the grader. And when an iterative loop is pitched against a baseline, insist the baseline spends the same budget on independent attempts.
"Execution is useful feedback only inside a valid measurement contract."
This will get over-read the moment someone applies it to repository tests. Everything here is competitive-programming puzzles in Python, and both models come from one family; the authors want an independent model tried next. A real codebase has a project suite, a build and reviewers, a far better contract than one program, and the study can't say how much of the effect survives that. What travels is the discipline. Freeze what you selected before you audit it, and never let the thing being optimised also define what winning means.
Sources
- Liang, Y., Gan, C., Ying, R., Wei, H., Cui, Z., Ni, S., "Auditing and Decomposing Feedback-Driven Evolution in LLM Test Generation under the Oracle Problem" — https://arxiv.org/abs/2608.19626