The standard way to test whether a language model understands a code change is to show it two versions of a program and ask a yes-or-no question: do these behave the same? Researchers at Shahid Beheshti University and ETH Zurich think that question lets models off too lightly, so they asked two harder ones. If the behaviour changed, which edit changed it? And can the model work out, in order, every edit that turned one version into the other?
To know the right answers, they made the edits themselves. They took a few hundred short Python functions from a well-known teaching set and applied between one and five small, mechanical edits to each. Some were harmless, like renaming a variable, adding a line that never runs, or rewriting a loop. Others broke the code on purpose by flipping a plus to a minus, an "and" to an "or" or a greater-than to a less-than. Every changed pair contained exactly one breaking edit, confirmed by a failing test, so there was always a single correct culprit. Five models were tested, from large open-weight models down to small, cheap ones.
The yes-or-no question did its job on the weaker models, which got worse as edits piled up. The strongest stayed close to perfect however many edits were stacked. Asked to name the culprit, the field spread out. Mistral's small model judged equivalence correctly about 95% of the time, close behind the leaders, but identified the breaking edit in only about half the changed pairs.
"High equivalence accuracy therefore does not imply that a model can identify which change altered the semantics."
No model managed the third question. The best one reconstructed the full edit history for 39% of pairs, and with five edits stacked, no model got the history right more than about 1% of the time, including the two that had been near-perfect on the first two questions.
That is the result most likely to be over-read. Rebuilding the exact order of five edits is not a job human reviewers are asked to do either, and the researchers note how unforgiving the scoring is, since one wrong step fails the whole answer. They also argue that tracking edit history matters for debugging, repair and code review, but nothing in the study puts a model in any of those settings, so that link is their argument, not a finding. The middle question is the one that maps onto real work. "Something here changes behaviour" and "this line is the bug" are different review comments, and only the second one saves anyone time.
The prompt moved the results too. On the largest model, removing the instruction to reason step by step before answering took its accuracy at naming the culprit from about 93% to about 44%, with the drop steepest as edits accumulated. Same model, same programs. On this evidence, a code-review tool's configuration can matter as much as the model it is built on, and a vendor quoting a model's benchmark score has not told anyone how its own product is set up.
The study also caught a cheaper kind of failure. GPT-4.1-nano answered "these differ" almost every time, so it caught nearly every real change and was right on barely more than one in ten unchanged pairs. A review bot that flags everything looks excellent if the only number reported is how many real bugs it caught, which suggests any evaluation of one should report its false alarms on untouched code alongside it.
The authors name the main limits. The programs are short teaching exercises, the edits come from a fixed list of nine, everything is Python, and each model answered once. "Equivalent" means the tests still passed, not that equivalence was proven. What the paper doesn't address is that none of the five models is a current frontier coding model of the kind most teams' agents run on, so whether the gap between spotting a change and explaining it survives at the top of the market is untested. Real pull requests also mix many edits of many kinds, which would more likely widen the gap than close it, though that too is a guess.
"A model may correctly judge two programs as non-equivalent without knowing which transformation is responsible for the semantic difference or which sequence of transformations produced the modified program from the original."
The practical lesson for anyone buying or building AI review is to stop accepting "it detects behavioural changes" as the headline capability. The more useful test, on this evidence, is whether the tool can point at the line responsible, and it is cheap to run in-house. Take a known-good change, plant one small breaking edit among harmless ones, and see whether the review names the right line or just says something is off.
Sources
- Saghatchian, O., Mahzoun, A., Etemadi, K., Kheradpisheh, S. R., "SCRBench: Beyond Equivalence Checking for Semantic Change Reasoning in Code" — https://arxiv.org/abs/2610.41562