Every codebase has a corner written for a version of the language nobody runs any more. It still works and it looks its age, and pasting a file of it into a chat assistant with the words "modernise this" is about the cheapest clean-up there is. A new study asks the question that shortcut skips: is the result actually better than the file you started with?
The researchers took 100 files from WordPress 4.0, released in 2014 and written before PHP 7, and asked five language models to upgrade each one to PHP 8.3. The instruction was explicit about the catch, telling the model to keep inputs, outputs, side effects and the way other code calls the file exactly as they were. Each model saw one file, with no surrounding project, no tests and no way to run anything, and only its first complete answer was scored. Two baselines sat alongside: the untouched original, and Rector, a rule-based tool that applies known upgrade patterns mechanically. Every version faced the same hidden behavioural checks, written before anyone saw a model's output.
Leaving the code alone turned out to be a strong position. Untouched, 83 of the 100 files already passed every check; the best model left 59, and Rector left 82. Every model broke several times as many working files as it fixed. Rector repaired about as many failing checks as most of the models did while breaking almost nothing, so its score did not come from simply leaving files alone.
"A modernized file is an improvement only if it beats the original."
Most of the breakage happened where nothing needed fixing. Only 7% of the outdated constructs the researchers catalogued had to change to run under PHP 8.3; the rest still ran, most of them without so much as a warning. In files that needed no required change at all, every model broke more than a third of the ones that had been working, and Rector broke very few. The model that did the most modernising by the study's own count was no more likely than its closest rivals to leave a file working.
Most of the damage was not subtle either. The largest share of failures came from files that no longer loaded at all, often because a model had added a type declaration PHP itself forbids, such as a return type on a magic method that the language requires to return nothing.
"Avoiding this error requires knowledge of PHP, not of WordPress."
The usual defence of a disappointing result in single-file work is missing context, on the grounds that a model which cannot see the callers is bound to guess wrong. Here, the researchers show that the most damaging failures could have been avoided using nothing but the file and PHP's own rules, and adding a basic syntax check that fell back to the original on failure recovered only part of the gap.
For a team, the implication runs beyond PHP. A modernisation diff is easy to approve, full of newer syntax and stricter types. On this evidence, a diff that looks more modern is no sign of a better file, and the newer syntax is where the risk sits. A reviewer who reads new type declarations as an improvement rather than as fresh claims about what the code accepts is waving through the kind of change that broke most often here. The safer brief is narrower than "bring this up to date": change only what the new runtime forces, and treat every optional edit as an unrequested rewrite.
It also changes what a vendor's modernisation claim should be measured against. A percentage of outdated patterns removed says nothing about whether the code still works, and this study found no sign that doing more of one meant more of the other. The question to put to any tool, or any internal pilot, is how many working files it broke against simply leaving them alone. If nobody has counted that, the number on the slide is measuring activity.
There are clear limits on how far this travels. The models tested are a generation behind today's, each produced a single answer per file under one prompt, and none could run its code or try again, which is not how a coding agent works in a real repository. The authors themselves say it is untested whether telling the models to make only required changes would close the gap. It is equally open whether an agent with tests and an execution loop would catch these failures or simply iterate around them. What the study does establish is the baseline any such claim has to beat, which is the code as it already stands.
Sources
- Mahmud, A., Khondaker, A. H., Ahmed, S. et al., "When Modernization Breaks Working Code: A Single-File Study of LLMs on Legacy PHP" — https://arxiv.org/abs/2610.40269