SciCode feeds an industry intelligence index, sits in the UK AI Security Institute's evaluation suite, and runs as a standing check at national laboratories. It hands a model a research problem in physics or chemistry and asks it to carry the derivation through and write numerical code that works. Its scores have been going nowhere, with today's strongest models bunched within a few points of each other, and one flagship tied the predecessor it was meant to beat.
"An instrument that cannot tell a flagship from its predecessor leaves only two explanations for its low scores: either scientific coding genuinely lies beyond today's models, or the instrument itself is miscalibrated."
A team of physicists took the second option seriously and re-derived every test problem by hand. They found more than 260 defects, and roughly three-quarters of them reject correct answers. The damage touched nine problems in ten. Most weren't typos. The single largest category was an answer key that silently demanded a convention the question never specified. Choose the other equally valid sign or index and you were marked wrong. Catching that requires knowing the physics, which is why the errors survived so long. One problem turned out to be unanswerable, since no method reproduces its expected answer, so every published score on this benchmark includes a question nobody could ever pass.
The scoring rules turned scattered errors into a wall. Each problem is a chain of steps, and it only counts if every step passes.
"Since every model meets the same broken steps, a defect shared by all models becomes a ceiling shared by all models—which is precisely the compression and stagnation observed on the leaderboard."
Corrected, the results are unrecognisable. Models that had been solving somewhere between a tenth and roughly a quarter of the whole problems now solve two-thirds to more than nine in ten. The order changed too, and the model that led on the original benchmark placed sixth on the corrected one.
The lesson travels well past scientific coding. A plateau is a statement about two things at once, the models and the ruler, and the industry habit is to read it as though only the first can move. When the top of a leaderboard compresses and a new flagship can't separate itself from last year's, the first suspect should be the test, not the models.
That matters more inside a company than outside it. Most teams running coding assistants now keep some golden set, a folder of tasks with expected outputs, used to decide whether to upgrade a model or switch vendors. It has exactly this failure mode, minus the physicists. Hand-written evals are built almost entirely from answer keys encoding one engineer's idea of the right structure and the right error message. When a model scores badly on that set, read the failures before the total, because a good share of them will be the test being wrong rather than the model.
It should also change how a vendor's benchmark claim lands. A score is not a property of a model; it's the output of software somebody has to maintain, and here an audit reshuffled the ranking outright.
There are two ways this finding will get over-read. It doesn't follow that every disappointing score is a broken test. The paper's harder condition, where the expert write-up is withheld and the model has to reconstruct the science itself, still leaves a wide gap, and that setting is the one closest to how anyone actually works. The audit also grades its own homework.
"Although the audit underwent multiple rounds of review, all reviews were conducted by the authors and therefore do not constitute external blind validation."
A team publishing a dramatic recovery has every reason to correct generously. Their answer is that they shipped the full change log, so anyone can check the work. That's the transferable practice here, a test that arrives with a record of why each answer is the answer. Benchmarks are code. They rot like code, and hardly anyone is paid to maintain them.
Sources
- Hu, S., Huang, L., Deng, Y. et al., "SciCode-Verified: How Benchmark Defects Underestimated the Scientific-Coding Ability of Language Models" — https://arxiv.org/abs/2608.04975