An agent handed a dependency upgrade makes the same small judgement over and over, deciding whether this version satisfies that constraint. Nobody had measured whether models can do it. Two researchers built 240 machine-checkable questions across npm, Python and Cargo, labelled each answer by running two independent resolver libraries and keeping only the cases where both agreed, then put six frontier models through them.
On the ordinary forms (a plain range, an exact pin, a simple wildcard) every model was at or near perfect. The failures were concentrated in a handful of rules. In the sharpest, a comparator missing its final digits rounds up, so >1 doesn't mean "anything above 1.0.0", it means "2.0.0 or higher". Every model tested, the strongest included, clustered near 60% on that rule in Cargo, barely above what answering "no" to every question would have scored, and for one model, at it. A version like 1.0.5 was waved into a range that excludes it.
"The rule is short and documented; the models simply do not apply it."
A second blind spot split by vendor. Python pads a bare 2 out to 2.0.0 before matching it against ==2.0.*, so the match succeeds. On a focused set of these corner cases, one model got all 26 wrong, calling every one a non-match. Another model got all 26 right. All six handled the plain form of that same rule without error. The authors read both traps as a failure to apply a rule rather than an absence of it, since pasting the rule into the prompt recovered most errors while instructing the model to reason in explicit steps did nothing. They say plainly that their design cannot separate reminding a model of something from teaching it.
The fix is almost embarrassing. Given the ecosystem's own resolver as a tool, every model answered essentially every question correctly. The library doing that work is free, exact, and already sitting on the machine the agent runs on. A couple of runs still came out wrong, for an instructive reason.
"The residual failure mode is thus not computation but ignoring or misreading a correct tool output: even with delegation, an agent must actually adopt the tool's answer."
None of this is about versions. Version constraints belong to a family that also holds cron expressions, glob patterns, timezone arithmetic and permission bits, all small formal languages with free, exact implementations, which an agent will cheerfully evaluate from memory instead. Nothing in a normal harness distinguishes a value the model computed from one it looked up. The output reads identically either way.
So the question for a team running agents on dependency work isn't which model to buy. It's which of these decisions the harness currently lets a model answer out of its own head. That's an audit of tool definitions rather than a model evaluation, and it's an afternoon's work. On this evidence, a vendor benchmark showing one model beating another at in-head version reasoning is measuring something that should never have been left to the model.
Demos won't surface it either, because the ordinary form came out right every time for every model. The failures live where a demo never goes, such as a bare major version, a post-release tag or a constraint someone typed without the patch digit.
One thing the study didn't do is follow a wrong answer through an agent into a broken build. It measured these judgements in isolation and says so, which leaves the error rate established and the cost of it unknown.
Sources
- Chen, Q., Liu, Z., "SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics" — https://arxiv.org/abs/2609.11180