Token spend has become a management number faster than anyone checked what it measures. Leaderboards of the heaviest users, spending targets and vendor dashboards that divide the AI bill by merged pull requests all assume the same thing: an engineer who spends more on coding agents produces more. Researchers at Carnegie Mellon, JetBrains and UC Irvine tested that assumption on a company's own usage data.

JetBrains routes every coding-agent call through an internal gateway that counts tokens per person. The team matched five weeks of those counts to the commits of 404 engineers, comparing each engineer's busy weeks with their quiet ones as well as engineers with each other. They also analysed an anonymous company survey about what people use the agents for. Use was voluntary, with no target or mandate.

Spend followed the person far more than the work. Who the engineer was explained most of the variation in weekly tokens, and the ranking of heavy and light spenders stayed largely the same from week to week. Among weeks with similar commit counts, the heaviest-spending weeks used 39 to 240 times the tokens of the lightest. Within the same engineer, a week with double the usual spend came with only about 10% more commits. Commits and lines changed together explained well under half of the differences in spend between engineers, and role and team on their own predicted it worse than simply assuming everyone spent the average.

"Token spend therefore measures how much engineers use agents across many kinds of work and not the outcome and value of that work."

Part of the reason is that commits are not what most agent use is for. In nearly nine in ten survey evaluations of the coding agents, technical staff said they had used them that week for something other than writing code, such as analysing data, research and planning. By the researchers' estimate, up to two fifths of spend does not move with the week's commits at all. They treat it as a ceiling, since it includes the fixed cost of using an agent at all and the spend behind commits written by hand. What the telemetry cannot say is how much of the rest was useful exploration and how much was the wrong, plausible-looking output that respondents named as their biggest time cost.

"A token spent on a prototype that informs a design counts the same as one spent on output that is discarded, so the count alone cannot separate the two."

The authors draw a firm conclusion from this: companies should not judge engineers' output by their spend, and should not expect spending targets to raise output much. The second half rests on an assumption the study cannot test. Engineers chose how much to use the agents, so the 10% figure is an association. A target imposed from above could do better or worse, and nobody in this data was under one.

The cost-per-commit dashboards come off worst, on this evidence. Because spend barely rises when commits do, tokens per commit fall automatically in a busy week, so an engineer looks more efficient whenever they ship more for any reason. The researchers also found that weeks on a newly released model used noticeably more tokens for the same number of commits, which means a company's ratio can worsen overnight without anyone changing how they work. A vendor reporting "return on token spend" is more likely reporting which model the team was on than anything about how it works.

The number still has uses. Because each person's spend is stable, the researchers note it is a reasonable basis for forecasting budgets, and it says plainly who is working through agents. The practical change is to stop letting it anywhere near performance conversations and to treat any efficiency ratio built on it with suspicion. If leadership wants to know what agent work is worth, the cheaper first step is the one the paper points towards, recording which ticket or task each agent session served so spend can be tied to outcomes rather than to commit counts. Capping heavy users is a blunter tool than it looks, on this data. The heaviest third account for most of the bill, and, as the researchers point out, the count gives no way to separate their productive spend from the rest.

This is one company that builds AI developer tools, so some metered use may be staff testing their own products. Output means commits only, which misses review, design and code quality, and five weeks is short. The study also cannot say whether the engineers who spend most produce better work, as distinct from more commits, which is the question a manager most wants answered.

Sources

  1. Agarwal, S., Lvova, O., Sergeyuk, A., Serova, A., Bryksin, T., Zimmermann, T., Vasilescu, B., "The AI Era's Lines of Code? A Case Study on What an Engineer's Token Spend Measures" — https://arxiv.org/abs/2611.13396