CMU and JetBrains Study Finds Token Spend Poorly Predicts AI Agent Output
A joint Carnegie Mellon and JetBrains study finds that engineers' token spend is only weakly linked to shipped output: for the same engineer, doubling spend corresponded to roughly 10% more commits. The work linked five weeks of coding-agent token usage to commits from 404 JetBrains engineers, backed by a 412-person company survey. The authors recommend using token spend only for budgeting and judging agent-assisted work by task and outcome evidence.
Full Report
Rohan Paul (@rohanpaul_ai) · 2026-11-11 17
New Carnegie Mellon and JetBrains paper finds that token spend shows how much engineers use AI agents, not how much they ship.
Companies now track engineers' token spend to set budgets and judge productivity, and some even rank top users. No study had followed both spend and output for the same engineers over time.
They linked 5 weeks of coding-agent token use to the commits of 404 JetBrains engineers. They also analyzed a company survey of 412 employees, where technical staff reported using agents beyond coding in nearly 9 of 10 evaluations.
For the same engineer, doubling spend went with only about 10% more commits. Among weeks with similar commit counts, the 90th-percentile week used 39 to 240 times the tokens of the 10th-percentile week.
Use token spend only for budgeting, and judge agent-assisted work by evidence about tasks and outcomes.
