Rohan Paul (@rohanpaul_ai) · 10/12/2026, 16:17:00
JetBrains Study: Coding-Agent Benchmarks Must Mirror Real User Task Flows; SWE-TaskFlow Doubles Cost Without Stable Success Gains
JetBrains Research argues coding-agent benchmarks should reflect real users' multi-turn mixed task flows rather than single-turn static issues. Analyzing 4,782 real sessions, they built SWE-TaskFlow to split SWE-Bench Pro tasks into multiple turns with code checkpoints, finding that on 700 tasks agent costs roughly doubled without stable success rate improvements, recommending benchmark customization based on actual user request distributions.
SOURCE COVERAGEFull Original Report
Rohan Paul (@rohanpaul_ai) · 2026-10-12 08
New JetBrains Research paper argues that coding-agent benchmarks should copy how their real users actually work, because real sessions mix and switch task types across many messages.
Real developers using coding agents ask questions, plan, review, and switch tasks mid-session, but benchmarks like SWE-Bench Pro hand the agent 1 detailed issue and grade 1 final patch. In that benchmark, new features and bug fixes make up 93.8% of tasks.
JetBrains researchers studied 4,782 real agent sessions, where features and bug fixes were well under half of requests. In longer sessions, users kept switching between questions, planning, reviews, and edits.
They built SWE-TaskFlow, which splits each benchmark issue into several turns and adds checkable questions about the code. On 700 SWE-Bench Pro tasks, splitting roughly doubled agent cost with no stable change in success rate.
If you evaluate coding agents, first measure what your own users ask for, then shape the benchmark's conversation to match it.
– arxiv. org/abs/2610.09633
Title: "Coding-Agent Benchmarks Should Match Their Users' Task Flows"
