Rohan Paul (@rohanpaul_ai) · 2026/10/12 16:17:00
JetBrains 研究指出编码智能体基准需匹配真实用户多轮任务流，SWE-TaskFlow 显示拆分对话致成本翻倍而成功率未稳
JetBrains Research 发布论文主张编码智能体基准应反映真实用户的多轮混合任务模式，而非单轮静态问题。基于 4782 个真实会话分析，研究者构建 SWE-TaskFlow 将 SWE-Bench Pro 任务拆分为多轮并增加代码检查点，结果显示在 700 个任务上智能体成本约翻倍但成功率无稳定提升，建议评估前先测量自身用户的实际请求分布以定制基准。
报道全文原始报道全文
Rohan Paul (@rohanpaul_ai) · 2026-10-12 08
New JetBrains Research paper argues that coding-agent benchmarks should copy how their real users actually work, because real sessions mix and switch task types across many messages.
Real developers using coding agents ask questions, plan, review, and switch tasks mid-session, but benchmarks like SWE-Bench Pro hand the agent 1 detailed issue and grade 1 final patch. In that benchmark, new features and bug fixes make up 93.8% of tasks.
JetBrains researchers studied 4,782 real agent sessions, where features and bug fixes were well under half of requests. In longer sessions, users kept switching between questions, planning, reviews, and edits.
They built SWE-TaskFlow, which splits each benchmark issue into several turns and adds checkable questions about the code. On 700 SWE-Bench Pro tasks, splitting roughly doubled agent cost with no stable change in success rate.
If you evaluate coding agents, first measure what your own users ask for, then shape the benchmark's conversation to match it.
– arxiv. org/abs/2610.09633
Title: "Coding-Agent Benchmarks Should Match Their Users' Task Flows"
