SWE-CoWork Benchmark Exposes Coding Agents' 8% Task Completion Rate
TokenWave AI has released SWE-CoWork, a benchmark that places coding agents in a simulated workplace where 82% of requirements live with teammates, forcing agents to ask the right person to learn what to build. Across 101 tasks drawn from real open-source repositories, the best models pass roughly 63% of individual checks but fully complete only 8 of 101, exposing long-horizon collaboration and information gathering as core weaknesses.
Related Entities
Full Report
Rohan Paul (@rohanpaul_ai) · 2026-11-17 16
Coding-agent benchmarks are starting to look like a new hire’s first week.
TokenWave AI has released SWE-CoWork, a new benchmark that tests whether AI coding agents can work like real team members.
Giving GPT-6 Astra the full spec up front lifted its passed checks by 16 points, yet it still fully finished only 8 of 101 tasks. SWE-CoWork puts coding agents in a simulated workplace modeled on real-world workflows, then links each failed check to a specific requirement.
Most coding benchmarks hand an AI agent a complete written task, but SWE-CoWork puts it in a simulated workplace where 82% of the requirements sit with teammates, so the agent only learns what to build by asking the right person. The best models pass about 63% of the individual checks yet fully finish fewer than 8% of 101 tasks.
↳ 引用 @TokenWaveai (TokenWave.AI): Can a coding agent take over a teammate’s work in progress?
Introducing SWE-CoWork, a benchmark for coding agents working as digital employees on long-horizon software engineering tasks.
Each assignment starts with a short handoff in a simulated workplace modeled on real-world engineering workflows. Agents search task boards and versioned docs, ask teammates for missing details, and revise their code when a reply changes what needs to be built.
Open questions and confirmed requirements can both be lost before the agent hands the work back. SWE-CoWork lets us trace them through the conversation, the patch, and the tests.
101 assignments across 101 real open-source repositories. 26 simulated teammates per workplace. 11 programming languages. 12 application domains. 17 workplace tools.
🌐 Website:
