AI Coding Agents Complete Under 10% of Simulated Workplace Tasks

By Jonathan KemperOriginal title: AI coding agents ace isolated tests but complete fewer than one in ten tasks in a simulated workplace
78AI Score

A new experiment places frontier coding agents such as Claude Opus 5.5 and GPT-6 Astra inside a simulated software company with colleagues, project boards, and chat histories to test them as digital employees. Despite near-saturated scores on isolated benchmarks like SWE-bench, the agents fully completed fewer than one in ten tasks, exposing a wide gap between benchmark performance and real workplace collaboration.

Full Report

AI coding agents ace isolated tests but complete fewer than one in ten tasks in a simulated workplace

Image description

ChatGPT-5.6 prompted by THE DECODER

AI coding agents are nearing top scores on standard benchmarks. A new experiment puts them to work for the first time as digital employees inside a simulated software company with colleagues, project boards, and chat histories. Even Claude Opus 5.5 and GPT-6 Astra fully complete fewer than one in ten tasks.

Success stories about AI coding agents keep piling up. Anthropic says Claude writes more than 80 percent of the company's production code, and standard benchmarks like SWE-bench are nearly saturated.

THE DECODER+