CloudOpsBench Reveals Reasoning as Key Bottleneck for AI Cloud Ops
The University of Michigan team released CloudOpsBench, an open-source benchmark evaluating AI agents on AWS DevOps tasks. Top models achieve only 50-60% success rates, with 80% of failures attributed to reasoning errors (e.g., misjudging scope, incomplete checks) rather than knowledge gaps. This highlights a critical weakness in current agents for complex infrastructure operations.
Full Report
New open-source benchmark reveals reasoning as the biggest bottleneck for AI in cloud operations
[Maximilian Schreiner](https://the-decoder.com/author/maximilian-schreiner/) [View the LinkedIn Profile of Maximilian Schreiner](https://www.linkedin.com/in/maximilian-schreiner-b6a5171aa/) Nov 4, 2026 CloudOpsBench is a new open-source benchmark that measures how well AI agents handle typical DevOps tasks in the cloud, from fixing services and removing resources to restricting access rights, rolling out updates, and decommissioning projects. Version 1.0 includes 45 tasks on AWS and was built by experienced DevOps engineers.
Even the best models fail at many of the tasks. Claude Fable 5.1 solved 60 percent, GPT-6 Astra managed 53 percent. According to the team led by Ang Chen at the University of Michigan, the main problem isn't a lack of knowledge but reasoning. About 80 percent of failures happen because agents misjudge the scope of a task, don't check all the relevant services, or stop too early. Roughly 20 percent of errors are wrong actions, some with irreversible consequences for the infrastructure.
No model completes every task, and in most cases the bottleneck is reasoning. | Image: CloudOpsBench
All tasks run identically in the real AWS cloud and in an emulator. Extensions for GCP, Azure, and OCI are planned. The project on GitHub is licensed under Apache 2.0, and a detailed failure analysis is also available.