Rohan Paul (@rohanpaul_ai)T2·78 pts
- 82% of requirements sit with simulated teammates, so agents must actively communicate to scope work, testing information gathering rather than pure coding
- Top models pass about 63% of individual checks but fully finish fewer than 8 of 101 tasks, making end-to-end completion the main bottleneck
- The benchmark spans 101 real open-source repositories, 26 simulated teammates, 11 languages, 12 domains and 17 workplace tools