Deepnote Launches Agent Initiative Benchmark for Proactive Agents
Deepnote released the Agent Initiative Benchmark, which drops proactive agents into a simulated company Slack and scores whether they know when to intervene, what to say, and when to stay silent. GPT-6 Astra led at 8.97 out of 10, with GPT-6.1 Sol close behind at 8.65 at just $0.11 per task, versus $1.87 per task for third-placed Claude Fable 5.2. The open-source dataset and benchmark code give builders a reproducible basis for choosing models for proactive agent use cases.
Related Entities
Full Report
Rohan Paul (@rohanpaul_ai) · 2026-11-13 15
This is a beautiful concept for a benchmark.
Deepnote (@deepnotehq), an agent workspace company, released the Agent Initiative Benchmark, a test that drops proactive agents into a company Slack and scores whether they know when to jump in, what to say, and when to stay quiet.
GPT-6 Astra led with 8.97 out of 10, and GPT-6.1 Sol followed at 8.65 for $0.11 per task, against $1.87 per task for third-placed Claude Fable 5.2.
It runs every model through the same loop:
A thread starts in a simulated company’s Slack The agent decides whether to act at all It picks which of 11 services to check and investigates It posts and tags the right people, or stays silent Two LLM evaluators score the run 0 to 10 against a reference answer
↳ 引用 @jakubjurovych (Jakub Jurovych): What’s the best model for proactive agents?
Today, we’re introducing the Agent Initiative Benchmark. We tested 12 frontier models on realistic workplace tasks inside a synthetic company with 6 years of highly detailed history: it has a financial model, CRM records, past-due invoices, Linear tickets, and real-life events affecting company development - competitor product announcements, SVB collapse, etc.
We used the company’s Slack to test how a proactive agent listening in on each conversation would assist the team in day-to-day tasks: from stepping in to align initiatives across teams to warning the team not to use an unapproved customer logo.
Highlights:
- 6.1 Sol and Astra from @OpenAIDevs beat Anthropic's latest Fable 5.2, with Sol crushing it at both performance and cost ($0.11 per task vs $1.87).
- The most expensive models aren't the best: Fable 5.1 is 430x more expensive than GPT-6 Luna, with similar performance.
- Most models try to help even if they're not needed. Interestingly, in these failure cases, they spend more time thinking, and the unnecessary work takes about 80% more time and money than if they stayed quiet.
If you’re building agents, keep reading for more tips on which model is best for your use case. Detailed breakdown, the open-source dataset, and benchmark code below:
