Stanford and DeepMind's TURTLE Evolves Tool-Use Benchmarks to Stay Hard

1 reports archived1 independent sourcesupdated 11/9/2026, 07:36:48

Synthesis & Latest Updates

1 Sources Cross-Validated

A new Stanford and Google DeepMind paper introduces TURTLE, which uses AlphaEvolve to mutate saturated tool-use tasks from AppWorld by adding filters, extra steps and hidden lookups, targeting roughly 50% solver success. Gemma 4 31B-it always solved 69% of original tasks, while nearly all 735 evolved tasks landed at 40%-60% completion, and an LLM judge rated 52.5% of evolved tasks hard versus 19.8% of originals.

LATEST/A new Stanford and Google DeepMind paper introduces TURTLE, which uses AlphaEvolve to mutate saturated tool-use tasks from AppWorld by adding filters, extra steps and hidden lookups, targeting roughly 50% solver success. Gemma 4 31B-it always solved 69% of original tasks, while nearly all 735 evolved tasks landed at 40%-60% completion, and an LLM judge rated 52.5% of evolved tasks hard versus 19.8% of originals.

Hourly heat curve

2 fully observed hours

Fewer than 3 fully observed hours; trend not plotted yet.

Coverage timeline

Total 1 reports · Latest first

Rohan Paul (@rohanpaul_ai)T2Nov 9, 07:36·78 pts
Stanford and DeepMind's TURTLE Evolves Tool-Use Benchmarks to Stay Hard
- TURTLE mutates existing tasks with AlphaEvolve instead of building new benchmarks from scratch, cutting human effort
- It targets roughly 50% solver success while requiring each task to be solved at least once
- Evolved tasks are markedly harder: LLM-judged hard tasks rise from 19.8% to 52.5%
View Related Reports & Analysis Original Source