Apple Study: Agent Generalization Driven by Data Diversity Over Task Volume
A new Apple paper reveals that increasing the diversity of synthetic apps and database states yields greater generalization gains for tool-use agents than simply adding more tasks within a fixed budget. The study introduces SynthAPI to demonstrate how data composition, rather than volume alone, drives performance improvements.
Full Report
Rohan Paul (@rohanpaul_ai) · 2026-10-29 04
New Apple paper finds that, as training data grows, tool-use agents gain more from diverse apps and database states than from more tasks.
that the mix of synthetic training data, not just its size, decides how well tool-use agents generalize to new benchmarks.
Teams train tool-use agents on synthetic environments, and most pipelines focus on generating more tasks. It was unclear whether new tools, new tasks, or new starting data matter most.
They built SynthAPI, which has an LLM code synthetic apps with real Python tools and databases, then write tasks with automatic success checks. They varied apps, tasks, and database versions across 849 trained models.
At a fixed 32,768-task budget, more apps and database versions lifted a 4B model on AppWorld's challenge split from 38.4% to 45.0%. Their scaling law, which tracks data mix, predicted larger held-out runs with 0.0285 error, versus 0.1706 for a task-count-only law.
