FrontierML Benchmark Shows AI Agents Can Buy Their Own Performance Gains
Researchers released FrontierML, a benchmark measuring how well agents convert capital into model performance across 21 end-to-end ML tasks. Each task gives a 12-hour CPU sandbox, a $10 to $200 wallet, and validation samples, and frontier systems already turn that budget into measurable self-improvement. The result highlights advancing AI R&D capability and hints at a machine economy where agents sub-contract work to one another.
Related Entities
Full Report
Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. If you’d like to support this, please subscribe. AI systems are getting good at sub-contracting to make themselves better:*…AI can now productively spend money to improve itself…*Researchers have developed FrontierML, a benchmark “that measures how well agents turn capital into model performance on challenging, end-to-end machine learning tasks.” It shows that today’s frontier systems are already able to, when given some money, turn this into improving their own performance on a basket of 21 distinct tasks.
21 long-horizon tasks “inspired by real work at frontier labs, neolabs, and academic institutions”. These tasks span six category areas, including LLM training, efficiency, vision & audio, computer use, applied NLP, and math & games. “Each task provides the agent with the same setup: a 12-hour CPU sandbox, a wallet with $10, $50, or $200, a few validation samples, and a task description.” Example task:Agents try to solve the tasks in pretty different ways. E.g, for one task about generating screenshots, the top-scoring agent fine-tuned a diffusion model using LoRA, while other high-scoring approaches involved an LLM writing a drawing program that is then rendered.
Key features:
- Scaling spending: “Give agents $10, $50, and $200 per task to spend on training compute and data, and evaluate whether agents can use the additional capital productively,” they write. “In our experiments, we find that stronger agents are able to reliably improve their submissions with greater access to capital.”
- Open-ended & end-to-end: “In each task, agents can choose architectures, source data, implement training, provision compute, evaluate checkpoints, and optimize inference,” they write. One notable observation is that agents do quite a lot of novel stuff; “in 28% of trajectories, agents add newly trained parts to pretrained models or train a new model from scratch.”
- Decent verification: “Agents submit an inference server that serves their model on an air-gapped H100 machine,” they write. “To disincentivize overfitting, we provide limited validation samples and evaluate on a large out-of-distribution held-out test set, not shown to the agent”.
Results: The authors measure performance by looking at the change in mean score over the 21 tasks when given additional money. Their baseline is a human researcher, which is able to get a 56 point boost when given $10 and a 66.1 boost when given $200. By comparison, the best performing system, Opus 5.5, gets a 45.4 boost with $10, and a 58.8 boost with $200. The highest ranking open weight model is Kimi K3.1 Pro which gets 25.5 with $10 and 29.6 with $200.
Why this matters - AI systems are not only going to be able to build themselves, but they can creatively sub-contract: When trying to solve complex problems, people don’t work alone. In fact, they often try to farm out bits of work to specialists around them to help make themselves more effective. FrontierML shows that today’s powerful AI systems are getting increasingly adept at doing this for ML research. This both highlights the advancing capabilities of these systems for AI R&D and also points towards a future machine economy where AI systems sub-contract out different bits of work to other systems.