Berkeley-Led Study Finds Self-Evolving Agent Harnesses Lose Solved Tasks
A new paper from UC Berkeley and other top labs finds that letting agents rewrite their own prompts, skills, tools, or code to raise benchmark scores hides tasks they lose along the way. Testing 4 evolution methods including GEPA and HyperAgents with GPT-5.4 across 2 harnesses and 2 benchmarks, all 16 evolved agents gained some tasks and lost others; one raised reliability by 5.2 points while dropping 10 previously covered tasks, and 8 of 11 more-reliable agents cost more per run. The authors urge tracking lost tasks, cost per run, and unseen-task results rather than average score alone.
Full Report
Rohan Paul (@rohanpaul_ai) · 2026-11-09 10
New University of California, Berkeley+other top labs paper finds that agents that rewrite their own harness to score higher also lose tasks they once solved, which the average score hides.
Harness evolution lets an agent edit its own prompts, skills, tools, or code while the model stays frozen. Results usually come as 1 higher benchmark score, which can hide lost tasks, higher costs, and gains that don't carry over.
They ran 4 evolution methods, including GEPA and HyperAgents, with GPT-5.4 on 2 harnesses and 2 benchmarks. Then they re-tested each evolved agent 8 times against its starting version.
All 16 evolved agents gained some tasks and lost others, and 1 raised reliability by 5.2 points while losing 10 tasks it used to cover. Of 11 agents that became more reliable, 8 also cost more per run.
When you evolve an agent harness, track tasks lost, cost per run, and results on unseen tasks, not just the average score.
