Termsslug: swe-bench-lite

SWE-bench Lite

Condensed benchmark of real GitHub software-engineering tasks, widely used to evaluate coding agents.

Curated Intelligence

2 items · Page 1/1
IT之家 · 智能时代T2·Agents
78 pts
Google Lets Agents Rewrite Their Own Harness to Boost Task Success Eightfold

Google researchers propose a self-improving harness architecture in which an optimizer model directly edits the agent's runtime framework code, keeping only changes that pass validation. On 40 SWE-bench Lite tasks, the modular approach lifted success from 5.0% to 40.0% at roughly 46.7K tokens per task, far below the 451.6K tokens needed for unconstrained full rewrites. The paper also flags risks of prompt bloat, benchmark overfitting and evaluator tampering.

Rohan Paul (@rohanpaul_ai)T2·Agents
78 pts
Google Study Finds Gated Modular Edits Cut Agent Harness Token Cost by 90%

A new Google study finds that self-improving agent harnesses work best when an optimizer model edits one code module at a time under validation and token caps, rather than performing an unchecked full rewrite. On 40 held-out SWE-bench Lite tasks, the gated approach reached 40.0% accuracy at 46.7k tokens per task, versus 42.5% at 451.6k tokens for a full rewrite, nearly 10x the cost. Enforcing rules via prompts instead of code collapsed accuracy to 2.5%.