LoLBench Reveals Claude's 14% Success Rate in Large-Scale Code Delivery
Researchers from Fudan University and others introduced LoLBench to evaluate AI Agents on delivering features within large software systems averaging 2.4 million lines of code. Tests revealed that even advanced models like Claude achieved only a 14% success rate in cross-module implementation, highlighting significant gaps in handling long-horizon complex engineering tasks.
Full Report
Today, AI agents can proficiently call tools, modify files, and run tests. But when faced with a large-scale software system that evolves over time, can they start from a new requirement and deliver a complete feature?
When product managers propose new requirements, developers rarely receive just a list of files or function signatures. They must first understand constraints, trace module dependencies, and then determine what to change, how to change it, and how to prove the changes are correct. These analyses directly impact subsequent delivery.
Researchers from Fudan University, City University of Hong Kong, and The Chinese University of Hong Kong have introduced LoLBench. This benchmark places agents into software systems averaging approximately 2.4 million lines of source code to evaluate the entire process from feature proposal to module delivery. It specifically tests the agent's perception capabilities in grounding requirements within the system and its implementation capabilities in executing cross-module modifications.
Paper Link: https://arxiv.org/abs/2609.37143
Dataset: https://huggingface.co/datasets/lolbench26/LoLBench
Experiments reveal critical breakpoints: Agents may correctly modify local components but miss necessary synchronized changes in other modules. Even when provided with implementation guidance, most tasks remain unsolved, indicating a significant gap remains between understanding requirements and completing delivery.
Redefining Long-Horizon Code Tasks
What kind of long-horizon code task holds actual engineering value?
"Long-horizon" cannot be defined solely by duration, number of turns, or token consumption. For example, asking an agent to output a number every minute, counting from 1 to 1000, takes a long time but requires only a simple loop. Similarly, tasks like those in SWE-bench involving localized bug fixes do not cover the challenges ranging from requirement analysis to module delivery.
The long-horizon code tasks addressed by LoLBench focus on module development for complex systems. Upon receiving a new requirement, developers repeatedly analyze: Which behaviors must be implemented? Can the existing architecture support them? Which dependencies will be affected? How can we verify the changes? LoLBench incorporates this analytical process—which usually happens inside a developer's head—into the evaluation, requiring agents to independently derive the path from user intent to implementation plan.
These tasks simultaneously exhibit perceptual complexity and implementation complexity:
- Perceptual Complexity refers to the difficulty of reading proposals, identifying implicit constraints and cross-module dependencies, and deriving the scope of modifications and implementation specifications.
- Implementation Complexity refers to the difficulty of translating specifications into correct cross-file modifications while ensuring existing functionality is not broken through testing and repair.
Figure 1: Comparison of Perceptual Complexity and Implementation Complexity.
How LoLBench Evaluates Both Types of Complexity
Deriving Implementation Specifications from Feature Proposals
LoLBench selects 100 tasks from community Enhancement Proposals and their corresponding merged Pull Requests (PRs), covering 29 systems, 8 programming languages, and 5 domains.
Agents are provided only with the proposal and the code repository state prior to implementation. The proposals average about 5,000 English words. Evaluation is conducted in an offline Docker environment to prevent agents from retrieving upstream implementations.
For example, PEP 617 [1] proposes replacing CPython's LL(1) parser with a PEG parser. Implementing this requires integrating syntax definitions, parser generation, Abstract Syntax Trees (AST), and error reporting. Agents must identify the entire implementation chain.
Regarding perceptual complexity, the average mention rate of target class names from reference implementations in LoLBench proposals is only 7.3%. This is less than half of SWE-bench Verified (15.9%) and less than one-fifth of FeatureBench (38.7%). Agents must infer the connection between requirements and code on their own.
Translating Specifications into Engineering Deliverables
Regarding implementation complexity, LoLBench repositories average about 2.4 million lines of source code, which is approximately 9.4 times larger than SWE-bench Verified and 4.9 times larger than FeatureBench. The average implementation involves modifying 5,499 lines of code, compared to 38 lines and 2,050 lines for the latter two benchmarks, respectively. Each task involves an average of 64.9 files, 40.1 classes, and 195.9 functions modified; the file count is 25 times and 4.2 times higher than the respective baselines. Cross-module coordination and verification are significantly more difficult.
Table 1: Comparison of LoLBench with Existing Code Task Benchmarks.
LoLBench verifies delivery correctness using two types of tests:
- Fail-to-Pass (F2P) tests require failure before implementation and success after. F2P tests verify behavior through public interfaces, allowing for different internal implementations.
- Pass-to-Pass (P2P) regression tests require success both before and after implementation.
Building upon the project's original test suite, the research team constructed erroneous implementations as mutants (such as omitted features or tampered requirements) to supplement test cases that identify missed errors, thereby increasing the test coverage of the reference patches.
After enhancement, the detection rate of 1,688 mutants rose from approximately 61% to 96%, and the average line coverage for F2P tests per task increased from approximately 60% to 79%.
Performance and Bottlenecks of 28 Agent Combinations
Even with Top Models or Scaffolds, Max Resolution Rate is Only 14%
The research team evaluated 28 available combinations of 6 frontier models and 5 scaffolds, with a time limit of 6 hours per task. Opus 5 paired with either Claude Code or mini-SWE-agent achieved the highest resolution rate of 14%. The median resolution rate across all combinations was only 2%.
Table 2: Key experimental results from the paper.
When using Opus 5, mini-SWE-agent and Claude Code achieved identical resolution rates. However, mini-SWE-agent took 3.2 times longer and generated 5 times more tokens than Claude Code, indicating that higher resource consumption did not yield better performance.
Significant Interaction and Token Usage Spent on Code Localization
In terms of token consumption, averaged across agents, the localization phase accounted for 52.4% of non-cached input tokens and 45.4% of generated tokens. Similarly, the number of interaction rounds during the localization phase was significantly higher than in other phases.
Figure 2: Average number of interaction rounds per phase.
Compared against the reference implementation's modified files and functions, the file modification precision across all 28 agent groups ranged from 38.0% to 80.4%, with recall ranging from 30.3% to 65.0%. For function modifications, precision ranged from 54.2% to 65.6%, while recall ranged from 20.8% to 53.4%. This indicates low accuracy in code localization and incomplete coverage.
Agents Still Struggle to Deliver Complete Solutions Even with Additional Modification Guidelines
The research team retained the original feature proposals but additionally provided a modification file tree and API implementation specifications derived from the reference implementation to help agents determine the scope of changes. With these additions, the resolution rate for Opus 5 + Claude Code increased from 14% to 34%, GPT-5.6 Sol + Codex rose from 3% to 25%, and DeepSeek V4 Flash + OpenCode improved from 1% to 17%.
Table 3: Experimental results after supplementing with modification file trees and API implementation specifications.
While additional modification guidelines can alleviate cognitive load, the best-performing combination still left 66% of tasks unresolved. Agents must still coordinate modules, implement interfaces, and pass validation, suggesting that the complexity of implementing large-scale systems remains high.
From Code Generation to Large-Scale Software System Evolution
These results provide testable directions for agent development: Can repository retrieval cover critical dependencies? Are implementation specifications complete before coding begins? Can validation promptly detect cross-module omissions?
The value of software engineering lies in reliably fulfilling user requirements. Agents need to understand why changes occur, how they impact the system, and fully implement those changes. LoLBench aims to shift the evaluation of agent capabilities from mere code generation to supporting continuous software evolution, making delivery outcomes the true measure of agent capability.
References:
PEP 617: https://peps.python.org/pep-0617/
Chasing ASI Instantly
⭐ Like, Share, and "Watching" – The One-Click Trio ⭐
Star us to lock in instant updates from New Intelligence!
