Today for AI
HOT RADAR
agentHEAT 10.2°

AI CURATED HOT STORY · 11/4/2026

LoLBench Reveals Claude's 14% Success Rate in Large-Scale Code Delivery

1 reports archived1 independent sourcesupdated 11/4/2026, 18:03:00
Synthesis & Latest Updates
1 Sources Cross-Validated

Researchers from Fudan University and others introduced LoLBench to evaluate AI Agents on delivering features within large software systems averaging 2.4 million lines of code. Tests revealed that even advanced models like Claude achieved only a 14% success rate in cross-module implementation, highlighting significant gaps in handling long-horizon complex engineering tasks.

LATEST/Researchers from Fudan University and others introduced LoLBench to evaluate AI Agents on delivering features within large software systems averaging 2.4 million lines of code. Tests revealed that even advanced models like Claude achieved only a 14% success rate in cross-module implementation, highlighting significant gaps in handling long-horizon complex engineering tasks.

Hourly heat curve

8 fully observed hours
Now
10.2
Peak
11.719:00
24h change
–

The line compares only sources observed throughout; gap hours are interpolated to keep the trend continuous.

Coverage timeline

Total 1 reports · Latest first
  1. 新智元 (微信公众号)T2·78 pts

    LoLBench Reveals Claude's 14% Success Rate in Large-Scale Code Delivery

    • LoLBench focuses on end-to-end requirement delivery in massive codebases (~2.4M LOC) rather than local bug fixes.
    • Mainstream models like Claude struggle with cross-module consistency, achieving an overall task completion rate of only ~14%.
    • Experiments show that adding implementation guidelines alone fails to address fundamental flaws in how agents understand system constraints and dependencies.
LoLBench Reveals Claude's 14% Success Rate in Large-Scale Code Delivery | Today for AI