Nvidia Proposes New Base Model Agent Evaluation: Scoring Only the First Passing Edit Yields 0.964 Correlation with SWE-bench

1 reports archived1 independent sourcesupdated 10/12/2026, 23:39:55

Synthesis & Latest Updates

1 Sources Cross-Validated

A new Nvidia paper reveals that scoring base models at every step of an agent run is ineffective, whereas evaluating them only on the specific edit that first makes tests pass accurately predicts post-training performance. By replaying successful trajectories to identify this decisive step, the method achieves a 0.964 rank correlation with SWE-bench Verified scores, offering a low-cost way to select base models for agentic fine-tuning.

LATEST/A new Nvidia paper reveals that scoring base models at every step of an agent run is ineffective, whereas evaluating them only on the specific edit that first makes tests pass accurately predicts post-training performance. By replaying successful trajectories to identify this decisive step, the method achieves a 0.964 rank correlation with SWE-bench Verified scores, offering a low-cost way to select base models for agentic fine-tuning.

HEAT TRENDHourly heat curve

4 fully observed hours

Now

15.3

Peak

16.601:00

24h change

–

The line compares only sources observed throughout; gap hours are interpolated to keep the trend continuous.

TIMELINECoverage timeline

Total 1 reports · Latest first

Rohan Paul (@rohanpaul_ai)T2·Oct 12, 23:3978 pts
Nvidia Proposes New Base Model Agent Evaluation: Scoring Only the First Passing Edit Yields 0.964 Correlation with SWE-bench
- Scoring base models at every step of an agent run yields poor discrimination and fails to predict capability effectively.
- Evaluating base models exclusively on the 'first passing edit' significantly improves prediction accuracy for post-training outcomes.
- The proposed method shows a 0.964 rank correlation with post-trained SWE-bench Verified scores, serving as a low-cost selection tool.
View Evidence Original Source