SWE-bench Verified Deemed Contaminated; Pro Recommended
Original title: Why we no longer evaluate SWE-bench Verified
68AI Score
Analysis reveals that the SWE-bench Verified benchmark suffers from significant data contamination and flawed tests, rendering it inaccurate for measuring frontier coding capabilities. Consequently, the research team recommends transitioning to the more rigorous SWE-bench Pro as the new evaluation standard.
Related Entities
Full Report
SWE-bench Verified is increasingly contaminated and mismeasures frontier coding progress. Our analysis shows flawed tests and training leakage. We recommend SWE-bench Pro.