Artificial Analysis released Intelligence Index v5, shifting 60% of weighting to private test sets to mitigate data contamination. The new FrontierCode 2.0 benchmark evaluates code based on maintainer merge willingness rather than just unit tests. DeepSeek V4.1 Flash achieved the highest score among open-source models (28%), signaling a paradigm shift in AI coding evaluation from 'runnability' to 'engineering mergeability'.
Data Contamination
数据污染
Training data inadvertently containing test set content, leading to inflated evaluation results.
Curated Intelligence
Artificial Analysis released Intelligence Index v5, its largest upgrade since v4, shifting 60% of evaluation weight to private test sets or held-out answers to enhance robustness and reflect real-world usage. The update also introduces the Progress Index for continuous capability tracking and adds new benchmarks like FrontierCode 2.0 and Terminal-Bench-Science.
Artificial Analysis released the biggest upgrade to its Intelligence Index since v4, now featuring 60% of evaluations based on private test sets or held-out answers to mitigate contamination. The update introduces two new benchmarks, FrontierCode 2.0 and Terminal-Bench-Science, alongside the launch of a new Progress Index to track model capability evolution over time.
Artificial Analysis launched a leaderboard for Cognition's FrontierCode 1.1, a benchmark requiring models to produce code changes in real open-source repositories that maintainers would actually merge. By using private tasks and dual verification (unit tests plus qualitative review), it addresses the overfitting issues of traditional coding benchmarks with a production-aligned 'mergeability' metric.
Analysis reveals that the SWE-bench Verified benchmark suffers from significant data contamination and flawed tests, rendering it inaccurate for measuring frontier coding capabilities. Consequently, the research team recommends transitioning to the more rigorous SWE-bench Pro as the new evaluation standard.
The article argues that mainstream AI benchmarks like MMLU-Pro are flawed due to data contamination, irrelevant questions, and poor calibration, failing to measure true intelligence. It advocates moving away from static 'human-like' testing towards dynamic, interview-style evaluation methods.
OpenAI introduces MLE-bench, a benchmark derived from Kaggle competitions to evaluate AI agents on real-world machine learning engineering tasks. Results show that the best setup (o1-preview with AIDE) achieves at least bronze medal level in only 16.9% of competitions, highlighting current limitations in automated ML engineering. The benchmark is open-sourced to facilitate research into agent resource scaling and pre-training contamination.
LiveCodeBench has launched a leaderboard for code LLMs designed to address data contamination in traditional benchmarks. By dynamically updating problems and strictly isolating training data, it offers more authentic and holistic evaluations of model capabilities.