Termsslug: data-contamination

Data Contamination

数据污染

Training data inadvertently containing test set content, leading to inflated evaluation results.

Curated Intelligence

8 items · Page 1/1
IT之家 · 智能时代T2·Models
78 pts
Artificial Analysis v5 Benchmark Launches; DeepSeek V4.1 Flash Tops Open-Source Coding

Artificial Analysis released Intelligence Index v5, shifting 60% of weighting to private test sets to mitigate data contamination. The new FrontierCode 2.0 benchmark evaluates code based on maintainer merge willingness rather than just unit tests. DeepSeek V4.1 Flash achieved the highest score among open-source models (28%), signaling a paradigm shift in AI coding evaluation from 'runnability' to 'engineering mergeability'.

Artificial Analysis (@ArtificialAnlys)T1·Models
78 pts
Artificial Analysis Launches Intelligence Index v5 with Private Benchmarks

Artificial Analysis released Intelligence Index v5, its largest upgrade since v4, shifting 60% of evaluation weight to private test sets or held-out answers to enhance robustness and reflect real-world usage. The update also introduces the Progress Index for continuous capability tracking and adds new benchmarks like FrontierCode 2.0 and Terminal-Bench-Science.

Artificial AnalysisT1·Models
78 pts
Artificial Analysis Launches Intelligence Index v5 with Private Datasets and New Code Benchmarks

Artificial Analysis released the biggest upgrade to its Intelligence Index since v4, now featuring 60% of evaluations based on private test sets or held-out answers to mitigate contamination. The update introduces two new benchmarks, FrontierCode 2.0 and Terminal-Bench-Science, alongside the launch of a new Progress Index to track model capability evolution over time.

Artificial Analysis (@ArtificialAnlys)T1·Models
68 pts
Artificial Analysis Launches FrontierCode 1.1 Leaderboard Evaluating Agents on Real-World Merge Standards

Artificial Analysis launched a leaderboard for Cognition's FrontierCode 1.1, a benchmark requiring models to produce code changes in real open-source repositories that maintainers would actually merge. By using private tasks and dual verification (unit tests plus qualitative review), it addresses the overfitting issues of traditional coding benchmarks with a production-aligned 'mergeability' metric.

OpenAI 官方博客T1·Models
68 pts
SWE-bench Verified Deemed Contaminated; Pro Recommended

Analysis reveals that the SWE-bench Verified benchmark suffers from significant data contamination and flawed tests, rendering it inaccurate for measuring frontier coding capabilities. Consequently, the research team recommends transitioning to the more rigorous SWE-bench Pro as the new evaluation standard.

One Useful Thing (Ethan Mollick)T2·Models
68 pts
Why AI Benchmarks Like MMLU-Pro Fail to Measure Intelligence

The article argues that mainstream AI benchmarks like MMLU-Pro are flawed due to data contamination, irrelevant questions, and poor calibration, failing to measure true intelligence. It advocates moving away from static 'human-like' testing towards dynamic, interview-style evaluation methods.

OpenAI 官方博客T1·Models
78 pts
OpenAI Releases MLE-bench to Evaluate ML Engineering Agents

OpenAI introduces MLE-bench, a benchmark derived from Kaggle competitions to evaluate AI agents on real-world machine learning engineering tasks. Results show that the best setup (o1-preview with AIDE) achieves at least bronze medal level in only 16.9% of competitions, highlighting current limitations in automated ML engineering. The benchmark is open-sourced to facilitate research into agent resource scaling and pre-training contamination.