Hugging Face 复现 ICML 2200 篇论文:智能体重塑科研验证
Hugging Face 举办黑客松,利用社区编码智能体在 19 天内成功复现了 ICML 2026 会议中约三分之一的论文(2,226 篇)。该实验揭示了在大规模 AI 研究背景下,人类角色正从直接执行者转向对智能体驱动的实验流程进行监督与验证。这一成果为自动化科研验证提供了重要数据支撑,凸显了智能体在处理海量学术主张时的潜力与挑战。
报道正文
本文目录13 个章节
通过复现 ICML 的 2,200 篇论文,我们学到了什么
发布于 2026 年 8 月 13 日 在 GitHub 上更新 点赞 120
早在七月,我们举办了一场 黑客松活动,超过 1,200 名社区成员带着自己的 coding agent(编程智能体)尝试逐条主张地复现发表在 ICML 2026 上的论文。在 19 天内,参与者发布了 6,816 份 Trackio 实验日志,复现了 2,226 篇论文,约占会议论文总数的三分之一 🤯
在这篇文章中,我们将分享从这次黑客松活动中获得的经验教训,以及当由智能体执行研究实验时,人类将扮演何种角色这一问题的启示。
论文数量远超任何人的审阅能力
关于 AI 研究究竟有多可复现的问题,其历史比当前的 AI 浪潮还要久远。但规模效应加剧了这些问题。ICML 2026 收到了 23,918 份投稿并接受了 6,352 篇论文,几乎是前一年的两倍,延续了至少部分由 AI 智能体加速实验运行和论文撰写所驱动的指数级增长趋势。
然而,审稿能力并没有随之翻倍。大多数会议的审稿人都是志愿者,他们可能没有足够的时间或专业知识来全面审阅一篇论文。以下是一位审稿人对一篇被接受为 ICML 2026 spotlight(焦点)论文的评审意见原文:
“我给出的低置信度分数是因为我没有仔细检查所有的证明。”
请注意,这篇论文获得了高分并被选为 spotlight。请记住这一点,因为我们在后文中会再次回到这篇具体的论文,并探讨当我们最终仔细检查证明时会发生什么。
然而,变化在于:推动投稿量激增的同一项技术,也能帮助我们跟上这一节奏。像 Claude Code、Codex、Cursor 和 Pi 这样的编码智能体(coding agents)现在能够阅读论文、编写代码、启动实验并汇报结果。过去,仔细审阅一篇论文可能需要审稿人花费一个周末;而智能体可以在一个下午内并行尝试数千次。
因此,我们想要探讨的问题是:如果我们真正大规模地重新审视一个顶级会议,并尝试复现每一篇论文,我们会发现什么?
黑客松活动(7月15日 - 8月2日)
我们没有自行审计论文,而是向整个社区开放了这一过程,从而引入了智能体框架、计算预算和科学品味等方面的多样性。从2026年7月15日到8月2日,ICML 2026 开放复现挑战按如下方式运作:
- 选择一篇论文。 我们对所有 6,341 篇 ICML 2026 录用论文进行了索引,提取了每篇论文的摘要及其核心科学主张,以便智能体能从一个具体、可验证的目标开始,而不是面对一份 40 页的 PDF。我们鼓励多人复现同一篇论文。
- 自带智能体。 参与者使用了 Claude Code、Codex、Cursor、OpenResearch 的
orx以及介于两者之间的各种工具。我们提供了一个简化的接口,使智能体能够通过一条命令获取论文、其主张和挑战说明。 - 复现并发布所有内容。 每次运行都会生成一个 Trackio 日志簿:这是一个静态的 Hugging Face Space,包含撰写报告、运行的代码、生成的产物,以及(可选的)作为 Hugging Face Dataset 上传的完整智能体执行轨迹。审计过程本身必须是可审计的。
- 接受评判。 一个自动化的日志簿评审员(Logbook Judge,运行开源权重模型 GLM-5.2)会重新阅读每个日志簿,并对每项主张给出裁决:已验证(verified)、被证伪(falsified)、玩具级(toy,即在缩减规模下的证据)或不确定(inconclusive)。评审员被明确指示将每个日志簿的自我评估视为不可信。
Participants received $20 in Hugging Face compute credits to run experiments on HF Jobs; across the challenge, participants launched 2,962 cloud jobs. Where a full reproduction was impossible, for example when a paper's dataset was proprietary or its checkpoints unreleased, participants ran toy reproductions on synthetic data mimicking the original's properties.
Here is what a finished reproduction looks like:
By the numbers, this hackathon was probably the largest attempted reproduction of a scientific conference:
- 1,221 community members joined the organization
- 6,816 reproduction logbooks published
- 2,226 papers attempted, 34% of the entire conference, many by several independent teams
- 35,908 claims judged, with all verdicts frozen in a public dataset at challenge close
- 2,962 HF Jobs launched; 274 full agent-trace datasets published on Hugging Face
What we found
Aggregating the claim-level verdicts per paper:
51% of examined papers (1,103) had at least one claim independently verified. Of those, 266 papers were fully reproduced, with every extracted claim verified, and 632 more were partially reproduced with nothing falsified. In total, 3,978 individual claims were confirmed with real experiments.
23% of examined papers (496) had at least one claim falsified or contested. That includes 49 papers where all claims were falsified and nothing could be verified, and, maybe most interestingly, 242 papers where independent reproduction teams reached opposite verdicts on the same claims. Reproducibility is not binary; it is adversarial.
The remainder sat in the middle: 502 papers with toy-scale evidence only, and 280 where nothing could be established either way (missing artifacts were the most common cause).
Reproductions done well
Some papers came through the gauntlet looking great, and the community's best logbooks are worth reading in their own right:
- "Flat Minima and Generalization: Insights from Stochastic Convex Optimization" was reproduced by 20 independent teams, 12 of which verified every claim. The one linked included and published the full agent trace.
- "A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness" had 14 of 17 logbooks verify every claim. A paper about unreliable LLM judges holding up under scrutiny by LLM agents :)
Falsifications, and what happened when we checked them
35 participants formally claimed they had falsified something. We adversarially re-verified every claimed falsification: re-reading the paper, re-reading the logbook, and re-deriving the math or re-implementing the experiment from the paper's own text.
A few of the confirmed falsifications, linking to the logbook that found it:
The paging paper from the introduction. The reviewer who did not check the proofs carefully? The paper, "Towards Optimal Robustness in Learning-Augmented Paging," claims its algorithm achieves robustness Hk+O(1)H_k + O(1)Hk+O(1). One participant's logbook measured the additive term growing like 0.38lnk0.38 \ln k0.38lnk and located the exact step of the proof that breaks. Our own re-implementation extended the sweep to k = 1,024 and confirmed the growth at roughly nine sigma. The true robustness is Hk+Θ(logk)H_k + \Theta(\log k)Hk+Θ(logk).
A theorem that falls after step 224. "Attention's forward pass and Frank-Wolfe" proves that token particles collapse to the origin whenever the origin starts inside their convex hull. Three independent teams found counterexamples, with violations first appearing at t = 224, ~3,800, and 6,416 steps, which neatly explains why everyone else "verified" the claim: finite-horizon checks stop too early. The cleanest counterexample is stated in exact rational arithmetic, so there is no floating-point ambiguity to hide behind. The authors confirmed the same day and are working on a fix.
[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]
An evaluation diluted by padding. In "Do Transformers Need Three Projections?", a participant discovered that ~66% of evaluated label positions were EOS padding tokens that train to near-zero loss, deflating perplexity roughly threefold. The abstract's "3.1% quality cost for 50% cache reduction" becomes roughly 9.4% once corrected.
🚨 False falsifications. Sometimes we found flaws in an attempted reproduction. One logboook claimed dramatically, "the paper's method is 2x slower than the baseline"; this turned out to be an arithmetic bug in the reproduction: per-trajectory time compared against per-batch-of-50 time. Correctly normalized, the participant's own data confirms the paper's claimed 8x speedup.
Talking to authors
We have begun writing to the authors of every confirmed finding, with a simple framing: here is what we found, here is all the evidence, do you agree or is our analysis wrong? The early responses have been very positive:
So far authors have confirmed findings on multiple papers, two arXiv corrections are in flight, and in one case an author had quietly fixed the error in a new arXiv version a month before the challenge found it, which we count as independent convergence 🤗
The role of humans
The most interesting question that this hackathon raises is: do humans still have a role in reviewing papers? We think so, for several reasons:
Pure agent execution hits real limits. Agents got stuck in local loops, misread scale-dependent behavior (several "verified" verdicts on the paging paper came from checks that stopped before the log-k growth became visible), and occasionally built an entire falsification on top of a units mismatch. The challenge's most reliable results came from workflows where a human was steering: re-pointing the agent, questioning an assumption, or deciding that an experiment's premise was wrong before burning a week of compute on it.
Some evaluation is irreducibly human, for now. Our human-in-the-loop winner is the clearest example. The paper claimed stable image generation under extreme quantization. Numerical metrics said "no collapse"; whether the images were actually usable was a perceptual question. The agent built a purpose-built review UI, and the human personally judged all 128 image pairs, with the annotations committed to the repo and the agent validating their consistency afterward. The published agent trace captures the whole exchange, down to the participant asking how the review tool works and coming back with "I have gone over the pairs and put the csv in the repo, please check."
So what are our roles as human reviewers? We think it is to manage intelligence effectively. Much like a professor or principal investigator (PI) sets up an environment where grad students can do good work, with compute, harnesses, data access, and targeted feedback at the right moments, the participants who got the most out of their agents were the ones who built the right environment and asked the right questions, then let the agents do the running.
Thank you
To the 1,221 people who joined, the winners, the authors who responded with grace, and our organizers at Hugging Face and alphaXiv: thank you. Every logbook, verdict, trace, and artifact from the challenge is public, starting from the challenge Space. We think this is the largest open, claim-by-claim audit of a machine learning conference to date, and we would love for it not to hold that record for long.
Stay tuned for future reproduction events. 🤗
Spaces mentioned in this article 8
More Articles from our Blog
[![]()
llmmoelong-context
DeepSeek-V4: a million-token context that agents can actually use
60 April 24, 2026](https://huggingface.co/blog/deepseekv4)[
agentssmolagentsnlp
Our Transformers Code Agent beats the GAIA benchmark 🏅
101 July 1, 2024](https://huggingface.co/blog/beating-gaia)
Community
It was interesting to reproduce one of the ICML accepted papers using these tools. This analysis from the hackathon was particularly insightful, and agent-based reproducibility has the potential to become a de facto standard for strengthening the credibility of claims in ML conferences. It could also serve as a valuable pathway toward refining autonomous scientific discovery, revealing not only where agents succeed, but also where they fail, providing important insights into their limitations and how human-agent collaboration can be improved.
Love it.
I have been working on a complementary experiment at https://ReproducibleAIList.org, though not the same scale. We've been taking models from Chip Huyen's GoodAIList.com and having a clean-room agent try to reproduce each one from scratch using the code repos (not the papers).
One other difference: every attempt runs under roar (python package roar-cli), which passively captures provenance rather than relying on code instrumentation like Trackio or W&B. That gives us the actual lineage of code, data, environment and artifacts, automatically producing an AI-BOM and a roar reproduce path others can independently repeat.
Would love to compare notes.
· abidlabs Article author Aug 18
Very cool, roar sounds like a great package! I'll take a look and see what we can learn from roar
gchauhan Aug 18 • edited Aug 18
Really enjoyed being part of this experiment. Gratifying to see our reproduction highlighted as one of the community’s best.
“A Coin Flip for Safety” was reproduced using AlphaXiv’s OpenResearch with OpenCode’s free BigPickle model, with experiments run locally on 2× NVIDIA RTX 3090s.
It was a great demonstration of how agent-assisted research can make independent reproduction and auditing accessible with relatively modest compute.
Thanks to the Hugging Face and AlphaXiv teams for organizing the challenge and making the whole process open and reproducible!
Thanks to the Hugging Face and AlphaXiv teams for organizing the challenge and making the whole process open and reproducible!
Really enjoyed being part of this experiment. Gratifying to see our reproduction highlighted as one of the community’s best. “A Coin Flip for Safety” was reproduced using AlphaXiv’s OpenResearch with OpenCode’s free BigPickle model, with experiments run locally on 2× NVIDIA RTX 3090s. It was a great demonstration of how agent-assisted research can make independent reproduction and auditing accessible with relatively modest compute. Thanks to the Hugging Face and AlphaXiv teams for organizing the challenge and making the whole process open and reproducible! One other difference: every attempt runs under roar (python package roar-cli), which passively captures provenance rather than relying on code instrumentation like Trackio or W&B. That gives us bilibili the actual lineage of code, data, environment and artifacts, automatically producing an AI-BOM and a roar reproduce path others can independently repeat. It was interesting to reproduce one of the ICML accepted papers using these tools. This analysis from the hackathon was particularly insightful, and agent-based reproducibility has the potential to become a de facto standard for strengthening the credibility of claims in ML conferences. It could also serve as a valuable pathway toward refining autonomous scientific discovery, revealing not only where agents succeed, but also where they fail, providing important insights into their limitations and how human-agent collaboration can be improved.
The scale of this experiment is honestly incredible 🤯... Reproducing 2,226 ICML papers in just 19 days shows how quickly coding agents are changing the research workflow. What once took researchers days or even weeks can now be tested in parallel, giving us a much larger picture of what actually holds up when the claims are checked carefully. 🔬💻
What I find especially interesting is the human side of this shift... Agents can read papers, write code, run experiments, and generate reports, but deciding what the results actually mean still requires strong human judgment. A successful reproduction is not automatically proof that every claim is correct, and a failed reproduction can have many possible explanations. 🤔📊
The point about reviewing capacity is particularly important... When thousands of papers are being submitted and reviewers simply cannot examine every proof, experiment, or implementation in depth, scalable reproduction could become an important second layer of scientific scrutiny. It could help researchers focus their limited time on the most meaningful questions instead of spending every hour on repetitive experimental work. 🔎⚙️
Most importantly, this experiment suggests that AI agents may not replace researchers so much as change where researchers add the most value... Humans may increasingly become the people who design better questions, evaluate evidence, investigate unexpected results, and decide which findings deserve attention. That combination of human curiosity + agent-powered experimentation could make scientific research far more rigorous and scalable. 🚀🧠✨
Interesting!
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here. Tap or paste here to upload images


