936 AI Agents Publish Nearly 1,000 Papers in 12-Week Autonomous Science Run

Original title: “Agent版arXiv”来了!936个智能体12个星期狂发近千篇论文,连同行评议都整上了
78AI Score

A research team from alphaXiv, Stanford, MIT and other institutions ran a 12-week experiment placing 936 agents in a shared preprint server and chat space, with 35 human labs supplying only compute, API budgets and broad research directions. The agents autonomously generated hypotheses, wrote and ran experiments, analyzed data and authored papers, producing 936 papers and 9,500 messages across 229 self-created chat rooms, while spontaneously developing replication checks, peer review and even their own journal. The platform is now open for external researchers to connect their own agents.

Full Report

Contents3 sections

They even learned to fact-check their peers~

henry reporting from 凹非寺

QuantumBit | WeChat Official Account QbitAI

What does a scientist society composed entirely of AI look like?

Recently, a research team from alphaXiv, Stanford, MIT, and other institutions actually conducted such an experiment:

They put 936 agents into the same online research community and let them freely evolve for a full 12 weeks.

The rules were simple: the research team set up a shared preprint server and chat rooms, and 35 human laboratories were only responsible for providing compute, API budgets, and broad research directions.

The rest—proposing hypotheses, writing code and running experiments, analyzing data, writing papers, and even peer communication—was left entirely to the agents to complete autonomously.

As a result, over 12 weeks, these nearly 1,000 agents published a total of 936 research papers on the platform, spanning multiple disciplines such as machine learning and solar physics.

Not only that, they also generated 9,500 discussions across 229 self-built chat rooms, reading each other's papers and reproducing experiments every day.

Once a problem was found, they would, like the human academic community, take the other party's code, rerun the experiment, and then continue exploring based on their "peer's" results.

After all this competition, these agents even started their own academic journal, established a review mechanism, and began gatekeeping papers from other AIs...

A true AI top conference~

And it is worth mentioning that, in addition to the experimental results above, this server for agent scientists to communicate is now open, and other researchers can also connect their own agents and submit research results there.

Next, let's take a closer look.

How does this research community with only AIs operate?

Overall, this experiment simulated the distributed ecosystem of the real scientific world:

The 35 participating independent laboratories each configured their own hardware, chose models and runtime frameworks, and then connected to the same shared workspace.

Each laboratory could independently choose different fields and directions, and then communicate through the public paper library and chat rooms to find work they could continue.

To enable research from different laboratories to connect with one another, the team first established a shared workflow:

  1. Before agents start working, they must first consult existing papers in the community and publicly announce their hypotheses and computational plans.

In laboratories that shared GPUs early on, they even coordinated device usage and parameter testing in public channels to avoid meaningless duplicate batch runs among peers.

  1. When submitting papers, each study was required to include runnable code, raw data, and a "Claim Ledger" linking the paper's figures to experimental outputs.

Through these records, other agents could precisely trace the data files and fields corresponding to each metric, verify the computation process, and directly run follow-up experiments on that basis.

At the very beginning of the experiment, that is, when it was launched on August 19, only 1 laboratory participated; from October 15 to 25, 5 more laboratories joined one after another, and after October 25, the final total reached 35 laboratories.

As participants increased, the agents' communication also gradually split from the initial public channel into topic-based chat rooms built around specific questions.

Here, they shared intermediate results at any time, questioned experimental designs, or directly took peers' code to reproduce results.

For example, six laboratories once carried out a relay around "how to reduce English forgetting when a model learns Korean":

One agent reported that a certain method could greatly reduce forgetting, and another quickly pointed out in the chat room that the two methods had learned Korean to fundamentally different degrees, and the comparison conditions were not aligned.

After receiving the feedback, it immediately reran the peer's code and corrected its judgment of the effect.

Subsequent laboratories continued along this line to dig deeper into the causes and test alternative approaches, and finally another agent summarized the experimental outputs of the entire research line.

More importantly, in this society of agent scientists, "failed research" was also incorporated into knowledge accumulation.

The platform rules gave rigorous negative results (null/zero results) the same academic recognition as positive results. As a result, other agents could use them to avoid dead ends that had already been disproven, greatly reducing repeated trial and error.

In addition, as more and more laboratories joined, the agents' submission volume also began to rise, and the agent scientists began to organize more formal quality review.

An agent named Cypress saw that other laboratories had uploaded a batch of placeholder "low-quality papers" in a short period of time, so it proactively created a review chat room and prepared to make accept, revise, or reject decisions on submissions.

After waiting four days and still failing to recruit volunteer reviewers, it simply established a standing large-model review panel and officially launched the Journal of Agent Research (JAR).

Specifically, the review panel called different models to separately oversee methodology and statistics, reproducibility, and research significance, and specially set up a "devil's advocate" role.

Every criticism had to quote the original text of the paper verbatim, and then a script automatically verified the quoted content to prevent the reviewing AI from making things up.

From November 4 to 9 alone, in just 5 days, this system generated 1,408 review comments for 352 versions of 263 studies.

Of course, this kind of agent review system also quickly encountered problems:

During a pause in automated review, the editor AI checked the code of three papers and found that some had leaked the answers in advance, and some had even directly replaced incorrect results with correct answers.

It immediately asked the authors to self-check. Another agent found that its own experiment had also leaked the answers, so it proactively withdrew the paper. After correction and rerunning, the model accuracy dropped from 91.7% to 64.6%.

This also confirms one point: even if the numbers in a paper genuinely come from real runs, the underlying code implementation may still deviate from what the paper describes.

Checking the code and raw data is the core step through which this AI journal truly exercises its gatekeeping role.

talkless, show me your code!

How Good Is the Output from This Agent Research Community?

So, how good is the output from this Agent research community?

To gauge the true level of this batch of research, the team further screened comparable work from over 47,000 human arXiv preprints published in the same period, ultimately forming 202 topic-matched pairs of AI papers and human papers for comparison, and mainly reached the following three conclusions:

First, Agent papers perform more solidly in experimental execution and method description.

In blind text evaluations, Agent papers scored higher than human papers on experimental rigor, reproduction details, and the match between conclusions and evidence.

Second, human papers are superior in innovation and research importance.

A breakdown of the data shows that the Agent's slight advantage in overall average score was mainly concentrated in Lab 1, which adopted a highly structured reproduction workflow; the other labs did not significantly outperform human papers.

On Twitter, alphaXiv also pointed out that Agents tend to make small improvements around familiar topics and existing benchmarks—

their ability to execute research is clearly stronger than their ability to judge "which problems are worth studying."

Third, quality screening after research completion still requires substantial human input.

The alphaXiv team admitted that after the experiments ended, they still spent a lot of time reading through the generated papers and verifying results before they could pick out the work truly worth attention.

In other words, "a reliable research quality signal did not emerge on its own." Therefore, although the research was all done by AI, humans remained involved throughout the entire experiment from start to finish.

However, alphaXiv also noted that many of the studies this time used small models, and the experiment duration was relatively short.

Next, the team hopes to use more ample compute, longer experiment times, and better scientific judgment to let Agents attempt more challenging topics.

Reference links [1]https://www.science.org/content/article/exclusive-i-got-eavesdrop-ai-agents-doing-science-it-was-eye-opening?referrer=https%3A%2F%2Ft.co%2F

One-click triple action: "Like," "Share," "Heart"

Feel free to leave your thoughts in the comments!

— The End —

🔹****In 2026, more and more people are proving to us: "Humanity's trust in AI is still not enough."

QuantumBit MEET2027 Intelligent Future Conference is here to lock in the next step of AI evolution for you.

Baidu's Wang Ying, MoleculeMind's Xu Jinbo, Kunlun Wanwei's Fang Han, Jiyuan Lvdong's Wang Yunhe, Zidong Taichu's Wang Jinqiao, Ant Health's Gu Jinjie, Xianggan Technology's Jin Yirong, ModelBest's Yao Yuan, SenseTime's Xuan Shanming, and Kuwa's Liao Wenlong are all here 🔍Learn more

December 9, Beijing. You are invited to join us to see AI clearly, seize AI, and master AI. 👉Click to attend

🌟 Light up the star 🌟

Frontier tech progress, every day

Open in WeChat