Anthropic Red Team Tests Self-Replicating AI: GLM-5.2 Succeeds in Simplified Environment, Small Models May Catch Up by Year-End

Anthropic Tests AI Self-Replication: Current Small Models Can't Do It Yet, But They May Catch Up by Year-End

Bottom line: There is currently no AI capable of autonomously replicating itself across the internet.

Anthropic tested a batch of open-source models in a simplified experimental environment. Only Zhipu's GLM-5.2 could complete the entire workflow: "transfer to a new machine, install and run the model, then propagate to the next." However, GLM-5.2 is too large to fit on small devices like TV boxes; smaller models that do fit failed to achieve this.

Based on the historical progress rate of small models, Anthropic estimates that by the end of this year, a small model will reach the level of GLM-5.2 in this specific test. This could lead to the emergence of AI capable of self-replication on cheap, connected devices.

This research was published on October 12 by Anthropic's Frontier Red Team, a group dedicated to assessing dangerous capabilities in models.

What Does AI Self-Replication Mean?

A computer worm is malicious software that copies itself from one machine to another without human intervention. Anthropic is concerned about an AI version of such worms: carrying model weights (the core model files) as part of the payload, running the model on each compromised device, and using that model to identify the next target.

Unlike traditional viruses, these AI worms can navigate unfamiliar devices. Traditional botnets (networks of hacked devices) typically use hardcoded attack code targeting specific device types. An AI worm encountering unknown hardware can investigate and experiment on its own. Furthermore, it lacks a central kill switch, making it difficult to shut down entirely.

How Far Are We from Practical Use?

Anthropic broke down replication into four steps: finding targets, gaining execution privileges, running the model on the new device, and propagating from that new device. The first two steps fall under general hacking, which prior studies have shown small models can perform; they were not the focus of this test.

The primary focus was step three: getting the model to run on an unfamiliar machine. This requires selecting the right inference software, installing the correct versions, and troubleshooting device-specific issues. A failure at any single step halts the entire process.

In testing, Qwen3.5-9B (a 9-billion-parameter model) successfully set up inference services on 8 out of 9 hardware categories. These categories mostly included cloud servers ranging from Macs and RTX 4090s to data center GPUs, along with two physical small development boards. No TV boxes were included in this specific subset.

Next came the full workflow. The research team set up eight cloud servers running simulated Android systems, introduced a worm, and observed whether it could autonomously scan the network, connect to other machines, transfer files, establish inference services, and continue propagating.

Models like Qwen3.5-9B and Gemma4-26B-A4B failed to propagate to even a single machine in every attempt. In contrast, GLM-5.2 successfully propagated to all 8 machines in 4 out of 5 tests, and reached 7 machines in the remaining test.

These results rely on three key assumptions:

1. Simplified Intrusion: The entry point was made nearly trivial. The machines' ADB ports (Android Debug Bridge) had no passwords; connecting granted immediate control. Anthropic acknowledges this represents the upper bound of real-world success rates. However, they note that the Mirai botnet spread by trying default vendor usernames and passwords, suggesting that security on many real-world IoT devices may not be significantly better than this setup.
2. External Brain: The worm's "brain," GLM-5.2, did not run on the infected machines because they lacked the capacity for such a large model. Instead, it ran on a separate server. On each infected machine, a smaller Llama3.1-8B model was deployed. The experiment aimed to demonstrate that if an 8-billion-parameter small model possessed GLM-5.2's capabilities, it could install itself on new machines.
3. Trend-Based Projection: The "catch-up by year-end" estimate is derived from a trend line of the General Capability Index (ECI). General capability only indirectly reflects replication ability, and Anthropic states this is a rough estimate.

Why Target TV Boxes?

For a model to run autonomously, it needs compatible hardware. Data centers are heavily secured, whereas consumer and small-business IoT devices are cheap, always online, and often lack robust security measures, making them easier targets. Anthropic cites white-label Android TV boxes as a typical example. These devices often run modified versions of Android Open Source Project code, with manufacturers adding no extra security features and providing only 1–4 GB of RAM. The previous Kimwolf botnet infected over 2 million devices, primarily these types of boxes.

While a single box cannot run a full model, the model can be split into segments and distributed across multiple boxes for relay computation. Anthropic purchased 15 white-label Android TV boxes costing approximately $55 each with 4 GB of RAM. They split Gemma4-26B-A4B across these devices and achieved a throughput of 0.39 tokens per second.

This demonstration was manually deployed by researchers to prove that aggregated compute power from such devices is sufficient. The article also notes that smart fridges, ovens, robot vacuums, and lawn mowers could share model segments, while surveillance cameras and video conferencing devices with built-in AI chips would be even more attractive targets.

What If It Happens?

Slow Propagation. Based on measured speeds on TV boxes, setting up an inference environment takes 28–51 hours for Qwen3.5-9B and 71–150 hours for Gemma4-26B-A4B. Anthropic calculated that if a worm infected 2 million boxes like Kimwolf, running one instance of Gemma4-26B-A4B on every 15 boxes, there would be roughly 133,000 replicas. The total compute power would equate to at most 40 H100 GPUs. Anthropic compares such a worm to slime mold or fruit fly swarms, where doubling the population takes days.

Experience Accumulation. The experiment allowed Qwen3.5-9B to pass notes on tried methods and encountered pitfalls to the next generation. Setup success rates rose from 45% in the first generation to 61–84% in subsequent generations. Assigning an Opus 4.8 model as a consultant provided no significant benefit, as the small models didn't know what questions to ask.

Partial Defensibility. The massive transfer of multi-gigabyte model files between devices and sustained high-volume communication would be conspicuous in ISP traffic monitoring. Faster propagation increases visibility. During the cleanup of Mirai, some ISPs isolated infected users' networks and notified them, clearing 92% of infections within 14 days. Without notification, only 74% were cleared.

Anthropic believes similar strategies could work against AI worms, but complete eradication would require cooperation from every ISP hosting infected devices. Device manufacturers can reduce the attack surface at the source; for instance, using default passwords on connected devices is already non-compliant in the UK market.

The original report does not offer specific advice for ordinary users. However, inferring from the described attack vectors, cheap TV boxes of unknown origin and connected devices with unchanged default passwords are the most critical risks to watch for.

↳ Quoting @willzhou97 (William Zhou): Excited to share a research blog post that Daniel Freeman (@bucketofkets) and I have been working on for the past few months:

https://www.anthropic.com/research/self-replicating-ai