AgentWorld Benchmark Reveals Top Models Achieve Only ~50% Success in Multi-Agent Collaboration Tasks

Powerful individual AI Agents are emerging at a rapid pace. From writing code and developing software to retrieving literature and assisting in scientific research, they are increasingly capable of completing tasks independently.

However, as these "super individuals" become stronger, an equally important question arises: Can they collaborate?

Looking back at human society, many complex achievements rely on the division of labor and collaboration among groups. If we were to establish a company to build rockets and land on the moon, the first consideration would likely not be finding an all-knowing genius, but rather how to organize a team: What professional talents are needed? How should work be divided? How is information shared? And how do different stages of work connect seamlessly?

From scientific discovery to large-scale engineering, while individual capability is crucial, organization and collaboration determine how far we can go.

This leads us to consider: As the individual capabilities of Large Language Models (LLMs) and Agents continue to improve, what is the current state of their collaborative abilities? When facing a common goal, will they proactively exchange information and coordinate tasks? When multiple agents work together, can they truly leverage their respective strengths to make complex tasks easier to complete?

With these questions in mind, researchers from OpenAgents, Columbia University, the University of Pennsylvania, Seoul National University, and Penn State University have jointly built AgentWorld: a benchmark for evaluating multi-agent collaboration capabilities.

Paper Link: https://arxiv.org/abs/2609.31590

Project Website: https://agentworld.io/

Code & Data: https://github.com/openagents-org/agentworld

The research team focuses on this scenario: When multiple agents possess different roles, capabilities, and resources, and must jointly complete a task, can they form effective divisions of labor, maintain information synchronization, and connect their individual actions to ultimately achieve the common goal?

What Does AgentWorld Add?

In recent years, the scientific community has explored enabling multiple LLMs to solve problems through dialogue, role division, and mutual feedback. Researchers have also proposed evaluation tasks focused on collaboration, as well as gaming and social simulation environments.

As these studies progress, a further question needs answering: When tasks span dozens of rounds, members hold different resources, and every step may impact teammates, can teams sustain effective collaboration?

The distinctive feature of AgentWorld lies in combining long-horizon tasks, asymmetric roles, and black-box interactions within a single collaborative evaluation environment.

High-level tools reduce interference from low-level controls such as navigation and operation; programmatic rule checks verify whether common goals are met. Furthermore, the proposed CCE metric traces which actions and messages are judged to contribute to the outcome. This allows the evaluation to observe not only whether the team succeeds but also to analyze how cross-role handoffs and coordination facilitate that success.

Introduction to AgentWorld

A team of 10 agents enters a Role-Playing Game (RPG) world.

Imagine a task like this: Create a magic wand for the mage in the party. The lumberjack is responsible for gathering wood, the carpenter processes the wood into a stick, and the mage uses the stick along with materials they already possess to complete the crafting. On the surface, it seems straightforward—everyone just needs to do their job.

But in actual execution, issues can arise at any handoff point: Who should receive the wood? Is the stick ready? Should steps already completed by teammates be repeated? Before the task is finished, does anyone prematurely declare "done"?

These are precisely the challenges that multi-agent collaboration systems must face.

AgentWorld is a benchmark designed to evaluate Agent collaboration capabilities: It enables multiple agents driven by Large Language Models (LLMs) to divide labor, communicate, share resources, and jointly complete tasks within the same continuously evolving environment.

In experiments involving four main models reported in the paper, the highest success rate on the primary task set was 52.0%. This result raises a significant question worth investigating: As the capabilities of individual Agents continue to improve, what is still missing in team-level collaboration?

Figure 1 | Example of the AgentWorld Environment: Different roles have their own perspectives and communicate via in-game chat. The screenshot shows a ten-agent team and does not correspond to the three-agent magic wand task described below.

Turning "Ability to Collaborate" into a Measurable Capability

"Multiple Agents participating" is a system structure, whereas "Multiple Agents collaborating effectively" is a capability that requires verification.

If we simply have several models answer questions separately and then aggregate the answers, it remains difficult to know: Do they understand each other's responsibilities? Can they adjust the division of labor when resources are insufficient? Can they modify plans based on the latest progress of their teammates?

AgentWorld places these questions into an MMORPG sandbox—a Massively Multiplayer Online Role-Playing Game environment. In this setting, agents must navigate a persistent world where their actions have consequences.

The paper reports that in experiments with four main models, the highest success rate on the primary task set was 52.0%. This result highlights a critical gap: even as individual agent capabilities advance, team-level collaboration remains a significant challenge.

In the magic wand task, logging, processing, and crafting form a dependency chain. Even if every member knows the final goal, the team still needs to resolve questions of "who goes first," "who hands it off to whom," and "when to hand it over."

Figure 2 | Another task example: Ten agents take on roles such as mining, smelting, forging, support, and coordination to jointly complete an item-crafting objective. The role configuration makes division of labor and resource transfer integral parts of the task.

Second is black-box interaction. Agents cannot directly read their teammates' internal reasoning states. They must infer what other members are doing through environmental observations, chat messages, and action outcomes. This makes "I thought my teammate had already finished" a genuine source of failure.

Third is multi-round execution. An initial plan cannot cover all subsequent changes. The team must continuously update its state during execution: Are resources in place? Does a teammate need support? Is the original division of labor still valid? The paper's main tasks set an execution budget of dozens of rounds, requiring agents to maintain coordination across multiple handoffs.

To reduce interference from low-level operations, AgentWorld provides 13 high-level API tools. For example, gathering or attack tools can encapsulate navigation and specific execution workflows, allowing models to focus more decision-making capacity on "what to do next" and "who to collaborate with."

This does not mean that planning and tool-use effects are completely excluded from evaluation, but it makes collaboration issues easier to observe and analyze.

The best teams completed about half of the tasks

More chat doesn't necessarily mean better performance

The study evaluated Gemini 3 Flash, Claude Haiku 4.5, GPT-5 Mini, and DeepSeek R1-70B under unified prompt templates, tool definitions, and interaction protocols.

On the main task set, the four models achieved task success rates of 52.0%, 45.0%, 36.0%, and 20.0%, respectively. On the enhanced task set, the corresponding success rates were 24.0%, 26.0%, 21.0%, and 10.0%.

Figure 3 | Redrawn based on the paper's main results table. This compares teams driven by each model under the settings specified in the paper; it does not represent a judgment of the capability ceiling for all agent system designs.

It is also worth noting the gap between "getting partway there" and "actually finishing." Gemini 3 Flash achieved a partial success rate of 71.5% and a full task success rate of 52.0% on the main task set. Teams advancing some intermediate goals does not guarantee that the final resource handoff or crafting steps will be completed.

Communication volume did not show a simple "more is better" relationship. GPT-5 Mini sent an average of 44.1 chat messages per main task but achieved a 36.0% success rate; Gemini 3 Flash sent an average of 11.0 messages and achieved a 52.0% success rate.

These results do not prove that less chatting improves success rates, but they remind us that message count cannot directly substitute for collaboration quality.

What deserves more attention is whether a message updates useful information, resolves ambiguities in the division of labor, and whether the recipient takes action based on it.

Beyond completing tasks, we must ask: Which actions actually helped?

Simply counting success rates is insufficient to describe how a team works.

Two teams might both complete a task, but one has clear division of labor and smooth handoffs, while the other experiences extensive redundant gathering, ineffective waiting, and misinterpreted information. If we only look at the final result, both receive the same success label.

To address this, the paper proposes Causal Collaboration Effectiveness (CCE).

It looks backward from the team's ultimately completed goal to calculate how many actions were judged as having contributed to that outcome.

Success rate answers "Did the team get the job done?" CCE further asks "Which actions helped along the way?" Its focus on contribution is not limited to the final step that completes the task, but also includes preparatory work that provided materials, conditions, or useful information for subsequent actions.

It starts from the action that completed the goal and traces back through time: What materials did this crafting depend on? Where did those materials come from via transfer? Who performed the gathering or processing that enabled that transfer? The algorithm uses large language models to judge causal dependencies between candidate actions, gradually building an action relationship graph.

Figure 4 | Schematic diagram of CCE from the paper: Actions by the lumberjack, carpenter, and mage are connected through cross-role dependencies. The 12/27 in the figure represents the proportion of contributing actions in this illustrative trajectory, not the average result across the entire dataset.

The calculation method is straightforward:

CCE for successful tasks = Number of actions judged as contributing to success ÷ Total number of actions executed by the team.

Taking the illustrative trajectory in Figure 4 as an example, three agents executed a total of 27 actions, of which 12 were included in the contribution chain leading to task success, so CCE = 12 ÷ 27 ≈ 44.4%.

The mage crafting the magic wand is the direct action completing the goal; the lumberjack gathering and handing over wood, and the carpenter processing and delivering the stick, may indirectly contribute to success through subsequent dependencies.

Chat messages can also be contributing actions. If a message provides information relied upon by subsequent actions, it may enter this contribution chain; whether it counts depends on the specific trajectory and judgment rules, rather than simply categorizing it as "chat" versus "tool call."

The paper defines CCE as 0 for failed tasks. This is a scoring convention for the metric and does not imply that every attempt during a failure was worthless.

Therefore, the average CCE across tasks is influenced by both the success rate and the proportion of contributing actions within successful trajectories. It should be read in conjunction with success rates and partial success rates, rather than treated as an independent "team synergy score."

Here, two concepts need to be distinguished: The average proportion of actions per task is not equal to the overall proportion calculated by aggregating actions across all tasks. Nor should actions not traced back as successful contributions be uniformly understood as meaningless; necessary exploration, attempts made under information scarcity, and judgment errors must all be considered.

The value of CCE lies in providing a checkable analytical perspective, helping researchers interrogate the team's working process. It still relies on model judgments and cannot alone prove that a system possesses general collaborative capabilities.

From redundant labor to premature completion

Failures occur in the details of collaboration

Analysis of communication trajectories in the paper identified several types of failures: repeatedly asking about already-resolved issues, mistaking roles or resource recipients, reporting incorrect item information, and declaring task completion before critical steps were finished.

These phenomena indicate that multi-agent systems need to maintain not just a "task plan," but also a constantly evolving team state.

For system developers, AgentWorld provides scenarios for further experimentation: Can clear role division reduce misunderstandings? Does shared memory help teams synchronize progress? What are the respective advantages of centralized planning versus distributed execution? Do new communication mechanisms actually improve completion rates, or do they merely increase message volume?

These questions should be answered through controlled experiments. The purpose of a collaboration benchmark is to ensure that methodological improvements can be evaluated against identical tasks, rules, and auditable execution logs.

From "Individual Competence" to "Collective Completion"

The experiments in AgentWorld are not sufficient to attribute collaboration difficulties to a universal ceiling for all large models. The results correspond to specific models, prompts, tool interfaces, and execution protocols; there remains room for exploration with more sophisticated designs in planning, memory, or communication.

However, it concretizes an important question: Evaluating agent teams requires looking not only at what individual members can do, but also at whether they can jointly complete tasks under conditions of mutual dependency.

As multi-agent systems are applied to increasingly complex workflows, the evaluation of collaboration capabilities must extend from dialogue quality to actual actions. AgentWorld provides a runnable and comparable experimental environment for this type of evaluation.

References:

https://arxiv.org/abs/2609.31590

Chasing ASI Instantly

⭐ Like, Share, and Watch – One-Click Triple Combo ⭐

Star us to lock in instant updates from New Intelligence Era!