Google et al. Release CLASHarness Benchmark for Multi-Agent Conflict Resolution

AI Agents are increasingly being deployed in teams to handle real-world workflows such as coding, cloud operations, financial reconciliation, and enterprise approvals.

However, existing Computer-Use benchmarks (such as OSWorld and WebArena) almost universally assume an idealized premise: environmental information is always consistent, and all teammate agents are 100% reliable.

Real-world scenarios clearly differ from this assumption. In multi-agent collaboration, sub-agents often report incorrect states due to reading stale caches, or they clash with the main agent over file locks and permissions.

For example: A sub-agent sends a message stating, "The online service is running normally; please switch traffic immediately," but the screenshot of the real-time monitoring dashboard actually shows massive errors. If the main agent blindly trusts its teammate's textual report, it will directly cause a production incident. It must proactively open the monitoring screenshot to verify the ground truth and take action to roll back configurations and release shared locks.

Research teams from Google, Emory, USC, UF, UChicago, UCSB, and Columbia have jointly released CLASHarness, conducting the first major evaluation of 15 mainstream large models on their performance under multi-agent conflicts.

Paper Link: https://arxiv.org/html/2610.19128

Open-source Code and Dataset: https://github.com/google-research/CLASHarness

Covering 20 real-world domains, 600 multi-turn executable conflict tasks, 7,970 4K visual evidence images, and 3,284 mandatory checkpoints.

Why is building a "Multi-Agent Conflict Benchmark" so difficult?

Constructing a conflict test set that includes codebases, terminal logs, multi-agent chat records, and high-resolution UI screenshots presents significant challenges: Purely manual creation cannot scale effectively; while having LLMs generate content in one go easily leads to plot inconsistencies, accidental leakage of answers in the prompt, or even generating "dead questions" where even the standard answer fails validation.

To address this, CLASHarness designed a "Three-Layer, Eight-Stage" automated generation and verification pipeline, ensuring 100% task solvability from the source:

Figure 1: The three-layer, eight-stage generation pipeline of CLASHarness. Green boxes represent zero-cost automatic code fixes, while red boxes represent local single-step retries with error feedback.

1. Step-by-Step Planning + Local Automatic Error Correction (Figure 1)

The pipeline first plans the global outline and agent role permissions, then generates specific tasks in rolling batches of every 3–4 turns to avoid context drift in long texts. Each turn's generation is subdivided into 8 small steps. If path or format issues arise, they are fixed automatically via code (0 model calls); if logical conflicts occur, only the erroneous step is retried without needing to start over. On average, it takes just 95 API calls to synthesize a complete 10-turn continuous task package.

Figure 2: Examples of 4K visual evidence generated by CLASHarness (covering monitoring dashboards, IAM permission reconciliation tables, compute scheduling monitors, and reimbursement bill details).

2. Eliminating Textual "Shortcuts": Key Evidence Hidden Only in 4K Images (Figure 2)

To prevent agents from bypassing image analysis by merely searching text, CLASHarness first defines the key ground truths for resolving conflicts (e.g., actual bill amounts, permission matrix states). It then renders realistic UI screenshots or live photos at 3840×2160 (4K) resolution and uses an automatic scanner to clear all answer clues from text files, forcing agents to extract evidence from images.

3. Validate Reference Solutions in Sandbox Before Generating Scorers

All task scoring criteria (34.9% precise code checks + 65.1% semantic majority-vote checks) are extracted and generated directly from reference trajectories successfully executed in real sandboxes. Additionally, 23 modular conflict plugins are built-in, allowing developers to synthesize new conflict test scenarios like building blocks.

Empirical Testing of 15 Frontier Models

The research team evaluated 15 mainstream models on 600 conflict tasks (including Claude Opus5 / 4.8, Sonnet 5, GPT-6 Astra, the GPT-5.6 series, the Gemini 3.x series, and open-source models Kimi K3, DeepSeek V4 Pro / Flash, GLM-5.2), accumulating over 9,000 real sandbox trajectories.

Figure 3: Overall leaderboard of 15 models on CLASHarness (600 tasks). The colored bars on the left represent end-to-end task pass rates, while the right side shows the mean score per individual check.

Finding 1: Individual scores are not low (~60%), but no model breaks the 30% threshold for end-to-end pass rate

As shown in Figure 3, under the standard limit of 20 steps, Claude Opus 5 (25.2%) and GPT-5.6 Sol (24.5%) rank highest; when the step limit is relaxed, GPT-6 Astra achieves the highest score of 28.7%. Among open-source models, Kimi K3 (21.8%, ranked 5th overall) and DeepSeek V4 Pro (19.8%) deliver standout performance, matching or even surpassing several closed-source commercial models.

Why do top-tier models achieve a Mean Score of 55%–67% on individual steps, yet fail to reach a 30% end-to-end Pass Rate? This is because resolving real-world conflicts requires completing an entire chain: identifying misleading teammate inputs → opening images to verify ground truth → modifying system configurations → finalizing handovers. Missing any single link in this chain causes the entire task to fail.

Finding 2: Weak models blindly trust textual reports from teammates, while strong models actively inspect images for verification.

When a sub-agent’s textual report conflicts with the actual environment state, weaker models are easily swayed, readily trusting incorrect textual reports. In contrast, stronger models proactively invoke tools to examine monitoring screenshots or billing images to verify ground truth (outperforming weaker models by 31–65 percentage points in visual grounding and evidence citation).

If image inputs are disabled (see Figure 4b), pass rates for all models plummet instantly by 87%–90%—even in tasks without strict requirements to view images, models that cannot see images retain only one-tenth of their original pass rate (e.g., Claude Opus 5 drops from 24.8% to 2.8%).

Finding 3: "Spotting contradictions" is easy, but "fixing issues and finalizing tasks" is extremely difficult.

To determine whether models fail due to inability to detect problems or inability to resolve them, the research team compared two types of task requirements within the same conflict scenarios:

• Investigation-only requirement: Models achieved an average pass rate of 28.1%.
• System-state modification requirement: The pass rate dropped directly to 11.0% (a net decrease of 18.2 percentage points).
• Full multi-agent collaboration workflow requirement: The pass rate fell further to just 4.4%.

The most common failure point lies in the "finalization phase": Even the strongest models, GPT-6 Astra and Claude Opus 5, experience 58%–72% failure rates in finalization actions.

Models frequently terminate tasks immediately after modifying main configuration files, forgetting to release shared locks, close tickets, or send messages notifying downstream sub-agents to take over.

Finding 4: More thinking tokens often lead to infinite loops; relaxing step limits unlocks deep exploration.

In math and coding benchmarks, increasing Thinking Tokens typically improves scores. However, in multi-agent interaction environments, the opposite occurs:

Figure 4: Three major ablation studies. (a) Relaxing step limits (20→30→40 steps); (b) Visual vs. non-visual input; (c) Ranking consistency across different generation models (Gemini 3.6 Flash vs. Claude Sonnet 5) (ρ=0.96).

1. The "Infinite Loop Trap" caused by excessive thinking.

Under the 20-step limit, the higher the proportion of tokens spent on internal reasoning, the lower the success rate (correlation ρ=-0.70). For example, increasing Gemini 3.5 Flash’s thinking intensity from Low to High caused its pass rate to crash from 13.5% to 3.0%, while the cost per solved problem skyrocketed to $39.54 (55 times higher than GPT-5.6 Luna’s $0.72). The reason is that the model gets stuck in repetitive internal deliberation and redundant tool calls, exhausting the 20-step budget before completing the core task in 89.2% of cases.

2. Relaxing step limits unlocks long-horizon exploration (Figure 4a).

Conversely, for models genuinely skilled at multi-step exploration, the 20-step cap restricts their potential. When the step limit was increased from 20 to 40 and 60 steps, Gemini 3.8 Flash’s pass rate surged from 12.8% to 29.2% and 30.7%, surpassing all baseline models operating under the 20-step limit, while simultaneously reducing the average cost per solved problem by 21% (to $3.50).

Open Source and Outlook

Beyond serving as a benchmark, CLASHarness is also a scalable multi-agent training data synthesis engine. Leveraging the framework’s built-in 20 domain seeds and 23 conflict plugins, researchers can low-costly and batch-synthesize 100% solvable multimodal conflict trajectories and Verifiable Rewards, enabling the training of next-generation multi-agent systems that are more reliable, less prone to blind obedience, and better at collaboration.

Currently, CLASHarness’s generation pipeline code, 600 evaluation tasks, and interactive trajectory viewer have been open-sourced on GitHub.

References:

https://arxiv.org/html/2610.19128

Chasing ASI Instantly

⭐ Like, Share, and Watch ⭐

Star us to lock in instant updates from New Intelligence!

Open in WeChat