ALL INTELLIGENCE
Arena.ai (@arena)T1·Agents

Arena Open-Sources Agent Interaction Evaluation Datasets

Original title: Agents are demonstrating increasingly strong capabilities in recreating games and applications. One ...
62AI Score

Arena.ai introduces Interaction-as-a-Judge, an evaluation paradigm in which computer-use agents directly play games and operate apps to judge quality, improving alignment with user preferences, error detection, and downstream generation. The team also open-sources two agent trace datasets, ArenaWebdevPreferences and ArenaWebdevCorrectness, to support future research.

Full Report

Arena.ai (@arena) · 2026-11-12 21

Agents are demonstrating increasingly strong capabilities in recreating games and applications. One likely reason is their growing ability to directly interact with these applications.

We studied Interaction-as-a-Judge, a growing evaluation paradigm where computer-use agents directly interact with games and applications to improve evaluation.

Interaction-as-a-Judge improves:

  • alignment with user preferences.
  • a model’s ability to detect errors in games.
  • downstream generation. We see improvements in inference-time scaling and the ability for interaction-only reinforcement learning.

It also scales with computer-use capabilities; the better the computer-use model and method, the better the judgments.

We’re also open-sourcing two agent trace datasets, ArenaWebdevPreferences and ArenaWebdevCorrectness, to help support future research. Dive into the thread to learn more.

↳ 引用 @arena (Arena.ai): Arena research intern and PhD candidate at Carnegie Mellon University, Wayne Chi, explains Interaction-as-a-Judge: letting computer-use agents play games and use apps to better improve evaluation and downstream generation.

A computer-use agent navigates an application, producing interaction traces for evaluation by an LLM judge or a trained reward model. The talk covers preference evaluation using real Code Arena battles and correctness evaluation using requirements mined from real user complaints in Agent Arena.

Watch the full talk:

推文配图