Arena Open-Sources Agent Interaction Evaluation Datasets
Arena.ai introduces Interaction-as-a-Judge, an evaluation paradigm in which computer-use agents directly play games and operate apps to judge quality, improving alignment with user preferences, error detection, and downstream generation. The team also open-sources two agent trace datasets, ArenaWebdevPreferences and ArenaWebdevCorrectness, to support future research.
Related Entities
Full Report
Arena.ai (@arena) · 2026-11-12 21
Agents are demonstrating increasingly strong capabilities in recreating games and applications. One likely reason is their growing ability to directly interact with these applications.
We studied Interaction-as-a-Judge, a growing evaluation paradigm where computer-use agents directly interact with games and applications to improve evaluation.
Interaction-as-a-Judge improves:
- alignment with user preferences.
- a model’s ability to detect errors in games.
- downstream generation. We see improvements in inference-time scaling and the ability for interaction-only reinforcement learning.
It also scales with computer-use capabilities; the better the computer-use model and method, the better the judgments.
We’re also open-sourcing two agent trace datasets, ArenaWebdevPreferences and ArenaWebdevCorrectness, to help support future research. Dive into the thread to learn more.
↳ 引用 @arena (Arena.ai): Arena research intern and PhD candidate at Carnegie Mellon University, Wayne Chi, explains Interaction-as-a-Judge: letting computer-use agents play games and use apps to better improve evaluation and downstream generation.
A computer-use agent navigates an application, producing interaction traces for evaluation by an LLM judge or a trained reward model. The talk covers preference evaluation using real Code Arena battles and correctness evaluation using requirements mined from real user complaints in Agent Arena.
Watch the full talk:
