Arena Proposes Interaction-as-a-Judge for Computer-Use Agents
Arena.ai research intern and CMU PhD candidate Wayne Chi introduced Interaction-as-a-Judge, a paradigm in which computer-use agents navigate real applications and games to produce interaction traces evaluated by an LLM judge or a trained reward model. The framework spans preference evaluation drawn from real Code Arena battles and correctness evaluation based on requirements mined from user complaints in Agent Arena.
Related Entities
Full Report
Arena.ai (@arena) · 2026-11-12 21
Arena research intern and PhD candidate at Carnegie Mellon University, Wayne Chi, explains Interaction-as-a-Judge: letting computer-use agents play games and use apps to better improve evaluation and downstream generation.
A computer-use agent navigates an application, producing interaction traces for evaluation by an LLM judge or a trained reward model. The talk covers preference evaluation using real Code Arena battles and correctness evaluation using requirements mined from real user complaints in Agent Arena.
Watch the full talk: