Arena.ai introduces Interaction-as-a-Judge, an evaluation paradigm in which computer-use agents directly play games and operate apps to judge quality, improving alignment with user preferences, error detection, and downstream generation. The team also open-sources two agent trace datasets, ArenaWebdevPreferences and ArenaWebdevCorrectness, to support future research.
Termsslug: interaction-as-a-judge
Interaction-as-a-Judge
交互即评判
Evaluation paradigm that scores agents' interaction traces in real applications via LLM judges or reward models.
Curated Intelligence
Arena.ai (@arena)T1·Agents
62 pts
Arena.ai (@arena)T1·Agents
58 pts
Arena.ai research intern and CMU PhD candidate Wayne Chi introduced Interaction-as-a-Judge, a paradigm in which computer-use agents navigate real applications and games to produce interaction traces evaluated by an LLM judge or a trained reward model. The framework spans preference evaluation drawn from real Code Arena battles and correctness evaluation based on requirements mined from user complaints in Agent Arena.