Termsslug: interaction-as-a-judge

Interaction-as-a-Judge

交互即评判

Evaluation paradigm that scores agents' interaction traces in real applications via LLM judges or reward models.

Curated Intelligence

2 items · Page 1/1
Arena.ai (@arena)T1·Agents
62 pts
Arena Open-Sources Agent Interaction Evaluation Datasets

Arena.ai introduces Interaction-as-a-Judge, an evaluation paradigm in which computer-use agents directly play games and operate apps to judge quality, improving alignment with user preferences, error detection, and downstream generation. The team also open-sources two agent trace datasets, ArenaWebdevPreferences and ArenaWebdevCorrectness, to support future research.

Arena.ai (@arena)T1·Agents
58 pts
Arena Proposes Interaction-as-a-Judge for Computer-Use Agents

Arena.ai research intern and CMU PhD candidate Wayne Chi introduced Interaction-as-a-Judge, a paradigm in which computer-use agents navigate real applications and games to produce interaction traces evaluated by an LLM judge or a trained reward model. The framework spans preference evaluation drawn from real Code Arena battles and correctness evaluation based on requirements mined from user complaints in Agent Arena.

Interaction-as-a-Judge | Today for AI