Rohan Paul (@rohanpaul_ai) · 10/14/2026, 17:52:33
AgentHorizon Benchmark Reveals LLM Judges Miss Subtle Errors in Long Computer-Use Tasks Even With Tools
ServiceNow and McGill University introduced AgentHorizon, a benchmark of 1,373 judging tasks derived from human screen recordings to detect 'near-miss' failures in long-horizon computer-use agents. The study reveals that even the top-performing GPT-5.5 achieves only 80.9% balanced accuracy on hard splits, while providing tools to models like Qwen 3.6 paradoxically reduces their performance from 77.6% (direct reading) to 70.4%, highlighting significant blind spots in current LLM-based automated evaluation.
SOURCE COVERAGEFull Original Report
Rohan Paul (@rohanpaul_ai) · 2026-10-14 09
New ServiceNow and McGill paper finds that AI judges still miss subtle mistakes in long computer-use tasks, and giving them tools does not reliably help.
Teams use LLMs to grade agent runs for training and evaluation. A run of 100+ steps can look finished while it drew the wrong shape or posted to the wrong Slack channel.
They built AgentHorizon, 1,373 judging tasks from human screen recordings. People recorded pairs of near-identical tasks and swapped the instructions, so each failure is a clean run of a slightly different request.
The best judge, GPT-5.5 in Codex, scored 80.9% balanced accuracy on the hard split, where always saying yes gets 50%. Qwen 3.6 27B did worse with tools, scoring 77.6% reading the whole run directly but 70.4% as an agent.
Before you trust an LLM judge with agent runs, test it on near-miss failures inside the exact agent setup you will use.
