Today for AI
HOT RADAR
llmHEAT 17.1°

AI CLUSTERED EVENT · 10/14/2026

AgentHorizon Benchmark Reveals LLM Judges Miss Subtle Errors in Long Computer-Use Tasks Even With Tools

1 reports archived1 independent sourcesupdated 10/14/2026, 17:52:33
Synthesis & Latest Updates
1 Sources Cross-Validated

ServiceNow and McGill University introduced AgentHorizon, a benchmark of 1,373 judging tasks derived from human screen recordings to detect 'near-miss' failures in long-horizon computer-use agents. The study reveals that even the top-performing GPT-5.5 achieves only 80.9% balanced accuracy on hard splits, while providing tools to models like Qwen 3.6 paradoxically reduces their performance from 77.6% (direct reading) to 70.4%, highlighting significant blind spots in current LLM-based automated evaluation.

LATEST/ServiceNow and McGill University introduced AgentHorizon, a benchmark of 1,373 judging tasks derived from human screen recordings to detect 'near-miss' failures in long-horizon computer-use agents. The study reveals that even the top-performing GPT-5.5 achieves only 80.9% balanced accuracy on hard splits, while providing tools to models like Qwen 3.6 paradoxically reduces their performance from 77.6% (direct reading) to 70.4%, highlighting significant blind spots in current LLM-based automated evaluation.

HEAT TRENDHourly heat curve

3 fully observed hours
Now
17.1
Peak
18.119:00
24h change
–

The line compares only sources observed throughout; gap hours are interpolated to keep the trend continuous.

TIMELINECoverage timeline

Total 1 reports · Latest first
  1. Rohan Paul (@rohanpaul_ai)T2·68 pts
    • AgentHorizon generates 'near-miss' failure cases by swapping instructions in paired screen recordings to test judges' ability to distinguish correct execution from subtle errors.
    • GPT-5.5 achieves 80.9% balanced accuracy on hard tasks, significantly above the 50% random baseline but still missing nearly 20% of subtle failures.
    • Counter-intuitive finding: Granting tool access to judge models did not improve reliability; it actually degraded performance for models like Qwen 3.6.
AgentHorizon Benchmark Reveals LLM Judges Miss Subtle Errors in Long Computer-Use Tasks Even With Tools | Today for AI