HOT RADAR
llmHEAT 12.3°

AI CURATED HOT STORY · 11/18/2026

Stanford Finds All 8 LLMs Favor Their Own Medical Answers as Judges

1 reports archived1 independent sourcesupdated 11/18/2026, 12:52:10
Synthesis & Latest Updates
1 Sources Cross-Validated

A new Stanford paper had eight models from OpenAI, Anthropic, Google and Qwen answer 620 real clinician questions, then rank all eight answers anonymously. Every judge favored its own answer, by 1.22 rank positions on average and nearly 3 for GPT-5.6 Sol. The bias persisted without a rubric, and revealing model names cut it by only 0.094 positions, implying LLM-as-judge medical benchmarks carry systematic self-preference. The authors recommend multi-family judge panels with per-judge reporting instead of one pooled score.

LATEST/A new Stanford paper had eight models from OpenAI, Anthropic, Google and Qwen answer 620 real clinician questions, then rank all eight answers anonymously. Every judge favored its own answer, by 1.22 rank positions on average and nearly 3 for GPT-5.6 Sol. The bias persisted without a rubric, and revealing model names cut it by only 0.094 positions, implying LLM-as-judge medical benchmarks carry systematic self-preference. The authors recommend multi-family judge panels with per-judge reporting instead of one pooled score.

Hourly heat curve

2 fully observed hours
Fewer than 3 fully observed hours; trend not plotted yet.

Coverage timeline

Total 1 reports · Latest first