Termsslug: llm-as-a-judge

LLM-as-a-Judge

LLM 作为裁判

Evaluation paradigm where LLMs score or rank model outputs instead of humans, prone to self-preference bias.

Curated Intelligence

21 items · Page 1/3
Rohan Paul (@rohanpaul_ai)T2·Models
78 pts
Stanford Finds All 8 LLMs Favor Their Own Medical Answers as Judges

A new Stanford paper had eight models from OpenAI, Anthropic, Google and Qwen answer 620 real clinician questions, then rank all eight answers anonymously. Every judge favored its own answer, by 1.22 rank positions on average and nearly 3 for GPT-5.6 Sol. The bias persisted without a rubric, and revealing model names cut it by only 0.094 positions, implying LLM-as-judge medical benchmarks carry systematic self-preference. The authors recommend multi-family judge panels with per-judge reporting instead of one pooled score.

Rohan Paul (@rohanpaul_ai)T2·Tools
62 pts
Voice Arena Launches First Open Benchmark for AI Dictation Apps

Voice Arena has released Dictation Bench, the first open benchmark for AI dictation apps. It fed 761 recordings from 87 speakers into 10 real Mac apps (not their APIs), with 5 LLM judges assessing whether each draft could be sent as-is; Wispr Flow scored 79% versus Apple's 35%. The benchmark publishes reference policies, app configurations, and judge instructions, giving vendors a reproducible standard for comparison.

Apple Machine Learning ResearchT1·Models
62 pts
LLM Judges Struggle With Personalized Writing Style Evaluation

A NeurIPS 2026 workshop paper systematically evaluates LLM judges on personalized writing style assessment. Judge quality correlates only weakly with general ability measured by MMLU (Pearson's ρ ≤ 0.2) and varies by writing task and evaluation criteria; paired comparisons prove more consistent than independent scoring, and the strongest models are not the most reliable judges.

AWS Machine Learning BlogT1·Agents
42 pts
AWS Bedrock AgentCore Integrates Decision Models for Agent Evaluation

Amazon Bedrock AgentCore Evaluations now supports 'decision models' to provide structured, calibrated scoring throughout the agent lifecycle. This feature enables developers to combine custom evaluators (LLM-as-a-judge or code-based) with decision models to quantify narrow-domain questions like policy compliance and tool safety, replacing unstable confidence scores from free-form text generation.

LangChain BlogT1·Agents
38 pts
LangChain Guide: Human Insight as Strategic Advantage for Agents

LangChain released a conceptual guide emphasizing that human expert judgment is critical for defining quality standards, contextual evaluation, and calibrating LLM judges in agent development. The article argues that structured human feedback forms the foundation for improving automated evaluations and recommends using LangSmith to integrate human review into workflows.

Apple Machine Learning ResearchT1·Agents
78 pts
Online RL Framework Enhances Robustness of iOS UI Navigation Agents

To address high inference costs of frontier VLMs and brittleness of SFT-trained small models, this work proposes an online reinforcement learning framework for training UI navigation agents directly on live iOS apps. By decoupling the agent into Planning and Grounding components and utilizing LLM-as-a-Judge for dense rewards, along with a novel GiGPPO value estimation method, the approach significantly improves error recovery and robustness in dynamic interfaces.

Artificial Analysis (@ArtificialAnlys)T1·Models
68 pts
MLCR-AA Update: Claude Sonnet 5.5 and GPT-6 Astra Tie for Top Medical Reasoning Score

Artificial Analysis updated the MLCR-AA leaderboard with refined prompts and a three-model judge panel to better reflect real-world medical record review. Claude Sonnet 5.5 and GPT-6 Astra tied for first place with a 41.7% pass rate, yet results indicate that medical long-context reasoning remains far from solved, with median models passing under 15%. Qwen3.8 2.4T A95B leads among open-weight models.

Artificial Analysis (@ArtificialAnlys)T1·Models
78 pts
Artificial Analysis Launches AA-PersonalityBench for LLM Personality and Style

Artificial Analysis introduces AA-PersonalityBench, evaluating LLMs across 30 personality traits and 21 communication characteristics using 900 curated questions. The benchmark aggregates subjective traits into eight indexes (e.g., Expressiveness, Transparency) and objectively quantifies style metrics like punctuation and emoji usage, aiding users in selecting models suited for specific contexts such as creative writing or legal tasks.

LangChain BlogT1·Agents
35 pts
Numan Details Eval-First Approach for AI Health Coach

UK digital health company Numan details the development of its AI health coach 'Nu', highlighting the shift from single-turn Q&A evaluation to multi-turn conversation quality assessment in a regulated medical context. The article emphasizes that rigorous evaluation criteria are more critical than automation itself, mentioning the use of LangGraph and LangSmith for debugging and deployment.

Hacker News AIT2·Models
42 pts
LLMs Fail to Distinguish Real vs. Fabricated Agent Memory in Kairos Experiment

The Kairos experiment demonstrates that LLM judges cannot effectively distinguish between an agent's real 30-day memory log and a fabricated autobiography generated by another model. With a score difference of only +0.032, the results fall within statistical equivalence. This highlights a fundamental flaw in current text-based AI memory metrics: they measure narrative style rather than verifying the authenticity of lived experience.