Termsslug: error-distribution-analysis

Error Distribution Analysis

错误分布分析

Inspecting which items a model fails and how wrong answers cluster to distinguish suppression from genuine incapacity.

Curated Intelligence

1 items · Page 1/1
Rohan Paul (@rohanpaul_ai)T2·Models
78 pts
Study Shows Low Benchmark Scores Can Be Manufactured Precisely

A new paper argues that a low score on dangerous-capability probes may reflect a deliberately suppressed model rather than genuine incapacity, so the number alone proves little about safety. The authors trained a LoRA dial that pulls a Qwen model toward a smaller Qwen model, hitting unseen target scores within 1.9 points, whereas prompting-based underperformance missed by 42 to 47 points. Crucially, up to 97% of the dial's new errors landed on a single multiple-choice letter, a distributional signature that can help detect manufactured low scores.