Max Planck Paper Finds Reasoning Makes LLMs Converge, Even on Shared Errors
A new Max Planck paper tested nine Qwen and GPT models on three prediction tasks built from US census surveys, with reasoning on and off. Enabling reasoning raised cross-model agreement, but the share of people all nine models got wrong also rose by 2.0 to 2.4 points; for the longest reasoning traces, adjusted agreement fell to roughly 0.31-0.39. The authors warn that ensemble-style error correction assumes uncorrelated mistakes, and reasoning may make errors more correlated.
Full Report
Rohan Paul (@rohanpaul_ai) · 2026-11-05 10
New Max Planck paper finds that turning on reasoning makes different LLMs agree more, even on mistakes, while long traces mark where they disagree.
When several LLMs judge the same person, a wrong answer from one model could be fixed by switching to another. That only works if the models do not all make the same mistakes.
The authors tested 9 Qwen and GPT models on 3 prediction tasks built from US census surveys, with reasoning on and off. They measured agreement beyond what each model's accuracy alone would predict.
With reasoning on, models agreed more, and the share of people all 9 models got wrong rose by 2.0 to 2.4 points. For people who drew the longest traces, adjusted agreement fell to about 0.31 to 0.39.
For decisions about people, check whether anyone gets a correct answer from your model set, since more agreement can also mean more shared errors.
