Max Planck Paper Finds Reasoning Makes LLMs Converge, Even on Shared Errors

1 reports archived1 independent sourcesupdated 11/5/2026, 18:25:34

Synthesis & Latest Updates

1 Sources Cross-Validated

A new Max Planck paper tested nine Qwen and GPT models on three prediction tasks built from US census surveys, with reasoning on and off. Enabling reasoning raised cross-model agreement, but the share of people all nine models got wrong also rose by 2.0 to 2.4 points; for the longest reasoning traces, adjusted agreement fell to roughly 0.31-0.39. The authors warn that ensemble-style error correction assumes uncorrelated mistakes, and reasoning may make errors more correlated.

LATEST/A new Max Planck paper tested nine Qwen and GPT models on three prediction tasks built from US census surveys, with reasoning on and off. Enabling reasoning raised cross-model agreement, but the share of people all nine models got wrong also rose by 2.0 to 2.4 points; for the longest reasoning traces, adjusted agreement fell to roughly 0.31-0.39. The authors warn that ensemble-style error correction assumes uncorrelated mistakes, and reasoning may make errors more correlated.

Hourly heat curve

2 fully observed hours

Fewer than 3 fully observed hours; trend not plotted yet.

Coverage timeline

Total 1 reports · Latest first

Rohan Paul (@rohanpaul_ai)T2Nov 5, 18:25·78 pts
Max Planck Paper Finds Reasoning Makes LLMs Converge, Even on Shared Errors
- Reasoning raises cross-model agreement, yet the share of cases all nine models got wrong rose 2.0-2.4 points
- On samples with the longest traces, adjusted agreement dropped to about 0.31-0.39, suggesting long reasoning does not yield more independent judgments
- Ensemble error correction assumes uncorrelated mistakes; before deployment, check whether any model in the set produces a correct answer
View Related Reports & Analysis Original Source