UChicago Study: LLM Self-Reported Decision Factors Are Unreliable
A new University of Chicago paper reveals that LLMs' self-reported accounts of their decision-making factors are often inaccurate and fail to identify the true variables driving answer variations. The study demonstrates that comparing multiple generated answers to find disagreement points (cruxes) is more effective than querying the model directly or analyzing a single response, while being significantly cheaper in compute than local LLM judges. This finding offers critical guidance for auditing AI decisions in high-stakes scenarios.
Full Report
Rohan Paul (@rohanpaul_ai) · 2026-10-24 09
New University of Chicago paper suggests that an LLM's own account of what matters to its decision is a poor guide to what actually varies across its answers.
That LLMs fail to predict the key questions their own answers split on, which only comparing many answers reveals.
When an LLM recommends something, you want its cruxes, the few sub-questions that drove its conclusion. Asking the model upfront, or reading 1 of its answers, is cheap but may miss what really drives it.
They had models forecast 59 March Madness games and 15 Metaculus events many times, then compared pairs of forecasts to find disputed facts. A reranker made that matching cheap, at an estimated 86 times less compute than a local LLM judge.
In March Madness, predicted cruxes shared 13.4% of clusters with the real ones, below the 39.2% a random shuffle gives. A single answer did better but covered only 28.6% of real cruxes, against a 37.8% baseline.
For high-stakes decisions, sample many answers and look where they disagree, instead of trusting the model's own list of what matters.
