Study Finds LLM Moral Choices Lack Stable Value Rankings
A study of five LLMs including GPT-5.2, Gemini and DeepSeek found that models lack a consistent value ranking when comparing pairs of moral violations: reversing option order changed the ranking in 26 of 30 test blocks. Critically, averaging responses produces a clean, human-like moral hierarchy that matches no individual test set, so moral alignment scores should only be trusted if rankings hold on individual prompts and both answer orders.
Full Report
Rohan Paul (@rohanpaul_ai) · 2026-11-11 04
This paper finds that LLM moral choices often lack a stable ranking of values, and averaging their answers creates a fake, human-like moral hierarchy.
Many studies conclude that LLMs hold sensible moral values because they do well on morality tests. But good answers do not show that a model weighs competing values against each other.
The researchers gave 5 LLMs, including GPT-5.2, Gemini, and DeepSeek, pairs of moral violations and asked which was worse. They read each answer's probability straight from the model's next-token output, then tested whether the choices fit a consistent ranking.
Reversing the order of the 2 options changed the ranking in 26 of 30 test blocks. After averaging, 4 of 5 models showed a clean, nearly identical ranking that matched no individual test set.
Before trusting a model's moral alignment scores, check that its choices hold a consistent ranking on individual prompts and both answer orders, without averaging.
