Tsinghua et al. Identify 'Thinking Inertia' in LLMs Despite Disabled Reasoning Modes

If you disable the "thinking mode" of a large language model via its API (e.g., setting enable_thinking to False for Qwen3) and instruct it to "skip reasoning and provide the answer directly," will it actually comply? The real-world output often defies our intuitive expectations.

Researchers from Tsinghua University, Oxford University, and Stanford University systematically tested this hypothesis. They discovered that even with thinking mode disabled, models may still generate calculations or derivations before the final answer. This phenomenon becomes more pronounced as questions become more open-ended. The researchers term this behavior Thinking Inertia.

Paper Link: https://arxiv.org/abs/2610.11765

Code Link: https://github.com/thinking-inertia/code

Trajectory Data: https://huggingface.co/datasets/thinking-inertia/llm-nothinking-trajectories

Figure 1: Even when explicitly instructed to output only the option, the model may still write out the solution process.

Deconstructing the "Answer"

What appears before the answer?

Determining whether a model is reasoning cannot rely solely on the thinking toggle or response length. A model with the thinking switch turned off may still write calculation steps within the standard response area; conversely, a lengthy text might merely repeat the question or follow a template without containing actual reasoning.

To address this, researchers decomposed responses into pre-answer text and the final answer, then evaluated them across three layers. ETR checks if the model truly provided only the answer: the format must be valid, with no substantive text before or after. QRel. measures the relevance of the pre-answer text to the question (note: repeating the question can also yield high scores). EIR further determines if explicit reasoning steps—such as calculations, comparisons, or eliminations—are present.

The evaluation model was blind to the identity of the tested model and experimental settings. On a sample of 600 outputs, the average agreement rate between five human annotators and the automated evaluator on the binary classification of "contains explicit reasoning" was 98.8%.

Figure 2: Three-layer metrics check respectively for answer-only compliance, textual relevance, and presence of explicit reasoning.

Six Models, Three Task Types, Six Benchmarks

The experiments covered six models from four families: Qwen3, DeepSeek, GPT, and Gemini. They tested the impact of different thinking settings and response instructions on outputs across six benchmarks. Questions were categorized into True/False, Multiple Choice, and Open-Ended Math problems requiring self-generated answers.

Overall, models are more likely to answer directly when only Yes/No is required. The proportion of explicit reasoning in multiple-choice tasks falls in the middle. For open-ended tasks, visible reasoning before the answer increases significantly. Stricter "answer-only" instructions reduce the proportion of explicit reasoning, but the overall ranking across task types persists: Open-Ended > Multiple Choice > True/False.

A striking example comes from DeepSeek-V4-Flash: after disabling thinking mode, its overall explicit reasoning rate remained at 99.9% on both GSM8K and MATH benchmarks.

This implies that turning off the thinking switch does not guarantee that the model will omit reasoning processes before the final answer.

Even if the final answer is correct, the model may fail to adhere to the requirement of "providing only the answer without reasoning." Answer correctness and instruction compliance regarding direct answering are two distinct dimensions that require separate evaluation.

Changing Only the Answer Format Alters Results

Does the difference stem from the question itself?

Researchers selected a batch of numerical answer questions from MMLU and MMLU-Pro, kept the question stems unchanged, modified the candidate answers and response requirements, and compared results using Qwen3-32B with thinking mode disabled.

For original multiple-choice questions, accuracy was 77.8%, and the answer-only rate was 10.5%. When converted to True/False questions where the model verifies a given answer, accuracy remained stable at 78.9%, but the answer-only rate jumped to 62.5%. When removing candidates and asking the model to generate the result independently, accuracy dropped to 52.6%, the answer-only rate fell to 0, and the explicit reasoning rate reached 90.1%.

With candidate answers available, the model can verify or select; without them, it must construct the result itself, making it more prone to writing out reasoning before the answer.

Figure 3: Accuracy and output behavior change when the same set of questions is reformatted into different response types.

Stronger Constraints Increase Costs for Open-Ended Tasks

Requiring models to minimize intermediate steps also impacts final accuracy. For five models supporting thinking mode disablement, adding strict answer-only instructions to open-ended math problems increased the average answer-only rate from near 0% to approximately 40%, while average accuracy dropped by about 15 percentage points. Changes in accuracy for True/False and Multiple Choice tasks were relatively stable.

Therefore, the ability to comply with instructions to skip reasoning and answer directly should be recognized as an important model capability that requires independent evaluation.

This capability is not automatically acquired simply by turning off the thinking switch. Evaluations must simultaneously assess output compliance and answer correctness.

This work has sparked subsequent verification within the community. Inspired by the paper, George Shostakovych conducted small-scale independent tests on Gemini-4-Argon, GPT-6-Astra, Kimi-K3, DeepSeek-V4-Flash, and Muse-Wonder-1.0. He observed similar phenomena where explicit reasoning persisted despite disabled thinking modes and reported these findings in his blog [2].

Research Team

First author Dianqiao Lei is a senior undergraduate student at Tsinghua University. Kevin Qinghong Lin, Philip Torr, and James Zou serve as co-corresponding authors.

Kevin Qinghong Lin is a postdoctoral researcher at the University of Oxford; Pan Lu is a postdoctoral researcher at Stanford University; Philip Torr is a Professor in the Department of Engineering Science at the University of Oxford; and James Zou is an Associate Professor in the Department of Computer Science and the Department of Electrical Engineering at Stanford University.

The paper has been accepted to NeurIPS 2026, and the paper, code, and data are publicly available.

References:

[1] Thinking Inertia: LLMs Keep Thinking When Told Not To https://arxiv.org/abs/2610.11765

[2] George Shostakovych. 'Answer only' is a request, not a switch. October 15, 2026. https://gshost.dev/writing/answer-only-is-a-request/

Chasing ASI Instantly

⭐ Like, Share, and Watch — One-Click Triple Combo ⭐

Star us to stay updated with New Intelligence's rapid delivery!