SpanUQ Introduces Span-Level Uncertainty Estimation for LLMs

SpanUQ advances Large Language Model (LLM) uncertainty estimation from the token level to the semantic span level, precisely identifying unreliable content within responses and achieving inference speeds 10–20 times faster than traditional multi-sampling methods.

In the process of truly deploying LLMs into products, we frequently encounter a problem that seems simple but is critically important: How reliable is the content generated by the model?

During actual product development at Amazon, we found that downstream systems often require not only the results generated by the LLM but also corresponding certainty or confidence information to optimize subsequent decision-making. For example, in an LLM-based information processing system, the generated content may be used for classification, reasoning, tool invocation, or even directly impact user experience. In such cases, downstream systems need to determine: Should the model's output be adopted directly, or does it require additional verification? Is it necessary to invoke a more powerful model? Or is the current information insufficient to support a reliable decision?

These questions all hinge on Uncertainty Estimation. Upon further reflection, we identified a more intriguing issue. When a segment of LLM-generated text contains multiple facts or semantic units, can we really measure the reliability of the entire content with a single score?

Suppose the model generates ten factual claims, nine of which are highly reliable while one exhibits significant uncertainty. For downstream systems, knowing only the average confidence of the entire response is clearly insufficient to support fine-grained decision-making. What we truly want to know is: Which specific parts of the content is the model confident about, and which parts remain uncertain?

This issue may become even more critical in future Multi-Agent systems. When multiple AI Agents collaborate, the output of one Agent may serve as the input for another. If Agents can exchange not only natural language information but also the uncertainty associated with each specific Claim, the receiving Agent can leverage these signals to decide whether additional verification, further reasoning, or adjustments to execution strategies are needed.

In other words, we believe that Uncertainty should not merely serve as a metric for evaluating model performance; it should also function as a decision signal that downstream systems can understand and utilize. With this perspective, we began to re-examine a fundamental yet often overlooked question in LLM Uncertainty Estimation: At what granularity should uncertainty be measured?

In our NeurIPS 2026 work, SpanUQ: Span-Level Uncertainty Quantification for Large Language Model Generation, we attempt to answer this question and propose a new research task: Span-Level Uncertainty Estimation (SLUE). We hope that this work will not only provide an effective method for uncertainty estimation but also offer a new starting point for future research into finer-grained and more actionable LLM Uncertainty.

• Paper: https://arxiv.org/abs/2607.05721
• Code: https://github.com/damon-demon/SpanUQ
• Model: https://huggingface.co/DamonDemon/SpanUQ
• Dataset: https://huggingface.co/datasets/DamonDemon/SpanUQ-Benchmark
• Session Time: Thu, Dec 10, 2026 • 3–6 PM in Hall C1 • Atlanta Poster Session 4

01 | From Token to Sequence: Granularity Limitations of Existing Uncertainty Estimation

Currently, there is extensive research on uncertainty estimation for LLMs, with two common granularities being Token-Level and Sequence-Level. However, in practical applications, we have found that both granularities have certain limitations.

1.1 | Token-Level Uncertainty: Fine-Grained but Semantically Incomplete

Token-Level methods typically utilize the probability distribution predicted by the model for each token—such as entropy or token probability—to estimate uncertainty during the generation process. These methods incur low additional computational overhead and provide fine-grained information. The problem, however, is that tokens do not naturally correspond to complete semantic units.

For example, when a model generates a date, number, or entity name, it may consist of multiple tokens. The uncertainty of a single token does not necessarily reflect the reliability of the entire factual claim.

Furthermore, probabilistic uncertainty in language generation is not entirely equivalent to factual uncertainty. A model may confidently generate an incorrect fact, or it may exhibit high token entropy for correct content due to the existence of multiple valid phrasings. Therefore, using Token-Level Uncertainty directly as a basis for downstream factual reliability judgments may introduce bias.

1.2 | Sequence-Level Uncertainty: Holistic View but Difficult to Localize Issues

Another line of research attempts to estimate the uncertainty of an entire response. For example, by performing multiple samplings, comparing semantic consistency, or using other methods, these approaches compute an uncertainty score for the complete generated sequence. While such methods can measure the overall reliability of a model's response, they introduce another problem: even if we know the entire response is unreliable, it is difficult to determine exactly which part contains errors. Consider the following simple example:

Marie Curie was a Polish physicist who won the Nobel Prize in 1901.

This passage contains multiple factual claims:

• Marie Curie was a Polish physicist.
• She won a Nobel Prize.
• She won the Nobel Prize in 1901.

The first two facts are correct, while the last one contains an incorrect year. If we use Sequence-Level methods, we might only obtain a single uncertainty score for the entire response. However, for downstream systems, what truly matters is identifying the specific problematic fact at the end, rather than rejecting the entire response. This leads us to ask: Is there a more natural granularity for uncertainty estimation that lies between Token-level and Sequence-level?

02 | Span-Level Uncertainty Estimation: Aligning Uncertainty with Semantics

Our core observation is that when humans understand, verify, and use information, they typically do not operate at the token level or treat the entire text as a monolithic unit. Instead, they process content in semantically complete chunks. For instance, given a text containing multiple factual claims, we can assess the reliability of each claim individually without viewing the entire response as an indivisible whole. Based on this observation, we formally propose a new research task in our paper:

2.1 | Span-Level Uncertainty Estimation (SLUE)

The goal of SLUE is to identify text segments within LLM-generated output that possess complete, independently evaluable semantics (Semantic Spans), and to estimate the corresponding uncertainty for each span.

Compared to traditional granularities, Span-Level Estimation aims to simultaneously achieve three characteristics.

First, Semantic Coherence: Each Span should correspond as closely as possible to a semantic unit that can be independently understood and evaluated, rather than being an arbitrary combination of tokens.

Second, Fine-Grained Localization: When only part of a response contains uncertain information, we want to precisely locate those specific contents, rather than assigning a single score to the entire response.

Third, Actionable Uncertainty: When uncertainty is associated with specific semantic content, downstream systems can adopt different processing strategies for different spans. For example, verifying only high-uncertainty factual claims instead of re-checking the entire response.

We believe the third point is particularly important because it implies that Uncertainty Estimation is not merely a model evaluation task but also has the potential to serve as a crucial bridge connecting LLM Generation with downstream decision-making. Of course, SLUE does not imply that Token-Level or Sequence-Level Uncertainty loses its value. Different granularities reflect information at different levels; through SLUE, we aim to address the limitations of existing methods regarding semantic localization and practical actionability.

2.2 | From Problem Definition to a Researchable Task

Proposing a new research task is only the first step. The real challenge lies in: How can we automatically identify semantic spans? How do we construct reasonable uncertainty labels for different spans? And how can we perform estimation without significantly increasing inference costs? To address these challenges, we further propose SpanUQ as a preliminary solution for exploring the SLUE task.

03 | SpanUQ: Achieving Span-Level Uncertainty Estimation via Single Forward Pass

In designing SpanUQ, we aimed to solve two key problems simultaneously:

• How to locate the semantic spans that need evaluation?
• How to efficiently and accurately estimate the uncertainty of these spans?

Existing Sampling-Based Uncertainty Estimation methods require generating multiple responses to the same question and then estimating model uncertainty by comparing these different outputs. Although such methods provide valuable insights, multiple sampling typically incurs additional inference overhead. For real-world systems that handle large volumes of requests and are sensitive to latency and cost, this can become a significant bottleneck. Therefore, we propose a different approach:

Can we extract and distill expensive Multi-Sample Uncertainty information into a lightweight model, enabling it to make predictions via a single forward pass?

This is the core design of SpanUQ. SpanUQ is a lightweight Probe with approximately 25M parameters that leverages the Hidden States of a pre-trained LLM to simultaneously perform semantic span detection and uncertainty estimation. The framework primarily consists of the following components.

3.1 | Extracting Uncertainty Information from Hidden States

Internal representations of LLMs contain rich information, and Hidden States from different layers may encode various types of semantic and predictive features. We first extract Hidden States from the pre-trained LLM and convert them into feature representations suitable for subsequent Span-Level prediction through Multi-Layer Fusion and a Token Encoder. During this process, the parameters of the original LLM remain frozen. This means we do not need to retrain the entire base model; instead, we use a relatively lightweight Probe to learn how to extract uncertainty-related signals from internal representations. It is worth noting that SpanUQ does not involve the model directly reading an existing internal variable that explicitly represents factual correctness. Rather, it learns the mapping between Hidden States and target uncertainty labels.

3.2 | DETR-Style Span Decoder: Simultaneously Identifying Multiple Semantic Spans

For Span Detection, an intuitive approach is to formulate it as Token Classification, for example, using BIO tagging to mark whether each token belongs to a specific span. However, this design faces limitations when handling overlapping spans, flexible segment boundaries, and joint prediction tasks. Inspired by the DETR architecture in object detection, we designed a DETR-Style Span Decoder.

The core idea is to use a set of learnable Span Queries that interact with text features via Cross-Attention, thereby predicting multiple semantic segments in parallel. Each query can predict the position of a candidate span and further estimate its corresponding uncertainty. This design frees Span Detection from complete reliance on traditional token-by-token annotation schemes and provides a more flexible structure for modeling overlapping semantic segments. In our experiments, our Span Detector achieved an F1 score of 0.910, representing a 39.4% relative improvement over the strongest heuristic method. This demonstrates that borrowing the Query-Based Prediction paradigm from object detection is a viable path for semantic segment localization.

3.3 | Mixture of Beta: Modeling Continuous Uncertainty

Once a semantic segment is successfully localized, the next challenge is predicting its uncertainty. A simple approach is to directly regress a score between 0 and 1. However, in actual generation tasks, the uncertainty distributions of different spans may exhibit complex shapes. For instance, some content may be highly reliable, some clearly unreliable, and some fall into intermediate states. To model such distributions more flexibly, we employed Mixture of Beta Distributions (MoB). The Beta Distribution is naturally defined on the [0, 1] interval, making it suitable for modeling bounded continuous values. By combining multiple Beta Components, we can express more complex uncertainty distributions rather than relying solely on a single point estimate. During training, we combined Beta Negative Log-Likelihood (Beta NLL) and Contrastive Ranking Objectives. The former helps the model learn continuous uncertainty labels, while the latter encourages the model to correctly distinguish the relative uncertainty of different spans. Both capabilities are crucial because, for practical systems, we care not only about how close the predicted scores are to the target values but also about the model's ability to reasonably rank the reliability of different content.

3.4 | Uncertainty-Conditioned Iterative Refinement

Additionally, we introduced Uncertainty-Conditioned Iterative Refinement (UCIR). The basic idea is to feed the uncertainty information obtained from the first round of predictions back into the shared Decoder to further refine span predictions. This design improves prediction quality with limited additional computational overhead. In our experiments, UCIR incurred less than 15% extra overhead and further enhanced the correlation of uncertainty rankings.

Overall, SpanUQ integrates Span Detection and Uncertainty Quantification into a unified framework. We aim to demonstrate through this approach that it is possible to obtain fine-grained uncertainty signals with semantic localization capabilities without relying on expensive multi-sample inference.

04 | SpanUQ-Bench: Providing a Standardized Evaluation Foundation for SLUE

Proposing a model alone is insufficient for a new research task. Without unified data and evaluation methods, researchers find it difficult to compare the performance of different approaches or systematically understand the challenges inherent in the task itself. Therefore, within SpanUQ, we constructed SpanUQ-Bench to provide a preliminary benchmark foundation for Span-Level Uncertainty Estimation. SpanUQ-Bench includes:

• 20K Prompts
• Approximately 293K Annotated Spans
• Continuous Uncertainty Labels

Unlike traditional binary correctness labels, we aim to further capture the degree of uncertainty for each span across multiple generations and verifications. To this end, we constructed continuous Soft Labels via Multi-Sample Claim Verification. A simple intuition is that if a factual claim exhibits strong consistency across different generations and verification processes for the same question, it likely corresponds to lower uncertainty.

Conversely, if related content shows high inconsistency across multiple samples or lacks stable verification support, its uncertainty may be higher. Of course, empirical signals derived from multiple sampling are not equivalent to absolute probabilities of factual correctness. However, they can provide valuable supervision for learning fine-grained uncertainty estimation. Through SpanUQ-Bench, we hope to transform SLUE from a conceptual research problem into a concrete task that can be trained, evaluated, and compared. We also hope that as more researchers participate, future benchmarks will emerge covering more task types, languages, model architectures, and definitions of uncertainty.

05 | Experimental Results: Accuracy, Efficiency, and Model Applicability

To evaluate the effectiveness of SpanUQ, we conducted experiments on five different LLM backbones, covering two model families including Dense and Mixture-of-Experts (MoE) architectures. The experiments were carried out across multiple dimensions, including Span-Level Uncertainty Quality, Span Detection, and inference efficiency.

5.1 | More Accurate Span-Level Uncertainty Estimation

Across five LLM backbones, SpanUQ achieves strong performance in span-level uncertainty estimation. Specifically, it attains an AUROC of 0.908–0.944 and an MAE of 0.110–0.129. Compared to baselines such as Token Entropy, MLP Probe, SelfCheckGPT-NLI, and P(True), SpanUQ demonstrates superior overall performance. Notably, we focus not only on whether the model can distinguish between high- and low-uncertainty spans but also on the error between continuous prediction scores and target labels. This is particularly critical for downstream systems that rely on uncertainty scores for further decision-making.

5.2 | 10–20× Inference Acceleration

Efficiency is another key advantage of SpanUQ. Traditional sampling-based methods often require multiple model calls, generating several candidate answers to estimate uncertainty. In contrast, SpanUQ leverages LLM hidden states and performs predictions via a lightweight probe. Under the evaluation settings described in the paper, SpanUQ achieves approximately 10–20× inference acceleration compared to sampling-based methods. This result indicates that fine-grained uncertainty estimation does not necessarily incur the high cost of multiple samplings. For real-world systems requiring scalable processing of model outputs, we believe efficient uncertainty estimation is a direction worth continued exploration.

5.3 | From Span-Level to Sequence-Level

Beyond directly evaluating span-level uncertainty, we explored an intriguing question: If uncertainties for multiple semantic spans are already obtained, can we derive the uncertainty of the entire response?

Experimental results indicate that sequence-level uncertainty exhibits a degree of decomposability. This implies that span-level estimation not only provides finer-grained information but may also help us understand higher-level uncertainty. We consider this a finding worthy of further research. It suggests that uncertainties at different granularities are not entirely independent; rather, there exists a structural relationship that can be further modeled.

06 | From Uncertainty Estimation to Uncertainty-Aware Decision Making

At this point, we have introduced the SLUE task, the SpanUQ method, and the corresponding experimental results. However, for us, the more interesting aspect of this research likely extends beyond the current performance improvements. Our true concern lies with a longer-term question: When LLMs can provide uncertainty signals for specific semantic spans within generated content, how can we leverage this information? We believe this may open new research opportunities for future uncertainty-aware AI systems. Below are three directions we find particularly compelling.

6.1 | Uncertainty-Aware Verification: Making Verification More Targeted

As LLMs are increasingly used for knowledge-intensive tasks, verifying the reliability of generated content has become a critical issue. A direct approach is to perform fact-checking on the entire response. However, if a response contains extensive factual information, comprehensive verification may require multiple retrievals, additional model calls, and significant computational costs. In reality, not all content requires the same level of verification. Suppose an LLM generates ten claims, but only two exhibit high uncertainty. If we can obtain an uncertainty score for each claim, downstream systems can invoke search tools or verification models specifically for those two high-uncertainty spans. This allows the system to concentrate limited computational resources on the content most in need of additional checks. Furthermore, different levels of uncertainty may correspond to different verification strategies. For low-uncertainty content, the system can prioritize direct usage; for medium-uncertainty content, lightweight verification can be applied; and for high-uncertainty content, stricter external retrieval or multi-model cross-validation may be considered. It is important to emphasize that low uncertainty does not guarantee factual correctness; therefore, specific verification strategies must still account for task risk, model calibration, and actual business requirements. Nevertheless, we believe span-level uncertainty provides a potential foundational signal for more refined verification strategies.

6.2 | Uncertainty-Guided Self-Refinement: Modifying Only What Truly Needs Modification

In recent years, self-refinement has emerged as a major research direction for improving LLM generation quality. The core idea involves having the model reflect on, evaluate, and correct its own generated content to progressively improve output quality. However, a thought-provoking question arises: When only local parts of a response are problematic, do we really need to regenerate the entire response?

Consider a long response containing multiple factual claims. If most of the content is already sufficiently reliable and only certain spans exhibit high uncertainty, regenerating the entire response may introduce unnecessary computational overhead and even alter originally correct content. Span-level uncertainty offers an alternative solution to this problem. We can first identify high-uncertainty semantic spans and then guide the model to verify, modify, or regenerate only those specific segments. This could shift self-refinement from whole-response regeneration toward more fine-grained, localized corrections. From a research perspective, many questions remain to be explored. For instance, how should we determine whether a span requires refinement based on its uncertainty? How can we maintain contextual consistency while modifying local content? And how can we assess whether refinement truly reduces error risk? These issues require further validation in subsequent studies. Nonetheless, we believe moving from whole-response refinement to targeted span-level refinement is a promising research direction.

6.3 | Uncertainty-Aware Multi-Agent Communication: Enabling Agents to Exchange Reliability Signals Alongside Information

We believe this could be one of the most promising application directions for Span-Level Uncertainty in future AI systems. As Multi-Agent Systems evolve, an increasing number of complex tasks are being completed through the collaboration of multiple agents with diverse capabilities. For instance, one agent might handle information retrieval, another performs analysis and reasoning, while a third formulates execution strategies based on the preceding results. In such systems, content generated by one agent is often utilized by others. However, current focus tends to be on what information is exchanged between agents, rather than systematically considering whether agents should also communicate how reliable that information is.

Consider a scenario where Agent A provides Agent B with an analytical result containing multiple factual claims. If Agent B only receives natural language text, it must independently assess which parts are trustworthy. But if Agent A simultaneously provides an uncertainty score for each claim, the situation changes significantly. Agent B can use these signals to decide whether to directly utilize the information, request additional evidence from Agent A, or invoke another agent for cross-validation. Furthermore, the entire Multi-Agent System could dynamically adjust task allocation, computational resource investment, and subsequent reasoning paths based on the uncertainty levels of different claims. This leads us to envision a broader mode of information interaction:

In the future, content passed between agents may not just be natural language text, but could also include reliability signals associated with specific semantic units.

Of course, realizing this vision involves several open challenges. For example, different agents may rely on entirely different base models; are their output uncertainty scores comparable? How do we ensure that a confidence score of 0.9 from one model carries a similar meaning as one from another? Moreover, how do we prevent miscalibrated uncertainty signals from propagating and amplifying within a Multi-Agent system? Therefore, we do not claim that Span-Level Uncertainty has already solved reliability issues in Multi-Agent Communication. Instead, we hope it serves as a starting point for further research into these problems.

07 | Conclusion: A Nascent Research Direction

As Large Language Models (LLMs) gradually evolve from standalone text generation tools into foundational components of complex AI systems, measuring, understanding, and conveying the reliability of model outputs is becoming an increasingly critical issue. In this work, we started from a relatively fundamental question: What is the appropriate granularity for Uncertainty Estimation?

We argue that while Token-Level Uncertainty offers fine-grained detail, it may lack complete semantic context. Sequence-Level Uncertainty provides holistic judgments but struggles to pinpoint specific problematic content. Associating uncertainty with text spans that possess complete semantics appears to be a natural way to bridge the gap between these two extremes. Based on this insight, we formally propose the Span-Level Uncertainty Estimation (SLUE) task, providing preliminary methods and evaluation foundations through SpanUQ and SpanUQ-Bench.

Naturally, we recognize that this research is still in its early stages. Currently, our primary focus is on efficiently estimating the uncertainty of semantic spans from LLM internal representations. However, there are many questions worth exploring in broader practical applications, such as:

• How can we further improve Uncertainty Calibration so that it becomes a decision signal truly worthy of reference for downstream systems?
• How can Span-Level Uncertainty guide models toward more efficient and targeted Verification and Self-Refinement?
• When multiple agents use different base models, how can fine-grained uncertainty information be reliably transmitted and understood among them?
• Beyond factual claims, can Span-Level Uncertainty extend to complex reasoning processes, code generation, and other types of structured outputs?

There are currently no simple answers to these questions. However, we believe that as LLMs and Multi-Agent Systems continue to develop in real-world applications, the goal of Uncertainty Estimation research will likely expand from mere model evaluation to broader Uncertainty-Aware Decision Making. In this process, Uncertainty Signals capable of localizing specific semantic content may play an increasingly important role.

We hope that SpanUQ serves not only as a method for solving the SLUE task but also as a starting point for the research community to explore fine-grained uncertainty estimation. To this end, we have open-sourced the relevant code, models, and benchmarks, aiming to facilitate future research and encourage more researchers to explore this direction together.

First Author Introduction:

Yimeng Zhang is currently an Applied Scientist at Amazon. His research interests include Generative AI, AI Agents, and Multimodal Intelligent Agents. He received his Ph.D. in Computer Science from Michigan State University in 2025, where he was a member of the OPTML Lab under the supervision of Associate Professor Sijia Liu. He previously studied at Columbia University and the University of Sheffield. His research focuses on efficient machine learning, model robustness, and safety, covering areas such as LLMs, diffusion models, and adversarial learning. He has published numerous papers at international conferences including CVPR, NeurIPS, ICLR, and ICML.

© THE END

Please contact this official account for authorization before reprinting.

For submissions or media inquiries: [email protected]