2506.07461v1

# From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centered **Authors**: Siddartha Devic Tejas Srinivasan Abstract Large Language Models (LLMs) are increasingly assisting users in the real world, yet their reliability remains a concern. Uncertainty quantification (UQ) has been heralded as a tool to enhance human-LLM collaboration by enabling users to know when to trust LLM predictions. We argue that current practices for uncertainty quantification in LLMs are not optimal for developing useful UQ for human users making decisions in real-world tasks. Through an analysis of 40 LLM UQ methods, we identify three prevalent practices hindering the community’s progress toward its goal of benefiting downstream users: 1) evaluating on benchmarks with low ecological validity; 2) considering only epistemic uncertainty; and 3) optimizing metrics that are not necessarily indicative of downstream utility. For each issue, we propose concrete user-centric practices and research directions that LLM UQ researchers should consider. Instead of hill-climbing on unrepresentative tasks using imperfect metrics, we argue that the community should adopt a more human-centered approach to LLM uncertainty quantification. 1 Introduction Classical machine learning and forecasting communities have long been focused on the tantalizing possibility of predicting outcomes and future events. A key, highly related goal is to accurately estimate when model predictions can safely be trusted or otherwise ignored by a human user (Guo et al.,, 2017; Valdenegro-Toro and Stoykova,, 2024; Marusich et al.,, 2024). This goal forms the basis for methods that estimate predictive uncertainty of machine learning models, which we will collectively refer to as uncertainty quantification (UQ) methods (Sullivan,, 2015). While UQ has traditionally been studied in classical machine learning models, the advent of large language models (LLMs) has precipitated a host of research creating and adapting UQ methods suited to this new paradigm. The primary reason for quantifying LLM uncertainty is user-centric: the motivation behind many proposed LLM UQ methods is to enable users to know when to trust LLM outputs (Xiong et al.,, 2023; Lin et al.,, 2024; Kuhn et al.,, 2023; Duan et al.,, 2023), improving the reliability and trustworthiness (Duan et al.,, 2023) of LLMs and allowing for their proliferation into high-stakes domains (Bakman et al.,, 2024). LLM UQ methods include calibrating confidence scores displayed to users for particular generations or claims within generations (Detommaso et al.,, 2024; Yang et al., 2024c, ), altering the language used in generations to better represent the uncertainty that the model has in its generations (Band et al.,, 2024; Stengel-Eskin et al.,, 2024), and generating predictions sets with coverage guarantees in a conformal prediction framework (Quach et al.,, 2024). Although LLM UQ methods have demonstrated the ability to quantitatively improve certain LLM capabilities —for example, via post-processing (Li et al.,, 2024) or sampling and aggregating multiple generations (Kuhn et al.,, 2023) —we argue that the original motivation to foster better reliance by users has fallen by the wayside. In particular, human studies are few and far between in the LLM UQ literature, and consequently we lack a comprehensive understanding of how useful LLM UQ methods are in assisting humans on real-world decision making tasks, or when these methods increase the transparency of recommendations and decisions generated by an LLM. Instead, many proposed LLM UQ methods are focused on hill-climbing flawed calibration metrics capturing only epistemic uncertainty on a narrow range of benchmarks which lack ecological validity (De Vries et al.,, 2020). Currently, LLM UQ is primarily a quantitative area, with new technical tools achieving improved benchmark scores (e.g. lower calibration errors) every month. In this position paper, we question the assumption that current practices for benchmarking LLM UQ methods moves us efficiently towards optimal human-AI collaboration. We identify various common practices in the community that hinder us from properly evaluating the utility of LLM UQ methods for LLM-assisted decision making. We then lay out recommendations which we believe will push LLM UQ towards providing value for real-world users, and potentially allow for its incorporation into the many LLM-based products with which millions of humans are now interacting. 1.1 Position and Main Claims We argue that the NLP community’s prevailing practices for evaluating LLM UQ methods are insufficient to benefit human users in real-world settings. From this user-centric viewpoint, we identify three major barriers towards widespread adoption of LLM UQ methods: 1. In Section 3, we argue that LLM UQ methods are primarily evaluated on benchmarks with low ecological validity which are not representative of real-world decision making scenarios. 1. In Section 4, we argue that benchmarks commonly used for LLM UQ evaluation do not consider aleatoric uncertainty, and to an extent, distributional uncertainty. These are important types of uncertainty that a user may encounter when using a large language model (LLM) as a decision aid. 1. In Section 5, we first argue that the interaction between LLM UQ methods and real-world users is heavily understudied and not well understood. Next, we claim that LLM UQ methods are evaluated using metrics that have little relation with downstream utility to users. Finally, possible uncertainty presentation schemes to LLM users on language tasks itself is understudied. For each issue, we highlight concrete remedies and research directions that LLM UQ methods researchers may find illuminating. Together, we hope that these pointed shortcomings of existing UQ practices can guide the field to providing useful algorithmic advice in real-world human-AI collaboration. We also discuss some alternative views in Section 6. 2 Preliminaries and Methodology The goal of uncertainty quantification (UQ) methods in both classical machine learning and LLMs is to extract an estimate across all sources of uncertainty in the system. UQ methods typically attempt to quantify three types of uncertainty: epistemic, aleatoric, and distributional (Hüllermeier and Waegeman,, 2021; Ben-David et al.,, 2010). Epistemic uncertainty, also known as model uncertainty, represents uncertainty about an outcome due to the limitations of the model. Epistemic uncertainty could arise from compute, data, or optimization limitations during training, or simply from choosing the wrong model for a particular task. Aleatoric uncertainty, on the other hand, represents uncertainty due to inherent randomness or noise in the data. The canonical example of aleatoric uncertainty is diagnosing patients with the flu: given two patients with identical symptoms, one may have the flu and the other may have allergies. The presence of aleatoric uncertainty supposes that outcomes assigned by nature are, to an extent, inherently random, and cannot be reduced by collecting additional training data. Distributional uncertainty represents the common scenario when the test and train distributions are different in some quantitative aspect. For example, a pre-trained LLM selected over other options because it achieved the highest evaluation scores on benchmarks such as MMLU or TriviaQA (Hendrycks et al.,, 2021; Joshi et al.,, 2017) may face a distribution shift in the presence of real-world user queries at deployment time. A more comprehensive overview of the types of uncertainty is deferred to He et al., 2023b ; Smith, (2024); Shorinwa et al., (2024). Uncertainty quantification methods usually attempt to estimate some subset or all of these sources of uncertainty. LLM UQ methods in particular fall into a few broad categories. First, there are methods which are unsupervised: some of these methods sample multiple generations in order to test consistency of the response, and measure uncertainty via variants of entropy (Xiong et al.,, 2023; Lin et al.,, 2024; Kuhn et al.,, 2023; Duan et al.,, 2023; Malinin and Gales,, 2021), and other methods simply prompt the LLM in different ways in order to probe its confidence in the answer (Tian et al.,, 2023; Kadavath et al.,, 2022; Yang et al., 2024c, ). There are also supervised methods which attempt to first elicit confidences via token level probabilities or special prompts, and then calibrate them in a post-hoc manner (Li et al.,, 2024; Ulmer et al.,, 2024; Chen et al.,, 2024; Detommaso et al.,, 2024; Mielke et al.,, 2022). Finally, there are methods which modify the LLM generation itself using tools like conformal prediction in order to better reflect the confidence of the LLM (Quach et al.,, 2024; Jiang et al.,, 2025; Cherian et al.,, 2024; Wang et al., 2025b, ). For a full discussion of related work, we defer to more comprehensive surveys (Shorinwa et al.,, 2024; Huang et al.,, 2024; Liu et al.,, 2025). Evaluation Methodology. We analyze 40 LLM UQ methods papers and their associated evaluation benchmarks to provide evidence for our claims. We collect methods papers and the references therein from two recent LLM UQ surveys (Shorinwa et al.,, 2024; Huang et al.,, 2024). In Appendix B, we provide a brief annotation of each selected paper, focusing on aspects relevant to our critiques such as “Does the paper test the robustness of the method towards distributional uncertainty?” and “What are the benchmarks that the paper uses to evaluate its LLM UQ method?”. Scope and Limitations. In this work, we only consider LLM UQ methods whose stated motivation is to improve user decision making by conveying uncertainty. We therefore do not consider UQ methods which primarily aim to improve capabilities of language models on reasoning, mathematical, or science benchmarks. For example, improving GRPO-style LLM reasoning chains with UQ (Ye et al.,, 2025) is out of scope. Furthermore, the scope of our claims does not extend to LLM UQ methods which may allow LLMs to act as better, fully autonomous agents in the wild (Liu et al., 2024b, ). For additional discussion on limitations of our analysis, we refer to Section 6. 3 Ecological Validity of UQ Benchmarks We argue that the LLM UQ methods primarily evaluate on benchmarks with low ecological validity (Bronfenbrenner,, 1977) and findings on such benchmarks are unlikely to generalize to queries posed by users in real-world settings. If we want LLM UQ methods to benefit users with decision making problems, then it is necessary to evaluate them on tasks that reflect real-world usage patterns. Ecological validity (Bronfenbrenner,, 1977) refers to the degree to which experimental results generalize to real-world settings. As LLMs are being deployed to assist people in the wild, the problem of ecological validity of NLP datasets has gained focus (De Vries et al.,, 2020), with several efforts being made to develop benchmarks that reflect realistic user-AI interactions (Lin et al.,, 2023; Zhao et al.,, 2024). De Vries et al., (2020) identify several ways in which NLP datasets lack ecological validity: synthetic language, artificial tasks, not working with prospective users, scripted interactions, and limiting to single-turn interactions. In this section, we assess the ecological validity of the 22 distinct benchmarks evaluated by the 40 LLM UQ methods we surveyed. We only consider benchmarks that were used by at least two methods papers. 3.1 LLM UQ benchmarks represent a narrow range of tasks <details> <summary>x1.png Details</summary>  ### Visual Description ## Bar Chart: Number of Benchmarks in Category ### Overview The chart displays a horizontal bar visualization comparing the number of benchmarks across eight categories. The x-axis represents the count of benchmarks (0–8), while the y-axis lists categorical labels. Bars are uniformly blue, with no legend present. The data suggests a hierarchical distribution of benchmark density across categories. ### Components/Axes - **Y-Axis (Categories)**: - Factual QA - Commonsense Reasoning - Medical QA - Sentiment Analysis - Open-book QA - Math - Topic Classification - Other reasoning - **X-Axis (Scale)**: - Labeled "# of Benchmarks in Category" - Range: 0 to 8 (discrete increments) - **Visual Elements**: - Horizontal bars (blue) aligned to y-axis categories - No gridlines or annotations beyond axis labels ### Detailed Analysis - **Factual QA**: Longest bar, extending to ~7 benchmarks (highest value). - **Commonsense Reasoning**: Second-longest bar, ~5 benchmarks. - **Medical QA** and **Sentiment Analysis**: Equal length, ~3 benchmarks each. - **Open-book QA**: ~2 benchmarks. - **Math**, **Topic Classification**, and **Other reasoning**: Shortest bars, each ~1 benchmark. ### Key Observations 1. **Dominance of Factual QA**: Factual QA accounts for over 50% of the total benchmarks (~7/13 total). 2. **Mid-Range Categories**: Commonsense Reasoning (~5) and Medical QA/Sentiment Analysis (~3) form the mid-tier. 3. **Low-Density Categories**: Math, Topic Classification, and Other reasoning each have only ~1 benchmark, indicating sparse representation. 4. **Symmetry in Lower Tiers**: The three lowest categories share identical benchmark counts, suggesting potential overlap in evaluation focus or resource allocation. ### Interpretation The data highlights a stark imbalance in benchmark development across categories. Factual QA and Commonsense Reasoning dominate, reflecting their perceived importance in AI evaluation. The near-absence of benchmarks in Math and Topic Classification may indicate either emerging fields or insufficient prioritization. The clustering of Medical QA and Sentiment Analysis at ~3 benchmarks suggests moderate but distinct evaluation needs. The uniformity of the lowest three categories raises questions about whether these represent niche areas or methodological constraints in benchmark creation. This distribution could inform resource allocation for future benchmark development, emphasizing underrepresented domains like Math and Topic Classification. </details> Figure 1: Category-wise distribution of LLM UQ benchmarks. Our first observation, illustrated in Figure 1, is that the majority of LLM UQ methods are evaluated on only factual question answering or commonsense reasoning tasks. In particular, seventeen of the benchmarks are either question-answering (QA) or common sense reasoning tasks. Real-world user queries are much more diverse; for instance, WildChat (Zhao et al.,, 2024) find that just 6.3% of their LLM queries are factual question answering. If a majority of LLM UQ method evaluations are on only these two domains, we will not know much about evaluated methods’ ability to provide useful uncertainty estimates on the wider range of decision making applications for which people use LLMs. 3.2 LLM UQ benchmarks do not reflect user-grounded applications | Benchmark | Category | C1: The benchmark has a meaningful connection to a real-world task | C2: The benchmark inputs are representative of real-world task inputs | C3: People find the benchmark to be challenging or require notable effort | C4: Undesirable events may occur due to bad decisions. | | --- | --- | --- | --- | --- | --- | | TriviaQA | Factual QA | ✗ | - | ✓ | ✗ | | Natural Questions | Factual QA | ✓ | ✓ | ✓ | ✗ | | MMLU | Factual QA | ✗ | - | ✓ | ✗ | | CoQA | Open-book QA | ✗ | - | ✓ | ✗ | | TruthfulQA | Factual QA | ✗ | - | ✗ | - | | SciQ | Factual QA | ✗ | - | ✓ | ✗ | | GSM8K | Math | ✗ | - | ✗ | ✗ | | BIGBench | Other reasoning | ✗ | - | ✓ | ✗ | | SST | Sentiment Analysis | ✗ | - | ✗ | ✗ | | WebQA | Open-book QA | ✓ | ✗ | ✗ | ✗ | | CommonSenseQA | Commonsense | ✗ | - | ✗ | ✗ | | IMDB | Sentiment Analysis | ✓ | ✓ | ✗ | ✗ | | AGNews | Topic Classification | ✓ | ✓ | ✗ | ✗ | | Yelp Reviews | Sentiment Analysis | ✓ | ✓ | ✗ | ✗ | | NQ-Open | Factual QA | ✓ | ✓ | ✗ | ✗ | | AmbigQA | Factual QA | ✗ | - | ✓ | ✗ | | MNLI | Commonsense | ✗ | - | ✗ | ✗ | | SNLI | Commonsense | ✗ | - | ✗ | ✗ | | SWAG | Commonsense | ✗ | - | ✗ | ✗ | | HellaSwag | Commonsense | ✗ | - | ✗ | ✗ | | BioASQ | MedicalQA | ✗ | - | ✓ | ✗ | | MedMCQA | MedicalQA | ✗ | - | ✓ | ✗ | Table 1: An analysis of our 22 benchmarks, categorized and annotated for the criteria in Section 3.2. In the context of human-centered explainable AI (XAI), Chaleshtori et al., (2024) introduce criteria for determining the suitability of a benchmark for evaluating the human utility of model explanations. These criteria are not specific to explanations, and hence we adopt them for evaluating the ecological validity of LLM UQ benchmarks: 1. The benchmark has a meaningful connection to a real-world task where a user might seek assistance from an LLM. For a benchmark to be ecologically valid, it must be grounded in a real-world application. Examples of benchmarks not satisfying this criterion include CommonsenseQA (Talmor et al.,, 2019), StrategyQA (Geva et al.,, 2021), and TriviaQA (Joshi et al.,, 2017). We consider the benchmark’s construction and input domain when deciding whether it satisfies this criterion. For instance, while users may use LLM assistants to answer factual questions, TriviaQA consists of niche trivia questions. The only application in which a person might ask a trivia question without knowing its answer would be if they were participating in a contest, where they would not be allowed to use an LLM. 1. The benchmark’s inputs are representative of task inputs in real-world LLM use. Even if the benchmark is grounded in a real-world application, its inputs must represent real inputs that a user might provide to the LLM. For instance, WebQA (Chang et al.,, 2022) consists of questions that are designed to yield a specific answer known ahead of time, which is unlikely to be the case when a user is actually collaborating with an LLM. 1. The benchmark instances require notable effort from people, or people are bad at the task (when provided reasonable resources). Tasks that a user can complete accurately with ease do not require them to collaborate with an LLM, rendering them unsuitable for studying LLM uncertainty quantification methods. For instance, one of the most popular benchmarks for studying UQ is GSM8K (Cobbe et al.,, 2021), which involves solving grade-school math problems which most users can solve with ease. 1. Some undesirable event may occur as a result of bad decisions. Quantifying LLM uncertainty is particularly important in tasks where the user is vulnerable to some undesirable consequences as a result of ineffective human-LLM collaboration. Chaleshtori et al., (2024) argue that only benchmarks satisfying criteria C1–C3 are suitable for studying user reliance, and among those, benchmarks also satisfying criteria C4 should be prioritized. In our annotation of whether LLM UQ benchmarks satisfy these criteria (Table 1), we find that only 6 out of 22 benchmarks (27.3%) satisfy C1 (grounded in a real application), and only 5 also satisfy C2 (realistic inputs). 9 out of 22 benchmarks satisfy criterion C3 (challenging for users). However, only 2 benchmarks satisfy all of C1–C3: Natural Questions (Kwiatkowski et al.,, 2019) and NQ-Open (Lee et al.,, 2019), which consist of information seeking questions from real Google search queries. None of the benchmarks satisfy C4 (high-stakes), indicating that popular benchmarks for evaluating LLM UQ methods tell us little of their applicability to high-stakes decision making tasks. 3.3 LLM UQ benchmarks reflect unrealistic task formats Finally, we find that 15 of the 22 benchmarks we analyzed (68.2%) are multiple choice problems. We consider classification tasks with a fixed set of classes, such as sentiment analysis or topic classification, to be multiple choice tasks for LLMs. MCQA questions are known to have many issues, as detailed in works such as Khatun and Brown, (2024); Zheng et al., (2024); Balepur et al., (2025). At a high level, the issues with MCQA evaluations include position-ordering bias, rigidity of the task format, presence of adversarially chosen distractors, brittleness to perturbations, and more. Most importantly, real-world user queries are hardly ever multiple choice, as demonstrated in ShareGPT (Ouyang et al.,, 2023) and WildChat (Zhao et al.,, 2024). Even for simple knowledge queries captured by QA benchmarks, real-world user queries are still short- or long-form generations. Evaluating LLM UQ methods primarily on MCQA benchmarks is not necessarily informative of their ability to estimate uncertain when the set of possible decisions is not enumerated. 3.4 Recommendations We make several recommendations for selecting more appropriate evaluation benchmarks: 1. Developers of LLM UQ methods should evaluate on a wider diversity of benchmarks that include applications beyond factual question answering and do not come with a fixed set of options guaranteed to contain the correct answer. When evaluating on MCQA benchmarks, we recommend using stable evaluation protocols as prescribed by Balepur et al., (2025). 1. When selecting benchmarks for evaluating LLM UQ methods, we recommend evaluating benchmarks against the criterion adopted from Chaleshtori et al., (2024) in Section 3.2. Benchmarks that satisfy these criteria will provide more meaningful insights towards the applicability of LLM UQ methods in real-world human-AI collaboration. 1. We call for the community to pay more attention towards understanding the nature of uncertainty in generation tasks, such as code generation (Vasconcelos et al., 2025a, ), summarization (He et al.,, 2024), planning (Hu et al.,, 2024), and other complex reasoning tasks that do not meet simplistic notions of “correctness” that factual QA tasks do. 4 Sources of Uncertainty Considered in LLM UQ In this section, we argue that many of the evaluation benchmarks used to measure the progress of LLM UQ methods fail to address two critical aspects: aleatoric uncertainty and distribution shift. From a user-centric viewpoint, an ideal LLM UQ method would be capable of representing all sources of uncertainty present, not only epistemic. However, we find a prevailing tendency in LLM UQ research to favor established QA benchmarks, despite their lack of aleatoric uncertainty. Furthermore, despite the importance of LLM UQ methods to detect out of distribution examples (Lin et al.,, 2022), we find that nearly half of all surveyed supervised LLM UQ methods are not evaluated on their performance under distribution shift. Together, these oversights suggest that some LLM UQ methods may lack reliability in real-world scenarios containing interacting forms of uncertainty. 4.1 Aleatoric Uncertainty Human decision makers are often faced with tasks that may be inherently uncertain. For example, the electronic patient health records available to a clinician in an emergency triage situation may not fully explain the state that the patient is in, but the clinician is still required to making a treatment decision (Alur et al.,, 2023). A bank may have only a few features about a potential client available — their credit score, income, etc. — and yet the bank needs to make a decision on whether to give out a mortgage loan or not. In each of these cases — due to the inherent randomness of nature, partial observability of features, or both — true outcomes may be non-deterministic and contain aleatoric uncertainty. Indeed, it is often in similar high stakes situations that many envision machine assistance to potentially be the most impactful (De-Arteaga et al.,, 2020; Brennan et al.,, 2009). Most LLM UQ methods have failed to address and evaluate on tasks with aleatoric uncertainty. The most popular datasets that LLM UQ papers are evaluated on are QA datasets. For example, benchmarks such GSM8k (Cobbe et al.,, 2021), MMLU (Hendrycks et al.,, 2021), and TriviaQA (Joshi et al.,, 2017) all collect questions with a single correct answer, labeled and checked by humans. The existence of a single gold label essentially eliminates the possibility of estimating aleatoric uncertainty, which relies on outcomes themselves not always having the same, correct answer. The only remaining possible source of aleatoric uncertainty is unintentional noise from annotator mistakes, and that is often removed in followup works seeking to improve a benchmark, e.g. Rücker and Akbik, (2023). In fact, of the 22 benchmarks used in LLM UQ papers that we evaluated, we found that only one of the 22 benchmarks we analyze contained aleatoric uncertainty intentionally (Min et al.,, 2020, AmbigQA). A lack of evaluation on benchmarks containing aleatoric uncertainty is not due to the fault of benchmark creators. Indeed, there exist a plethora of benchmarks designed with aleatoric uncertainty in mind. For example, SST-5 (with the full five classes, Socher et al., (2013)), AmazonReviews (Hou et al.,, 2024), toxic text classification datasets such as ToxiGen (Hartvigsen et al.,, 2022), and opinion aggregation datasets like ChaosNLI (Nie et al.,, 2020) contain multiple label annotations per instance with label variation, thus intrinsically encoding the aleatoric uncertainty in the underlying population. One recent dataset proposed specifically to evaluate LLM UQ methods under aleatoric uncertainty is FolkTexts (Cruz et al.,, 2024), which asks an LLM to answer questions about an individual based on open US Census data. Since the dataset has hundreds of thousands of datapoints for each task, given a particular task and set of features, the dataset often contains multiple, possibly conflicting outcomes. A calibrated LLM, therefore, would output reasonable probabilistic estimates on every individual. Importantly, (Cruz et al.,, 2024) finds that not all LLM uncertainty elicitation and calibration methods on LLMs perform well under aleatoric uncertainty, further highlighting the need to test for it explicitly. Further, autoregressive LLMs have been shown to exhibit inconsistent label prediction and probability distributions in such multi-label scenarios (Ma et al.,, 2025). Ambiguity and Selective Prediction. Areas such as ambiguous question answering (Cole et al.,, 2023; Zhang and Choi,, 2021) and selective prediction (Geifman and El-Yaniv,, 2017; Chen et al.,, 2023) also incorporate elements of aleatoric uncertainty in LLMs. For example, ambiguous QA datasets such as AmbigQA (Min et al.,, 2020) and Ambiguous Trivia QA (Kuhn et al.,, 2022) introduce aleatoric uncertainty through linguistic ambiguity of queries. However, evaluation is usually tied to identifying clarifying questions that would disambiguate the original query into a new query with a single correct answer, rather than demanding that an LLM represent the probabilistic ambiguity of the original query. Additional work in selective prediction also highlights the need for more LLM UQ methods to be tested and verified against aleatoric uncertainty. For example, Vazhentsev et al., (2023) demonstrate that UQ methods which are designed and tested on only capturing epistemic uncertainty can under-perform on selective prediction tasks which contain aleatoric uncertainty. These related developments underscore an important point: if LLM UQ methods are only ever evaluated against epistemic uncertainty — and not tested on their ability to quantify and express irreducible, data-inherent uncertainty present in many real-world scenarios — then UQ methods may end up less useful for user-centric downstream applications like selective prediction or ambiguous QA. This blind spot reinforces the need to test against aleatoric uncertainty as a central tenet. 4.2 Distribution Shift Real-world LLM users are diverse in many ways, including geographically (Zhao et al.,, 2024; Dunn et al.,, 2024), and culturally (Hershcovich et al.,, 2022), with tangible impacts on downstream applications. For example, Xing et al., (2022) demonstrate that the way that users interact with conversational search agents differs in complexity and length by user age. Robustness to these variations is often not measurable when evaluating a LLM UQ method on a train / test split of a single benchmark. Instead, robustness to query variations requires that LLM UQ methods remain useful in the face of a third kind of uncertainty: distribution shift. Many LLM UQ methods are focused on post-processing the outputs of the model in a way which improves calibration or uncertainty estimation. Such methods usually fall within the regime of supervised parameter learning, i.e., the method requires learning some parameters from a set of data. For example, the well-known Temperature or Platt scaling UQ methods (Guo et al.,, 2017; Platt et al.,, 1999) —- variants of which are often applied in LLM UQ (Ulmer et al.,, 2024; Xie et al.,, 2024) — require hold-out data in order to fit re-scaling parameters for model logits or probabilistic predictions. Even unsupervised parameter-free LLM UQ methods such as verbalized confidence scoring (Tian et al.,, 2023; Xiong et al.,, 2023) often apply post-hoc calibration to improve the final uncertainty estimate on a particular task (Wang et al., 2025a, ). For such methods, the exact hold-out data used makes a difference in the learned parameters, and hence, the performance, of the UQ method. It is therefore vital to assess performance on data which is out-of-distribution (OOD): i.e., queries which look different from the training data used to learn the original parameters of the LLM UQ method. Consistent failure of LLM UQ methods to recognize or adapt to distribution shifts may potentially undermine user trust in the LLM’s reliability, which can result in decreased utility of the joint human-AI system (Srinivasan and Thomason,, 2025). Unfortunately, we find that robustness to distribution shift was discussed and tested in only 10 out of the 19 supervised, parameter-learning LLM UQ methods papers we evaluated. We believe that testing post-hoc calibration methods under distribution shift should be a requirement of any LLM UQ paper. Evaluating under distribution shift as a standard practice would benefit LLM UQ methods in real-world settings where user-queries may change over time or shift with new demographics, and also when LLM UQ methods are used in downstream tasks — such as hallucination detection (Detommaso et al.,, 2024; Zhang et al.,, 2023) or selective prediction (Chen et al.,, 2023) — where OOD performance is essential for reliability. 4.3 Recommendations To address the concerns raised in this section, we provide two simple and actionable recommendations: 1. Evaluate methods on datasets with intrinsic, intentional aleatoric uncertainty (like FolkTexts (Cruz et al.,, 2024) or ChaosNLI (Nie et al.,, 2020)). 1. For works proposing supervised LLM UQ methods, ensure that you are testing on settings with distribution shift. Evaluating under distribution shift is especially important for methods utilizing conformal prediction guarantees (Cherian et al.,, 2024; Quach et al.,, 2024), since theoretical conformal guarantees usually do not hold under distribution shift (Aolaritei et al.,, 2025). 5 Metrics and Underscoring the Need for Human Studies We argue that metrics for optimizing and evaluating LLM UQ methods may not sufficiently capture the ability of LLMs to foster effective and meaningful human-AI collaboration. In particular, most papers we surveyed focus on standard calibration measures such as Brier score, ECE, or AURUC on simple QA tasks. In Appendix A, we also argue a switch from ECE to smECE (Błasiok and Nakkiran,, 2024). We argue for a focus on metrics that correlate with human uplift on real-world decision tasks. This user-centric shift requires advancement in both the theory of human-AI collaboration and practical human studies on the utility of algorithmic assistance with quantified uncertainty. Although some studies have already been conducted in classical UQ settings (Corvelo Benz and Gomez Rodriguez,, 2025; Alur et al.,, 2023), the interplay between generated language, UQ, and usefulness to users has been drastically under-considered. Of the issues and directions that we raise in each section, this issue requires the most thought, focus, and careful research to execute. 5.1 Inspiration from Classical UQ: Calibration Metrics for Real-world Usefulness When an LLM assists a doctor in diagnosing a complex case or a software engineer in evaluating potential bugs in a section of code, the fundamental goal of UQ extends beyond achieving good ECE or Brier score on the task. Instead, a common desiderata of the LLM UQ literature is to make language models more useful and transparent as a human aide (Yang et al., 2024c, ; Tian et al.,, 2023; Sung et al.,, 2025; Li et al.,, 2024). Computing the uncertainty of a model on a particular task or data point is one facet of this goal. However, as the aforementioned examples demonstrate, simply computing a calibrated probability is often not the end of the story. In order to declare success, LLM UQ — like many other research areas related to modern LLM applications (Kapoor and Narayanan,, 2025) — requires human uplift studies. In particular, do humans with access to algorithmic advice with quantified uncertainty perform better on real-world tasks than humans with (i) algorithmic advice but no quantified uncertainty; and (ii) no algorithmic advice at all? Both the theory and the human-computer interaction (HCI) communities have explored aspects of how uncertainty should be presented to users in order to provide uplift; i.e., help users improve their decision making by accessing algorithmic advice. In classical (non-LLM) UQ, results are still emerging, but they already point to a somewhat surprising trend: simply calibrating the probabilities does not seem to be the best way to present uncertainties to users. For example, Donahue et al., (2022) prove a necessary and sufficient condition on the complementariness of human and algorithmic predictions in order for uplift to occur in a simple theoretical framework. Importantly, model calibration (or accuracy) does not always satisfy the required condition. There is a host of additional work in UQ which echoes similar results: Vodrahalli et al., (2022) show that humans on a real task perform better when given access to algorithmic advice which is uncalibrated rather than calibrated. Cao et al., (2024) also show that model calibration is not a sufficient condition for uplift in human studies, and focus on designing uncertainty presentation schemes which are more effective. Finally, Corvelo Benz and Rodriguez, (2023) prove that calibrated models will always be suboptimal when aggregating human and algorithmic predictions. Instead, they propose obtaining models which are not necessarily calibrated overall, but instead are human-aligned in a formal sense defined by how human and algorithmic predictions relate to one another. They then demonstrate that human-alignment empirically correlates with improved algorithmic assisted human performance in human studies ran on a synthetic task (Corvelo Benz and Gomez Rodriguez,, 2025). Lastly, Vasconcelos et al., 2025b show that LLM UQ methods may need to be further modified to provide uplift for programmers on real-world coding tasks. Together, these works reveal a striking disconnect between research on human-AI collaboration and LLM UQ. At face value, LLM UQ research claims to be focused on the goal of making language models more reliable and trustworthy (Li et al.,, 2024). At a deeper level, however, there exists very little research which has directly confirmed whether LLM UQ is actually useful to humans on real decision making tasks. In particular, human-uplift studies from both the HCI and theory communities seem to demonstrate that calibrated probabilities — the gold standard of LLM UQ — may not be the correct metric for optimal human-AI collaboration. Nonetheless, many LLM UQ papers remained as focused as ever on providing simple, calibrated probabilities on tasks like factual question answering or knowledge retrieval. While optimizing LLM UQ metrics like ECE and Brier score may already be useful for applications like selective prediction (Li et al.,, 2023) or pure capabilities research like reasoning (Taubenfeld et al.,, 2025), optimizing these same metrics not necessarily imply that we will get better collaborative human-LLM performance “for free” out of the box. New Uncertainty Presentation Schemes for LLMs. As articulated in Liao and Vaughan, (2023, Section 3.5), in addition to simply presenting probabilities or confidences to users, LLM UQ techniques can look very different than classical UQ ones through taking specific advantage of the medium of language. For example, LLM UQ methods may utilize unique ways of introducing vaguer claims in order to refrain from complete abstention when highly uncertain (Jiang et al.,, 2025), or use linguistic hedging (Mielke et al.,, 2022) and anthropomorphic language (Kim et al.,, 2024) to convey uncertainty in non-numeric ways to users. Such methods are unique to LLM UQ and should be further investigated to understand how they impact human uplift on decision tasks. 5.2 Recommendations To ameliorate the aforementioned shortcomings of current LLM UQ evaluation metrics, we advocate for three research directions, which we outline below. Via these proposed approaches, we push for better cohesion between NLP researchers developing LLM UQ methods and the HCI community. Ideally, this will create and accelerate feedback loops between both communities, allowing for more rapid iteration and investigation into LLM UQ approaches which provide measurable human uplift. 1. Adapt findings from existing studies in classical UQ on human-aligned calibration (Corvelo Benz and Gomez Rodriguez,, 2025; Vodrahalli et al.,, 2022) to uncertainty-augmented LLM advice. 1. Develop studies which attempt to understand the relationship between LLM UQ metrics (like ECE, Brier score) and human-AI team performance. 1. Incentivize more studies on how the presentation of LLM uncertainty impacts user reliance. Recommendation R6 is fairly straightforward and builds upon the numerous UQ papers with human studies mentioned in Section 5.1. It is possible that presenting probabilities of generated text via LLM UQ methods may have a differing effect on users when compared with the classical UQ setting. Language responses may offer explanations and anthropomorphic cues, whereas predictive classical models used in prior human studies do not. Understanding the effect of presenting uncertainty estimates alongside LLM-generated responses on human-AI collaboration outcomes is vital. Our next recommendation R7 is more fundamental: it articulates a necessity to better understand the metrics that LLM UQ seeks to optimize. The community stands to benefit from comprehensive analyses on whether metrics like ECE or Brier score correlate with increased downstream performance by users. Understanding this relationship between metrics and downstream utility could inform the LLM UQ community on whether to continue optimizing for metrics such as ECE, or potentially switch to other measurable metrics identified as important by human studies. As an example, the seminal natural language summarization metric ROUGE was shown to correlate with human summary judgments, making researchers more confident that it is a reasonable metric to hillclimb on (Lin,, 2004). Furthermore, other metrics like BLEU in machine translation were shown to have the opposite behavior — that is, they were shown to at times correlate poorly with human judgments (Callison-Burch et al.,, 2006) — shifting attention to the development of new metrics (Freitag et al.,, 2022). Without evidence that calibration metrics are correlated with utility to users, our community risks "flying blind" by choosing ad hoc evaluation metrics. Lastly, R8 points to the need to better understand the effect of LLM uncertainty presentation on user reliance. It is important for researchers to understand what types of LLM UQ uncertainty presentation schemes — e.g., displaying claim-level numeric probabilities (Detommaso et al.,, 2024), hedged linguistic uncertainty (Band et al.,, 2024), hedges with anthropomorphic cues (Kim et al.,, 2024) — perform best in different human decision contexts. Such research would directly steer further research in the LLM UQ community, and as such, should be encouraged to be published in ML / NLP conferences, as opposed to being restricted to only HCI venues such as CHI. 6 Alternative Views and Conclusions We present two potential alternative views that a researcher in LLM UQ may make regarding our call for a re-focusing of the subfield on human-centric utility. Current LLM UQ research is laying essential groundwork with technical metrics and simpler benchmarks. Complex evaluations can come later. Rebuttal. Relying only on particular metrics or simplistic benchmarks in the nascent stages of a research subfield potentially risks establishing a trajectory which may not exactly align with the ultimate goal of human-centric utility in down-stream decision making. As an example of this in action, we lean on the developments over the past decade in explainable-AI (XAI). In XAI, what started primarily as a technical and quantitative research area transformed into one with human-AI interaction at its core. This perspective change occurred only once a host of work realized that i) always providing model explanations may sometimes reduce the performance of the AI-assisted human (Paleja et al.,, 2021; Buçinca et al.,, 2020; Alufaisan et al.,, 2021; Liao et al.,, 2022; Joshi et al.,, 2023); and 2) learning when and how to jointly model AI assistance and human behavior cannot always be solved with purely technical solutions, and is highly context dependent (Rong et al.,, 2023). Indeed, LLM UQ can be considered as one avenue of research within the broader field of XAI, and as such, we should apply many of the same insights from XAI here. Current benchmarks are scalable and allow rapid iteration. Your proposed evaluations are too resource-intensive and would slow innovation and evaluation. Rebuttal. While the cost of evaluating current LLM UQ benchmarks is certainly cheaper than running human studies, the long-term cost of deploying UQ solutions which erode user trust or lead to lower decision system utility is arguably far greater. Over-zealous and rushed deployments of new technologies have historically had lasting impact on user trust. For example, in 2021, Wong et al., (2021) showed that healthcare software company Epic’s sepsis classification tool performed poorly at deployment time (in part due to distribution shift similar to that described in Section 4.2). In addition, because of how the system was implemented, clinicians and nurses ended up having such low trust in the system that they tried to minimize its involvement in decisions (Gichoya et al.,, 2023; Habib et al.,, 2021). Such examples underscore the need for human-centric technology development in order to yield effective and efficient human-uplift on important real-world tasks (Shneiderman,, 2020). Conclusion. In this work, we have laid out what we believe are foundational issues in the evaluation of LLM UQ methods. In particular, we highlight three problematic practices: 1) evaluating on benchmarks with low ecological validity; 2) only considering epistemic uncertainty; and 3) hill-climbing on metrics that are not necessarily indicative of downstream utility. We further provide concrete recommendations for overcoming each of these issues, either through revised practices or new research directions. Although numerous, we believe these problems can be addressed by the many researchers pursuing this subfield through the adoption of a more human-centered perspective. References - Aichberger et al., (2025) Aichberger, L., Schweighofer, K., Ielanskyi, M., and Hochreiter, S. (2025). Improving uncertainty estimation through semantically diverse language generation. In The Thirteenth International Conference on Learning Representations. - Alufaisan et al., (2021) Alufaisan, Y., Marusich, L. R., Bakdash, J. Z., Zhou, Y., and Kantarcioglu, M. (2021). Does explainable artificial intelligence improve human decision-making? In Proceedings of the AAAI Conference on Artificial Intelligence. - Alur et al., (2023) Alur, R., Laine, L., Li, D., Raghavan, M., Shah, D., and Shung, D. (2023). Auditing for human expertise. Advances in Neural Information Processing Systems, 36:79439–79468. - Aolaritei et al., (2025) Aolaritei, L., Jordan, M. I., Marzouk, Y., Wang, Z. O., and Zhu, J. (2025). Conformal prediction under l $\backslash$ ’evy-prokhorov distribution shifts: Robustness to local and global perturbations. arXiv preprint arXiv:2502.14105. - Bakman et al., (2024) Bakman, Y. F., Yaldiz, D. N., Buyukates, B., Tao, C., Dimitriadis, D., and Avestimehr, S. (2024). MARS: meaning-aware response scoring for uncertainty estimation in generative llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024. - Balepur et al., (2025) Balepur, N., Rudinger, R., and Boyd-Graber, J. L. (2025). Which of these best describes multiple choice evaluation with llms? a) forced b) flawed c) fixable d) all of the above. arXiv preprint arXiv:2502.14127. - Band et al., (2024) Band, N., Li, X., Ma, T., and Hashimoto, T. (2024). Linguistic calibration of long-form generations. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net. - Ben-David et al., (2010) Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W. (2010). A theory of learning from different domains. Machine learning, 79:151–175. - Błasiok et al., (2023) Błasiok, J., Gopalan, P., Hu, L., and Nakkiran, P. (2023). A unifying theory of distance from calibration. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing, pages 1727–1740. - Błasiok and Nakkiran, (2024) Błasiok, J. and Nakkiran, P. (2024). Smooth ECE: principled reliability diagrams via kernel smoothing. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. - Brennan et al., (2009) Brennan, T., Dieterich, W., and Ehret, B. (2009). Evaluating the predictive validity of the compas risk and needs assessment system. Criminal Justice and behavior, 36(1):21–40. - Bronfenbrenner, (1977) Bronfenbrenner, U. (1977). Toward an experimental ecology of human development. American psychologist, 32(7):513. - Buçinca et al., (2020) Buçinca, Z., Lin, P., Gajos, K. Z., and Glassman, E. L. (2020). Proxy tasks and subjective measures can be misleading in evaluating explainable ai systems. In Proceedings of the 25th international conference on intelligent user interfaces, pages 454–464. - Callison-Burch et al., (2006) Callison-Burch, C., Osborne, M., and Koehn, P. (2006). Re-evaluating the role of bleu in machine translation research. In 11th Conference of the European Chapter of the Association for Compuational Linguistics, pages 249–256. Association for Computational Linguistics. - Cao et al., (2024) Cao, S., Liu, A., and Huang, C.-M. (2024). Designing for appropriate reliance: The roles of ai uncertainty presentation, initial user decision, and user demographics in ai-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 8(CSCW1):1–32. - Chaleshtori et al., (2024) Chaleshtori, F. H., Ghosal, A., Gill, A., Bambroo, P., and Marasović, A. (2024). On evaluating explanation utility for human-ai decision making in nlp. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7456–7504. - Chang et al., (2022) Chang, Y., Narang, M., Suzuki, H., Cao, G., Gao, J., and Bisk, Y. (2022). Webqa: Multihop and multimodal qa. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition. - Chen and Mueller, (2023) Chen, J. and Mueller, J. (2023). Quantifying uncertainty in answers from any language model and enhancing their trustworthiness. arXiv preprint arXiv:2308.16175. - Chen et al., (2023) Chen, J., Yoon, J., Ebrahimi, S., Arik, S. O., Pfister, T., and Jha, S. (2023). Adaptation with self-evaluation to improve selective prediction in llms. arXiv preprint arXiv:2310.11689. - Chen et al., (2024) Chen, L., Perez-Lebel, A., Suchanek, F., and Varoquaux, G. (2024). Reconfidencing llm uncertainty from the grouping loss perspective. In The 2024 Conference on Empirical Methods in Natural Language Processing. arXiv. - Cherian et al., (2024) Cherian, J., Gibbs, I., and Candes, E. (2024). Large language model validity via enhanced conformal prediction methods. Advances in Neural Information Processing Systems, 37:114812–114842. - Chidambaram and Ge, (2025) Chidambaram, M. and Ge, R. (2025). Reassessing how to compare and improve the calibration of machine learning models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. - Cobbe et al., (2021) Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. - Cole et al., (2023) Cole, J. R., Zhang, M. J., Gillick, D., Eisenschlos, J. M., Dhingra, B., and Eisenstein, J. (2023). Selectively answering ambiguous questions. arXiv preprint arXiv:2305.14613. - Corvelo Benz and Rodriguez, (2023) Corvelo Benz, N. and Rodriguez, M. (2023). Human-aligned calibration for ai-assisted decision making. Advances in Neural Information Processing Systems, 36:14609–14636. - Corvelo Benz and Gomez Rodriguez, (2025) Corvelo Benz, N. L. and Gomez Rodriguez, M. (2025). Human-alignment influences the utility of ai-assisted decision making. arXiv preprint arXiv:2501.14035. - Cruz et al., (2024) Cruz, A. F., Hardt, M., and Mendler-Dünner, C. (2024). Evaluating language models as risk scores. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. - De-Arteaga et al., (2020) De-Arteaga, M., Fogliato, R., and Chouldechova, A. (2020). A case for humans-in-the-loop: Decisions in the presence of erroneous algorithmic scores. In Proceedings of the 2020 CHI conference on human factors in computing systems, pages 1–12. - De Vries et al., (2020) De Vries, H., Bahdanau, D., and Manning, C. (2020). Towards ecologically valid research on language user interfaces. arXiv preprint arXiv:2007.14435. - Detommaso et al., (2024) Detommaso, G., Lopez, M. B., Fogliato, R., and Roth, A. (2024). Multicalibration for confidence scoring in llms. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. - Donahue et al., (2022) Donahue, K., Chouldechova, A., and Kenthapadi, K. (2022). Human-algorithm collaboration: Achieving complementarity and avoiding unfairness. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1639–1656. - Duan et al., (2023) Duan, J., Cheng, H., Wang, S., Zavalny, A., Wang, C., Xu, R., Kailkhura, B., and Xu, K. (2023). Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. arXiv preprint arXiv:2307.01379. - Dunn et al., (2024) Dunn, J., Adams, B., and Madabushi, H. T. (2024). Pre-trained language models represent some geographic populations better than others. arXiv preprint arXiv:2403.11025. - Freitag et al., (2022) Freitag, M., Rei, R., Mathur, N., Lo, C.-k., Stewart, C., Avramidis, E., Kocmi, T., Foster, G., Lavie, A., and Martins, A. F. (2022). Results of wmt22 metrics shared task: Stop using bleu–neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 46–68. - Geifman and El-Yaniv, (2017) Geifman, Y. and El-Yaniv, R. (2017). Selective classification for deep neural networks. Advances in neural information processing systems, 30. - Geva et al., (2021) Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., and Berant, J. (2021). Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics. - Gichoya et al., (2023) Gichoya, J. W., Thomas, K., Celi, L. A., Safdar, N., Banerjee, I., Banja, J. D., Seyyed-Kalantari, L., Trivedi, H., and Purkayastha, S. (2023). Ai pitfalls and what not to do: mitigating bias in ai. The British Journal of Radiology, 96(1150):20230023. - Groot and Valdenegro-Toro, (2024) Groot, T. and Valdenegro-Toro, M. (2024). Overconfidence is key: Verbalized uncertainty evaluation in large language and vision-language models. arXiv preprint arXiv:2405.02917. - Guo et al., (2017) Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR. - Habib et al., (2021) Habib, A. R., Lin, A. L., and Grant, R. W. (2021). The epic sepsis model falls short—the importance of external validation. JAMA Internal Medicine, 181(8):1040–1041. - Hansen et al., (2024) Hansen, D., Devic, S., Nakkiran, P., and Sharan, V. (2024). When is multicalibration post-processing necessary? In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024. - Hartvigsen et al., (2022) Hartvigsen, T., Gabriel, S., Palangi, H., Sap, M., Ray, D., and Kamar, E. (2022). Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022. - (43) He, G., Chen, J., and Zhu, J. (2023a). Preserving pre-trained features helps calibrate fine-tuned language models. arXiv preprint arXiv:2305.19249. - He et al., (2024) He, J., Yang, R., Yu, L., Li, C., Jia, R., Chen, F., Jin, M., and Lu, C.-T. (2024). Can we trust the performance evaluation of uncertainty estimation methods in text summarization? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16514–16575. - (45) He, W., Jiang, Z., Xiao, T., Xu, Z., and Li, Y. (2023b). A survey on uncertainty quantification methods for deep learning. arXiv preprint arXiv:2302.13425. - Hendrycks et al., (2021) Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2021). Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. - Heo et al., (2024) Heo, J., Xiong, M., Heinze-Deml, C., and Narain, J. (2024). Do llms estimate uncertainty well in instruction-following? arXiv preprint arXiv:2410.14582. - Hershcovich et al., (2022) Hershcovich, D., Frank, S., Lent, H. C., de Lhoneux, M., Abdou, M., Brandl, S., Bugliarello, E., Piqueras, L. C., Chalkidis, I., Cui, R., Fierro, C., Margatina, K., Rust, P., and Søgaard, A. (2022). Challenges and strategies in cross-cultural NLP. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022. Association for Computational Linguistics. - Hou et al., (2024) Hou, Y., Li, J., He, Z., Yan, A., Chen, X., and McAuley, J. (2024). Bridging language and items for retrieval and recommendation. arXiv preprint arXiv:2403.03952. - Hu et al., (2024) Hu, Z., Liu, C., Feng, X., Zhao, Y., Ng, S.-K., Luu, A. T., He, J., Koh, P. W. W., and Hooi, B. (2024). Uncertainty of thoughts: Uncertainty-aware planning enhances information seeking in llms. Advances in Neural Information Processing Systems, 37:24181–24215. - Huang et al., (2024) Huang, H.-Y., Yang, Y., Zhang, Z., Lee, S., and Wu, Y. (2024). A survey of uncertainty estimation in LLMs: Theory meets practice. arXiv preprint arXiv:2410.15326. - Hüllermeier and Waegeman, (2021) Hüllermeier, E. and Waegeman, W. (2021). Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine learning, 110(3):457–506. - Jiang et al., (2025) Jiang, Z., Liu, A., and Van Durme, B. (2025). Conformal linguistic calibration: Trading-off between factuality and specificity. arXiv preprint arXiv:2502.19110. - Joshi et al., (2023) Joshi, B., Liu, Z., Ramnath, S., Chan, A., Tong, Z., Nie, S., Wang, Q., Choi, Y., and Ren, X. (2023). Are machine rationales (not) useful to humans? measuring and improving human utility of free-text rationales. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023. - Joshi et al., (2017) Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. (2017). Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers. - Kadavath et al., (2022) Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022). Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. - Kapoor and Narayanan, (2025) Kapoor, S. and Narayanan, A. (2025). Ai as normal technology. 25-09 Knight First Amend. Inst. - Khatun and Brown, (2024) Khatun, A. and Brown, D. G. (2024). A study on large language models’ limitations in multiple-choice question answering. arXiv preprint arXiv:2401.07955. - Kim et al., (2024) Kim, S. S., Liao, Q. V., Vorvoreanu, M., Ballard, S., and Vaughan, J. W. (2024). " i’m not sure, but…": Examining the impact of large language models’ uncertainty expression on user reliance and trust. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 822–835. - Kuhn et al., (2022) Kuhn, L., Gal, Y., and Farquhar, S. (2022). Clam: Selective clarification for ambiguous questions with generative language models. arXiv preprint arXiv:2212.07769. - Kuhn et al., (2023) Kuhn, L., Gal, Y., and Farquhar, S. (2023). Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. - Kumar et al., (2024) Kumar, A., Morabito, R., Umbet, S., Kabbara, J., and Emami, A. (2024). Confidence under the hood: An investigation into the confidence-probability alignment in large language models. arXiv preprint arXiv:2405.16282. - Kwiatkowski et al., (2019) Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019). Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466. - Lee et al., (2019) Lee, K., Chang, M.-W., and Toutanova, K. (2019). Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. - Li et al., (2023) Li, M., Shi, T., Ziems, C., Kan, M.-Y., Chen, N. F., Liu, Z., and Yang, D. (2023). Coannotating: Uncertainty-guided work allocation between human and large language models for data annotation. arXiv preprint arXiv:2310.15638. - Li et al., (2025) Li, Y., Qiang, R., Moukheiber, L., and Zhang, C. (2025). Language model uncertainty quantification with attention chain. arXiv preprint arXiv:2503.19168. - Li et al., (2024) Li, Y., Wang, S., Huang, L., and Liu, L.-P. (2024). Graph-based confidence calibration for large language models. arXiv preprint arXiv:2411.02454. - Liao and Vaughan, (2023) Liao, Q. V. and Vaughan, J. W. (2023). Ai transparency in the age of llms: A human-centered research roadmap. arXiv preprint arXiv:2306.01941, 10. - Liao et al., (2022) Liao, Q. V., Zhang, Y., Luss, R., Doshi-Velez, F., and Dhurandhar, A. (2022). Connecting algorithmic research and usage contexts: a perspective of contextualized evaluation for explainable ai. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 10, pages 147–159. - Lin et al., (2023) Lin, B. Y., Deng, Y., Chandu, K., Ravichander, A., Pyatkin, V., Dziri, N., Le Bras, R., and Choi, Y. (2023). Wildbench: Benchmarking llms with challenging tasks from real users in the wild. In The Thirteenth International Conference on Learning Representations. - Lin, (2004) Lin, C.-Y. (2004). ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. - Lin et al., (2022) Lin, S., Hilton, J., and Evans, O. (2022). Teaching models to express their uncertainty in words. Trans. Mach. Learn. Res., 2022. - Lin et al., (2024) Lin, Z., Trivedi, S., and Sun, J. (2024). Generating with confidence: Uncertainty quantification for black-box large language models. Trans. Mach. Learn. Res., 2024. - (74) Liu, L., Pan, Y., Li, X., and Chen, G. (2024a). Uncertainty estimation and quantification for llms: A simple supervised approach. arXiv preprint arXiv:2404.15993. - (75) Liu, O., Fu, D., Yogatama, D., and Neiswanger, W. (2024b). Dellma: A framework for decision making under uncertainty with large language models. arXiv preprint arXiv:2402.02392, 1. - Liu and Wu, (2024) Liu, T. and Wu, Z. S. (2024). Multi-group uncertainty quantification for long-form text generation. arXiv preprint arXiv:2407.21057. - Liu et al., (2025) Liu, X., Chen, T., Da, L., Chen, C., Lin, Z., and Wei, H. (2025). Uncertainty quantification and confidence calibration in large language models: A survey. arXiv preprint arXiv:2503.15850. - Ma et al., (2025) Ma, M., Chochlakis, G., Pandiyan, N. M., Thomason, J., and Narayanan, S. (2025). Large language models do multi-label classification differently. arXiv. - Malinin and Gales, (2021) Malinin, A. and Gales, M. J. F. (2021). Uncertainty estimation in autoregressive structured prediction. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. - Manakul et al., (2023) Manakul, P., Liusie, A., and Gales, M. J. F. (2023). Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023. - Manggala et al., (2025) Manggala, P., Mastakouri, A. A., Kirschbaum, E., Kasiviswanathan, S., and Ramdas, A. (2025). Qa-calibration of language model confidence scores. In The Thirteenth International Conference on Learning Representations. - Marusich et al., (2024) Marusich, L., Bakdash, J. Z., Zhou, Y., and Kantarcioglu, M. (2024). Using AI uncertainty quantification to improve human decision-making. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. - Mielke et al., (2022) Mielke, S. J., Szlam, A., Dinan, E., and Boureau, Y.-L. (2022). Reducing conversational agents’ overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10:857–872. - Min et al., (2020) Min, S., Michael, J., Hajishirzi, H., and Zettlemoyer, L. (2020). AmbigQA: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). - Nie et al., (2020) Nie, Y., Zhou, X., and Bansal, M. (2020). What can we learn from collective human opinions on natural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). - Nikitin et al., (2024) Nikitin, A., Kossen, J., Gal, Y., and Marttinen, P. (2024). Kernel language entropy: Fine-grained uncertainty quantification for llms from semantic similarities. Advances in Neural Information Processing Systems, 37:8901–8929. - Ouyang et al., (2023) Ouyang, S., Wang, S., Liu, Y., Zhong, M., Jiao, Y., Iter, D., Pryzant, R., Zhu, C., Ji, H., and Han, J. (2023). The shifted and the overlooked: A task-oriented investigation of user-gpt interactions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. - Paleja et al., (2021) Paleja, R., Ghuy, M., Ranawaka Arachchige, N., Jensen, R., and Gombolay, M. (2021). The utility of explainable ai in ad hoc human-machine teaming. Advances in neural information processing systems, 34:610–623. - Platt et al., (1999) Platt, J. et al. (1999). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61–74. - Quach et al., (2024) Quach, V., Fisch, A., Schuster, T., Yala, A., Sohn, J. H., Jaakkola, T. S., and Barzilay, R. (2024). Conformal language modeling. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. - Rong et al., (2023) Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., and Kasneci, E. (2023). Towards human-centered explainable ai: A survey of user studies for model explanations. IEEE transactions on pattern analysis and machine intelligence, 46(4):2104–2122. - Rücker and Akbik, (2023) Rücker, S. and Akbik, A. (2023). Cleanconll: A nearly noise-free named entity recognition dataset. arXiv preprint arXiv:2310.16225. - Shneiderman, (2020) Shneiderman, B. (2020). Human-centered artificial intelligence: Reliable, safe & trustworthy. International Journal of Human–Computer Interaction, 36(6):495–504. - Shorinwa et al., (2024) Shorinwa, O., Mei, Z., Lidard, J., Ren, A. Z., and Majumdar, A. (2024). A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions. arXiv preprint arXiv:2412.05563. - Smith, (2024) Smith, R. C. (2024). Uncertainty quantification: theory, implementation, and applications. SIAM. - Socher et al., (2013) Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., and Potts, C. (2013). Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642. - Srinivasan and Thomason, (2025) Srinivasan, T. and Thomason, J. (2025). Adjust for trust: Mitigating trust-induced inappropriate reliance on ai assistance. arXiv preprint arXiv:2502.13321. - Stengel-Eskin et al., (2024) Stengel-Eskin, E., Hase, P., and Bansal, M. (2024). Lacie: Listener-aware finetuning for confidence calibration in large language models. arXiv preprint arXiv:2405.21028. - Sullivan, (2015) Sullivan, T. J. (2015). Introduction to uncertainty quantification, volume 63. Springer. - Sung et al., (2025) Sung, Y. Y., Fleisig, E., Hou, Y., Upadhyay, I., and Boyd-Graber, J. L. (2025). Grace: A granular benchmark for evaluating model calibration against human calibration. arXiv preprint arXiv:2502.19684. - Talmor et al., (2019) Talmor, A., Herzig, J., Lourie, N., and Berant, J. (2019). Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158. - Taubenfeld et al., (2025) Taubenfeld, A., Sheffer, T., Ofek, E., Feder, A., Goldstein, A., Gekhman, Z., and Yona, G. (2025). Confidence improves self-consistency in llms. arXiv preprint arXiv:2502.06233. - Tian et al., (2023) Tian, K., Mitchell, E., Zhou, A., Sharma, A., Rafailov, R., Yao, H., Finn, C., and Manning, C. D. (2023). Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arXiv preprint arXiv:2305.14975. - Ulmer et al., (2024) Ulmer, D., Gubri, M., Lee, H., Yun, S., and Oh, S. J. (2024). Calibrating large language models using their generations only. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024. - Valdenegro-Toro and Stoykova, (2024) Valdenegro-Toro, M. and Stoykova, R. (2024). The dilemma of uncertainty estimation for general purpose ai in the eu ai act. arXiv preprint arXiv:2408.11249. - (106) Vasconcelos, H., Bansal, G., Fourney, A., Liao, Q. V., and Wortman Vaughan, J. (2025a). Generation probabilities are not enough: Uncertainty highlighting in ai code completions. ACM Trans. Comput.-Hum. Interact. - (107) Vasconcelos, H., Bansal, G., Fourney, A., Liao, Q. V., and Wortman Vaughan, J. (2025b). Generation probabilities are not enough: Uncertainty highlighting in ai code completions. ACM Transactions on Computer-Human Interaction, 32(1):1–30. - Vazhentsev et al., (2024) Vazhentsev, A., Fadeeva, E., Xing, R., Panchenko, A., Nakov, P., Baldwin, T., Panov, M., and Shelmanov, A. (2024). Unconditional truthfulness: Learning conditional dependency for uncertainty quantification of large language models. arXiv preprint arXiv:2408.10692. - Vazhentsev et al., (2023) Vazhentsev, A., Kuzmin, G., Tsvigun, A., Panchenko, A., Panov, M., Burtsev, M., and Shelmanov, A. (2023). Hybrid uncertainty quantification for selective text classification in ambiguous tasks. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11659–11681. - Vodrahalli et al., (2022) Vodrahalli, K., Gerstenberg, T., and Zou, J. Y. (2022). Uncalibrated models can improve human-ai collaboration. Advances in Neural Information Processing Systems, 35:4004–4016. - (111) Wang, P., Lam, B. D., Liu, Y., Asgari-Targhi, A., Panda, R., Wells, W. M., Kapur, T., and Golland, P. (2025a). Calibrating expressions of certainty. International Conference on Learning Representations. - (112) Wang, Z., Wang, Q., Zhang, Y., Chen, T., Zhu, X., Shi, X., and Xu, K. (2025b). Sconu: Selective conformal uncertainty in large language models. arXiv preprint arXiv:2504.14154. - Wong et al., (2021) Wong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., et al. (2021). External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA internal medicine, 181(8):1065–1070. - Xiao et al., (2022) Xiao, Y., Liang, P. P., Bhatt, U., Neiswanger, W., Salakhutdinov, R., and Morency, L.-P. (2022). Uncertainty quantification with pre-trained language models: A large-scale empirical analysis. arXiv preprint arXiv:2210.04714. - Xie et al., (2024) Xie, J., Chen, A. S., Lee, Y., Mitchell, E., and Finn, C. (2024). Calibrating language models with adaptive temperature scaling. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024. - Xing et al., (2022) Xing, Z., Yuan, X., and Mostafa, J. (2022). Age-related difference in conversational search behavior: Preliminary findings. In CHIIR ’22: ACM SIGIR Conference on Human Information Interaction and Retrieval, Regensburg, Germany, March 14 - 18, 2022. - Xiong et al., (2023) Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., and Hooi, B. (2023). Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306.13063. - Yaldiz et al., (2024) Yaldiz, D. N., Bakman, Y. F., Buyukates, B., Tao, C., Ramakrishna, A., Dimitriadis, D., Zhao, J., and Avestimehr, S. (2024). Do not design, learn: A trainable scoring function for uncertainty estimation in generative llms. arXiv preprint arXiv:2406.11278. - (119) Yang, A., Chen, C., and Pitas, K. (2024a). Just rephrase it! uncertainty estimation in closed-source language models via multiple rephrased queries. arXiv preprint arXiv:2405.13907. - (120) Yang, A. X., Robeyns, M., Wang, X., and Aitchison, L. (2024b). Bayesian low-rank adaptation for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. - (121) Yang, D., Tsai, Y.-H. H., and Yamada, M. (2024c). On verbalized confidence scores for llms. arXiv preprint arXiv:2412.14737. - Ye et al., (2025) Ye, Z., Melo, L. C., Kaddar, Y., Blunsom, P., Staton, S., and Gal, Y. (2025). Uncertainty-aware step-wise verification with generative reward models. arXiv preprint arXiv:2502.11250. - Yona et al., (2024) Yona, G., Aharoni, R., and Geva, M. (2024). Can large language models faithfully express their intrinsic uncertainty in words? arXiv preprint arXiv:2405.16908. - Yuksekgonul et al., (2023) Yuksekgonul, M., Zhang, L., Zou, J. Y., and Guestrin, C. (2023). Beyond confidence: Reliable models should also consider atypicality. Advances in Neural Information Processing Systems, 36:38420–38453. - Zhang and Choi, (2021) Zhang, M. J. and Choi, E. (2021). Situatedqa: Incorporating extra-linguistic contexts into qa. arXiv preprint arXiv:2109.06157. - Zhang et al., (2023) Zhang, T., Qiu, L., Guo, Q., Deng, C., Zhang, Y., Zhang, Z., Zhou, C., Wang, X., and Fu, L. (2023). Enhancing uncertainty-based hallucination detection with stronger focus. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023. - Zhao et al., (2024) Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., and Deng, Y. (2024). Wildchat: 1m chatgpt interaction logs in the wild. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. - Zheng et al., (2024) Zheng, C., Zhou, H., Meng, F., Zhou, J., and Huang, M. (2024). Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations. Appendix A ECE and Measuring Calibration in Practice There may be other reasons to shy away from solely optimizing ECE when measuring calibration error for LLMs. A recent sequence of related work has investigated the fundamental problem of measuring calibration error (Błasiok et al.,, 2023; Błasiok and Nakkiran,, 2024; Chidambaram and Ge,, 2025). At a high level, these works together demonstrate that ECE — the most popular calibration error notion — can be highly unstable and not-robust. This is due to a discretization / binning procedure that is necessary when computing ECE because of measurability issues with real-valued probabilistic predictions. Błasiok and Nakkiran, (2024) instead propose a related but robust notion termed SmoothECE (smECE), which gets around the binning requirement by applying kernel smoothing. Hansen et al., (2024) further demonstrate that using smECE rather than ECE can change conclusions about what model is better calibrated on a particular task, especially within the small sample regime (<5k datapoints). ECE, however, remains a popular metric for measuring calibration error of LLM UQ methods. We found that 16 papers used ECE to measure calibration error, and further, many of these papers had evaluation sets which were smaller than 5k datapoints. For example, both Li et al., (2024) and Yang et al., 2024b used ECE to measure performance on evaluation sets of around 2k datapoints. At this scale of dataset size, smECE should be preferred. Among all papers we analyzed, the only paper which utilized smECE to measure calibration error was Ulmer et al., (2024). Indeed, Ulmer et al., (2024, Table 3) demonstrates that measuring calibration error with smECE vs. ECE can result in a different ordering in the quality of LLM UQ methods applied to Vicuna and GPT-3.5. Due to the difficulty and cost of obtaining LLM predictions at larger dataset scales, we believe that the community stands to benefit from utilizing smECE as a drop-in replacement for ECE. Doing so may help ensure that any technical advancements in calibration error remain robust and statistically significant even with test dataset size restrictions. Appendix B LLM UQ Methods Annotation In this section, we provide our annotation for each LLM UQ method paper which we include in our survey. 1. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (Xiong et al.,, 2023) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Verbalized confidence via prompting, sampling, and aggregation techniques. 1. Datasets used to evaluate proposed LLM UQ method. - SportUND (commonsense), StrategyQA (commonsense), GSM8K, SVAMP, Date Understanding (symbolic reasoning), Object Counting (symbolic reasoning), MMLU-Law, MMLU-BizEthics 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - N/A 1. Supervised, semi-supervised, or unsupervised method? - Unsupervised, but has hyperparameters which can influence the outcome of the method. 1. Metrics used to evaluate performance. - ECE, AUROC, AUPRC 1. Human uplift study? - No 1. Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models (Lin et al.,, 2024) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - A simple measure for the “semantic dispersion” of a set of generated short-form responses. Can be a reliable predictor of the quality of LLM responses. 1. Datasets used to evaluate proposed LLM UQ method. - CoQA, TriviaQA, Natural Questions 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - N/A 1. Supervised, semi-supervised, or unsupervised method? - Unsupervised. 1. Metrics used to evaluate performance. - AUROC, AUARC 1. Human uplift study? - No 1. Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation (Kuhn et al.,, 2023) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Short form UQ via clustering multiple generations and calculating entropy. 1. Datasets used to evaluate proposed LLM UQ method. - CoQA, TriviaQA 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - N/A 1. Supervised, semi-supervised, or unsupervised method? - Unsupervised 1. Metrics used to evaluate performance. - AUROC 1. Human uplift study? - No 1. Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models (Duan et al.,, 2023) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Short form UQ via multiple generations and re-weighing at the token / sentence level. 1. Datasets used to evaluate proposed LLM UQ method. - CoQA, TriviaQA, SciQ, MedQA, MedMCQA 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - N/A 1. Supervised, semi-supervised, or unsupervised method? - Unsupervised 1. Metrics used to evaluate performance. - AUROC 1. Human uplift study? - No 1. Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities (Nikitin et al.,, 2024) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Short form UQ via clustering multiple generations and calculating entropy in a more refined approach than semantic entropy (is a generalization of that work) 1. Datasets used to evaluate proposed LLM UQ method. - TriviaQA, SQuAD, BioASQ, NQ, SVAMP 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - N/A 1. Supervised, semi-supervised, or unsupervised method? - Unsupervised 1. Metrics used to evaluate performance. - AUROC, AUARC 1. Human uplift study? - No 1. Mars: Meaning-aware response scoring for uncertainty estimation in generative llms (Bakman et al.,, 2024) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Unsupervised UQ for short QA responses. 1. Datasets used to evaluate proposed LLM UQ method. - TriviaQA, Natural Questions, WebQA 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - N/A 1. Supervised, semi-supervised, or unsupervised method? - Unsupervised 1. Metrics used to evaluate performance. - AUROC 1. Human uplift study? - No 1. Uncertainty Estimation in Autoregressive Structured Prediction (Malinin and Gales,, 2021) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Unsupervised UQ for short QA responses. 1. Datasets used to evaluate proposed LLM UQ method. - newstest14 and LibriSpeech (translation datasets) 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - Yes (out of pre-training domain) 1. Supervised, semi-supervised, or unsupervised method? - Unsupervised 1. Metrics used to evaluate performance. - NLL, AUPR, PRR 1. Human uplift study? - No 1. Quantifying Uncertainty in Answers from Any Language Model and Enhancing Their Trustworthiness (Chen and Mueller,, 2023) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Unsupervised UQ for short QA responses. 1. Datasets used to evaluate proposed LLM UQ method. - GSM8k, SVAMP, CSQA, TriviaQA 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - N/A 1. Supervised, semi-supervised, or unsupervised method? - Unsupervised 1. Metrics used to evaluate performance. - AUROC 1. Human uplift study? - No 1. Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus (Zhang et al.,, 2023) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Unsupervised, reference free UQ method for hallucination detection at the sentence level. 1. Datasets used to evaluate proposed LLM UQ method. - WikiBio 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - N/A 1. Supervised, semi-supervised, or unsupervised method? - Unsupervised 1. Metrics used to evaluate performance. - AUC-PR 1. Human uplift study? - No 1. Language Models (Mostly) Know What They Know (Kadavath et al.,, 2022) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - P(True) token probabilities in short form UQ, also UQ fine-tuning. 1. Datasets used to evaluate proposed LLM UQ method. - TriviaQA, Lambada, Arithmetic, GSM8k, and Codex HumanEval 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - Yes 1. Supervised, semi-supervised, or unsupervised method? - Supervised and Unsupervised 1. Metrics used to evaluate performance. - ECE 1. Human uplift study? - No 1. SELFCHECKGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (Manakul et al.,, 2023) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Detect hallucinations in long form generated text via UQ methods. 1. Datasets used to evaluate proposed LLM UQ method. - WikiBio 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - N/A 1. Supervised, semi-supervised, or unsupervised method? - Unsupervised 1. Metrics used to evaluate performance. - AUC-PR 1. Human uplift study? - No 1. Graph-based confidence calibration for large language models (Li et al.,, 2024) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Confidence of short form generation. 1. Datasets used to evaluate proposed LLM UQ method. - CoQA, TriviaQA 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - Yes 1. Supervised, semi-supervised, or unsupervised method? - Supervised 1. Metrics used to evaluate performance. - ECE, AUROC, Brier 1. Human uplift study? - No 1. Calibrating Large Language Models Using Their Generations Only (Ulmer et al.,, 2024) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Train auxilary model to predict confidence of short form generations of LLM 1. Datasets used to evaluate proposed LLM UQ method. - TriviaQA 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - No 1. Supervised, semi-supervised, or unsupervised method? - Supervised 1. Metrics used to evaluate performance. - Brier, ECE, smECE, AUROC 1. Human uplift study? - No 1. Bayesian Low-Rank Adaptation for Large Language Models (Yang et al., 2024b, ) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Train LoRA Bayesian ensembles fine-tuned on target task. 1. Datasets used to evaluate proposed LLM UQ method. - MC / common sense reasoning tasks: WG-S, ARC-C, ARC-E, OBQA, WG-M, BoolQ, MMLU 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - Yes 1. Supervised, semi-supervised, or unsupervised method? - Supervised 1. Metrics used to evaluate performance. - Acc, ECE, NLL 1. Human uplift study? - No 1. Reconfidencing LLM Uncertainty from the Grouping Loss Perspective (Chen et al.,, 2024) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Short form uncertainty quantification on answers 1. Datasets used to evaluate proposed LLM UQ method. - Propose new dataset, short answer but derived for fairness purposes (has group information) 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - No 1. Supervised, semi-supervised, or unsupervised method? - Supervised 1. Metrics used to evaluate performance. - Brier, CL, GL 1. Human uplift study? - No 1. Reducing conversational agents’ overconfidence through linguistic calibration (Mielke et al.,, 2022) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Train a "calibrator" that tries to predict the correctness of the LLM based on its internal state. 1. Datasets used to evaluate proposed LLM UQ method. - TriviaQA 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - No 1. Supervised, semi-supervised, or unsupervised method? - Supervised 1. Metrics used to evaluate performance. - ECE, Max CE, NLL 1. Human uplift study? - No 1. Multicalibration for Confidence Scoring in LLMs (Detommaso et al.,, 2024) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Supervised UQ for short QA respones 1. Datasets used to evaluate proposed LLM UQ method. - BigBench, MMLU, triviaQA, OpenBookQA, TruthfulQA, and MathQA 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - No 1. Supervised, semi-supervised, or unsupervised method? - Supervised 1. Metrics used to evaluate performance. - Group average squared calibration error (gASCE), ASCE (similar to ECE). 1. Human uplift study? - No 1. Teaching Models to Express Their Uncertainty in Words (Lin et al.,, 2022) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Prompts LLM to generate confidence as a sequence of tokens 1. Datasets used to evaluate proposed LLM UQ method. - CalibratedMath 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - Yes 1. Supervised, semi-supervised, or unsupervised method? - Supervised 1. Metrics used to evaluate performance. - Brier, ECE 1. Human uplift study? - No 1. Do Not Design, Learn: A Trainable Scoring Function for Uncertainty Estimation in Generative LLMs (Yaldiz et al.,, 2024) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Trainable / learnable response scoring function for LLMs. Follow-up work on (Bakman et al.,, 2024). 1. Datasets used to evaluate proposed LLM UQ method. - TriviaQA NaturalQA WebQA GSM8K 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - Yes 1. Supervised, semi-supervised, or unsupervised method? - Supervised 1. Metrics used to evaluate performance. - AUROC, PRR 1. Human uplift study? - No 1. Conformal Linguistic Calibration: Trading-off between Factuality and Specificity (Jiang et al.,, 2025) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Modify the output to make language used more vague to better represent the uncertainty of the model. 1. Datasets used to evaluate proposed LLM UQ method. - SimpleQA, Natural Questions 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - No 1. Supervised, semi-supervised, or unsupervised method? - Supervised 1. Metrics used to evaluate performance. - Conformal coverage / factuality requirement. 1. Human uplift study? - No 1. Calibrating Expressions of Certainty (Wang et al., 2025a, ) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Calibrating linguistic expressions of certainty, e.g., “Maybe” and “Likely”, in both humans and language models. 1. Datasets used to evaluate proposed LLM UQ method. - SciQ, TruthfulQA 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - No 1. Supervised, semi-supervised, or unsupervised method? - Supervised 1. Metrics used to evaluate performance. - ECE and Brier Score. 1. Human uplift study? - Test methods on humans. 1. QA-Calibration of Language Model Confidence Scores (Manggala et al.,, 2025) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Group-wise calibration guarantees for QA 1. Datasets used to evaluate proposed LLM UQ method. - MMLU, TriviaQA, SciQ, BigBench, OpenBookQA 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - No 1. Supervised, semi-supervised, or unsupervised method? - Supervised 1. Metrics used to evaluate performance. - AUAC, calibration error 1. Human uplift study? - No 1. Unconditional Truthfulness: Learning Conditional Dependency for Uncertainty Quantification of Large Language Models (Vazhentsev et al.,, 2024) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Learn conditional dependency between generation steps in order to estimate uncertainty. 1. Datasets used to evaluate proposed LLM UQ method. - Text generation: CNN/DailyMail, XSUM, SamSum Long-form QA: MedQUAD, PubMedQA, TruthfulQA Short-form QA: SciQ, CoQA, TriviaQA 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - No 1. Supervised, semi-supervised, or unsupervised method? - Supervised 1. Metrics used to evaluate performance. - Prediction Rejection Ratio (PRR, selective prediction metric). 1. Human uplift study? - No 1. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback (Tian et al.,, 2023) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Unsupervised prompting methods, and also supervised method which uses a calibration hold-out set to choose confidence levels. 1. Datasets used to evaluate proposed LLM UQ method. - TriviaQA, SciQA, TruthfulQA 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - No 1. Supervised, semi-supervised, or unsupervised method? - Mix of supervised / unsupervised 1. Metrics used to evaluate performance. - ECE, ECE-t, BS-t, AUC 1. Human uplift study? - No 1. Do LLMs Estimate Uncertainty Well in Instruction Following? (Heo et al.,, 2024) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Systematic evaluation of LLM UQ in instruction following. 1. Datasets used to evaluate proposed LLM UQ method. - IFEval 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - N/A 1. Supervised, semi-supervised, or unsupervised method? - Unsupervised 1. Metrics used to evaluate performance. - AUROC 1. Human uplift study? - No 1. Improving Uncertainty Estimation through Semantically Diverse Language Generation (Aichberger et al.,, 2025) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Estimate aleatoric semantic uncertainty via steering multiple generations 1. Datasets used to evaluate proposed LLM UQ method. - TruthfulQA, CoQA, TriviaQA 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - N/A 1. Supervised, semi-supervised, or unsupervised method? - Unsupervised 1. Metrics used to evaluate performance. - Rouge-L, Rouge-1, BLEURT, AUROC 1. Human uplift study? - No 1. Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words? (Yona et al.,, 2024) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Evaluate LLM UQ with a new metric called faithfulness. Argue that LLMs ignore intrinsic uncertainty. Also test a few-shot prompting-based LLM UQ method Uncertainty+. 1. Datasets used to evaluate proposed LLM UQ method. - Natural Questions, PopQA 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - N/A 1. Supervised, semi-supervised, or unsupervised method? - N/A 1. Metrics used to evaluate performance. - Faithfulness 1. Human uplift study? - N/A 1. Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models (Kumar et al.,, 2024) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Connect confidence stated by LLM (conveyed in model response) to the confidence quantified by token probabilities. 1. Datasets used to evaluate proposed LLM UQ method. - CommonsenseQA, QASC, RiddleQA, OpenBookQA, ARC 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - N/A 1. Supervised, semi-supervised, or unsupervised method? - Unsupervised 1. Metrics used to evaluate performance. - Spearman correlation 1. Human uplift study? - No 1. Language Model Uncertainty Quantification with Attention Chain (Li et al.,, 2025) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Uses attention weights to find semantically critical tokens in a backtracking procedure. Allows estimation of the marginal probabilities of answer tokens. 1. Datasets used to evaluate proposed LLM UQ method. - GSM8K, MATH, BigBench 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - No 1. Supervised, semi-supervised, or unsupervised method? - Unsupervised 1. Metrics used to evaluate performance. - AUROC, ECE 1. Human uplift study? - No 1. Preserving Pre-trained Features Helps Calibrate Fine-tuned Language Models (He et al., 2023a, ) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Calibrating fine-tuned language models results in poor performance under distribution shift. Propose method to preserve pre-trained features in order to improve calibration. 1. Datasets used to evaluate proposed LLM UQ method. - WikiText-103, SWAG, HellaSWAG, MNLI, SNLI 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - Yes 1. Supervised, semi-supervised, or unsupervised method? - Supervised 1. Metrics used to evaluate performance. - ECE + reliability diagram 1. Human uplift study? - No 1. Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approach (Liu et al., 2024a, ) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Use hidden layer features for UQ in a supervised manner. 1. Datasets used to evaluate proposed LLM UQ method. - CoQA, TriviaQA, MMLU, WMT14 Machine Translation 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - Yes 1. Supervised, semi-supervised, or unsupervised method? - Supervised 1. Metrics used to evaluate performance. - AUROC 1. Human uplift study? - No 1. Just rephrase it! Uncertainty estimation in closed-source language models via multiple rephrased queries (Yang et al., 2024a, ) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Rephrase the prompt to sample multiple outputs, aggregate to get confidence. 1. Datasets used to evaluate proposed LLM UQ method. - ARC, openbook QA, MMLU 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - N/A 1. Supervised, semi-supervised, or unsupervised method? - Unsupervised 1. Metrics used to evaluate performance. - Acc, ECE, TACE, Brier, AUROC 1. Human uplift study? - No 1. Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis (Xiao et al.,, 2022) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Benchmark various LLM UQ methods on NLP tasks. 1. Datasets used to evaluate proposed LLM UQ method. - Sentiment analysis: IMDB, Yelp reviews. NLI: MNLI, SNLI. Commonsense reasoning: SWAG, HellaSWAG 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - Yes 1. Supervised, semi-supervised, or unsupervised method? - Supervised 1. Metrics used to evaluate performance. - ECE 1. Human uplift study? - No 1. Beyond Confidence: Reliable Models Should Also Consider Atypicality (Yuksekgonul et al.,, 2023) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Estimate the atypicality of inputs in order to improve LLM UQ. 1. Datasets used to evaluate proposed LLM UQ method. - IMDB, TREC, AG News 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - Yes 1. Supervised, semi-supervised, or unsupervised method? - Supervised 1. Metrics used to evaluate performance. - ECE 1. Human uplift study? - No 1. Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Models (Groot and Valdenegro-Toro,, 2024) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Prompts LLM to output confidence intervals 1. Datasets used to evaluate proposed LLM UQ method. - SST (sentiment analysis) GSM8k (math) CoNLL (NER) Japanese Uncertain Images (for VLMs only) 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - N/A 1. Supervised, semi-supervised, or unsupervised method? - Unsupervised 1. Metrics used to evaluate performance. - ECE, MaxCE, NCE 1. Human uplift study? - No 1. Multi-group Uncertainty Quantification for Long-form Text Generation (Liu and Wu,, 2024) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Uncertainty quantification for factual correctness in long form text generation 1. Datasets used to evaluate proposed LLM UQ method. - BIO-NQ (subset of natural questions corresponding to biographies) 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - No 1. Supervised, semi-supervised, or unsupervised method? - Supervised 1. Metrics used to evaluate performance. - ASCE (average squared calibration error), max gASCE, average gASCE, Brier score 1. Human uplift study? - No 1. LACIE: Listener-Aware Finetuning for Confidence Calibration in Large Language Models (Stengel-Eskin et al.,, 2024) 1. Type of uncertainty quantification (short / long form generation, conformal, etc.). - Cast calibration as preference optimization problem with a speaker and listener. 1. Datasets used to evaluate proposed LLM UQ method. - TriviaQA TruthfulQA 1. Test performance under distribution shift? (yes / no). Can also be N/A for methods which are unsupervised. - Yes 1. Supervised, semi-supervised, or unsupervised method? - Supervised 1. Metrics used to evaluate performance. - ECE, AUROC, Precision, Recall of user 1. Human uplift study? - Yes, human evaluation with annotators accepting or rejecting LLM answers.