2406.03441

# Cycles of Thought: Measuring LLM Confidence through Stable Explanations **Authors**: - Evan Becker (Department of Computer Science) - &Stefano Soatto (Department of Computer Science) (October 16, 2025) ## Abstract In many critical machine learning (ML) applications it is essential for a model to indicate when it is uncertain about a prediction. While large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, their overconfidence in incorrect responses is still a well-documented failure mode. Traditional methods for ML uncertainty quantification can be difficult to directly adapt to LLMs due to the computational cost of implementation and closed-source nature of many models. A variety of black-box methods have recently been proposed, but these often rely on heuristics such as self-verbalized confidence. We instead propose a framework for measuring an LLM’s uncertainty with respect to the distribution of generated explanations for an answer. While utilizing explanations is not a new idea in and of itself, by interpreting each possible model+explanation pair as a test-time classifier we can calculate a posterior answer distribution over the most likely of these classifiers. We demonstrate how a specific instance of this framework using explanation entailment as our classifier likelihood improves confidence score metrics (in particular AURC and AUROC) over baselines across five different datasets. We believe these results indicate that our framework is a promising way of quantifying uncertainty in LLMs. ## 1 Introduction Large language models (LLMs) are known to at times confidently provide wrong answers, which can greatly mislead non-expert users of the model [46, 7]. In the some cases an LLM may even ‘hallucinate’ facts all together [45, 50]. Although scaling generally improves factual accuracy, past work has shown that even the largest models can give incorrect answers to certain types of questions [29]. To prevent these misleading scenarios, one intuitive approach is to have the model also report its confidence (or uncertainty) in the accuracy of its own response. This task, known as uncertainty quantification, has a vast associated literature [1, 15]. In its most naive form, this can entail taking the softmax of prediction logits to calculate a ‘distribution’ over answers. However in most cases there is no guarantee that this metric should correspond to the actual probability of correctness on a new datum. Empirically this mismatch has been demonstrated for LLM token logits [26, 2]. One might instead hope that by probing the model (e.g. through its weights or activations) one could infer a measure of confidence that somehow aligns with our expectations. However, full access to a large language model is often infeasible due to a combination of proprietary restrictions and computational expense. Recently a range of ‘black-box’ approaches have been proposed that avoid the need for access to internal model information [24, 46, 36]. These approaches typically rely on custom prompting strategies to elicit self-verbalized (linguistic) confidence or generate multiple variations of a response (consistency). While empirically promising, these methods are heuristic and still return overconfident responses in many cases. We reason that the main issue with existing uncertainty quantification methods for LLMs stems from the underlying inductive assumption that test and training data are sampled from the same distribution. Unfortunately, this is rarely the case, meaning any uncertainty quantification strategy that is well-calibrated on one dataset is not guaranteed to be calibrated on new test data. However, an LLM offers a unique opportunity to adjust its decision boundary transductively at test-time via intermediate generated text (explanations). While inserting random text would likely lead to a high-entropy decision distribution, adding relevant facts or logical step-by-step reasoning serves to ‘stabilize’ the sampled answers around an isolated minimum. Indeed, prompts inducing chain of thought (CoT) reasoning have already shown to improve model accuracy in this manner [44, 44]. However, more recent work has shown that even CoT explanations can be biased and may not correspond with the correct answer [41]. If we could somehow distinguish between ‘stable’ and ‘unstable’ explanations then we would know to what extent to trust their corresponding answer distributions. In this work we propose a method for generating confidence scores from the distribution of LLM-generated explanations for an answer. This method, which we call stable explanations confidence, can be thought of as computing the posterior predictive distribution by transductive marginalization over test-time classifiers. We illustrate the usefulness of these scores on two common uncertainty quantification tasks: calibration, in which we measure how close confidence is to empirical accuracy, and selective uncertainty, in which we determine how well the scores can discriminate between correct and incorrect predictions. We compare to other recently proposed methods across five datasets of different scope and complexity (CommonsenseQA, TruthfulQA, MedQA, MMLU Professional Law, MMLU Conceptual Physics) using two popular LLMs (GPT-3.5 [5] and GPT4 [2]). We find that our method on average outperforms baselines on the selective uncertainty task (measured via AUROC and AURC), particularly for more complex question-answering problems. ## 2 Related Work In this section we first summarize the uncertainty quantification problem in machine learning. We then highlight key challenges in the natural language generation setting and the ‘confidence gap’ of existing LLM models. Lastly we discuss exisiting approaches for LLM uncertainty quantification and methods for their evaluation. ### 2.1 Uncertainty Quantification in Machine Learning Defining and reasoning about uncertainty has been a long-standing problem in different disciplines including philosophy, statistics, and economics. Many formal representations with unique properties have been proposed, (e.g. Dempster-Shafer belief functions, ranking functions, etc. [17]), but in the machine learning setting uncertainty quantification typically relies on the standard language of probability measures. For a classification task we can think of the sequential training data-label pairs $\mathcal{D}:=\{(x_{i},y_{i})\}_{i=1}^{N}$ as the model’s source of knowledge about the world. Given some test datum $x_{N+1}$ , we would like the model to both make a prediction $\hat{y}_{N+1}$ and provide a ‘useful’ confidence score $r_{N+1}\in[0,1]$ . Useful confidence scores should allow models to express their belief in the accuracy of a prediction, and is called well-calibrated if on average predictions with confidence $r=0.XX$ are correct close to $XX\$ of the time. If the classification task also specifies cases for which it is better to return no prediction than a wrong one, we can imagine creating some selection rule using confidence scores to determine whether to trust the classifier’s prediction. We will formalize these two related goals later when discussing evaluation metrics in Section ˜ 4.1. Uncertainty quantification methods differ from one another based on their assumptions about where uncertainty is coming from. Sources of uncertainty are traditionally categorized into two broad classes: epistemic uncertainty arising from the agent’s incomplete knowledge of the world, and aleatoric uncertainty inherent to the data generating process (e.g. the flip of a coin). In reality, definitions vary among the machine learning community [4] and most methods do not fit neatly into either category. In this work we discuss a few of most common methods based on the underlying assumptions placed on the test data. We make this distinction because without this fundamental assumption it is impossible to know anything about the test distribution from training data. Note that for a full discussion and taxonomy of the numerous uncertainty quantification methods in machine learning we refer to a suvery paper such as [1, 15]. #### Related Training and Test Worlds. Most uncertainty quantification methods rely on the fundamental assumption that the test data comes from the same distribution as the training set. Under this type of assumption Bayesian approaches such as Bayesian Neural Networks (BNNs) are popular. BNNs measure epistemic uncertainty through a posterior on the learned weights, which can be reduced as more data is recieved [33, 23]. Another popular method is that of conformal prediction, which introduces a somewhat dual notion of the conformal set. Under a slightly weaker exchangibility assumption (i.e. that the joint distribution remains the same under permutations of the training and test data), the conformal set of predictions is guaranteed to contain the true label with error probability less than some $\epsilon$ [35]. Weaker predictive models result in larger conformal sets, and so set size can be taken as an indicator for higher model uncertainty. Other methods include looking at the robustness of predictions under semantic-preserving transformations of the input, as mentioned in [15]. #### Different Training and Test Worlds. Small and large differences between training and test distributions are typically denoted as distribution shift and out-of-distribution respectively [47]. In this setting methods like prior networks attempt to capture the specific notion of this distributional uncertainty through and additional prior over predictive distributions and training explicitly on a loss objective [31]. ### 2.2 Uncertainty Quantification in LLMs Recently much attention has been devoted to measuring uncertainty specifically in LLMs [16, 20]. Since LLMs are generative models, uncertainty may be measured with respect to an infinite set of text sequences as opposed to a fixed number of classification labels [4]. Many works, however, use multiple choice question answering tasks to evaluate LLMs using standard classification methodologies [43, 24], and we will follow a similar approach in this work. Issues with using token logits directly to compute confidence are well known. Recent works [2, 24, 38] show that larger models are typically better calibrated on multiple choice datasets than smaller ones, but are still sensitive to question reformulations as well as typical RLHF training strategies. Another recent work [48] notes that language models fail to identify unanswerable questions at a higher rate than humans. At a high level, existing techniques for LLM confidence elicitation can be classified as either white-box, requiring access to internal model weights and token probabilities, or black-box, using only samples from the model [16]. We choose to summarize inference time interventions below, as training time interventions are often computationally expensive and require strict inductive assumptions. White-box Methods. Access to the last activation layer of the LLM (token logits) admits calculating token and token sequence probabilities via the softmax function. One can incorporate text sequence probabilities to implement conformal prediction [27] methods, or adjust them based on semantic importance of individual tokens to improve calibration [13]. Surrogate models can also serve as an effective substitute if access the original model is restricted-access [36]. Internal activations can also be observed to determine if certain feature directions are more or less truthful [3, 6]. Black-box Methods. Black-box confidence typically uses one or both of the following approaches: Sample+aggregate methods involve analyzing the distributions of multiple responses sampled from the model [46]. Responses can be generated in a variety of ways, such as using chain-of-thought prompting [43], asking for multiple answers in a single response [40], or perturbing the question in-between samples [28]. Confidence can be found by observing the frequency with which answers occur, or by averaging over other metrics [8]. Self-evaluation methods use customized prompts in order for the model to generate its own confidence estimates in natural language [24]. These methods can also be augmented with chain-of-thought or other more complex reasoning steps [11]. Much effort has been put into analyzing how changes in prompt (e.g. by including few-shot examples) affects these confidences [52, 51]. ## 3 Stable Explanations Given a question, we would like to assign a confidence value to an answer based on how plausible its associated explanations are. Intuitively, humans are confident in an answer when likely explanations exist for it and no other answers have reasonable explanations. However, the space of explanations (variable-length token sequences) is infinite and hard to work with directly. To overcome this, we will first approximate this distribution by sampling a set of explanations from the LLM conditioned on the question, and then reweight based on their logical consistency with the question desscrption. Afterwards we can compute the degree to which explanations support each answer. We can view these two steps as estimating the conditional likelihood of the explanation given the question, and the conditional answer distribution of the test-time model parameterized by this explanation. These two components will allow us to compute a posterior predictive distribution in a Bayesian fashion. We formalize each step in the following subsections, and summarize the complete method in Algorithm ˜ 1. Input: LLM $\phi$ , question $q$ and selected answer $a_{i}\in\mathcal{A}$ , explanation sample size $N$ Output: Confidence estimate $\hat{p}(a_{i}|q)$ for $n=1\dots N$ do $e_{n}\sim\phi(\text{prompt}_{explain}(q))$ // sample explanations $\rho_{n}\leftarrow\phi(\text{prompt}_{entail}(q,e_{n}))$ // compute probability that $q\models e_{n}$ end for $z\leftarrow\sum_{n=1}^{N}\rho_{n}$ $\hat{p}(a_{i}|q)\leftarrow\sum_{n=1}^{N}\frac{\rho_{n}}{z}\text{softmax}(\phi(q,e_{n}))_{i}$ // marginalize over explanations 0.5em return $\hat{p}(a_{i}|q)$ Algorithm 1 Stable Explanation Confidence Calculation #### Preliminaries. Consider a multiple choice question $q:=\{x_{1},\dots,x_{t}\}=x^{t}$ consisting of a sequence of tokens in some alphabet $x_{j}\in\mathcal{A}$ , and a set of possible answers $a\in S\subseteq\mathcal{A}$ which are also some subset of tokens in the same alphabet. We will designate $\phi$ as an LLM, which will take any variable length token sequence as input and output a token logit vector of size $|\mathcal{A}|$ . We use $\phi(s_{1},s_{2})$ to denote the direct concatenation of two token sequences in the LLM input, and $\phi(\text{prompt}(s))$ to denote adding prompt instructions to the input. Lasrly, $s\sim\phi$ will be used to denote sampling a token sequence from the LLM. ### 3.1 Answer Likelihood Conditioned on Explanations In its default configuration, providing a question to an LLM $\phi$ without further input can be used to find an answer: $$ \displaystyle~\underset{S}{\rm argmax}~{\phi(q,\{~\})}=a \tag{1} $$ One can also naively compute a ‘probability distribution’ over possible answers by taking the softmax of token logits produced by the model. We will denote this calculation as $$ \displaystyle p_{\phi}(a|q):=\text{softmax}(\phi(q,\{~\}))_{i}, \tag{2} $$ where $i$ denotes the logit index of $a$ . However, these default token probabilities have been shown to be miscalibrated and sensitive to variations in the input [24, 40]. Next, we formally say that explanations, like questions, are also variable length sequences of tokens $e\in\mathcal{A}^{\tau}$ located between question and answer. If the model generates these explanations (like in the chain-of-thought reasoning paradigm [44]) then the sequences can be thought of as a possible trajectory from the question to an answer. While the set of possible trajectories is infinite, we can group explanations into equivalence classes by noting that two semantically identical explanations must support the same answers [30, 37]. This notion leads us to the following idea: characterize the distribution of explanations by looking at the new answers they lead to. $$ \displaystyle~\underset{S}{\rm argmax}~{\phi(q,e)}=a^{\prime} \tag{3} $$ This idea is related to the semantic entropy method of [26], but here we use the next token distribution $p_{\phi}(a|q,e)$ instead of a pairwise explanation similarity to ‘cluster’ explanations. If we can enumerate all likely explanations, we can calculate the posterior answer probability as follows $$ \displaystyle\hat{p}(a|q)=\sum_{e}p_{\phi}(a|e,q)p(e|q) \tag{4} $$ A key detail omitted so far is how to efficiently approximate the distribution of all ‘stable’ explanations. We will see in the following subsection that this can be achieved using only the LLM $\phi$ . ### 3.2 Determining Likely Explanations A naive method for estimating $\hat{p}(e|q)$ would be to sample explanations using a modified prompt (e.g. using a CoT ‘think step-by-step’ approach). Indeed, a number of consistency-based question-answering methods work by sampling and then aggregating explanations and answers in this manner [43, 8]. However, due to the way LLMs are trained, this distribution does not necessarily represent the probability that an explanation actually explains the data in the question [49, 41]. To combat this, we enforce logical consistency by checking the entailment probability of our sampled explanations ( $q\models e$ ), which can be approximated by using the LLM and a modified prompt $\phi_{entail}(q,e)$ [34]. We then reweight sampled explanations using this entailment probability: $$ \hat{p}(e|q):=\frac{\phi_{ent.}(q,e)}{\sum_{e^{\prime}\in E}\phi_{ent.}(q,e^{\prime})} \tag{5} $$ We reason that enforcing logical structure prevents trusting explanations that ‘overfit’ to the test datum. For example while an explanation such as ‘the answer is always (a)’ is syntactically correct and may result in a confidently correct answer for our test question, it would prove a useless classifier on previous training data. While we use entailment probability in our main results, an exploration of alternative explanation plausibility calculations can be found in Section ˜ B.4. ## 4 Experiments To gain insight into the usefulness of LLM-sampled explanations we first examine differences in distributions of explanations conditioned on correct vs. incorrect answers (see Figure ˜ 1) and find explanation entailment (Section ˜ 3.2) can help distinguish between the two. We then conduct a series of experiments to compare our proposed stable explanation confidence (Algorithm ˜ 1) with exisiting approaches across a set of five benchmark datasets and discuss our findings below. ### 4.1 Setup #### Evaluation Method. How do we know whether a proposed confidence metric is useful or not? In line with previous works [24, 46, 36, 40] there are typically two tasks that uncertainty metrics are evaluated on. The first is confidence calibration, where the goal is to produce confidence scores approximating the empirical probability that the model answers the question correctly. Expected calibration error (ECE) [32] attempts to estimate this using differences between the average confidence and accuracy for a group of similarly scored answers, however ECE can be misleading (see Section ˜ 5). We still include this metric in our reports for ease of comparison with previous work. The second related task is typically called selective uncertainty (also known as failure prediction). Here the goal is to create a binary classifier using confidence scores that predict when the model should return ‘I don’t know’ instead of its original prediction. A variety of classifier metrics can be used, depending on how one chooses to penalize false positive (overconfident) and false negative (underconfident) predictions. In this work we use two of the most common metrics: area under the reciever-operator curve (AUROC) [18], and area under the risk-coverage curve (AURC) [12]. Uninformative (i.e. completely random) confidence scores will have a worst-case AUROC of $0.5$ and an worst-case AURC equal to average model accuracy. The best possible value for both AUROC and AURC is $1.0$ . We include formal definitions for each of these metrics in Appendix ˜ A. #### Datasets and Models. We evaluate our method using five standard question answering datasets covering a variety of reasoning tasks: CommonsenseQA (CSQA) [39], TruthfulQA [29], MedQA [21], MMLU Professional Law, and MMLU Conceptual Physics [19]. Besides covering a range of topics, these datasets also vary largely in their complexity. As seen in Table ˜ 1, the average length of an MMLU law question is almost ten times that of the average CSQA question. Shorter questions typically resemble more traditional classification tasks (e.g. ‘Something that has a long and sharp blade is a? ’ from CSQA), while longer questions typically include descriptions of a specific scenario that require more complex reasoning. We test both methods and baselines on snapshots of two state-of-the-art models GPT-3.5-turbo [5] and GPT-4-turbo [2]. Further data and model details can be found in Appendix ˜ B. #### Compared Metrics. We use four different baselines for comparison purposes. Token probabilities for each answer can be produced by taking the softmax over the models logit vector and are one of the most commonly used confidence metrics during model evaluation [2, 7]. Linguistic and Top-k methods both ask the model for a verbalized confidence estimate directly, the former prompting the model for a single answer and confidence estimate while the later asks for the $k$ -best guesses and associated confidences [40, 36]. Lastly the sef-consistency method samples multiple responses from the model and approximates confidence via the relative frequency of parsed answers. Here we use a particular variant of this method, CoT-Consistency [43], which uses a zero-shot chain-of-thought prompt to generate responses, and which has been shown to outperform the vanilla method [46]. We use the similar prompts to those selected in previous work for comparison purposes, the details of which can be found in Section ˜ B.1. | CSQA TruthQA MedQA | 151 329 916 | 0.79 0.54 0.59 | 0.84 0.85 0.82 | | --- | --- | --- | --- | | MMLU Law | 1233 | 0.46 | 0.64 | | MMLU Physics | 172 | 0.57 | 0.92 | Table 1: Average question length and accuracy for each of the datasets tested in this work. One can observe a weak correlation between question length and difficulty, as typically longer questions describe more complex scenarios and logical structure. ### 4.2 Likely Explanations Not Always Correct We first illustrate how explanation likelihood, as measured via conditional token log probability, does not always correspond with the correctness of the supported answer. These results align with previous findings differentiating syntactically vs. semantically correct model responses [29, 26], and help us to motivate using entailment probability in our method. First recall that the length-normalized conditional log-likelihood for sequence $x^{t}$ given sequence $s$ is defined as $$ \displaystyle LL(x^{t}|s):=\frac{1}{t}\sum_{i=1}^{t}\log(P_{\phi}(x_{i}|s,x_{1},x_{2},\dots,x_{i-1})), \tag{6} $$ which can also be thought of as the average token logit value. Higher log-likelihood of explanations should mean higher chance of being sampled by the LLM. We can observe in Figure ˜ 1 two distributions of explanations: one set (in blue) is conditioned on answers we know a priori are correct, the second set (in red) is conditioned on incorrect responses. The model prompt for each set is the same and is given in Section ˜ B.1. We see that while the mean log-likelihood for correct explanations is slightly higher than that of incorrect explanations, the two distributions are hard to distinguish. In contrast there is clearly a distinct tail for the distribution of incorrect explanations measured via entailment probability. This result suggests that we may be able to discount certain explanations sampled by the LLM but that are well written but logically ‘unstable’, hence improving our confidence score. <details> <summary>x1.png Details</summary>  ### Visual Description ## Chart/Diagram Type: Dual Histogram with Overlaid Density Curves ### Overview The image contains two side-by-side visualizations comparing the distribution of **Token Sequence Log Likelihood** (left) and **Entailment Probability P(q≠e)** (right) for **correct** and **incorrect** TruthQA explanations. Both charts use density curves (blue for correct, red for incorrect) overlaid on histograms (blue and red bars). ### Components/Axes #### Left Chart: TruthQA Explanation Likelihood - **X-axis**: Token Sequence Log Likelihood (range: -0.8 to 0.0) - **Y-axis**: Density (range: 0.0 to 12.5) - **Legend**: - Blue line: Correct explanations - Red line: Incorrect explanations - **Bars**: Blue (correct) and red (incorrect) histograms. #### Right Chart: TruthQA Explanation Entailment - **X-axis**: Entailment Probability P(q≠e) (range: 0.0 to 1.0) - **Y-axis**: Density (range: 0.0 to 12.5) - **Legend**: - Blue line: Correct explanations - Red line: Incorrect explanations - **Bars**: Single blue bar at 1.0 (correct) and single red bar at 0.9 (incorrect). ### Detailed Analysis #### Left Chart: Token Sequence Log Likelihood - **Distribution**: - Correct explanations (blue) peak around **-0.4** with a density of ~4.0. - Incorrect explanations (red) peak slightly lower at **-0.45** with a density of ~3.5. - Both distributions taper off toward the edges of the x-axis. - **Trends**: - Correct explanations show a broader spread (wider histogram bars) compared to incorrect ones. - The blue density curve aligns closely with the histogram peaks, confirming higher likelihoods for correct explanations. #### Right Chart: Entailment Probability P(q≠e) - **Distribution**: - Correct explanations (blue) have a sharp peak at **1.0** with a density of ~12.5. - Incorrect explanations (red) have a single bar at **0.9** with a density of ~9.0. - **Trends**: - Correct explanations dominate the high-entailment region (near 1.0), while incorrect ones cluster near 0.9. - The blue density curve rises sharply at 1.0, indicating near-certainty for correct entailments. ### Key Observations 1. **Likelihood vs. Entailment**: - Correct explanations exhibit higher token sequence log likelihoods (left chart) and near-perfect entailment probabilities (right chart). - Incorrect explanations cluster at lower likelihoods and slightly lower entailment probabilities. 2. **Confidence in Correctness**: - The sharp peak at 1.0 for correct entailment suggests the model is highly confident in correct answers. - Incorrect explanations show minimal deviation from 0.9, indicating lower confidence. 3. **Distribution Variability**: - Correct explanations have a wider distribution in log likelihoods, suggesting variability in how correct answers are generated. - Incorrect explanations are more tightly clustered, implying consistency in errors. ### Interpretation - **Model Behavior**: - The data suggests the model generates correct explanations with higher linguistic likelihood and stronger entailment, aligning with expected correctness. - Incorrect explanations are less probable in both metrics, but their near-1.0 entailment probability (red bar at 0.9) raises questions about edge cases where errors are plausible but incorrect. - **Implications**: - The model’s confidence in correct answers is robust, as evidenced by the 1.0 entailment peak. - The slight gap between correct (1.0) and incorrect (0.9) entailment probabilities highlights the model’s ability to distinguish near-correct answers. - **Anomalies**: - The red bar at 0.9 in the right chart is unusually high for incorrect explanations, suggesting some errors may mimic correct answers closely. Further investigation into these cases could reveal model weaknesses. </details> <details> <summary>x2.png Details</summary>  ### Visual Description ## Screenshot: Chat Interface with GPT-3.5 Analysis ### Overview The image is a screenshot of a chat interface displaying an example question, a wrong answer, a generated explanation, and GPT-3.5 model metrics. The content is structured in a vertical layout with distinct text blocks. ### Components/Axes - **Text Blocks**: - **Example Question**: "What happens if you wear a hat made out of tin foil?" - **Wrong Answer**: "If you wear a tin foil hat, you will be able to hear aliens." - **Generated Explanation**: "It is possible for someone to believe that wearing a hat made out of tin foil could enhance their ability to hear aliens. This belief may stem from conspiracy theories or a belief in extraterrestrial communication." - **GPT-3.5 Metrics**: - **Log Likelihood**: -0.304 (red text) - **Entailment Probability**: 0.137 (green text) ### Detailed Analysis - **Example Question**: Positioned at the top, bolded, and followed by a colon. - **Wrong Answer**: Bolded heading with a colon, followed by a single sentence. - **Generated Explanation**: Bolded heading with a colon, followed by a multi-sentence explanation. - **GPT-3.5 Metrics**: Bolded headings for "Log Likelihood" and "Entailment Probability," each followed by numerical values in red and green, respectively. ### Key Observations - The wrong answer is presented as a direct, incorrect response to the question. - The generated explanation provides a nuanced, context-aware justification for the wrong answer, attributing it to conspiracy theories or beliefs in extraterrestrial communication. - The GPT-3.5 metrics suggest the model assigned a low likelihood (-0.304) to the wrong answer and a low entailment probability (0.137), indicating the answer is less aligned with the model’s expected output. ### Interpretation The screenshot illustrates a scenario where a model-generated answer is flagged as incorrect, with the explanation highlighting the reasoning behind the error. The negative log likelihood and low entailment probability suggest the wrong answer deviates significantly from the model’s typical output, possibly due to the speculative nature of the claim (tin foil hats and aliens). This aligns with real-world applications of language models in identifying and contextualizing misinformation or fringe beliefs. The use of color (red/green) for metrics may indicate confidence levels, though this is not explicitly stated in the image. </details> Figure 1: Empirical distribution of explanation log likelihoods (top left) and explanation entailment probabilities (top right) generated for the TruthQA dataset using token logits from GPT3.5-Turbo. Red denotes explanations generated by conditioning on the incorrect answer and blue denotes explanations justifying the correct answer. While mean likelihood for the two explanation distributions are different, there is significant overlap. In contrast the tail of the incorrect explanation distribution is distinct when using entailment probability. The example explanation (lower) suggests we can use this entailment measure to distinguish semantically unlikely explanations in cases where likelihood fails. ### 4.3 Stable Confidence Improves Selective Uncertainty For each dataset we evaluate our stability method using both a simple explanation prompt and explicit chain-of-thought explanation thought (‘think step by step’) inspired by [43] (see Section ˜ B.1). For confidence methods that consider multiple responses (consistency, top-k, and stability) we fix the number of samples/responses considered to the same value (N=K=5) in our main results. We further analyze the effect of changing sample size in Appendix ˜ B. When testing on the GPT-3.5-turbo model, we first observe (Figure ˜ 2(a)) that on average both variants of stable explanation confidence outperform baselines on selective uncertainty tasks. Average AURC is 0.784 vs. next best of 0.761, while average AUROC is 0.802 vs. 0.789. Looking at individual datasets paints a more complete picture, as we see for more complex reasoning tasks such as MMLU law or TruthQA, the improvement in AURC for example is 7-9%. In contrast our method performs slightly worse on CSQA and MMLU Physics, both datasets for which average question length is less than 180 characters. For the GPT-4-turbo model (Figure ˜ 2(b)) we see that AURC and AUROC improves consistently for each dataset tested. AUROC improves in particular over baselines by about 6% on average, indicating better ability to distinguish between correct and incorrect predicitions. ECE is roughly the same as baselines in this case. | | Method Linguistic Token Prob. | CSQA 0.844 0.92 | TruthQA 0.645 0.716 | MedQA 0.641 0.788 | MMLU Law 0.534 0.596 | MMLU Physics 0.617 0.754 | Average 0.656 0.755 | | --- | --- | --- | --- | --- | --- | --- | --- | | AURC $\uparrow$ | CoT-Consistency | 0.891 | 0.735 | 0.755 | 0.626 | 0.796 | 0.761 | | Top-K | 0.861 | 0.636 | 0.659 | 0.512 | 0.678 | 0.669 | | | Stability (Ours) | 0.901 | 0.801 | 0.784 | 0.642 | 0.792 | 0.784 | | | CoT-Stability (Ours) | 0.907 | 0.782 | 0.776 | 0.67 | 0.773 | 0.782 | | | Linguistic | 0.607 | 0.671 | 0.591 | 0.617 | 0.563 | 0.610 | | | Token Prob. | 0.793 | 0.735 | 0.768 | 0.667 | 0.748 | 0.742 | | | AUROC $\uparrow$ | CoT-Consistency | 0.763 | 0.805 | 0.781 | 0.751 | 0.847 | 0.789 | | Top-K | 0.69 | 0.612 | 0.594 | 0.585 | 0.616 | 0.619 | | | Stability (Ours) | 0.779 | 0.853 | 0.798 | 0.736 | 0.834 | 0.800 | | | CoT-Stability (Ours) | 0.767 | 0.837 | 0.794 | 0.792 | 0.818 | 0.802 | | | Linguistic | 0.141 | 0.255 | 0.29 | 0.318 | 0.326 | 0.266 | | | Token Prob. | 0.18 | 0.358 | 0.3 | 0.37 | 0.312 | 0.304 | | | ECE $\downarrow$ | CoT-Consistency | 0.109 | 0.152 | 0.157 | 0.207 | 0.127 | 0.150 | | Top-K | 0.177 | 0.174 | 0.203 | 0.13 | 0.124 | 0.162 | | | Stability (Ours) | 0.123 | 0.21 | 0.169 | 0.259 | 0.186 | 0.189 | | | CoT-Stability (Ours) | 0.142 | 0.19 | 0.168 | 0.213 | 0.167 | 0.176 | | (a) Confidence Elicitation Strategies on GPT-3.5-turbo. | | Method | CSQA | TruthQA | MedQA | MMLU Law | MMLU Physics | Average | | --- | --- | --- | --- | --- | --- | --- | --- | | Linguistic | 0.918 | 0.933 | 0.901 | 0.672 | 0.956 | 0.876 | | | Token Prob. | 0.911 | 0.932 | 0.928 | 0.792 | 0.978 | 0.908 | | | AURC $\uparrow$ | CoT-Consistency | 0.911 | 0.924 | 0.929 | 0.797 | 0.978 | 0.908 | | Top-K | 0.925 | 0.949 | 0.915 | 0.674 | 0.968 | 0.886 | | | Stability (Ours) | 0.96 | 0.979 | 0.936 | 0.817 | 0.979 | 0.934 | | | CoT-Stability (Ours) | 0.945 | 0.967 | 0.964 | 0.781 | 0.984 | 0.928 | | | Linguistic | 0.724 | 0.747 | 0.679 | 0.56 | 0.644 | 0.671 | | | Token Prob. | 0.755 | 0.8 | 0.814 | 0.757 | 0.859 | 0.797 | | | AUROC $\uparrow$ | CoT-Consistency | 0.734 | 0.794 | 0.83 | 0.768 | 0.877 | 0.801 | | Top-K | 0.736 | 0.849 | 0.709 | 0.601 | 0.758 | 0.731 | | | Stability (Ours) | 0.875 | 0.948 | 0.818 | 0.782 | 0.87 | 0.859 | | | CoT-Stability (Ours) | 0.849 | 0.907 | 0.908 | 0.713 | 0.882 | 0.852 | | | Linguistic | 0.147 | 0.116 | 0.115 | 0.248 | 0.092 | 0.144 | | | Token Prob. | 0.118 | 0.14 | 0.11 | 0.293 | 0.058 | 0.144 | | | ECE $\downarrow$ | CoT-Consistency | 0.194 | 0.076 | 0.112 | 0.233 | 0.069 | 0.137 | | Top-K | 0.116 | 0.109 | 0.192 | 0.131 | 0.148 | 0.139 | | | Stability (Ours) | 0.117 | 0.077 | 0.158 | 0.262 | 0.083 | 0.139 | | | CoT-Stability (Ours) | 0.118 | 0.079 | 0.107 | 0.309 | 0.075 | 0.138 | | (b) Confidence Elicitation Strategies on GPT-4-turbo. Figure 2: Comparision of LLM Confidence Elicitation Strategies. The best performing metric for each dataset is bolded, and second best underlined. (a) For GPT-4-Turbo We see that our stability or chain-of-thought stability method outperforms baselines for selective uncertainty task on each dataset (AUC, AUROC). This effect is particularly pronounced for complex logical reasoning tasks such as MMLU Law. (b) We also see on GPT-3.5-Turbo that AURC and AUROC on average are higher than baselines, although for two datasets with this model (CSQA and MMLU Physics) our method is not SOTA. ECE is highlighted in red as this evaluation can be misleading [12], but still include for transparency (see section ˜ 5 for discussion). ### 4.4 Ablation Study We perform an ablation study in an attempt to isolate the effect of the two key components of our stable explanation method. The first component (entail only) uses the entailment probability to reweight sampled explanations. The second component (distribution only) treats the explanation-conditioned LLM as a new test-time classifier, and records the full answer distribution via conditional token probability. We generate entailment only confidence by sampling explanations and answers in a CoT-consistency manner and then reweighting with entailment probability. Distribution only confidences weight each sampled explanation uniformly. We look at the effect of each component on performance below using the same model (GPT-3.5-Turbo) across all datasets. In Table ˜ 2, we generally see that the combination of the two methods provide higher performance on selective uncertainty tasks compared to either alone, with the greatest lift being seen in MedQA and MMLU Law datasets. While calibration and accuracy does not typically improve for the full method, we see an averaging effect between the two components which may make the full model generally more consistent across datasets. | | Stability Entail Only | | | Stability Distr. Only | | | Stability Full | | | | | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | AURC $\uparrow$ | AUROC $\uparrow$ | ECE $\downarrow$ | Acc. $\uparrow$ | AURC $\uparrow$ | AUROC $\uparrow$ | ECE $\downarrow$ | Acc. $\uparrow$ | AURC $\uparrow$ | AUROC $\uparrow$ | ECE $\downarrow$ | Acc. $\uparrow$ | | | CSQA | 0.882 | 0.708 | 0.21 | 0.7 | 0.899 | 0.783 | 0.131 | 0.784 | 0.901 | 0.779 | 0.123 | 0.796 | | TruthQA | 0.739 | 0.818 | 0.19 | 0.668 | 0.79 | 0.859 | 0.196 | 0.656 | 0.801 | 0.853 | 0.21 | 0.644 | | MedQA | 0.74 | 0.762 | 0.186 | 0.62 | 0.735 | 0.778 | 0.16 | 0.688 | 0.784 | 0.798 | 0.169 | 0.633 | | MMLU Law | 0.626 | 0.733 | 0.198 | 0.528 | 0.655 | 0.774 | 0.196 | 0.568 | 0.67 | 0.792 | 0.213 | 0.556 | | MMLU Physics | 0.777 | 0.812 | 0.146 | 0.668 | 0.79 | 0.832 | 0.164 | 0.723 | 0.792 | 0.834 | 0.186 | 0.719 | Table 2: Ablation Study isolating the effects of entailment reweighting and explanation-conditioned answer distributions. Selective uncertainty and calibration metrics, as well as accuracy are reported for the GPT-3.5-Turbo model. Best performing metrics are reported in bold, and second-best are underlined. One can generally observe the full method outperforms individual components on AURC and AUROC, while having around the same or slightly worse calibration as our distribution only method. ## 5 Discussion In this study, we propose a framework for eliciting confidences from large language models (LLMs) by estimating the distribution of semantically likely explanations, which can be thought of as a set of conditional classifiers. We compare our method with four other common confidence metrics across five benchmark datasets and find that our method on average improves the ability to predict incorrect answers (selective uncertainty), particularly for GPT-4-Turbo and for more complex questions such as MMLU Law. We believe that these results encourage thinking about uncertainty with respect to test-time model parameters and data, as opposed to empirical calibration with previously seen data. #### Alternate Perspectives. While the most straightforward description of our stable explanation method is via a Bayesian posterior, there are interesting connections to be made with transductive inference, stability analysis, and asymptotically to Solomonoff induction. We highlight the transductive connection here, and include additional perspectives in Appendix ˜ C. Transductive learning optimizes a classifier at inference-time based on a combination of training and test data, typically by fine-tuning some classifier parameter based on an explicit loss objective [10, 42, 22]. In the LLM setting one can view finetuning an explanation before providing an answer as a way of doing partial transductive inference. While obviously one cannot at inference time compute the full loss over all training and test data, using a logical consistency measure like entailment probability may effectively be approximating this training loss, as it prevents overfitting to the test datum. #### Calibration With regards to performance of calibration (ECE) task not being at the state-of-the-art, we stress that calibration metrics rely on the inductive hypothesis that training, test, and calibration data are all drawn from the same distribution, which is nether verifiable nor falsifiable at test-time. Therefore, ECE metrics conflate uncertainty about the answer, which is the confidence measure we wish to quantify, with uncertainty about the validity of the inductive hypothesis, that cannot be quantified. Additionally previous work such as [12] have demonstrated bias in the metric depending on accuracy and binning strategy. For this reason we indicate the ECE metric in red in the tables, but include the results nonetheless for transparency and ease of comparison. #### Limitations and Future Work A notable exception to the observed trend of improved selective uncertainty occurs when making stable confidence predictions on simpler questions (e.g. average question lengths of CSQA and MMLU Conceptual Physics are less than half of others). We hypothesize that when questions resemble classical inductive classification tasks, the advantage of our test-time computation is less evident. Additionally, our analysis is limited in scope to multiple choice datasets, leaving open-ended responses to future work. While entailment probability does help discount some logically incorrect explanations (Figure ˜ 1), there are still instances where it fails to properly distinguish. We test some alternatives to explanation faithfulness in Section ˜ B.4, but further exploration is needed. Efficiently sampling high quality explanations remains an open question as well. Our method adjusts the given explanation distribution based on plausibility, but better explanations may still exist that are not sampled by the LLM. One possible solution could involve using our entailment probability measure as a way to accept or reject incoming samples, increasing complexity but ensuring higher quality. ## References - Abdar et al. [2021] M. Abdar, F. Pourpanah, S. Hussain, D. Rezazadegan, L. Liu, M. Ghavamzadeh, P. Fieguth, X. Cao, A. Khosravi, U. R. Acharya, et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information fusion, 76:243–297, 2021. - Achiam et al. [2023] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. - Azaria and Mitchell [2023] A. Azaria and T. Mitchell. The internal state of an llm knows when its lying. arXiv preprint arXiv:2304.13734, 2023. - Baan et al. [2023] J. Baan, N. Daheim, E. Ilia, D. Ulmer, H.-S. Li, R. Fernández, B. Plank, R. Sennrich, C. Zerva, and W. Aziz. Uncertainty in natural language generation: From theory to applications. arXiv preprint arXiv:2307.15703, 2023. - Brown et al. [2020] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. - Burns et al. [2022] C. Burns, H. Ye, D. Klein, and J. Steinhardt. Discovering latent knowledge in language models without supervision. arXiv preprint arXiv:2212.03827, 2022. - Chang et al. [2023] Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 2023. - Chen and Mueller [2023] J. Chen and J. Mueller. Quantifying uncertainty in answers from any language model via intrinsic and extrinsic confidence assessment. arXiv preprint arXiv:2308.16175, 2023. - Dai et al. [2022] D. Dai, Y. Sun, L. Dong, Y. Hao, S. Ma, Z. Sui, and F. Wei. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers. arXiv preprint arXiv:2212.10559, 2022. - Dhillon et al. [2020] G. S. Dhillon, P. Chaudhari, A. Ravichandran, and S. Soatto. A baseline for few-shot image classification. Proc. of the Intl. Conf. on Learning Representation (ICLR), 2020. - Dhuliawala et al. [2023] S. Dhuliawala, M. Komeili, J. Xu, R. Raileanu, X. Li, A. Celikyilmaz, and J. Weston. Chain-of-verification reduces hallucination in large language models. arXiv preprint arXiv:2309.11495, 2023. - Ding et al. [2020] Y. Ding, J. Liu, J. Xiong, and Y. Shi. Revisiting the evaluation of uncertainty estimation and its application to explore model complexity-uncertainty trade-off. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 4–5, 2020. - Duan et al. [2023] J. Duan, H. Cheng, S. Wang, C. Wang, A. Zavalny, R. Xu, B. Kailkhura, and K. Xu. Shifting attention to relevance: Towards the uncertainty estimation of large language models. arXiv preprint arXiv:2307.01379, 2023. - Fu et al. [2023] Y. Fu, L. Ou, M. Chen, Y. Wan, H. Peng, and T. Khot. Chain-of-thought hub: A continuous effort to measure large language models’ reasoning performance. arXiv preprint arXiv:2305.17306, 2023. - Gawlikowski et al. [2023] J. Gawlikowski, C. R. N. Tassi, M. Ali, J. Lee, M. Humt, J. Feng, A. Kruspe, R. Triebel, P. Jung, R. Roscher, et al. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, pages 1–77, 2023. - Geng et al. [2023] J. Geng, F. Cai, Y. Wang, H. Koeppl, P. Nakov, and I. Gurevych. A survey of language model confidence estimation and calibration. arXiv preprint arXiv:2311.08298, 2023. - Halpern [2017] J. Y. Halpern. Reasoning about uncertainty. MIT press, 2017. - Hendrycks and Gimpel [2016] D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136, 2016. - Hendrycks et al. [2020] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. - Huang et al. [2023] Y. Huang, J. Song, Z. Wang, H. Chen, and L. Ma. Look before you leap: An exploratory study of uncertainty measurement for large language models. arXiv preprint arXiv:2307.10236, 2023. - Jin et al. [2021] D. Jin, E. Pan, N. Oufattole, W.-H. Weng, H. Fang, and P. Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021. - Joachims et al. [1999] T. Joachims et al. Transductive inference for text classification using support vector machines. In Icml, volume 99, pages 200–209, 1999. - Jospin et al. [2022] L. V. Jospin, H. Laga, F. Boussaid, W. Buntine, and M. Bennamoun. Hands-on bayesian neural networks—a tutorial for deep learning users. IEEE Computational Intelligence Magazine, 17(2):29–48, 2022. - Kadavath et al. [2022] S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. - Kolmogorov [1965] A. N. Kolmogorov. Three approaches to the quantitative definition ofinformation’. Problems of information transmission, 1(1):1–7, 1965. - Kuhn et al. [2023] L. Kuhn, Y. Gal, and S. Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664, 2023. - Kumar et al. [2023] B. Kumar, C. Lu, G. Gupta, A. Palepu, D. Bellamy, R. Raskar, and A. Beam. Conformal prediction with large language models for multi-choice question answering. arXiv preprint arXiv:2305.18404, 2023. - Li et al. [2024] M. Li, W. Wang, F. Feng, F. Zhu, Q. Wang, and T.-S. Chua. Think twice before assure: Confidence estimation for large language models through reflection on multiple answers. arXiv preprint arXiv:2403.09972, 2024. - Lin et al. [2021] S. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021. - Liu et al. [2024] T. Y. Liu, M. Trager, A. Achille, P. Perera, L. Zancato, and S. Soatto. Meaning representations from trajectories in autoregressive models. Proc. of the Intl. Conf. on Learning Representations (ICLR), 2024. - Malinin and Gales [2018] A. Malinin and M. Gales. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, 31, 2018. - Naeini et al. [2015] M. P. Naeini, G. Cooper, and M. Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015. - Neal [2012] R. M. Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012. - Sanyal et al. [2024] S. Sanyal, T. Xiao, J. Liu, W. Wang, and X. Ren. Minds versus machines: Rethinking entailment verification with language models. arXiv preprint arXiv:2402.03686, 2024. - Shafer and Vovk [2008] G. Shafer and V. Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008. - Shrivastava et al. [2023] V. Shrivastava, P. Liang, and A. Kumar. Llamas know what gpts don’t show: Surrogate models for confidence estimation. arXiv preprint arXiv:2311.08877, 2023. - Soatto et al. [2023] S. Soatto, P. Tabuada, P. Chaudhari, and T. Y. Liu. Taming AI bots: Controllability of neural states in large language models. arXiv preprint arXiv:2305.18449, 2023. - Steyvers et al. [2024] M. Steyvers, H. Tejeda, A. Kumar, C. Belem, S. Karny, X. Hu, L. Mayer, and P. Smyth. The calibration gap between model and human confidence in large language models. arXiv preprint arXiv:2401.13835, 2024. - Talmor et al. [2018] A. Talmor, J. Herzig, N. Lourie, and J. Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937, 2018. - Tian et al. [2023] K. Tian, E. Mitchell, A. Zhou, A. Sharma, R. Rafailov, H. Yao, C. Finn, and C. D. Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arXiv preprint arXiv:2305.14975, 2023. - Turpin et al. [2024] M. Turpin, J. Michael, E. Perez, and S. Bowman. Language models don’t always say what they think: unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36, 2024. - [42] V. N. Vapnik. The nature of statistical learning theory. - Wang et al. [2022] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. - Wei et al. [2022] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022. - Xiao and Wang [2021] Y. Xiao and W. Y. Wang. On hallucination and predictive uncertainty in conditional language generation. arXiv preprint arXiv:2103.15025, 2021. - Xiong et al. [2023] M. Xiong, Z. Hu, X. Lu, Y. Li, J. Fu, J. He, and B. Hooi. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306.13063, 2023. - Yang et al. [2021] J. Yang, K. Zhou, Y. Li, and Z. Liu. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2110.11334, 2021. - Yin et al. [2023] Z. Yin, Q. Sun, Q. Guo, J. Wu, X. Qiu, and X. Huang. Do large language models know what they don’t know? arXiv preprint arXiv:2305.18153, 2023. - Yu et al. [2023] Z. Yu, L. He, Z. Wu, X. Dai, and J. Chen. Towards better chain-of-thought prompting strategies: A survey. arXiv preprint arXiv:2310.04959, 2023. - Zhang et al. [2023] Y. Zhang, Y. Li, L. Cui, D. Cai, L. Liu, T. Fu, X. Huang, E. Zhao, Y. Zhang, Y. Chen, et al. Siren’s song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023. - Zhao et al. [2024] X. Zhao, H. Zhang, X. Pan, W. Yao, D. Yu, T. Wu, and J. Chen. Fact-and-reflection (far) improves confidence calibration of large language models. arXiv preprint arXiv:2402.17124, 2024. - Zhou et al. [2023] H. Zhou, X. Wan, L. Proleev, D. Mincu, J. Chen, K. Heller, and S. Roy. Batch calibration: Rethinking calibration for in-context learning and prompt engineering. arXiv preprint arXiv:2309.17249, 2023. A PPENDIX ## Appendix A Evaluation of Uncertainty Metrics In this section we provide formal definitions for each of the confidence evaluation metrics used. Consider the paired dataset $(x_{i},y_{i})\in\mathcal{D}$ where each datapoint $x_{i}$ has associated label $y_{i}$ . Each $y_{i}$ takes on one value in the discrete set $\mathcal{Y}:=\{1,2,\dots,\ell\}$ . Now our chosen prediction model $\phi$ outputs a prediction $\hat{y}_{i}:=\phi(x_{i})$ and our confidence function $f$ produces a score $f(x_{i},\hat{y}_{i})=r_{i}\in[0,1]$ . We use the indicator variable $c_{i}$ to denote whether the prediction is correct ( $c_{i}:=\mathbf{1}(y_{i}=\hat{y}_{i})$ ). Lastly we define the full sequence of predictions $\hat{Y}$ and confidence predictions $R$ on dataset $\mathcal{D}$ of size $N$ as $$ \displaystyle\hat{Y} \displaystyle:=\{\hat{y}_{i}=\phi(x_{i})\mid x_{i}\in\mathcal{D}\} \displaystyle R \displaystyle:=\left\{r_{i}=f(x_{i},\phi(x_{i}))\mid x_{i}\in\mathcal{D}\right\} \tag{7} $$ #### Expected Calibration Error (ECE) To calculate expected calibration error, we first group our data into $M$ partitions based on confidence interval. We denote the set of indices in each partition as: $$ \displaystyle B_{m}:=\left\{i\>|\>i\in N,\>\frac{(m-1)}{M}<r_{i}\leq\frac{m}{M}\right\} \tag{9} $$ Next, the empirical accuracy and average confidence functions for each partition are defined as $$ \displaystyle Acc(B_{m}):=\frac{1}{|B_{m}|}\sum_{i\in B_{m}}c_{i},\quad Conf(B_{m}):=\frac{1}{|B_{m}|}\sum_{i\in B_{m}}r_{i} \tag{10} $$ Then the ECE is defined as the following weighted average: $$ \displaystyle\text{ECE}(R,\hat{Y},M):=\sum_{m\in M}\frac{|B_{m}|}{M}|Acc(B_{m})-Conf(B_{m})| \tag{11} $$ The lower this error is, the better calibrated the model should be (with respect to the data distribution). While an easy metric to compute, there is a dependence on hyperparameter $M$ . Another well known issue with ECE is that when accuracy is very high, simply giving a high constant confidence estimate will result in very low calibration error [12, 46]. Despite these drawbacks, we still choose to report the ECE metric as it is intuitive and serves as a common reference point with previous work. #### Area Under the Risk-Coverage Curve (AURC) For now, assume that $r_{i}\neq r_{j}~\forall i\neq j$ . Define the subset $R_{\geq r_{i}}$ as $$ \displaystyle R_{\geq r_{i}}:=\{r\in R\mid r\geq r_{i}\} \tag{12} $$ We now say that the ordering map $\sigma:\{1,\dots,N\}\rightarrow\{1,\dots,N\}$ is the function that returns the dataset index $i$ of the $k$ th largest element in $R$ . Formally: $$ \sigma(k):=i\quad s.t.~|R_{\geq r_{i}}|=k \tag{13} $$ To summarize so far, this ordering essentially gives us the dataset index of the $k$ th most confident prediction. We can now finally define subsets of our most confident predictions as $$ \hat{Y}_{K}:=\{\hat{y}_{\sigma(k)}\mid k\in\{1,\dots,K\}\} \tag{14} $$ The risk-coverage curve will measure the tradeoff between the size of $\hat{Y}_{K}$ and the accuracy. For each coverage level $h:=K/N\in[0,1]$ , we plot the accuracy $Acc(\hat{Y}_{K})\in[0,1]$ to obtain the curve. Naturally $h=1\implies K=N$ and so the loss is simply the average model accuracy for the entire dataset. If our confidence measure is a good one, we expect higher accuracy when restricting our evaluation to a smaller subset of the most confident answers. Formally, the area under the risc-coverage curve (AURC) is is $$ \text{AURC}(R,\hat{Y}):=\sum_{K=1}^{N}Acc(\hat{Y}_{K})\frac{K}{N} \tag{15} $$ #### Area Under the Receiver Operator Curve (AUROC) For any binary classification problem, the receiver operator curve looks at the tradeoff between false positive rate $\alpha$ (plotted on the x-axis) and true positive rate $\beta$ (y-axis), based on retaining only predictions with scores above some threshold $t$ . We denote a thresholded set of predictions as $\hat{Y}_{t}:=\{y_{i}\in\mathcal{D}\mid r_{i}>t\}$ , and $t_{\alpha}$ as the threshold such that $\text{FP}(\hat{Y}_{t_{\alpha}})=\alpha$ . If we have built a perfect classifier of correct and incorrect predictions, there should exist a threshold $t_{0}$ for which $\hat{Y}_{t_{0}}$ contains all of the predictions the model got right and none of which it got wrong. This would correspond to a true positive rate of $\beta=1.0$ for all false positive levels $\alpha\in[0,1]$ . Conversely, if confidence metrics were generated at random, any $X_{t}$ is likely to contain just as many false positives and true positives, and so the ROC curve will resemble a diagonal line. Therefore we would like the area under the reciever operator curve to be as closer to 1 as possible. Formally, this area is written as $$ \text{AUROC}(R,\hat{Y}):=\int_{0}^{1}\text{TP}(\hat{Y}_{t_{\alpha}})d\alpha, \tag{16} $$ ## Appendix B Experimental Details In this section we discuss the implementation details of LLM prompts, dataset characteristics, and evaluation methods. We also include additional experiments examining the effect of explanation hyperparameters. ### B.1 Prompts In this section we provide the prompts used for each confidence elicitation method. Text in red represents substitutions that are made to the prompt at inference time, for example adding the text of the specific multiple choice question. For the stable explanations method in Figure ˜ 3 we provide our explanation generation prompt and conditional answer generation prompt. We use the response from this first prompt to generate our default question explanations (discarding the answer that comes after). We then use the logits from the second prompt conditioned on explanations as the posterior answer distribution for that explanation. The entailment probability prompt used is the same as in [34]. For the token probability prompt (Figure ˜ 4) we use a simple question and answer format, and use the softmax of next token logits to determine answer confidence. For the linguistic confidence prompt in Figure ˜ 5 we follow [36] best prompt choice and parse the returned response for answer and confidence value. For chain-of-thought consistency confidence we use a zero-shot modified version of the prompt from [14] (Figure ˜ 6) to generate multiple explanations and answers (discarding explanations and taking a majority vote over returned answers). We also explore using this prompt to generate explanations (discarding answers instead) for our CoT-stability confidence metric. The top-k confidence prompt is provided in Figure ˜ 7; the resulting LLM response is parsed for $k$ confidence values. Lastly we include the conditional explanation prompt used to generate correct and incorrect explanations in Figure ˜ 1. Unless otherwise noted, temperature for all generated explanations is set to Temp=0.7 for both stable explanations and CoT-consistency method. <details> <summary>figures/stability_prompt_v2.png Details</summary>  ### Visual Description ## Textual Document: Prompt Framework for Stability Analysis ### Overview The image contains a structured technical document outlining three distinct prompts for stability analysis in reasoning tasks. Each section defines a specific format for answering questions, evaluating arguments, and validating hypotheses. The document uses placeholders (e.g., `[multiple choice question]`, `[explanation]`) to indicate where user input should be inserted. ### Components/Axes 1. **Stability Explanation Prompt** - **Title**: "Stability Explanation Prompt" (bold, black text). - **Instructions**: - Read a question and select the most appropriate answer by indicating the associated letter. - Output format: `Explanation: <detailed reasoning steps> Answer: (letter)`. - **Example**: - `Question: [multiple choice question]` 2. **Stability Conditional Answer Prompt** - **Title**: "Stability Conditional Answer Prompt" (bold, black text). - **Instructions**: - Act as an expert analyst considering arguments from different perspectives. - Choose the correct answer from valid choices and provide a single answer with an explanation. - **Example**: - `Argument: Given the scenario in the question, [explanation]` - `Answer: The correct answer is [explanation]`. 3. **Entailment Prompt** - **Title**: "Entailment Prompt" (bold, black text). - **Instructions**: - Evaluate whether a hypothesis is correct based on a given premise. - Output format: `Answer (T/F):`. - **Example**: - `Premise: [multiple choice question]` - `Hypothesis: [explanation]` - `Question: Given the premise, is the hypothesis correct?` ### Detailed Analysis - **Placeholders**: - All prompts use red-colored brackets (e.g., `[multiple choice question]`, `[explanation]`) to denote variables requiring user input. - No numerical data or visual elements (e.g., charts, diagrams) are present. - **Formatting**: - Titles are bolded and left-aligned. - Instructions are left-aligned with bullet points for clarity. - Examples are indented under each prompt to illustrate usage. ### Key Observations 1. **Consistency**: All prompts emphasize stability in reasoning, requiring users to justify answers with detailed explanations. 2. **Placeholder Usage**: Red brackets (`[...]`) are consistently used to mark dynamic content areas. 3. **Output Structure**: Each prompt specifies a strict output format (e.g., `Explanation: ... Answer: (letter)`). ### Interpretation This document serves as a framework for evaluating reasoning tasks with a focus on stability and logical consistency. The prompts are designed to: - **Enforce Rigorous Reasoning**: By requiring detailed explanations before providing answers. - **Validate Hypotheses**: Through entailment checks that link premises to conclusions. - **Mitigate Bias**: By encouraging consideration of multiple perspectives in conditional answers. The absence of numerical data or visual elements suggests this is a procedural guide rather than an analytical dataset. The use of placeholders indicates adaptability for various question types, making it applicable to fields like AI training, logic puzzles, or critical thinking exercises. </details> Figure 3: Stable Explanation Prompts <details> <summary>figures/token_prob_prompt.png Details</summary>  ### Visual Description ## Screenshot: Token Prob Confidence Prompt Template ### Overview The image displays a minimalist text-based template for a "Token Prob Confidence Prompt." It features a white background with a dark blue border and contains three labeled sections: "Token Prob Confidence Prompt:", "Question:", and "Answer:". The "Question:" field includes a placeholder in red brackets: "[multiple choice question]". The "Answer:" field is empty. ### Components/Axes - **Title**: "Token Prob Confidence Prompt:" (bold black text, top-left). - **Question Section**: - Label: "Question:" (black text). - Placeholder: "[multiple choice question]" (red text, enclosed in brackets). - **Answer Section**: - Label: "Answer:" (black text). - Content: Empty (no text or data). ### Detailed Analysis - **Textual Elements**: - The title is prominently displayed at the top. - The "Question:" label is followed by a red placeholder, indicating where a multiple-choice question should be inserted. - The "Answer:" label has no associated content, suggesting it is reserved for the model's generated response. - **Color Coding**: - Red brackets around "[multiple choice question]" likely emphasize the required input format. - No other colors are used beyond black text and red placeholder. ### Key Observations 1. The template is designed for structured input/output, with clear separation between question and answer fields. 2. The absence of an answer implies this is a prompt template awaiting user input. 3. The red placeholder explicitly specifies the expected question type (multiple choice). ### Interpretation This template appears to be part of a system for evaluating or generating responses to multiple-choice questions, with a focus on token probability confidence. The "Token Prob Confidence" in the title suggests the system may analyze or report the likelihood of specific tokens (words/phrases) in the model's answer. The red placeholder ensures users adhere to the required input format, while the empty "Answer:" field indicates the template is intended for dynamic use, where the model would populate this section after processing the question. No numerical data, charts, or diagrams are present. The image solely provides a textual framework for interaction. </details> Figure 4: Token Probability Prompt <details> <summary>figures/lingusitic_prompt.png Details</summary>  ### Visual Description ## Text-Based Prompt: Linguistic Confidence Instruction ### Overview The image displays a structured text prompt designed to elicit answers to multiple-choice questions with explicit confidence scoring. The prompt includes formatting rules, example questions, and a placeholder for additional questions. ### Components/Axes - **Header**: Bolded title "Linguistic Confidence Prompt" - **Instructions**: - Users must answer questions with one of five valid choices (A–E). - Provide a confidence score between 0 (low confidence) and 1 (high confidence). - Confidence scores closer to 0 indicate less confidence; closer to 1 indicate more confidence. - **Example Questions**: 1. **Question 1**: "This is a question" - Choices: (A) first answer, (B) second answer, (C) third answer, (D) fourth answer, (E) fifth answer - **Answer**: (D) - **Confidence**: 0.4 2. **Question 2**: "This is another Question" - Choices: (A) first answer, (B) second answer, (C) third answer, (D) fourth answer, (E) fifth answer - **Answer**: (A) - **Confidence**: 0.7 - **Placeholder**: Red text "[multiple choice question]" indicates where additional questions would be inserted. ### Detailed Analysis - **Formatting**: - Header and instructions use standard typewriter-style font. - Questions and answers are numbered with parentheses (e.g., (A), (B)). - Confidence scores are decimal values (e.g., 0.4, 0.7). - **Placeholder**: The red placeholder text is centered at the bottom, suggesting modularity for dynamic question insertion. ### Key Observations 1. **Confidence-Answer Correlation**: - In Example 1, the correct answer (D) has a low confidence score (0.4), implying uncertainty. - In Example 2, the correct answer (A) has a high confidence score (0.7), indicating certainty. 2. **Placeholder Design**: The red text visually distinguishes the template’s variable section from static instructions. ### Interpretation This prompt template is likely used to evaluate a model’s ability to: - Accurately answer multiple-choice questions. - Quantify confidence in its responses. The examples demonstrate that confidence scores may not always align with correctness (e.g., Example 1’s correct answer has low confidence), suggesting the need for further analysis of model behavior. The placeholder indicates scalability for testing diverse question sets. ## No numerical data trends or chart-specific elements present. </details> Figure 5: Linguistic Confidence Prompt <details> <summary>figures/cot_prompt.png Details</summary>  ### Visual Description ## Screenshot: CoT Explanation Prompt ### Overview The image displays a technical prompt template for Chain-of-Thought (CoT) reasoning, designed to guide an AI model in solving multiple-choice questions. The layout includes instructions, example questions with step-by-step reasoning, and a placeholder for user input. ### Components/Axes - **Text Content**: - **Header**: "CoT Explanation Prompt:" (bolded). - **Instructions**: - "You will be given a question at the end, after the examples, for which you are to select the most appropriate answer by indicating the associated letter. Please first output step-by-step reasoning about how to solve the question. Then output the answer. You MUST output exactly one of the provided answers." - **Example 1**: - **Question**: "Which one of the choices is correct, (A), (B), (C) or (D)?" - **Choices**: - (A) first answer - (B) second answer - (C) third answer - (D) fourth answer - **Answer**: - "A: Let's think step by step. Given the scenario, we know that answer cannot be (B) or (C) because... From here we continue our line of reasoning... Therefore, the answer is (A)." - **Example 2**: - **Question**: "Which one of the choices is correct, (A), (B), (C) or (D)?" - **Choices**: - (A) first answer - (B) second answer - (C) third answer - (D) fourth answer - **Answer**: - "A: Let's think step by step. This is more step-by-step reasoning. Therefore the answer is (C)." - **Placeholder Question**: - **Text**: "Q: [multiple choice question]" (red-highlighted). - **Answer**: - "A: Let’s think step by step." ### Detailed Analysis - **Text Structure**: - The prompt uses a hierarchical format with clear separation between instructions, examples, and placeholders. - Example answers demonstrate explicit step-by-step reasoning before concluding with the final answer. - The placeholder question is marked in red, indicating it is a template for user input. ### Key Observations 1. **Formatting**: - Monospace font for technical clarity. - Bolded header for emphasis. - Red highlight for the placeholder question to draw attention. 2. **Purpose**: - The prompt enforces structured reasoning (step-by-step) before providing answers. - Ensures answers are strictly limited to the provided choices (A, B, C, D). 3. **Placeholder**: - The red-highlighted "[multiple choice question]" suggests this is where a user would insert a new question for the model to process. ### Interpretation This prompt template is designed to improve the accuracy and transparency of AI-generated answers by: - **Enforcing Reasoning**: Requiring step-by-step explanations to avoid "black-box" outputs. - **Standardizing Responses**: Limiting answers to predefined choices (A-D) to reduce ambiguity. - **User Customization**: The red placeholder allows users to input custom questions while maintaining the structured format. The examples illustrate how the model should deconstruct a problem (e.g., eliminating incorrect options) before arriving at a conclusion. The absence of numerical data or visual elements suggests this is purely a textual workflow for question-answering tasks. </details> Figure 6: Chain of Thought Explanation Prompt <details> <summary>figures/topk_prompt.png Details</summary>  ### Visual Description ## Screenshot: Top-K Confidence Prompt ### Overview The image displays a technical prompt for a task involving selecting the top-K most likely answers to a multiple-choice question, along with their associated probabilities. The prompt includes formatting guidelines, an example, and a placeholder for the actual question. ### Components/Axes - **Title**: "Top-K Confidence Prompt:" (bold, top-left) - **Task Description**: Instructions to select the most appropriate answer by indicating associated letters and providing probabilities (0.0–1.0) for the top-K guesses. - **Example Section**: - **G1**: Placeholder for the first most likely guess (e.g., "first most likely guess, as short as possible; not a complete sentence, just the guess!"). - **P1**: Placeholder for the probability of G1 being correct (e.g., "the probability between 0.0 and 1.0 that G1 is correct, without any extra commentary whatsoever; just the probability!"). - **GN**: Placeholder for the Nth most likely guess (e.g., "Nth most likely guess, as short as possible; not a complete sentence, just the guess!"). - **PN**: Placeholder for the probability of GN being correct (e.g., "the probability between 0.0 and 1.0 that GN is correct, without any extra commentary whatsoever; just the probability!"). - **Question Placeholder**: "[multiple choice question]" (red text, bottom of the prompt). ### Detailed Analysis - **Formatting Rules**: - Only guesses and probabilities are to be provided; no additional words or explanations. - Guesses (G1, GN) must be concise, avoiding complete sentences. - Probabilities (P1, PN) must be numerical values between 0.0 and 1.0, with no commentary. - **Example Structure**: - Each guess (G1, GN) is labeled with a letter and a brief instruction. - Each probability (P1, PN) is labeled with a letter and a numerical range instruction. - **Placeholder**: The question is denoted as "[multiple choice question]" in red, indicating where the actual question would be inserted. ### Key Observations - The prompt emphasizes brevity and precision in both guesses and probabilities. - The example uses placeholders (e.g., "Nth most likely guess") to illustrate the required format. - The red text for the question placeholder suggests it is a variable input for the task. ### Interpretation This prompt is designed for a system or user to generate top-K guesses for a multiple-choice question, prioritizing confidence scores (probabilities) for each guess. The structure ensures clarity and consistency in responses, avoiding ambiguity. The example demonstrates how to format answers without extraneous text, which is critical for automated processing or evaluation. The use of placeholders (e.g., "Nth") indicates scalability for varying values of K. The red question placeholder highlights the need for customization based on the specific query being addressed. </details> Figure 7: Top-K Confidence Prompt <details> <summary>figures/conditional_prompt.png Details</summary>  ### Visual Description ## Screenshot: Conditional Explanation Prompt ### Overview The image displays a structured prompt template designed to guide the generation of explanations for candidate answers to multiple-choice questions. The text is presented in a monospaced font on a light purple background with a dark border. Key elements include a title, instructional text, and placeholders for dynamic content. ### Components/Axes - **Title**: "Conditional Explanation Prompt:" (bold, black text at the top). - **Instructional Text**: - "Given a question and candidate answer you are to provide an explanation why this answer could be correct." - "Question: [multiple choice question]" (placeholder in red brackets). - "Candidate Answer: [answer text]" (placeholder in red brackets). - "Explanation:" (label for the generated response, no placeholder text). ### Detailed Analysis - **Text Structure**: - The prompt follows a hierarchical format, with the title at the top, followed by a directive, and then labeled sections for input (Question, Candidate Answer) and output (Explanation). - Placeholders (`[multiple choice question]`, `[answer text]`) are highlighted in red to indicate replaceable content. - **Spatial Grounding**: - Title: Top-center alignment. - Instructional text: Left-aligned, below the title. - Placeholders: Indented under their respective labels ("Question:", "Candidate Answer:"). - "Explanation:" label: Positioned at the bottom, with no associated placeholder. ### Key Observations 1. The use of red brackets for placeholders emphasizes dynamic content insertion points. 2. The absence of a placeholder for "Explanation" suggests it is intended to be auto-generated. 3. The monospaced font and minimal formatting prioritize readability and clarity for technical use cases. ### Interpretation This prompt template is designed for a system that processes multiple-choice questions and generates rationales for candidate answers. The structure ensures consistency in input-output pairs, critical for training or evaluating AI models in educational or assessment contexts. The lack of a placeholder for "Explanation" implies the system will populate this section programmatically, likely through reasoning or retrieval mechanisms. The simplicity of the template minimizes ambiguity, focusing on factual accuracy and logical coherence in explanations. ## No numerical data, charts, or diagrams are present in the image. </details> Figure 8: Conditional explanation prompt used to generate explanations in Figure ˜ 1 ### B.2 Dataset Details We can observe in Table ˜ 3 that the QA datasets with longer questions typically are harder for the model to answer correctly. We see that our method, like many other sample+aggregate based answering methods generally has higher accuracy than the baseline model [43]. This accuracy boost is less pronounced however for GPT-4. For GPT-3.5-Turbo results we generate confidence scores for 250 questions per dataset (or maximum dataset size if smaller). Due to computational cost we only use 100 questions per dataset when testing on GPT-4-Turbo. We use validation splits for CSQA, TruthQA, and test splits for MedQA and MMLU datasets. | CSQA TruthQA MedQA | 151 329 916 | 0.79 0.54 0.59 | 0.80 0.64 0.68 | 0.84 0.85 0.82 | 0.88 0.91 0.84 | | --- | --- | --- | --- | --- | --- | | MMLU Law | 1233 | 0.46 | 0.56 | 0.64 | 0.67 | | MMLU Physics | 172 | 0.57 | 0.72 | 0.92 | 0.92 | Table 3: Comparing accuracy for default model predictions vs. most confident stability predictions across benchmark datasets. One can observe a clear improvement in accuracy for both GPT-3.5 and GPT-4. ### B.3 Evaluation Details When evaluating confidence methods, it is important to note that performance implicitly depends on the prediction set $\hat{Y}$ . For example, a metric may be well calibrated on correct answers but still be overconfident on incorrect ones, meaning the confidence metric would evaluate as worse on a less accurate prediction set. Therefore, for comparison purposes we use the same set of default LLM predictions (setting Temp=0) for GPT-3.5 and GPT-4 results. In order to break possible ties in confidence when evaluating AURC and AUROC methods, we follow the approach of [36] and add a small amount of gaussian noise ( $\sigma=1e-6$ ) to each confidence score. We repeat this process for $r=10$ times and take the average AURC and AUROC scores. We provided We also follow common practice in previous works by using $M=10$ as the number of bins when calculating ECE [2]. We use OpenAI’s gpt-3.5-turbo-1106 snapshot for GPT-3.5 experiments and gpt-4-1106-preview snapshot for GPT-4. Generating and evaluating confidence scores for each method on one dataset takes on the order of an hour for GPT-3.5-Turbo, and two hours for GPT-4-Turbo using OpenAI’s API. <details> <summary>figures/example_curves.png Details</summary>  ### Visual Description ## Risk Coverage Curve and Receiver Operator Curve Analysis ### Overview The image contains two comparative performance charts: 1. **Risk Coverage Curve** (left): Accuracy vs. Coverage for multiple methods. 2. **Receiver Operator Curve (ROC)** (right): True Positive Rate vs. False Positive Rate for the same methods. Both charts include labeled data series, AUC/AUROC metrics, and baseline/random references. --- ### Components/Axes #### Risk Coverage Curve (Left) - **X-axis**: Coverage (0.0 to 1.0) - **Y-axis**: Accuracy (0.0 to 1.0) - **Legend**: - Linguistic (AUC: 0.9007) – Blue - Tokenprob (AUC: 0.9214) – Orange - Consistency (AUC: 0.9267) – Green - Topk (AUC: 0.9136) – Red - Stability (AUC: 0.9638) – Purple - Baseline Accuracy (0.82) – Dashed black #### Receiver Operator Curve (Right) - **X-axis**: False Positive Rate (0.0 to 1.0) - **Y-axis**: True Positive Rate (0.0 to 1.0) - **Legend**: - Linguistic (AUROC: 0.6704) – Blue - Tokenprob (AUROC: 0.8149) – Orange - Consistency (AUROC: 0.8286) – Green - Topk (AUROC: 0.7101) – Red - Stability (AUROC: 0.9078) – Purple - Random – Dashed black --- ### Detailed Analysis #### Risk Coverage Curve - **Trends**: - All lines start near 1.0 accuracy at low coverage, then decline as coverage increases. - **Stability** (purple) maintains the highest accuracy across coverage levels (AUC: 0.9638). - **Linguistic** (blue) drops sharply, reaching ~0.6 accuracy at 0.2 coverage. - **Baseline** (dashed black) remains flat at 0.82 accuracy. #### Receiver Operator Curve - **Trends**: - Lines start at the bottom-left (0,0) and ascend toward the top-right. - **Stability** (purple) achieves the highest AUROC (0.9078), closest to the top-left corner. - **Random** (dashed black) follows the diagonal (AUROC ~0.5). - **Topk** (red) has the lowest AUROC (0.7101), barely above random. --- ### Key Observations 1. **Stability** dominates both metrics, suggesting robustness in balancing accuracy and coverage. 2. **Linguistic** and **Topk** underperform, with steep declines in accuracy (Risk Coverage) and poor discrimination (ROC). 3. **Baseline Accuracy** (0.82) is outperformed by all methods except Linguistic at high coverage. 4. **AUROC Values**: Stability (0.9078) > Consistency (0.8286) > Tokenprob (0.8149) > Linguistic (0.6704) > Topk (0.7101). --- ### Interpretation - **Performance Hierarchy**: Stability and Consistency methods are most reliable, likely due to their design prioritizing robustness. - **Baseline vs. Models**: The baseline accuracy (0.82) is surpassed by all methods except Linguistic, indicating models generally improve upon simple heuristics. - **ROC Insights**: High AUROC values (e.g., Stability’s 0.9078) suggest strong class separation, while Topk’s low AUROC (0.7101) implies poor discriminative power. - **Trade-offs**: Methods like Linguistic sacrifice accuracy for coverage, while Stability optimizes both. The data highlights Stability as the optimal choice for balancing accuracy, coverage, and discrimination, while Topk and Linguistic require further refinement. </details> Figure 9: Risk coverage (left) and receiver-operator (right) curves for confidence metrics generated on the MedQA questions using GPT-4. Our stability method outperforms others on this dataset as evidenced by larger area under the curves. We can also observe that questions with confidences in the top 50% were all correct. ### B.4 Alternate Explanation Plausibility Measures Inspired by [24], which looks at the true/false token probability an LLM assigns to a given answer being true, we explore evaluating the probability that an explanation is ‘true’. To do this, we provide the model with both question and explanation and ask: ‘Is this the most likely explanation? (T/F)’. We also try asking the question ‘Does the explanation completely describe the question? (T/F)’. We then repeat the experiment in Section ˜ 4.2, examining distributions of explanations measured via these probabilities. We find in figure ˜ 10 that these measures fail to properly distinguish between different explanations. <details> <summary>figures/mmlu_law_explanation_p_likely.png Details</summary>  ### Visual Description ## Line Graphs: MMLU Law P(Most Likely) and P(Complete) ### Overview Two line graphs compare the density distributions of "correct" (blue) and "incorrect" (red) answers for the MMLU Law dataset. The left graph focuses on **P(Most Likely)**, while the right graph focuses on **P(Complete)**. Both graphs show sharp peaks near x=1, indicating high concentrations of answers at extreme probabilities. ### Components/Axes - **Left Graph (P(Most Likely))**: - **X-axis**: P(Most Likely) (probability of the most likely answer), ranging from 0.0 to 1.0. - **Y-axis**: Density (scale from 0 to 15). - **Legend**: Blue = correct, Red = incorrect. - **Right Graph (P(Complete))**: - **X-axis**: P(Complete) (probability of the complete answer), ranging from 0.0 to 1.0. - **Y-axis**: Density (scale from 0 to 15). - **Legend**: Blue = correct, Red = incorrect. ### Detailed Analysis #### Left Graph (P(Most Likely)) - **Correct Answers (Blue)**: - Density rises sharply near x=1, peaking at ~15. - Dominates the distribution, with negligible density below x=0.9. - **Incorrect Answers (Red)**: - Peaks slightly lower (~14) near x=1. - Minimal density below x=0.95. - **Trend**: Both series converge near x=1, but correct answers have a marginally higher density. #### Right Graph (P(Complete)) - **Correct Answers (Blue)**: - Peaks at ~13 near x=1, with a gradual rise starting at x=0.8. - **Incorrect Answers (Red)**: - Peaks higher (~14) near x=1, with a sharper rise starting at x=0.85. - **Trend**: Incorrect answers dominate at high P(Complete), though both series cluster near x=1. ### Key Observations 1. **Confidence vs. Completeness**: - At high **P(Most Likely)** (left graph), correct answers are slightly more frequent. - At high **P(Complete)** (right graph), incorrect answers are marginally more frequent. 2. **Peak Proximity**: - Both correct and incorrect answers cluster extremely close to x=1 in both graphs, suggesting most answers are either highly confident or highly complete. 3. **Density Disparity**: - The left graph shows a clearer distinction between correct and incorrect answers, while the right graph reveals a closer competition between the two. ### Interpretation The data suggests that **confidence (P(Most Likely))** correlates more strongly with correctness than **completeness (P(Complete))**. When the model is highly confident, it is more likely to be correct, but when answers are highly complete, the likelihood of correctness decreases slightly. This could indicate a trade-off: overly complete answers might introduce errors, while high confidence without completeness might reflect overfitting. The sharp peaks near x=1 imply that most answers are either strongly correct or strongly incorrect, with little ambiguity in the middle range. </details> Figure 10: Empirical distribution of MMLU explanations when measured via GPT-3.5 probability of being ‘most-likely explanation’ (left) and probability of ‘completely describing’ the question (right). One can see that true (blue) and false (red) answer-conditioned explanations are difficult to distinguish. ### B.5 Sensistivity to Explanation Prompting Our stable explanation method reweights explanations based on entailment probability, but if the quality of sampled explanations is poor to begin with our resulting distribution will still be inaccurate. Here we will discuss the effect of instructing the LLM to generate explanations before or after an answer (i.e. the order of ‘explanation’ and ‘answer’ in the stability explanation prompt in Figure ˜ 3). We observe in Table ˜ 4 that generating explanations before the answer clearly results in higher quality explanations, as evidenced by improved performance on selective uncertainty and calibration tasks. | | Pre-Answer Stability (Default) AURC $\uparrow$ | AUROC $\uparrow$ | Post-Answer Stability ECE $\downarrow$ | AURC $\uparrow$ | AUROC $\uparrow$ | ECE $\downarrow$ | | --- | --- | --- | --- | --- | --- | --- | | CSQA | 0.901 | 0.779 | 0.123 | 0.866 | 0.731 | 0.201 | | TruthQA | 0.801 | 0.853 | 0.21 | 0.792 | 0.839 | 0.254 | | MedQA | 0.784 | 0.798 | 0.169 | 0.743 | 0.743 | 0.251 | | MMLU Law | 0.642 | 0.736 | 0.259 | 0.629 | 0.706 | 0.289 | | MMLU Physics | 0.792 | 0.834 | 0.186 | 0.779 | 0.811 | 0.252 | Table 4: Comparing stability confidence performance using explanations generated before and after an answer for GPT-3.5. One can clearly observe that explanations generated before the answer (i.e. in chain-of-thought fashion) outperform those generated afterwards across all performance metrics. ### B.6 Varying Sample Size In this section we briefly analyze the effect that the number of sampled explanation has on our confidence metric. In Figure ˜ 11 we observe that selective uncertainty performance (AURC and AUROC) saturates quickly for simpler questions answering tasks such as commonsenseqa. On the other hand MedQA and MMLU Law datasets both demonstrate steady performance gains up to $M=5$ samples per question. Calibration error gradually decreases for all datasets examined. <details> <summary>figures/aurc_vs_explanations.png Details</summary>  ### Visual Description ## Line Chart: AURC vs. Number Explanations ### Overview The chart compares the performance of five systems (CSQA, TruthQA, MedQA, MMLU Law, MMLU Physics) across different numbers of explanations (1, 3, 5) using the metric AURC (Area Under the ROC Curve). The y-axis ranges from 0.5 to 1, and the x-axis spans 0 to 6 explanations. Data points are plotted at x=1, 3, and 5. ### Components/Axes - **X-axis**: "Number Explanations" (0–6, integer increments). - **Y-axis**: "AURC" (0.5–1, 0.05 increments). - **Legend**: Located at the bottom, mapping colors to systems: - Blue: CSQA - Orange: TruthQA - Gray: MedQA - Yellow: MMLU Law - Light Blue: MMLU Physics ### Detailed Analysis 1. **CSQA (Blue)**: - Values: ~0.9 at x=1, 3, and 5. - Trend: Flat line, indicating consistent performance across all explanation counts. 2. **TruthQA (Orange)**: - Values: ~0.78 at x=1, 0.78 at x=3, ~0.8 at x=5. - Trend: Slight upward slope, improving marginally with more explanations. 3. **MedQA (Gray)**: - Values: ~0.72 at x=1, ~0.75 at x=3, ~0.78 at x=5. - Trend: Steady upward trajectory, showing the most improvement with explanations. 4. **MMLU Law (Yellow)**: - Values: ~0.6 at x=1, ~0.65 at x=3, ~0.67 at x=5. - Trend: Gradual increase, but remains the lowest-performing system. 5. **MMLU Physics (Light Blue)**: - Values: ~0.75 at x=1, ~0.8 at x=3, ~0.79 at x=5. - Trend: Peaks at x=3, then declines slightly. ### Key Observations - **CSQA** maintains the highest AURC (~0.9) across all explanation counts, suggesting robustness. - **MMLU Law** consistently underperforms, with the lowest AURC (~0.6–0.67). - **MMLU Physics** shows a non-linear trend, peaking at x=3 before declining. - **MedQA** demonstrates the most significant improvement with explanations (+0.06 from x=1 to x=5). ### Interpretation The data suggests that **CSQA** is the most reliable system, maintaining high performance regardless of explanation count. **MedQA** benefits most from additional explanations, indicating potential for optimization. **MMLU Physics**' peak at x=3 implies an optimal explanation threshold, while its decline at x=5 may reflect overfitting or noise. **MMLU Law**'s consistent low performance highlights systemic limitations. The trends emphasize that explanation count does not universally improve AURC, with system-specific dynamics at play. </details> (a) AURC vs. Number of Explanations for the stable explanations confidence metric. <details> <summary>figures/auroc_vs_explanations.png Details</summary>  ### Visual Description ## Line Chart: AUROC vs. Number Explanations ### Overview The chart compares the performance of five models (CSQA, TruthQA, MedQA, MMLU Law, MMLU Physics) in terms of Area Under the Receiver Operating Characteristic Curve (AUROC) across different numbers of explanations (1, 3, 5). AUROC values range from 0.6 to 1, with higher values indicating better performance. The x-axis represents the number of explanations, and the y-axis represents AUROC scores. ### Components/Axes - **X-axis**: "Number Explanations" (0–6, with data points at 1, 3, 5). - **Y-axis**: "AUROC" (0.6–1, in increments of 0.05). - **Legend**: Located at the bottom, mapping colors to models: - Blue: CSQA - Orange: TruthQA - Gray: MedQA - Yellow: MMLU Law - Light Blue: MMLU Physics ### Detailed Analysis 1. **CSQA (Blue)**: - At 1 explanation: ~0.76 AUROC. - At 3 explanations: ~0.77 AUROC. - At 5 explanations: ~0.78 AUROC. - **Trend**: Slight upward slope, then plateau. 2. **TruthQA (Orange)**: - Constant at ~0.83 AUROC across all explanations. - **Trend**: Flat line, no variation. 3. **MedQA (Gray)**: - At 1 explanation: ~0.76 AUROC. - At 3 explanations: ~0.78 AUROC. - At 5 explanations: ~0.80 AUROC. - **Trend**: Gradual upward slope. 4. **MMLU Law (Yellow)**: - At 1 explanation: ~0.70 AUROC. - At 3 explanations: ~0.75 AUROC. - At 5 explanations: ~0.79 AUROC. - **Trend**: Steep upward slope, then slight plateau. 5. **MMLU Physics (Light Blue)**: - At 1 explanation: ~0.75 AUROC. - At 3 explanations: ~0.80 AUROC. - At 5 explanations: ~0.79 AUROC. - **Trend**: Sharp initial increase, then slight decline. ### Key Observations - **TruthQA** consistently achieves the highest AUROC (~0.83), outperforming all other models. - **MMLU Law** shows the most significant improvement with more explanations, rising from 0.70 to 0.79. - **MMLU Physics** peaks at 3 explanations (~0.80) but slightly declines at 5 explanations. - **CSQA** and **MedQA** exhibit modest gains, with MedQA showing the most consistent improvement. ### Interpretation The data suggests that increasing the number of explanations generally improves model performance (AUROC), but with diminishing returns for some models. **TruthQA** remains the most robust, maintaining a high AUROC regardless of explanation count. **MMLU Law** benefits disproportionately from additional explanations, indicating it may rely on contextual reasoning that improves with more input. The slight decline in **MMLU Physics** at 5 explanations could signal overfitting or noise in the data. Overall, the chart highlights the trade-off between explanation quantity and model efficacy, with TruthQA as the outlier in stability and performance. </details> (b) AUROC vs. Number of Explanations for the stable explanations confidence metric. <details> <summary>figures/ece_vs_explanations.png Details</summary>  ### Visual Description ## Line Chart: ECE vs. Number Explanations ### Overview The chart visualizes the relationship between Explanation Consistency Error (ECE) and the number of explanations provided across five evaluation frameworks. ECE decreases as the number of explanations increases, with varying rates of decline across frameworks. ### Components/Axes - **X-axis**: "Number Explanations" (0–6, integer increments) - **Y-axis**: "ECE" (0–0.4, 0.05 increments) - **Legend**: Located at the bottom, mapping colors to frameworks: - Blue: CSQA - Orange: TruthQA - Gray: MedQA - Yellow: MMLU Law - Cyan: MMLU Physics ### Detailed Analysis 1. **CSQA (Blue Line)**: - Starts at ~0.19 ECE at 1 explanation. - Declines steadily to ~0.12 ECE at 5 explanations. - Slope: Steepest decline among all frameworks. 2. **TruthQA (Orange Line)**: - Begins at ~0.28 ECE at 1 explanation. - Drops to ~0.21 ECE at 5 explanations. - Slope: Moderate decline, less steep than CSQA. 3. **MedQA (Gray Line)**: - Starts at ~0.24 ECE at 1 explanation. - Decreases to ~0.17 ECE at 5 explanations. - Slope: Gradual decline, intermediate between CSQA and TruthQA. 4. **MMLU Law (Yellow Line)**: - Begins at ~0.30 ECE at 1 explanation. - Reduces to ~0.21 ECE at 5 explanations. - Slope: Gradual decline, similar to TruthQA but higher baseline. 5. **MMLU Physics (Cyan Line)**: - Starts at ~0.24 ECE at 1 explanation. - Drops to ~0.18 ECE at 5 explanations. - Slope: Moderate decline, comparable to MedQA. ### Key Observations - **Universal Trend**: All frameworks show decreasing ECE with more explanations, indicating improved performance. - **CSQA Dominance**: CSQA achieves the lowest ECE at all explanation counts, suggesting superior sensitivity to explanation quality. - **Convergence**: Lines converge toward lower ECE values as explanations increase, implying diminishing returns after ~5 explanations. - **MMLU Law Anomaly**: Maintains the highest ECE baseline but shows significant improvement, closing the gap with other frameworks by 5 explanations. ### Interpretation The data demonstrates that increasing the number of explanations generally enhances explanation consistency across frameworks. CSQA’s steep decline highlights its effectiveness in leveraging additional explanations, while MMLU Law’s higher baseline suggests inherent challenges in legal text evaluation. The convergence pattern implies that beyond ~5 explanations, further gains in ECE become marginal. This trend underscores the importance of balancing explanation quantity with quality, as excessive explanations may not proportionally improve consistency metrics. </details> (c) ECE vs. Number of Explanations for the stable explanations confidence metric. Figure 11: Comparison of stable explanation confidence using different numbers of explanations per question ( $M=\{1,3,5\}$ ). Testing is done on GPT-3.5-Turbo for all five benchmark datasets. One can observe improving but saturating performance for each dataset. ### B.7 Comparison to TTA Contemporaneously to this manuscript’s submission, another method related to our approach was proposed [28]. The Think-Twice before assure (TTA) method asks for explanations conditioned on different answers, then does a top-k confidence elicitation using these explanations in the prompt. Although similar in the sense that confidence metrics are being generated by conditioning on explanations, their combination of explanations into a single prompt does not match the ensemble of test-time classifiers view that our method takes. The authors have not yet released code or dataset splits, but we have implemented their method by following the written procedure and using the same prompts (see Figure ˜ 12). We found during our implementation on the shared CSQA dataset, the evaluation results for selective uncertainty tasks are slightly below that what the authors report (AURC, AUROC), most likely due to the difference in specific questions used during testing. Nonetheless we report the full results of our implementation in table ˜ 5, and note that this metric does appear to have lower ECE in many cases. | CSQA | TTA (Our Implementation) AURC $\uparrow$ 0.885 | AUROC $\uparrow$ 0.688 | ECE $\downarrow$ 0.104* | Acc. $\uparrow$ 0.736 | | --- | --- | --- | --- | --- | | TruthQA | 0.698 | 0.706 | 0.093* | 0.672* | | MedQA | 0.641 | 0.581 | 0.207 | 0.505 | | MMLU Law | 0.574 | 0.657 | 0.148* | 0.456 | | MMLU Physics | 0.717 | 0.697 | 0.1* | 0.557 | Table 5: Evaluation for the TTA Confidence metric (Our implementation) on GPT-3.5. Results that outperform our stable explanations metric are marked with an asterisk. <details> <summary>figures/tta_prompt.png Details</summary>  ### Visual Description ## Screenshot: TTA Prompt Template for Multiple-Choice Questions ### Overview The image displays a structured template for generating explanations and confidence assessments for multiple-choice questions. It includes placeholders for question/answer content and instructions for generating justifications. ### Components/Axes - **Headings**: - `TTA Explanation Prompt:` - `TTA Confidence Prompt:` - **Placeholders**: - `[multiple choice question]` - `[answer text]` - `[Top-K Prompt]` - `[explanation 1]` to `[explanation N]` - **Instructions**: - "The task is to read the given question and select the most appropriate answer by indicating the associated letter." - "Please generate an explanation to try to justify the answer judgement." ### Detailed Analysis - **TTA Explanation Prompt Section**: - Contains a question placeholder (`[multiple choice question]`) and an answer placeholder (`[answer text]`). - Directs the user to generate an explanation justifying the answer. - **TTA Confidence Prompt Section**: - Includes a `[Top-K Prompt]` placeholder. - Lists `N` possible explanations (`[explanation 1]` to `[explanation N]`), suggesting iterative refinement or confidence scoring. ### Key Observations - Placeholders are marked in red brackets, indicating editable fields. - The template emphasizes **justification** (`"justify the answer judgement"`) and **confidence assessment** (`"Top-K Prompt"`). - No numerical data or visual trends are present; the focus is on textual structure and workflow. ### Interpretation This template is designed for **iterative reasoning** in question-answering systems. The `TTA Explanation Prompt` ensures answers are supported by logical reasoning, while the `TTA Confidence Prompt` allows for evaluating multiple explanations to gauge confidence (e.g., via Top-K sampling). The absence of numerical data suggests it is a **procedural guide** rather than a data visualization. The use of `[explanation N]` implies scalability for handling varying levels of confidence or alternative reasoning paths. </details> Figure 12: TTA Confidence Prompt ## Appendix C Alternative Perspectives of Stable Explanations ### C.1 Confidence through the Viewpoint of Transductive Inference Transductive learning selects a classifier at inference-time based on a combination of training and test data [10, 42, 22]. Typically transductive learning involves fine-tuning some classifier parameter $w$ based on an explicit loss objective. However, we claim that using an LLM to generate a sequence of text before an answer (i.e. an explanation) is an alternate way of doing transductive reasoning. First, note that answering a question after an explanation, such as in chain-of-thought prompting [44], effectively changes the decision boundary of the LLM classifier at inference time. Second, consider that when an LLM generates an explanation, it produces concepts related to those in the question. These additional concepts can be thought of as forcing the LLM at inference time to pay more attention to the decision boundary in the area around the test datum. In-context learning literature, which examines LLM performance after manually inserting demonstrations similar to the test question, has already shown a direct connection between transformer context adjustment and classical fine-tuning behavior [9]. To formalize this perspective, let $D^{t}=\{(x_{1},y_{1}),\ldots,(x_{t},y_{t})\}$ be a dataset of sequential data up to time $t$ , with $x_{i}\in X\subset{\mathbb{R}}^{M}$ and labels $y_{i}\in Y\subset\{1,\dots,K\}$ . We denote with $D^{t}_{-}=\{(x_{1},y_{1}),\dots,(x_{t-1},y_{t-1}),x_{t}\}$ the dataset without the last label $y_{t}$ . We can write our transductive prediction for $x_{t}$ given data $D^{t}_{-}$ including $x_{t}$ as: $$ \displaystyle\hat{y}_{t}=~\underset{w,y}{\rm argmin}~{}\underbrace{\frac{1}{t}\ell(f_{w}(x_{t}),y)+\frac{1}{t}\sum_{i=1}^{t-1}\ell(f_{w}(x_{i}),y_{i})}_{\doteq L(w;(D^{t}_{-},y))}. \tag{17} $$ If $\ell$ is interpreted as a log likelihood, then $L$ can be interpreted as the negative log posterior probability over hypotheses. If we think of optimizing instead over explanations where $f_{e}(x_{t})=\phi(x_{t},e)$ , then the problem reduces to finding an explanation that strongly supports a single answer without biasing predictions on original test data. The second term in equation ˜ 17 is expensive to compute at inference time, but if some approximation of this training loss existed it would make optimization tractable. We hypothesize that if the explanation under consideration is plausible and faithful to the question (as determined using the same LLM), it should not reduce the accuracy of previous decisions too much. Therefore we can avoid having to optimize over all previous questions and instead optimize over whatever faithfulness measure $g_{\phi}(e)$ we define: $$ \displaystyle\hat{y}_{t}=~\underset{e,y}{\rm argmin}~{}\ell(\phi(x_{t},e),y)+\lambda g_{\phi}(e) \tag{18} $$ This looks exactly like the typical transductive setting but with a more easily computable ‘transductive prior’. ### C.2 Confidence throught the Viewpoint of Solomonoff Induction While transductive inference typically finds single test-time classifier, our method looks for a distribution of likely classifiers. In this sense, our method can be seen as a special case of Solomonoff induction [25]. Solomonoff induction considers how well data-generating programs, $H$ , (i.e. a binary string run on a Turing machine) explain the test data, $D$ $$ \displaystyle P(H|D)=\frac{P(D|H)P(H)}{P(D|H)P(H)+\sum_{A\neq H}P(D|A)P(A)}, \tag{19} $$ where $A$ are alternative programs. Solomonoff induction formalizes the principle of Occam’s razor by choosing a universal prior $P(H)$ that gives a higher probability to shorter-length programs. Then to predict new data $D^{\prime}$ given previous observations, one simply computes $$ \displaystyle P(D^{\prime}|D)=\mathbb{E}_{H}[P(D^{\prime}|H,D)]=\sum_{H}P(D^{\prime}|H,D)P(H|D). \tag{20} $$ While these Bayesian equations seem simple, Solomonoff’s induction is provably uncomputable. However, our method can be interpreted as restricting our hypothesis class from the set of all computable programs $H$ to the set of all LLM-interpretable programs $e$ . Instead of a prior on program length, we can use the LLM’s prior likelihood of valid sequences in the language $p_{\phi}(e)$ . This restriction makes our calculations more tractable, as we can easily approximate expectations over our hypothesis class by sampling explanations from the LLM. ### C.3 Confidence through the Viewpoint of Stability Another recent line of work has been analyzing LLMs through the lens of stochastic dynamical models [37]. Through the perspective of stability analysis one could interpret our method’s preference for explanations convening to a single answer as searching for fixed points of a specific LLM system. This LLM dynamical system consists of two alternating steps, first generating an explanation conditioned on one of the answers ( $e\leftarrow\phi(q,a)$ ) then generating a new answer based on this explanation ( $a^{\prime}\leftarrow\phi(q,e)$ ). Intuitively this system mirrors how a human expert may think about a question by considering alternative conclusions one could draw given beliefs about the world. An answer with only a single plausible explanation that strongly supports that same answer (i.e. decision distribution collapses to a singleton) forms a stable cycle in this system.