2411.13343

# Fact-Level Confidence Calibration and Self-Correction ## Abstract Confidence calibration in LLMs, i.e., aligning their self-assessed confidence with the actual accuracy of their responses, enabling them to self-evaluate the correctness of their outputs. However, current calibration methods for LLMs typically estimate two scalars to represent overall response confidence and correctness, which is inadequate for long-form generation where the response includes multiple atomic facts and may be partially confident and correct. These methods also overlook the relevance of each fact to the query. To address these challenges, we propose a fact-level calibration framework that operates at a finer granularity, calibrating confidence to relevance-weighted correctness at the fact level. Furthermore, comprehensive analysis under the framework inspired the development of confidence-guided fact-level self-correction (ConFix), which uses high-confidence facts within a response as additional knowledge to improve low-confidence ones. Extensive experiments across four datasets and six models demonstrate that ConFix effectively mitigates hallucinations without requiring external knowledge sources such as retrieval systems Code is available at https://github.com/yuanyige/fact-calibration. Fact-Level Confidence Calibration and Self-Correction Yige Yuan 1,2, Bingbing Xu 1, Hexiang Tan 1,2, Fei Sun 1, Teng Xiao 3, Wei Li 1, Huawei Shen 1,2, Xueqi Cheng 1,2 1 Institute of Computing Technology, Chinese Academy of Sciences 2 University of Chinese Academy of Sciences, 3 Penn State University ## 1 Introduction Large Language Models (LLMs) have recently shown remarkable capabilities in understanding and generating language that closely resembles human communication Brown et al. (2020); OpenAI (2023). Nonetheless, a major obstacle to their reliability is the prevalence of hallucinations Lin et al. (2021); Zhang et al. (2023); Li et al. (2023a); Golovneva et al. (2022); Bang et al. (2023), a phenomenon where the models generate incorrect and unreliable outputs. This issue not only undermines user trust but also restricts the application of LLMs in domains where reliability is crucial, such as in the legal, financial, and educational fields. Echoing the ancient adage that “To know what you know and what you do not know, that is true wisdom”, confidence calibration in LLMs emerges as an effective method to mitigate hallucinations Li et al. (2024); Liu et al. (2023); Huang et al. (2024). By confidence calibrating, models can better align their self-assessed confidence with the actual accuracy of responses, empowering them to self-evaluate the correctness of outputs. This mechanism offers an effective way to identify hallucinations by using the model’s confidence as a basis to either trust or question the model’s response. <details> <summary>x1.png Details</summary>  ### Visual Description ## Diagram: Confidence-Based Response Analysis for "Deep Blue" Query ### Overview The diagram illustrates a conversational interaction between a user and a chatbot discussing "Deep Blue," a chess-playing computer developed by IBM. It visualizes confidence scores for different components of the chatbot's response, using color-coding (red for high confidence, blue for low confidence) and directional arrows to indicate confidence enhancement pathways. ### Components/Axes 1. **User Input**: - Text: "What is the Deep Blue?" - Position: Top-left corner. 2. **Chatbot Response**: - Text: "Deep Blue is a go computer developed by IBM that famously defeated world champion Garry Kasparov in 1997." - Confidence Scores: - "developed by IBM": 0.30 (blue) - "famously defeated world champion Garry Kasparov": 0.90 (red) - "in 1997": 0.85 (red) - "a go computer": 0.80 (red) - Position: Center-left. 3. **Response Breakdown**: - Four segmented statements with confidence scores: - "Deep Blue is developed by IBM." (confidence: 0.90, red) - "Deep Blue defeated world champion Garry Kasparov." (confidence: 0.85, red) - "The defeat is in 1997." (confidence: 0.80, red) - "Deep Blue is a go computer." (confidence: 0.30, blue) - Position: Center-right. 4. **Confidence Enhancement Arrows**: - Red arrow labeled "enhance" pointing from "Low Confidence" (blue) to "High Confidence" (red). - Position: Bottom-center. 5. **Legend**: - Red: High Confidence (0.80–0.90) - Blue: Low Confidence (0.30) - Position: Bottom-left. 6. **Chatbot Icon**: - Visual representation of a chatbot (speech bubble with robot face). - Position: Right of the chatbot response. ### Detailed Analysis - **Confidence Distribution**: - High-confidence statements (red) dominate the response, covering core facts (e.g., "developed by IBM," "defeated Garry Kasparov"). - Low-confidence statements (blue) include ambiguous phrasing ("a go computer") and temporal specificity ("in 1997"). - **Color Consistency**: - Red consistently marks statements with confidence ≥0.80. - Blue marks the lowest-confidence statement ("a go computer," 0.30). - **Flow**: - The user's question triggers the chatbot's response, which is then decomposed into verifiable components. - Arrows suggest that improving low-confidence elements (e.g., correcting "go computer" to "chess computer") could enhance overall accuracy. ### Key Observations 1. **Ambiguity in "Go Computer"**: - The chatbot initially mislabels Deep Blue as a "go computer" (confidence: 0.30), likely due to confusion with IBM's later "AlphaGo" system. 2. **Temporal Specificity**: - The year "1997" has moderate confidence (0.80), reflecting potential uncertainty in precise event dating. 3. **Confidence Enhancement Pathway**: - The diagram implies that refining low-confidence elements (e.g., correcting "go" to "chess") would strengthen the response's reliability. ### Interpretation The diagram highlights the interplay between factual accuracy and model confidence in natural language processing. The chatbot's initial error ("go computer") underscores challenges in disambiguating homonymous terms (e.g., "Deep Blue" vs. "AlphaGo"). The confidence scores and color-coding provide a visual framework for identifying knowledge gaps, while the enhancement arrows suggest iterative improvement mechanisms. This analysis emphasizes the importance of contextual grounding (e.g., linking "Deep Blue" to chess history) to resolve ambiguities and improve response reliability. </details> Figure 1: Motivation of our fact-level confidence calibration and confidence-guided self-correction. However, current confidence calibration methods for LLMs Guo et al. (2017a); Nguyen and O’Connor (2015) typically estimate two scalars to represent the overall confidence and correctness for the entire response. This approach is unreasonable for long-form generation, where responses may contain multiple atomic facts (illustrated in Fig. 1). In such cases, considering the varying of facts in one response, the confidence and correctness should also be diverse, capable of reflecting higher certainty in some facts and greater uncertainty in others. Furthermore, within long-form responses Xiao et al. (2024b, a), certain facts exist that may indeed be correct but lack relevance to the query. Previous calibration methods predominantly Huang et al. (2024); Liu et al. (2023) focus on correctness while neglecting the relevance. To address these challenges, we propose a novel framework for confidence calibration operating at a finer fact-level granularity. Within this framework, the confidence assessment of each fact incorporates two key aspects: correctness and relevance. Correctness indicates the factual accuracy of the fact, while relevance measures the extent to which the fact is related to the query. Calibration of a response is defined as the degree of alignment between confidence and correctness weighted by relevance across all facts. This framework endows the model with the capability to exhibit partial confidence and correctness in individual facts. Extensive analysis based on the aforementioned framework yields three interesting findings: (1) Calibration on fact-level imposes a stricter standard than calibration on response-level. (2) Overconfidence issue stems from implicit facts. (3) High variance exists in fact-level confidence within a response. The aforementioned three observations inspire the development of Confidence-Guided Fact-Level Self-Correction (ConFix) to enhance the generation and mitigate hallucinations (illustrated in Fig. 1). For a response, ConFix first leverages the aforementioned framework to segment the response into multiple facts and evaluate their confidence vector. It then uses the high-confidence facts and their associated confidence score as additional knowledge to augment low-confidence facts, with the aim of all facts within the response achieving high confidence. ConFix can self-enhance to mitigate hallucinations without the need for external knowledge sources such as retrieval systems. Experiments with ConFix across four datasets and six models reveal that it can significantly reduce the occurrence of hallucinations, thereby increasing the models’ reliability and enabling their real-world applications. In summary, our main contributions are: Fact-Level Calibration Framework: The proposed fact-level calibration framework operates at a finer level of granularity to align the confidence with the correctness weighted by relevance across all facts. This framework enables LLMs to exhibit partial confidence and accuracy in individual facts. Insightful Observations: We uncover insightful observations regarding the model’s scale and its calibration capability on fact- and response-levels. Self-Correction Method: We propose ConFix based on the fact-level calibration framework to enhance the generation and reduce hallucinations without relying on external knowledge sources. ## 2 Related Work and Formulation #### Related Work Several prior works, such as SelfCheckGPT Manakul et al. (2023); Wang et al. (2024); Zhang et al. (2024) and so on Mündler et al. (2023), focus on uncertainty-based hallucination detection. While these methods share a common goal with ours, there are several key distinctions: (1) Their focus is primarily on detection, whereas our work goes beyond that by investigating the mechanisms of fine-grained confidence estimation and identifying three novel phenomena. (2) We introduce a self-correction method that functions without relying on external knowledge, in contrast to their approaches, which are limited to detection. (3) While their methods operate at the sentence level, we take a more granular approach by breaking down text into atomic facts, enabling a deeper analysis of confidence at the fact level. (4) Furthermore, we propose a general framework for confidence estimation, with their methods representing specific instances within this broader approach. There are also works like FactScore Min et al. (2023) that focus on fact-level factuality evaluation by leveraging external knowledge. In contrast, our method centers on the model’s internal confidence and calibration, providing a self-correction solution that operates without the external knowledge. #### Problem Formulation Let the dataset be $D=\{x_1,x_2,…,x_N\}$ , where $x_i$ represents the $i$ -th query, and the model’s responses are $A=\{y_1,y_2,…,y_N\}$ . Each query-answer pair $(x_i,y_i)$ is associated with a confidence score $conf_i$ , indicating the model’s certainty to the answer, and a correctness score $corr_i$ , measuring the objective accuracy of the answer. In long-form generation, we extend this to the fact level, where each response $y_i$ contains $M_i$ individual facts $\{f_i^j\}_j=1^M_i$ , with each fact $f_i^j$ assigned a relevance score $rel_i^j$ , a correctness score $corr_i^j$ , and a confidence score $conf_i^j$ . The aim is to align confidence with relevance-weighted correctness across all facts in each response. <details> <summary>x2.png Details</summary>  ### Visual Description ## Process Diagram: Deep Blue Evaluation Framework ### Overview The diagram illustrates a multi-stage evaluation framework for assessing the accuracy and relevance of factual claims about Deep Blue, a chess-playing computer developed by IBM. It includes fact extraction, confidence estimation, external evaluation, and a mathematical metric for overall performance. ### Components/Axes 1. **Fact Extraction** - Lists four factual claims about Deep Blue, each color-coded: - **Blue**: "Deep Blue is a go computer." (Confidence: 0.30) - **Pink**: "Deep Blue is developed by IBM." (Confidence: 0.90) - **Yellow**: "Deep Blue defeated world champion Garry Kasparov." (Confidence: 0.85) - **Green**: "The defeat is in 1997." (Confidence: 0.80) 2. **Confidence Estimation** - Subdivided into **Correctness** and **Relevance** scores for each fact: - **Correctness**: - Blue: 0.00 (incorrect) - Pink: 1.00 (correct) - Yellow: 1.00 (correct) - Green: 1.00 (correct) - **Relevance**: - Blue: 1.00 (irrelevant) - Pink: 1.00 (relevant) - Yellow: 1.00 (relevant) - Green: 0.70 (partially relevant) 3. **External Evaluation** - Symbols: A globe (global context) and a puzzle piece (external data integration), connected by a "+" sign. 4. **Evaluation Metric** - Formula: $$ \text{F-ECE} = \sum_{n=1}^N \frac{|B_k|}{N} \left| \text{corr}_{\text{avg}} - \text{conf}_{\text{avg}}^k \right| $$ - Aggregates confidence, correctness, and relevance scores into a single metric. ### Detailed Analysis - **Fact Extraction**: - The first claim ("go computer") is factually incorrect (correctness = 0.00) but marked as relevant (relevance = 1.00). - The remaining claims are correct (correctness = 1.00) and relevant, except the year claim (relevance = 0.70). - **Confidence Estimation**: - Confidence scores (0.30–0.90) reflect uncertainty in fact validity. Higher confidence correlates with higher correctness (e.g., IBM development: 0.90 confidence, 1.00 correctness). - **External Evaluation**: - The globe and puzzle piece suggest incorporating external validation (e.g., historical records, domain expertise). ### Key Observations 1. **Mismatch in Relevance**: The "go computer" claim is irrelevant despite high confidence. 2. **Year Relevance**: The 1997 defeat claim has lower relevance (0.70), possibly due to contextual ambiguity. 3. **Formula Complexity**: The F-ECE metric balances confidence gaps between correctness and relevance, penalizing overconfident incorrect claims. ### Interpretation The framework prioritizes factual accuracy while accounting for contextual relevance. The "go computer" claim highlights the risk of overconfidence in incorrect facts, while the year claim’s partial relevance suggests nuanced historical context. The F-ECE metric quantifies this balance, emphasizing the need for external validation (symbolized by the globe and puzzle piece) to refine evaluations. This approach ensures robustness in AI systems handling ambiguous or domain-specific queries. </details> Figure 2: An illustration of our fact-level confidence calibration framework for fine-grained LLM calibration. ## 3 Fact-Level Confidence Calibration In this section, we first outline the motivation and provide a overview of the Fact-Level Confidence Calibration framework. Following this, we discuss three key observations arising from our framework that offer deeper insights. Finally, we conclude by summarizing how these observations inform and shape our approach to conduct self-correction. ### 3.1 Motivation Compared to the confidence calibration for short-form generation or traditional classification problems, a significant challenge in calibrating long-form text generation is that a response may contain multiple facts, making it unreasonable to assign a single correctness measure and a single confidence score to the entire response. The reason is that the answer might be partially correct, and the model might also be partially confident in only a subset of the facts of a response. Meanwhile, some facts in response are irrelevant to the query, so the calibration based solely on correctness is insufficient. Based on the above motivation, our proposed calibration framework aims to calibrate the confidence to relevance-weighted correctness on the fact level, which leads to the following two advantages: (1) Finer Granularity: we assign a confidence vector rather than a scalar to a response, where each item represents confidence for a single fact. This fine-grained framework allows for more nuanced and precise calibration. (2) Relevance Awareness: we assess both the correctness and the relevance of each fact, which ensures that the confidence score attributed to each fact can reflect its significance and appropriateness within the given context. Table 1: Comparison of confidence estimation and calibration performance across five base models for three confidence estimation methods (Verbalization, isTrueLogit, Consistency) on the ASQA and LongFact datasets. Dataset ( $→$ ) ASQA LongFact Metric ( $→$ ) $/$ Model ( $↓$ ) Confidence Calibration Confidence Calibration Method ( $↓$ ) $/$ Level ( $→$ ) Fact Response Fact Response Fact Response Fact Response Verbalization Llama-2-7b 0.6946 0.7110 0.2605 0.2508 0.6753 0.7842 0.2109 0.1415 Llama-2-13b 0.7629 0.8289 0.2404 0.2606 0.7510 0.8418 0.1556 0.1310 Vicuna-7b 0.8054 0.8425 0.3365 0.3524 0.7567 0.8591 0.1505 0.1369 Vicuna-13b 0.8661 0.9339 0.2540 0.2688 0.8838 0.9299 0.1134 0.1094 GPT-3.5-turbo 0.9620 0.9673 0.1789 0.1855 0.9659 0.9554 0.0856 0.0935 Comparison 0.8182 0.8567 0.0040 0.0043 0.8065 0.8741 0.0025 0.0006 isTrueLogit Llama-2-7b 0.6639 0.8088 0.2972 0.3748 0.5081 0.8624 0.3525 0.1998 Llama-2-13b 0.9741 0.9483 0.3723 0.3764 0.8693 0.9696 0.1635 0.1932 Vicuna-7b 0.5448 0.5672 0.2867 0.2806 0.5607 0.6186 0.2998 0.2539 Vicuna-13b 0.8116 0.6342 0.2486 0.2837 0.8648 0.7947 0.1240 0.2213 GPT-3.5-turbo - - - - - - - - Comparison 0.7486 0.7396 0.0018 0.0000 0.7007 0.8113 0.0167 0.0003 Consistency Llama-2-7b 0.8714 0.8334 0.3224 0.3331 0.9276 0.9760 0.2167 0.1660 Llama-2-13b 0.8903 0.8320 0.3101 0.3326 0.8940 0.8980 0.1385 0.1804 Vicuna-7b 0.8834 0.8227 0.3312 0.3373 0.9397 0.9767 0.1946 0.1477 Vicuna-13b w 0.9250 0.9007 0.2473 0.2500 0.9727 0.9767 0.1020 0.1085 GPT-3.5-turbo 0.8859 0.8393 0.1572 0.1807 0.9613 0.9480 0.0715 0.0957 Comparison 0.8912 0.8456 0.0080 0.0069 0.9391 0.9551 0.0047 0.0014 ### 3.2 Architecture To calibrate confidence with relevance-weighted correctness at fact level, our framework comprises four components, as shown in fig. 2: (1) fact extraction, (2) correctness and relevance evaluation, (3) confidence estimation, (4) calibration evaluation. #### Fact Extraction Given query-answer pair $(x_i,y_i)$ , the first step is to extract the individual facts from response. This extraction can be efficiently carried out using a powerful external language model, such as GPT-4 Brown et al. (2020), producing a set of facts $\{f_i^j\}_j=1^M_i$ for $y_i$ . #### Correctness and Relevance Evaluation Once facts are extracted, we evaluates both the correctness and relevance of each fact in relation to the query. Correctness is assessed by verifying the factual accuracy of each fact using GPT-4 in combination with retrieval methods, such as search engines and ground truth answers from relevant datasets, yielding correctness scores $\{corr_i^j\}_j=1^M_i$ . Similarly, relevance scores $\{rel_i^j\}_j=1^M_i$ can also be measured using GPT-4 to capture each fact’s pertinence to the query within the context of the overall response. #### Confidence Estimation Confidence estimation quantifies the certainty of the LLM for each fact in its response. To derive a confidence vector $\{conf_i^j\}_j=1^M_i$ for each response, we employ three widely used confidence estimation methods: (1) Verbalization-based method Tian et al. (2023): model is prompted to verbally express confidence score for each fact in the response. (2) Logit-based method Kadavath et al. (2022a): confidence is determined by the probability assigned to the “True” token when evaluating the response as True or False. (3) Consistency-based method Manakul et al. (2023): confidence is determined by the consistency across multiple sampled outputs. Formally, the confidence of $f_i^j$ can be represented as: $$ \displaystyleconf_i^j=C≤ft(LLM(·),p_c( f_i^j,x_i,y_i)\right), \tag{1} $$ where $LLM$ is the model to be evaluated, $p_c$ is the prompt, $C$ denotes the method used for estimation. #### Calibration Evaluation We define Fact-Level Expected Calibration Error (F-ECE) as the evaluation metric used to quantify the discrepancy between a model’s confidence and the relevance-weighted correctness across all responses and their corresponding facts. For each fact within a response, we calculate the relevance-weighted correctness as the product of the fact’s correctness score and its relevance score. The response-level relevance-weighted correctness and confidence are then computed by averaging these values across all facts within a response, as detailed in eq. 2. $$ \begin{split}corr_i&=\frac{1}{M_i}∑_j=1^M_icorr _i^j×rel_i^j,\\ conf_i&=\frac{1}{M_i}∑_j=1^M_iconf_i^j,\end {split} \tag{2} $$ F-ECE is then computed by assessing the difference between the average relevance-weighted correctness and the average confidence within each bin $k$ , where $B$ is the number of bins for grouping confidence scores, and $B_k$ is the set of responses in the $k$ -th bin. Let $corr^k_avg=\frac{1}{|B_k|}∑_i∈ B_{k} corr_i$ and $conf^k_avg=\frac{1}{|B_k|}∑_i∈ B_{k} conf_i$ , $$ F-ECE=∑_n=1^N\frac{|B_k|}{N}≤ft|corr^k_ avg-conf^k_avg\right| \tag{3} $$ <details> <summary>x3.png Details</summary>  ### Visual Description ## Bar Chart: Model Performance Across Confidence Levels ### Overview The image contains six grouped bar charts comparing model performance (accuracy and sample counts) across confidence thresholds (0.0–1.0). Models are categorized into "Optimal," "GPT-3.5-turbo," and "Llama-2-13b/7b" across two domains: "Fact-Level" and "Response-Level." Each chart uses color-coded bars to represent model performance metrics. --- ### Components/Axes - **X-Axis (Confidence)**: Labeled "Confidence" with increments at 0.0, 0.2, 0.4, 0.6, 0.8, and 1.0. - **Y-Axes**: - **Left (Accuracy)**: Labeled "Accuracy" with increments from 0.0 to 1.0. - **Right (#Samples)**: Labeled "#Samples" with increments from 0 to 400. - **Legends**: - **Optimal**: Gray bars. - **GPT-3.5-turbo**: Purple bars. - **Llama-2-13b/7b**: Red bars. - **Subplots**: - **Top Row**: Fact-Level performance (GPT-3.5-turbo and Llama-2-13b). - **Middle Row**: Fact-Level performance (Llama-2-13b). - **Bottom Row**: Response-Level performance (Llama-2-7b). --- ### Detailed Analysis #### Fact-Level (GPT-3.5-turbo vs. Llama-2-13b) - **Optimal**: Accuracy increases steadily from ~0.1 at 0.0 confidence to ~0.9 at 1.0 confidence. Sample counts rise from ~50 to ~400. - **GPT-3.5-turbo**: Accuracy peaks at ~0.75 at 0.8 confidence. Sample counts plateau at ~300. - **Llama-2-13b**: Accuracy peaks at ~0.6 at 0.8 confidence. Sample counts peak at ~150. #### Response-Level (Llama-2-7b) - **Optimal**: Accuracy reaches ~0.95 at 1.0 confidence. Sample counts rise to ~400. - **GPT-3.5-turbo**: Accuracy peaks at ~0.85 at 0.8 confidence. Sample counts stabilize at ~350. - **Llama-2-7b**: Accuracy peaks at ~0.7 at 0.8 confidence. Sample counts peak at ~250. --- ### Key Observations 1. **Optimal Models Dominate**: Optimal models consistently achieve the highest accuracy and require the most samples across all confidence levels. 2. **GPT-3.5-turbo vs. Llama**: GPT-3.5-turbo outperforms Llama models in both domains, with larger margins in Response-Level tasks. 3. **Confidence-Accuracy Correlation**: Accuracy improves with confidence for all models, but Optimal models show the steepest improvement. 4. **Sample Efficiency**: Higher confidence predictions correlate with fewer required samples (e.g., Optimal models at 1.0 confidence use ~400 samples vs. ~50 at 0.0 confidence). --- ### Interpretation The data suggests that **Optimal models** are the most reliable, achieving near-perfect accuracy at high confidence while requiring substantial sample sizes. **GPT-3.5-turbo** balances performance and efficiency, outperforming Llama models but lagging behind Optimal. **Llama models** underperform in both domains, particularly in Response-Level tasks, where their accuracy gaps widen. The inverse relationship between confidence and sample counts implies that models become more certain with fewer data points as confidence thresholds increase, though this trend is less pronounced in Llama models. The distinction between Fact-Level and Response-Level performance highlights the complexity of response generation tasks compared to factual recall. </details> Figure 3: Comparison of calibration measures between fact-level and response-level across models of three different scales: LLaMA-7B/13B, and GPT-3.5, indicating fact-level imposes a stricter standard (Observation 1). ## 4 Key Observation This section discusses three important phenomena observed under our fact-level calibration.These findings not only demonstrate the superiority of our framework over traditional response-level calibration, but also inspire the development of a confidence-guided fact-level self-correction method based on these insights. ### 4.1 Experiment Setup This section outlines the experimental setups, including the datasets, models, and evaluations. #### Datasets We utilize two long-form response datasets: (1) LongFact Wei et al. (2024): a dataset contains prompts that evaluate model’s factuality in open-end responses. (2) ASQA Stelmakh et al. (2022): a dataset centers on ambiguous factoid questions with multiple reference answers. #### Models We use five popular models from different families and scales, including: (1) LLaMA Touvron et al. (2023): include LLaMA-7b-chat and LLaMA-13b-chat. (2) Vicuna Chiang et al. (2023): include Vicuna-7b and Vicuna-13b. (3) GPT Brown et al. (2020): GPT-3.5-turbo. #### Correctness and Relevance Estimation We use the Search-Augmented Factuality Evaluator (SAFE), a pipeline proposed by Wei et al. (2024) that employs LLMs as agents to automatically evaluate the factuality of long-form responses. It utilizes a multi-step reasoning process that includes sending search queries to Google Search Hillis et al. (2012) to verify the information provided. #### Confidence Estimation We utilize three popular confidence estimation methods from different categories Geng et al. (2024): Verbalization Tian et al. (2023), isTrueLogit Kadavath et al. (2022a), and Consistency Manakul et al. (2023). #### Calibration Evaluation Metrics We use Expected Calibration Error (ECE) Guo et al. (2017a); Naeini et al. (2015) at response-level, and F-ECE at the fact-level, as introduced in section 3.2. ### 4.2 Results on Fact-Level Calibration This section highlights three key phenomena observed during our fact-level calibration. These findings not only underscore the advantages of our framework compared to traditional response-level’s, but also inform the design of a confidence-guided fact-level self-correction approach. <details> <summary>x4.png Details</summary>  ### Visual Description ## Violin Plot Comparison: Model Performance Across Metrics ### Overview The image presents a grid of violin plots comparing the performance distributions of five AI models (Llama-2-7b, Llama-2-13b, Vicuna-7b, Vicuna-13b, GPT-3.5-turbo) across two metrics: **LongFact** (top row) and **ASQA** (bottom row). Each plot visualizes the distribution of values for a specific model-metric combination, with color-coded distributions (purple for LongFact, gray for ASQA) and grid-aligned axes. --- ### Components/Axes - **X-axis**: Labeled "Value" (no numerical scale provided, but distributions suggest values range approximately between 0 and 1). - **Y-axis**: - Top row: "LongFact" (purple distributions). - Bottom row: "ASQA" (gray distributions). - **Legend**: Located in the top-right corner, mapping: - Purple: LongFact - Gray: ASQA - **Grid**: Light gray horizontal and vertical lines for reference. - **Background**: White. --- ### Detailed Analysis #### LongFact (Top Row) 1. **Llama-2-7b**: - Wide distribution with a peak near 0.5. - Long tails extending to ~0.1 and ~0.9. 2. **Llama-2-13b**: - Narrower distribution, peaking sharply at ~0.4. - Minimal spread compared to Llama-2-7b. 3. **Vicuna-7b**: - Broad distribution with a peak at ~0.6. - Extended tails to ~0.2 and ~0.8. 4. **Vicuna-13b**: - Similar to Vicuna-7b but slightly narrower. - Peak at ~0.55. 5. **GPT-3.5-turbo**: - Very narrow distribution, concentrated near 0.4. - Minimal variability. #### ASQA (Bottom Row) 1. **Llama-2-7b**: - Bimodal distribution with peaks at ~0.3 and ~0.7. - Significant spread between modes. 2. **Llama-2-13b**: - Single peak at ~0.4. - Narrower than Llama-2-7b. 3. **Vicuna-7b**: - Sharp peak at ~0.5. - Minimal spread. 4. **Vicuna-13b**: - Similar to Vicuna-7b but slightly broader. - Peak at ~0.52. 5. **GPT-3.5-turbo**: - Narrow distribution centered at ~0.2. - Extremely low variability. --- ### Key Observations 1. **LongFact Performance**: - Vicuna-7b and Vicuna-13b exhibit the highest central tendency (~0.55–0.6). - GPT-3.5-turbo shows the lowest and most consistent performance (~0.4). - Llama-2 models display bimodal or wide distributions, suggesting variability in outcomes. 2. **ASQA Performance**: - Vicuna-7b and Vicuna-13b achieve the highest central tendency (~0.5–0.52). - GPT-3.5-turbo underperforms significantly (~0.2). - Llama-2-7b’s bimodal distribution indicates inconsistent results. 3. **Model Size Trends**: - Larger models (e.g., Llama-2-13b, Vicuna-13b) generally show narrower distributions in LongFact, implying greater consistency. - In ASQA, larger models (Vicuna-13b) maintain high performance but with slightly more spread than smaller Vicuna models. 4. **Color Consistency**: - Purple (LongFact) and gray (ASQA) distributions align precisely with the legend. No mismatches observed. --- ### Interpretation The data suggests that **Vicuna models outperform others in LongFact**, particularly the 7b variant, which achieves the highest peak (~0.6). However, their performance in ASQA is more variable, with Llama-2-7b showing bimodal results. GPT-3.5-turbo consistently underperforms in both metrics, with the lowest central tendency (~0.2–0.4). The narrow distributions of Llama-2-13b and GPT-3.5-turbo indicate lower variability, while wider distributions (e.g., Vicuna-7b in LongFact) suggest greater diversity in outcomes. This could reflect differences in training data, architecture, or task-specific optimization. The ASQA metric appears more sensitive to model size, with larger models (Vicuna-13b) maintaining higher performance than smaller ones (Llama-2-7b). Notably, the absence of numerical scales limits precise quantification, but the relative trends are clear. The grid alignment and legend placement ensure clarity in distinguishing metrics and models. </details> Figure 4: Comparison of confidence distribution across different responses between fact-level and response-level, with the purple plots representing fact-level distribution under different statistical metrics and the gray plots showing the response-level distribution, highlighting the over-confidence issue at the response level, which stems from the dominance of the implicit high-confidence facts hidden within the response (Observation 2). <details> <summary>x5.png Details</summary>  ### Visual Description ## Box Plot Chart: Model Performance Comparison Across Metrics ### Overview The image displays a grouped box plot comparing the performance of five language models (Llama-2-7b, Llama-2-13b, Vicuna-7b, Vicuna-13b, GPT-3.5-turbo) across two metrics: **LongFact** (top row) and **ASQA** (bottom row). Each box plot shows the distribution of values, with red lines representing medians, green boxes indicating interquartile ranges (IQRs), and open circles denoting outliers. ### Components/Axes - **X-axis**: Model names (Llama-2-7b, Llama-2-13b, Vicuna-7b, Vicuna-13b, GPT-3.5-turbo). - **Y-axis**: Labeled "Value" (no explicit scale, but values range approximately from -0.5 to 1.0 based on box plot positions). - **Legend**: Not explicitly labeled, but visual conventions apply: - **Red lines**: Medians. - **Green boxes**: Interquartile ranges (IQR). - **Open circles**: Outliers. - **Rows**: - **Top row**: LongFact metric. - **Bottom row**: ASQA metric. ### Detailed Analysis #### LongFact Metric (Top Row) 1. **Llama-2-7b**: - Median ≈ 0.5 (red line). - IQR spans ~0.3 to 0.7. - No outliers. 2. **Llama-2-13b**: - Median ≈ 0.6. - IQR spans ~0.4 to 0.8. - One outlier at ~0.9. 3. **Vicuna-7b**: - Median ≈ 0.5. - IQR spans ~0.3 to 0.7. - One outlier at ~0.9. 4. **Vicuna-13b**: - Median ≈ 0.6. - IQR spans ~0.4 to 0.8. - One outlier at ~0.9. 5. **GPT-3.5-turbo**: - Median ≈ 0.7. - IQR spans ~0.5 to 0.9. - No outliers. #### ASQA Metric (Bottom Row) 1. **Llama-2-7b**: - Median ≈ 0.4. - IQR spans ~0.2 to 0.6. - One outlier at ~0.8. 2. **Llama-2-13b**: - Median ≈ 0.5. - IQR spans ~0.3 to 0.7. - One outlier at ~0.9. 3. **Vicuna-7b**: - Median ≈ 0.5. - IQR spans ~0.3 to 0.7. - One outlier at ~0.9. 4. **Vicuna-13b**: - Median ≈ 0.6. - IQR spans ~0.4 to 0.8. - One outlier at ~0.9. 5. **GPT-3.5-turbo**: - Median ≈ 0.7. - IQR spans ~0.5 to 0.9. - One outlier at ~1.0. ### Key Observations - **Median Trends**: - GPT-3.5-turbo consistently shows the highest medians in both metrics. - Llama-2-13b and Vicuna-13b outperform smaller models (7b variants) in both metrics. - **Outliers**: - Outliers are rare but present in Llama-2-13b, Vicuna-7b, and GPT-3.5-turbo. - Outliers in ASQA for GPT-3.5-turbo reach the upper bound of the y-axis (~1.0). - **Variability**: - Larger models (13b) exhibit tighter IQRs, suggesting more consistent performance. - Smaller models (7b) show wider IQRs, indicating higher variability. ### Interpretation The data suggests that **larger models (13b)** generally outperform smaller models (7b) in both **LongFact** and **ASQA** metrics, with GPT-3.5-turbo achieving the highest median performance. The presence of outliers in some models (e.g., Llama-2-13b, GPT-3.5-turbo) may indicate edge cases or anomalies in specific evaluations. The tighter IQRs for larger models imply greater reliability, while smaller models show more variability, potentially due to limited capacity or training data. The absence of a legend complicates direct interpretation of colors, but standard box plot conventions were applied. </details> Figure 5: Confidence distribution within individual responses at fact level, with red bar indicating the response-level confidence score, indicating high variance exists in fact-level confidence within a single response (Observation 3). #### Observation 1: Fact-level calibration imposes a stricter standard. We compare fact-level and response-level calibration following the protocol outlined in Guo et al. (2017a), where reliability histograms and statistical metrics for is summarized. For both fact and response level, we estimate confidence through all three confidence estimation method, where the correctness/relevance evaluation for facts are described in section 3.2. The detailed prompt template is available in Appendix C. To generate the reliability histograms, we divide the model’s predictions into ten bins based on confidence scores and calculate the average accuracy for each bin. Ideally, an optimally calibrated model would produce a diagonal pattern in the histogram. As shown in fig. 3, a comparison of the histograms on the left and right demonstrates that our fact-level framework can better distinguish calibration performance across models of varying scales. Notably, models such as Llama-2-7b, which appear well-calibrated under traditional response-level calibration, perform significantly worse under fact-level calibration. A similar trend is consistently observed in table 1, where, as highlighted in the table, the variance in fact-level calibration across five different models is nearly an order of magnitude higher than that of response-level calibration. This ability to reveal discrepancies arises from fact-level calibration’s consideration of fine-grained correctness for each fact, as well as the relevance of each fact to the query. The alignment with the model’s capabilities highlights both the superiority and validity of our method, indicating that fact-level calibration imposes a stricter standard. #### Observation 2: Understanding over-confidence through fact-level analysis. We examine the distribution of sentence confidence for both fact-level and response-level calibrations. For the response-level, we directly plot its confidence scores distribution in a grey violin plot. For the fact-level, since confidence for a single response is represented as a vector rather than a scalar, we compute three statistical measures: the mean, maximum, and minimum of the vector. These measures are then visualized as three separate purple violin plots. The results, shown in fig. 4, reveal two noteworthy phenomena: (1) The confidence distribution at the response level is narrow, concentrated around higher confidence values, while the distribution of the mean confidence in the fact-level calibration is wider and indicates lower overall confidence. (2) The response-level confidence distribution closely mirrors the distribution of the maximum confidence values in the fact-level method. Furthermore, a similar phenomenon is also corroborated in table 1, where the average confidence at the fact level across five different models is generally lower than that at the response level. These observations suggest that response-level confidence is largely influenced by the fact with the highest implicit confidence, which can result in overconfidence. In contrast, fact-level framework breaks down the response into individual facts, assessing confidence for each one. This approach highlights less confident aspects of the response, helping to mitigate the issue of overconfidence. #### Observation 3: High variance exists in fact-level confidence within a response. We examine the variance in fact-level confidence within each individual responses. Specifically, for each response, we extract its confidence vector and visualize its distribution using box plots. Due to space constraints, we have visualized 10 responses for each model across both datasets, as shown in fig. 5. Red bar represents the response-level confidence score. Two key observations can be made: (1) Fact-level confidence exhibits significant variance within individual responses, whereas response-level confidence tends to be concentrated at a higher level. (2) Outlier facts typically have much lower confidence scores. The numerous white dots in the box plots indicate these outliers, which generally correspond to facts with substantially lower confidence, falling below the main distribution. This suggests that certain facts are generated by the model with considerably less confidence compared to others within the same response. ## 5 Application: Confidence-Guided Fact-Level Self-Correction <details> <summary>x6.png Details</summary>  ### Visual Description ## Flowchart: Knowledge Correction Process ### Overview The diagram illustrates a multi-stage process for extracting, correcting, and re-estimating confidence in factual statements. It shows how initial answers are refined through confidence thresholds, factor analysis, and iterative corrections. ### Components/Axes 1. **Sections**: - **(1) Fact Extraction & Confidence Estimation**: Initial fact identification with confidence scores (0.90, 0.85, 0.80, 0.30). - **(2) Factor Extraction & Fact Correction**: Component analysis and modification (e.g., "Go Computer" → "Chess Computer"). - **(3) Confidence Re-Estimation**: Updated confidence scores post-correction. 2. **Color Coding**: - **Pink**: High confidence facts (>0.85). - **Blue**: Low confidence facts (<0.85). - **Green**: Corrected facts with re-estimated confidence. 3. **Flow Arrows**: - Red: High confidence facts → Low confidence facts. - Blue: Factor extraction → Fact correction. - Green: Confidence re-estimation. ### Detailed Analysis 1. **Section (1) Fact Extraction & Confidence Estimation**: - High confidence facts (0.90, 0.85, 0.80): - "Deep Blue is developed by IBM." - "Deep Blue defeated world champion Garry Kasparov." - "The defeat is in 1997." - Low confidence fact (0.30): - "Deep Blue is a go computer." (flagged for correction). 2. **Section (2) Factor Extraction & Fact Correction**: - **Factor Extraction**: Identifies components ("Deep Blue, Go Computer"). - **Modification**: Changes "Go Computer" to "Chess Computer". - **Corrected Fact**: "Deep Blue is a chess computer." 3. **Section (3) Confidence Re-Estimation**: - New confidence score (0.90) assigned to the corrected fact. - Final output: "Deep Blue is a chess computer." (high confidence). ### Key Observations - **Thresholds**: Confidence scores above 0.85 are considered high confidence; below 0.85 are flagged for correction. - **Component Isolation**: The system separates "Go Computer" (incorrect) from "Chess Computer" (correct) during factor extraction. - **Iterative Refinement**: Confidence is re-estimated after correction, ensuring final outputs meet quality standards. ### Interpretation This diagram demonstrates a systematic approach to handling uncertainty in knowledge extraction. By combining confidence thresholds with component analysis, the process ensures that only well-supported facts are presented as final answers. The correction of "Go Computer" to "Chess Computer" highlights the system's ability to address factual errors through structured analysis. The re-estimation of confidence (0.30 → 0.90) underscores the value of iterative refinement in improving accuracy. </details> Figure 6: An illustration of our confidence-guided fact-level self-correction framework (ConFix), consisting of three steps: (1) fact extraction and confidence estimation; (2) fact correction using high-confidence facts; and (3) confidence re-estimation. These three steps can be iterated until the desired level of enhancement is achieved. In this section, we introduce the motivation and architecture of Confidence-Guided Fact-Level Self-Correction, dubbed ConFix. ConFix utilizes facts with high confidence as references to revise facts with low confidence, thereby enhancing the generation quality and mitigating hallucinations. Importantly, ConFix operates in real-time during the generation process, eliminating the need for additional fine-tuning or retraining, which reduces computational costs and enhances its adaptability. Furthermore, it functions without relying on external knowledge sources, significantly broadening its applicability across diverse scenarios. ### 5.1 Motivation The development of Confidence-Guided LLM Self-Correction is driven by the three observations discussed earlier. The rationale behind these observations supporting self-correction is as follows: (1) Observations 1 and 2 demonstrate that, even under strict conditions, the fact-level framework reduces overconfidence and enhances the model’s calibration, aligning confidence more closely with factual accuracy. This improved calibration is crucial for enabling effective confidence-guided self-correction. (2) Observation 3 reveals that high-confidence and low-confidence facts often coexist within the same response. Even when the overall confidence is relatively consistent, outliers tend to be low-confidence facts. This enables high-confidence facts to serve as some kind of references, providing the necessary information to refine and correct the low-confidence facts. ### 5.2 Architecture The overall architecture of ConFix is illustrated in fig. 6, including three steps: (1) fact extraction & confidence estimation, (2 )factor extraction & fact correction, and (3) fact confidence re-estimation. #### S1: Fact Extraction & Confidence Estimation Given a response $y_i$ , ConFix first conducts fact extraction and confidence estimation for each extracted fact, following the same process as described in section 3.2. After obtaining the facts $\{f_i^j\}_j=1^M_i$ for $y_i$ and their corresponding confidence scores $\{conf_i^j\}_j=1^M_i$ , we then split the facts into two groups: high-confidence and low-confidence, based on a confidence threshold $τ$ . The high-confidence group shown in eq. 4 is used as a form of internal knowledge base, whose knowledge is leveraged to reinforce and augment facts within the low-confidence group, which is shown in eq. 5, $$ \displaystyle\mathit{f_h}=\{f_i^j\midconf_i^j≥τ\} \displaystyle\mathit{f_l}=\{f_i^j\midconf_i^j<τ\}, \tag{4} $$ where the threshold is defined as the mean confidence score across facts $τ=\frac{1}{M_i}∑_j=1^M_iconf_i^j$ . #### S2: Factor Extraction & Fact Correction To ensure that only the erroneous parts of the low-confidence facts are modified without changing the overall meaning, we restrict the modifiable parts. Specifically, we first parse the key factors through factor extraction. Let $\{\mathit{fa}_i^j,k\}_k=1^K_i^{j}$ represent the $K_i^j$ factors extracted from the target fact $f_i^j∈ f_l$ , $$ \displaystyle\{\mathit{fa}_i^j,k\}_k=1^K_i^{j}=F( LLM(·),p_f(f_i^j)), \tag{6} $$ where $p_f$ is the prompt, which includes: (1) A clear task description. (2) Several instances. (3) The task containing the input sentence. The model is expected to output its extracted factors. After extracting factors, we then perform fact correction, targeting only the extracted factors for modification. This process can be represented as, $$ \displaystyle\hat{f}_i^j=R(LLM(·),p_r(f_i^j, \{\mathit{fa}_i^j,k\}_k=1^K_i^{j},f_h) \tag{7} $$ where $p_r$ is the prompt, which includes: (1) A clear task description. (2) Several instances. (3) The task containing the input target fact, the extracted factors and the high-confidence reference facts. The model is expected to output the modified target fact, noting that the model allows for returning “NoError” to make no modifications to the input. #### S3: Fact Confidence Re-Estimation Finally, the modified facts undergo the confidence estimation process again to obtain new confidence scores: $$ \displaystyle\hat{conf}_i^j=C≤ft(LLM(·), p_c(\hat{f}_i^j,x_i,y_i)\right), \tag{8} $$ where $\hat{conf}_i^j$ represents the confidence score of the modified fact $\hat{f}_i^j$ . Finally, if $\hat{conf}_i^j>conf_i^j$ , the modification is deemed successful and is accepted. Otherwise, ConFix will repeat the process of factor extraction, fact correction, and confidence re-estimation. This iterative process continues until either a satisfactory confidence score is achieved or a predetermined maximum number of iterations $N$ is reached, where the model return “NoError” and make no modifications to the input. ### 5.3 Experiment on Self-Correction #### Experiment Setup We adopt a two-fold evaluation. First, we assess error detection performance using standard metrics such as Accuracy, Precision, and Recall Powers (2020). Subsequently, for evaluating self-correction, we leverage GPT-4 in a zero-shot pairwise evaluation framework (see prompts in Appendix C), measuring improvement ratio, consistency ratio, and regression ratio to capture the performance of correction. #### Error Detection Table 2 shows the error detection results of our proposed method across five different base models. As seen from the table, larger models, such as GPT and the 13B model, outperform smaller models like the 7B model in terms of Accuracy and Precision. However, all models show lower performance in Recall, suggesting that a significant number of erroneous facts go undetected. This indicates a persistent overconfidence issue, with models often incorrectly considering false information as correct. #### Error Correction Table 3 presents the error correction results of our method across five different base models. The results show that, among the three possible outcomes: Improved, Same, and Regressed, our method consistently achieves the highest proportion of improve for almost all models, with the exception of LLaMA-2-7B. The performance drop observed in LLaMA-2-7B is interesting, as it aligns with findings from previous experiments: ConFix depends on the model’s calibration capabilities, functioning effectively when confidence is well-aligned with correctness. However, LLaMA-2-7B is inadequately calibrated, as indicated by its higher ECE/F-ECE scores in table 1. Therefore, the self-correction performance drop in LLaMA-2-7B does not undermine our ConFix, but rather highlights the importance of adequate calibration. Under the right conditions, our method can significantly improve model performance without the need for external knowledge, and these conditions are easily met, as calibration can be enhanced using various existing techniques Liu et al. (2023); Geng et al. (2024). Table 2: Acc., Precision and Recall of Error-Detection. Base Model Accuracy (%) Precision (%) Recall (%) GPT-3.5-turbo 83.29 87.89 13.71 Vicuna-7b 60.06 99.90 0.15 Vicuna-13b 74.81 77.68 8.46 Llama-2-7b 64.26 67.86 13.35 Llama-2-13b 70.45 77.45 30.62 Table 3: GPT-4 Evaluation of Self-Correction. Base Model Improved (%) Same (%) Regressed (%) #Revised GPT-3.5-turbo 46.30 24.07 29.63 108 Vicuna-7b 50.00 50.00 0.00 2 Vicuna-13b 49.40 28.92 21.69 83 Llama-2-7b 6.76 12.56 80.68 207 Llama-2-13b 53.35 19.59 27.07 418 ## 6 Conclusion This paper presents a novel fact-level calibration and self-correction framework designed to address hallucination issues in long-form responses generated by large language models. Traditional response-level methods fall short when handling complex outputs containing multiple facts. Our approach evaluates each fact’s correctness and relevance individually, both externally and internally, enabling fine-grained confidence assessments. This framework imposes a stricter standard than response-level methods, reduces overconfidence, and highlights significant confidence variance across facts within a response. By leveraging high-confidence facts for in-context learning, our self-correction method ConFix can effectively mitigate hallucinations without the need for external knowledge, as demonstrated across various models. ## 7 Limitations and Broader Impacts In this work, we propose a fact-level calibration framework and, based on this framework, introduce a confidence-guided fact-level self-correction method. However, for this self-correction method to be effective, the model itself must possess a certain level of calibration ability. In our paper, we discuss how our calibration framework can alleviate over-confidence. In future work, we will further explore ways to enhance calibration ability within the calibration framework, paving the way for more effective confidence-guided self-correction. ## References - Bang et al. (2023) Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023. - Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. - Cheng et al. (2023) Silei Cheng, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and Lijuan Wang. 2023. Prompting gpt-3 to be reliable. In International Conference on Learning Representations (ICLR 23). - Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. - Dan and Roth (2021) Soham Dan and Dan Roth. 2021. On the effects of transformer size on in-and out-of-domain calibration. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2096–2101. - Desai and Durrett (2020) Shrey Desai and Greg Durrett. 2020. Calibration of pre-trained transformers. arXiv preprint arXiv:2003.07892. - Dubois et al. (2024) Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475. - Dubois et al. (2023) Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacafarm: A simulation framework for methods that learn from human feedback. Preprint, arXiv:2305.14387. - Geng et al. (2024) Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6577–6595. - Golovneva et al. (2022) Olga Golovneva, Moya Peng Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2022. Roscoe: A suite of metrics for scoring step-by-step reasoning. In The Eleventh International Conference on Learning Representations. - Guo et al. (2017a) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017a. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR. - Guo et al. (2017b) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017b. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1321–1330. PMLR. - Hillis et al. (2012) Ken Hillis, Michael Petit, and Kylie Jarrett. 2012. Google and the Culture of Search. Routledge. - Hu et al. (2023) Mengting Hu, Zhen Zhang, Shiwan Zhao, Minlie Huang, and Bingzhe Wu. 2023. Uncertainty in natural language processing: Sources, quantification, and applications. arXiv preprint arXiv:2306.04459. - Huang et al. (2024) Yukun Huang, Yixin Liu, Raghuveer Thirukovalluru, Arman Cohan, and Bhuwan Dhingra. 2024. Calibrating long-form generations from large language models. arXiv preprint arXiv:2402.06544. - Kadavath et al. (2022a) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022a. Language models (mostly) know what they know. CoRR, abs/2207.05221. - Kadavath et al. (2022b) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022b. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. - Kuhn et al. (2023) Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. - Li et al. (2023a) Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023a. Halueval: A large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6449–6464. - Li et al. (2024) Moxin Li, Wenjie Wang, Fuli Feng, Fengbin Zhu, Qifan Wang, and Tat-Seng Chua. 2024. Think twice before assure: Confidence estimation for large language models through reflection on multiple answers. arXiv preprint arXiv:2403.09972. - Li et al. (2023b) Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023b. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval. - Lin et al. (2021) Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958. - Liu et al. (2023) Xin Liu, Muhammad Khalifa, and Lu Wang. 2023. Lightweight language model calibration for open-ended question answering with varied answer lengths. In The Twelfth International Conference on Learning Representations. - Manakul et al. (2023) Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9004–9017, Singapore. Association for Computational Linguistics. - Min et al. (2023) Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. CoRR, abs/2305.14251. - Mündler et al. (2023) Niels Mündler, Jingxuan He, Slobodan Jenko, and Martin Vechev. 2023. Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation. arXiv preprint arXiv:2305.15852. - Naeini et al. (2015) Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using bayesian binning. Proceedings of the … AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence, 2015:2901–2907. - Nguyen and O’Connor (2015) Khanh Nguyen and Brendan O’Connor. 2015. Posterior calibration and exploratory analysis for natural language processing models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 1587–1598. The Association for Computational Linguistics. - OpenAI (2023) OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774. - Powers (2020) David MW Powers. 2020. Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation. arXiv preprint arXiv:2010.16061. - Stelmakh et al. (2022) Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022. Asqa: Factoid questions meet long-form answers. arXiv preprint arXiv:2204.06092. - Tian et al. (2023) Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arXiv preprint arXiv:2305.14975. - Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288. - Wang et al. (2024) Ante Wang, Linfeng Song, Baolin Peng, Ye Tian, Lifeng Jin, Haitao Mi, Jinsong Su, and Dong Yu. 2024. Fine-grained self-endorsement improves factuality and reasoning. arXiv preprint arXiv:2402.15631. - Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. - Wei et al. (2024) Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, et al. 2024. Long-form factuality in large language models. arXiv preprint arXiv:2403.18802. - Xiao et al. (2024a) Teng Xiao, Mingxiao Li, Yige Yuan, Huaisheng Zhu, Chao Cui, and Vasant G Honavar. 2024a. How to leverage demonstration data in alignment for large language model? a self-imitation learning perspective. arXiv preprint arXiv:2410.10093. - Xiao et al. (2024b) Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, and Vasant G Honavar. 2024b. Cal-dpo: Calibrated direct preference optimization for language model alignment. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. - Zhang et al. (2024) Caiqi Zhang, Fangyu Liu, Marco Basaldella, and Nigel Collier. 2024. Luq: Long-text uncertainty quantification for llms. arXiv preprint arXiv:2403.20279. - Zhang et al. (2023) Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023. Siren’s song in the ai ocean: A survey on hallucination in large language models. arXiv preprint arXiv:2309.01219. ## Appendix A Datasets #### LongFact Wei et al. (2024): A dataset consisting of prompts designed to assess a model’s factuality in long-form responses created by GPT-4. This dataset includes a diverse range of topics and ensures that the prompts require detailed and nuanced answers, making it a robust benchmark for evaluating the factual accuracy of language models in generating extended text. The dataset is particularly valuable for testing the capabilities of models in maintaining factual consistency over longer passages, which is crucial for applications such as content creation, summarization, and complex question answering. #### ASQA Stelmakh et al. (2022): A dataset designed for long-form question answering that uniquely centers on ambiguous factoid questions. ASQA provides a challenging testbed for models as it includes questions that can have multiple valid answers depending on the interpretation of the ambiguity. This dataset emphasizes the need for models to not only retrieve accurate information but also to handle the inherent uncertainty and provide comprehensive responses. ASQA is instrumental in pushing the boundaries of model performance in scenarios where clarity and precision are essential, such as in education and interactive AI systems. ## Appendix B Models #### LLaMA-7b-chat & LLaMA-13b-chat Touvron et al. (2023): These models are part of the LLaMA family, known for their strong performance in various natural language processing tasks. The “chat” versions are particularly fine-tuned for conversational contexts, making them suitable for generating coherent and contextually appropriate responses in dialogue settings. LLaMA models are designed to balance performance and computational efficiency, making it a popular choice for research and application in interactive AI systems. #### Vicuna-7b and Vicuna-13b Chiang et al. (2023) Vicuna is an open-source chatbot trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT. Preliminary evaluation using GPT-4 as a judge shows Vicuna-13B achieves more than 90% quality of ChatGPT and Bard while outperforming other models like LLaMA and Stanford Alpaca Li et al. (2023b); Dubois et al. (2024, 2023) in more than 90% of cases. #### GPT-3.5-turbo Brown et al. (2020) This model is part of OpenAI’s well-known GPT series. GPT-3.5-turbo is an enhanced version of GPT-3, offering improved performance and efficiency. It is designed to handle a wide range of language tasks, from text generation to comprehension and translation. The "turbo" variant is optimized for faster inference and lower latency, making it ideal for real-time applications where quick response times are crucial. GPT-3.5-turbo is widely used in both research and industry due to its versatility and high-quality output. ## Appendix C Prompts ### C.1 Prompt for Fact-Level Confidence Estimation The specific prompts used for fact-level confidence estimation are detailed below. | Instructions: | | --- | | 1. The following STATEMENT has been extracted from the broader context of the given RESPONSE to the given QUESTION. | | 2. Indicate how confident you are in the accuracy of the STATEMENT when answering the QUESTION, based on your knowledge. | | 3. The confidence evaluation should be a value between 0 and 1 (with two decimal places retained), based on the following scoring criterion: | | {Criterion} 4. Your task is to do this for the STATEMENT, RESPONSE and QUESTION under "Your Task". | | Some examples have been provided for you to learn how to do this task. | | {Some Examples} | | Your Task: | | QUESTION: | | {Question} | | RESPONSE: | | {Response} | | STATEMENT: | | {Statement} | Table 4: Prompt for fact-level confidence estimation {Criterion}, {Question}, {Response} and {Statement} are placeholders. ### C.2 Prompt for Response-Level Confidence Estimation The specific prompts used for response-level confidence estimation are detailed below. | Instructions: | | --- | | 1. The following RESPONSE is the answer to the given QUESTION. | | 2. Indicate how confident you are in the accuracy of the RESPONSE when answering the QUESTION, based on your knowledge. | | 3. The confidence evaluation should be a value between 0 and 1 (with two decimal places retained), based on the following scoring criterion: | | {Criterion} | | 4. Your task is to do this for the RESPONSE and QUESTION under ‘‘Your Task’’. | | Some examples have been provided for you to learn how to do this task. | | {Some Examples} | | Your Task: | | QUESTION: | | {Question} | | RESPONSE: | | {Response} | Table 5: Prompt for response-level confidence estimation {Criterion}, {Question}, {Response} and {Statement} are placeholders. ### C.3 Prompt for Factor Extraction The specific prompts used for factor extraction are detailed below. | Instructions: | | --- | | You are to read a sentence and identify the key factors within it. | | The task involves pinpointing the essential elements or aspects that significantly influence or characterize the situation, event, or subject described. | | Return the identified key factors using the format <[factor1, factor2, ...]> | | Some examples have been provided for you to learn how to do this task. | | {Some Examples} | | Your Task: | | SENTENCE: | | {Sentence} | Table 6: Prompt for factor extraction {Sentence} is placeholders. ### C.4 Prompt for Fact Correction The specific prompts used for fact correction are detailed below. | Instructions: | | --- | | You have been provided with a sentence and some reference knowledge. | | The sentence has been analyzed, and its factors have been identified. | | However, it is acknowledged that there may be errors or inaccuracies in the identified factors. | | Your task is to first review the identified factors and check for any errors or inaccuracies. | | If there are no errors, simply return ‘‘NoError’’ to indicate that no corrections are needed. | | If errors are present, proceed to make the necessary corrections. | | Ensure that the corrections are limited to the existing factors without adding new content. | | Use the format <old factor -> new factor> for each correction. | | {Some Examples} | | Your Task: | | SENTENCE: | | {Sentence} | | FACTORS: | | {Factor} | | REFERENCE: | | {Reference} | Table 7: Prompt for fact correction {Sentence}, {Factor} and {Reference} are placeholders. ### C.5 GPT-4 Judgments for Self-Correction For the self-correction, we utilize GPT-4 for zero-shot pair-wise evaluation. We use gpt-4-0314 for all our experiments. The specific prompts used for GPT-4 evaluation are detailed below. | You will be provided with a QUESTION, its RESPONSE, and all facts extracted from the RESPONSE under the heading "ALL FACTS". You will also be provided with a specific fact under the heading "TARGET FACT 1", which is included in ALL FACTS. Additionally, you will be given a modified version of this target fact under the heading "TARGET FACT 2". | | --- | | Based on your knowledge, evaluate whether the modification of the target fact is an improvement, the same, or a regression. | | An improvement implies: | | 1. More accurate information. | | 2. Greater relevance to the question. | | 3. Minimal overlap with other facts in ALL FACTS. | | A regression implies: 1. Introduction of erroneous or inaccurate information. | | 2. Lower relevance to the question. | | 3. Repetition or introduction of information that is already provided with other facts in ALL FACTS. | | QUESTION: | | {Question} | | RESPONSE: | | {Response} | | ALL FACTS: | | {All Facts} | | TARGET FACT 1: | | {Original Fact} | | TARGET FACT 2: | | {New Fact} | | First, provide a one-sentence comparison of the two facts and explain whether you think the modification is an improvement, the same, or a regression. Second, on a new line, the state only "IMPROVED", "SAME", or "REGRESSED" to indicate the effectiveness of the modification. Your response should use the following format: | | COMPARISON: <one-sentence comparison and explanation> | | REVISION: <"IMPROVED", "SAME", or "REGRESSED"> | Table 8: Prompt for GPT-4 evaluation for the self-correction {Question}, {Response}, {All Facts}, {All Facts}, {Original Facts} and {New Fact} are placeholders. ## Appendix D Related Work The concept of confidence calibration was first introduced to nerual networks by Guo et al. (2017a) to prevent logits from making incorrect classifications with high probability. This concept has since been extended to NLP models Desai and Durrett (2020); Dan and Roth (2021); Hu et al. (2023). Common methods for estimating confidence scores include logit-based methods, consistency-based methods, and verbalization-based methods. Logit-based methods (Guo et al., 2017b; Cheng et al., 2023; Kadavath et al., 2022b) assess model confidence by examining the logits predicted by the model. Consistency-based methods (Wang et al., 2023; Kuhn et al., 2023) rely on the principle that language models tend to produce similar outputs consistently when they are confident. Recently, research has indicated that verbalization-based methods Tian et al. (2023) might offer superior confidence estimation.