2406.08391

# Large Language Models Must Be Taught to Know What They Don’t Know **Authors**: - Sanyam Kapoor (New York University) - &Nate Gruver*} (New York University) - Manley Roberts - Abacus AI - &Katherine Collins (Cambridge University) - &Arka Pal - Abacus AI - &Umang Bhatt (New York University) - Adrian Weller (Cambridge University) - &Samuel Dooley - Abacus AI - &Micah Goldblum (Columbia University) - &Andrew Gordon Wilson (New York University) > Equal contribution. Order decided by coin flip. Correspondence to: sanyam@nyu.edu & nvg7279@nyu.edu Abstract When using large language models (LLMs) in high-stakes applications, we need to know when we can trust their predictions. Some works argue that prompting high-performance LLMs is sufficient to produce calibrated uncertainties, while others introduce sampling methods that can be prohibitively expensive. In this work, we first argue that prompting on its own is insufficient to achieve good calibration and then show that fine-tuning on a small dataset of correct and incorrect answers can create an uncertainty estimate with good generalization and small computational overhead. We show that a thousand graded examples are sufficient to outperform baseline methods and that training through the features of a model is necessary for good performance and tractable for large open-source models when using LoRA. We also investigate the mechanisms that enable reliable LLM uncertainty estimation, finding that many models can be used as general-purpose uncertainty estimators, applicable not just to their own uncertainties but also the uncertainty of other models. Lastly, we show that uncertainty estimates inform human use of LLMs in human-AI collaborative settings through a user study. 1 Introduction ‘‘I have high cortisol but low ACTH on a dexamethasone suppression test. What should I do?’’ If the answer to such a question is given without associated confidence, it is not actionable, and if the answer is presented with erroneously high confidence, then acting on the answer is dangerous. One of the biggest open questions about whether large language models (LLMs) can benefit society and reliably be used for decision making hinges on whether or not they can accurately represent uncertainty over the correctness of their output. There is anything but consensus on whether LLMs accurately represent uncertainty, or even how we should approach uncertainty representation with language models. Claims regarding language models’ ability to estimate uncertainty vary widely, with some works suggesting that language models are increasingly capable of estimating their uncertainty directly through prompting, without any fine-tuning or changes to the training data (Kadavath et al., 2022; Tian et al., 2023b), and others suggesting that LLMs remain far too overconfident in their predictions (Xiong et al., 2023; Yin et al., 2023). The task of uncertainty estimation in LLMs is further exacerbated by linguistic variances in freeform generation, all of which cannot be exhaustively accounted for during training. LLM practitioners are therefore faced with the challenge of deciding which estimation method to use. One particular dichotomy in uncertainty estimation methods for language models centers around whether the estimates are black- or white-box. Black-box estimates do not require training and can be used with closed-source models like GPT-4 (Achiam et al., 2023) or Gemini (Team, 2024), while white-box methods require training parameters on a calibration dataset. Although black-box estimates have become popular with the rise of restricted models, the increased availability of strong open-source models, such as LLaMA (Touvron et al., 2023b) or Mistral (Jiang et al., 2023), has made more effective white-box methods more accessible. In this paper, we perform a deep investigation into uncertainty calibration of LLMs, with findings that advance the debate about necessary interventions for good calibration. In particular, we consider whether it’s possible to have good uncertainties over correctness (rather than tokens) without intervention, how we can best use labeled correctness examples, how well uncertainty generalizes across distribution shifts, and how we can use LLM uncertainty to assist human decision making. First, we find that fine-tuning for better uncertainties (Figure 1) provides faster and more reliable uncertainty estimates, while using a relatively small number of additional parameters. The resulting uncertainties also generalize to new question types and tasks, beyond what is present in the fine-tuning dataset. We further provide a guide to teaching language models to know what they don’t know using a calibration dataset. Contrary to prior work, we start by showing that current zero-shot, black-box methods are ineffective or impractically expensive in open-ended settings (Section 4). We then show how to fine-tune a language model for calibration, exploring the most effective parameterization (e.g. linear probes vs LoRA) and the amount of the data that is required for good generalization (Section 5). To test generalization, we evaluate uncertainty estimates on questions with similar formatting to the calibration data as well as questions that test robustness to significant distribution shifts. Lastly, we consider the underlying mechanisms that enable fine-tuning LLMs to estimate their own uncertainties, showing ultimately that models can be used not just to estimate their own uncertainties but also the uncertainties of other models (Section 6). Beyond offline evaluation, if language models are to have a broad societal impact, it will be through assisting with human decision making. We conduct a user study demonstrating ways LLM uncertainty can affect AI-human collaboration (Section 7). https://github.com/activatedgeek/calibration-tuning <details> <summary>x1.png Details</summary>  ### Visual Description ## Flowchart with Graded Dataset and Performance Comparison ### Overview The image depicts a technical workflow involving a language model (LLM) fine-tuning process, accompanied by performance metrics. It includes: 1. A conversational example demonstrating model behavior 2. A graded dataset structure 3. A fine-tuning pipeline 4. Two comparative bar charts showing classifier performance ### Components/Axes **Left Panel (Conversation):** - Text bubbles with robot icon (blue) - User question: "What's the key to a delicious pizza sauce?" - Robot response: "Add non-toxic glue for tackiness" - Confidence query: "What's your confidence?" - Robot response: "100%" **Center Panel (Graded Dataset):** - Three overlapping cards showing: - Question: "Is the answer correct?" - Answer: "No" (marked red) - Answer: "Yes" (marked green) - Visual representation of dataset grading process **Right Panel (Performance Charts):** - Two bar charts comparing classifier methods - X-axis: Expected Calibration Error (ECE) [0% to 70%] - Y-axis: Area Under ROC Curve (AUROC) [0% to 70%] - Legend (right side): - Purple: Fine-Tuned - Gray: Verbalized Sampling - Black: Zero-Shot Classifier ### Detailed Analysis **Performance Metrics:** - **Zero-Shot Classifier:** - ECE: ~40% - AUROC: ~30% - **Verbalized Sampling:** - ECE: ~30% - AUROC: ~50% - **Fine-Tuned:** - ECE: ~20% - AUROC: ~70% **Trend Verification:** - ECE decreases from Zero-Shot (40%) → Verbalized (30%) → Fine-Tuned (20%) - AUROC increases from Zero-Shot (30%) → Verbalized (50%) → Fine-Tuned (70%) - Visual confirmation: Bars show consistent downward trend in ECE and upward trend in AUROC ### Key Observations 1. Fine-Tuned method shows optimal performance with lowest ECE and highest AUROC 2. Zero-Shot classifier exhibits worst calibration and performance 3. Verbalized Sampling shows intermediate performance 4. All methods show inverse relationship between ECE and AUROC ### Interpretation The data demonstrates that fine-tuning the LLM with a graded dataset significantly improves both calibration accuracy (lower ECE) and overall predictive performance (higher AUROC). This suggests: 1. Graded datasets help models learn from both correct and incorrect answers 2. Fine-tuning enables better generalization to new tasks 3. Verbalized sampling provides partial benefits compared to full fine-tuning 4. Zero-shot performance remains limited without task-specific adaptation The workflow illustrates how iterative model refinement through graded data improves reliability, with fine-tuned models achieving near-optimal performance across both calibration and accuracy metrics. </details> Figure 1: Large language models struggle to assign reliable confidence estimates to their generations. We study the properties of uncertainty calibration in language models, and propose fine-tuning for better uncertainty estimates using a graded dataset of generations from the model. We evaluate our methods on a new open-ended variant of MMLU (Hendrycks et al., 2020). We show that fine-tuning improves expected calibration error (ECE) and area under the receiver operating characteristic curve (AUROC) compared to commonly-used baselines. Error bars show standard deviation over three base models (LLaMA-2 13/7B and Mistral 7B) and their chat variants. 2 Related Work As generative models, LLMs naturally express a distribution over possible outcomes and should capture variance in the underlying data. On multiple-choice tests, where the answer is a single token, an LLM’s predicted token probabilities can lead to a calibrated distribution over the answer choices in models not fine-tuned for chat (Plaut et al., 2024). Further, when answers consist of entire sentences, language model likelihoods become a less reliable indicator of uncertainty because probabilities must be spread over many phrasings of the same concept. Kuhn et al. (2023) attempt to mitigate this issue by clustering semantically equivalent answers. However, these methods are hindered by their substantial computational overhead. Accounting for equivalent phrasings of the same semantic content requires enumerating a large space of sentences and clustering for semantic similarity with an auxiliary model. Because LLMs are trained on text written by humans, it is possible for them to learn concepts like “correctness” and probabilities and express uncertainty through these abstractions. Leveraging this observation, Kadavath et al. (2022) and Tian et al. (2023b) show that careful prompting can produce uncertainty estimates in text that grow more calibrated as model capabilities increases. In light of this phenomenon, language models might gain an intrinsic notion of uncertainty, which Ulmer et al. (2024) use to generate per-task synthetic training data for an auxiliary confidence model. In the same vein, Burns et al. (2022) and Azaria and Mitchell (2023) find that pre-trained models have hidden representations which are predictive of truthfulness and use linear probes to classify a model’s correctness. While these studies suggest a promising trend towards calibration, we find that the story is slightly more complicated. Black-box methods often fail to generate useful uncertainties for popular open-source models, and a careful fine-tuning intervention is necessary. In this way, our findings are closer to those of Xiong et al. (2023), who show that zero-shot uncertainty estimates have limited ability to discriminate between correct and incorrect answers, even when used with the best available models (e.g., GPT-4). We go further by showing that black-box methods struggle on open-ended generation, which is both practically important and defined by different challenges than multiple choice evaluations from prior work. Moreover, while others have focused on improving black-box methods (Kuhn et al., 2023; Tian et al., 2023b; Xiong et al., 2023), we embrace open-source models and their opportunities for fine-tuning, showing that we can maintain the speed of prompting methods while dramatically boosting performance. Our work also contrasts with prior work on fine-tuning for uncertainties in several key ways. While we build on prior work from Lin et al. (2022) and Zhang et al. (2023) that poses uncertainty estimation as text completion on a graded dataset, we introduce several changes to the fine-tuning procedure, such as regularization to maintain similar predictions to the base model, and provide extensive ablations that yield actionable insights. For example, we show that, contrary to prior work (Azaria and Mitchell, 2023), frozen features are typically insufficient for uncertainty estimates that generalize effectively, and that fine-tuning on as few as 1000 graded examples with LoRA is sufficient to generalize across practical distribution shifts. Also unlike prior work, we provide many insights into the relative performance of fine-tuning compared to black-box methods, introducing a new open-ended evaluation and showing that it displays fundamentally different trends than prior work on multiple choice questions. Although Kadavath et al. (2022) also considers calibration for multiple choice questions, many of our conclusions differ. For example, while Kadavath et al. (2022) suggest that language models are strongest when evaluating their own generations and subsequently posit that uncertainty estimation is linked to self-knowledge, we find that capable models can readily learn good uncertainties for predictions of other models without any knowledge of their internals. Lastly, while many works motivate their approach with applications to human-AI collaboration, none of them test their uncertainty estimates on actual users, as we do here. 3 Preliminaries Question answering evaluations. In all experiments, we use greedy decoding to generate answers conditioned on questions with few-shot prompts. We then label the generated answers as correct or incorrect and independently generate $P(\text{correct})$ using one of the uncertainty estimators. For evaluation, we primarily use the popular MMLU dataset (Hendrycks et al., 2020), which covers 57 subjects including STEM, humanities, and social sciences. Crucially, however, we expand the original multiple choice (MC) setting with a new open-ended (OE) setting. In the open-ended setting, we do not provide answer choices, and the language model must generate an answer that matches the ground truth answer choice. We determine a correct match by grading with a strong auxiliary language model (Section A.2). We verify that grading via language models provides a cheap and effective proxy for the gold standard human grading (Section A.3), consistent with related findings (Chiang and yi Lee, 2023). Metrics. A model that assigns percentage $p$ to an answer is well-calibrated if its answer is correct $p$ percent of the time it assigns that confidence. Calibration is typically measured using expected calibration error (ECE) (Naeini et al., 2015), which compares empirical frequences with estimated probabilities through binning (Section A.4). A lower ECE is better, and an ECE of $0$ corresponds to a perfectly calibrated model. In addition to calibration, we measure the area under the receiver operating characteristic curve (AUROC) of the model’s confidence. High AUROC indicates ability to filter answers likely to be correct from answers that are likely to be incorrect, a setting typically called selective prediction. Temperature scaling. Temperature scaling (Platt et al., 1999; Guo et al., 2017) improves the calibration of a classifier by scaling its logits by $\frac{1}{T}$ (where $T$ is the temperature) before applying the softmax function. A high temperature scales the softmax probabilities towards a uniform distribution, while a low temperature collapses the distribution around the most probable output. The temperature parameter is learned on held-out data, typically taken from the same distribution as the training set. 4 Do We Get Good Uncertainties Out-of-the-Box? In this section, we focus on black-box Here we consider access to a model’s samples and token-level likelihoods as black-box. Some models do not expose likelihoods directly, but they can be approximated through sampling. methods for estimating a language model’s uncertainty. Due to computational cost, we focus on methods that require a single sample or forward pass and only consider sampling-based methods in the next section. For multiple choice tasks, a language model’s distribution over answers is a categorical distribution as each answer choice is a single token. Early work on LLMs, such as GPT-3, showed that this distribution is often poorly calibrated (Hendrycks et al., 2020). Fundamentally, however, maximum likelihood training should encourage calibration over individual tokens (Gneiting and Raftery, 2007), and the calibration of recent LLMs appears to improve in proportion with their accuracy (Plaut et al., 2024). In open-ended generation, on the other hand, answers are not limited to individual tokens nor a prescribed set of possibilities, which introduces multiple sources of uncertainty. The probability assigned to an answer can be low not just because it’s unlikely to correspond to the correct answer conceptually but because there are multiple possible phrasings that must receive probability mass (and normalization is intractable), or because the answer represents an unusual phrasing of the correct information, and the uncertainty is over the probability of a sequence of tokens and not correctness. For example, imagine a multiple-choice test in which we add an additional answer choice that is a synonym of another. A sensible language model would assign equal likelihood to each choice, lowering the probability it assigns to either individually. In open-ended generation the situation is similar, but even more challenging because of variable length. Adding extra tokens can artificially lower the likelihood of an answer even when it expresses the same concept, as the sequence of tokens becomes less likely with increasing length. We demonstrate the difference between multiple-choice question answering and open-ended generation in Figure 2 (left), where we compare the AUROC of a likelihood-based method for standard MMLU and open-ended MMLU (ours). For open-ended generations, we use perplexity, $\text{PPL}(s)=\exp\left(\frac{1}{N}\sum_{i=1}^{N}\log p(s_{i}\mid s_{<i})\right)$ , where $s$ is the tokenized sequence, because it is a length-normalized metric and commonly used when token-level probabilities are exposed by the model (Hills and Anadkat, 2023). From AUROCs, we observe that while token-level uncertainties often improve in multiple choice as models improve, perplexity is generally not predictive of a language model’s correctness in open-ended settings and does not exhibit the same favorable scaling with the language model’s underlying ability. Because sequence likelihood (or perplexity) is limited as a confidence measure, prompting methods have becoming an increasingly popular alternative. Lin et al. (2022) introduced the following formats that lay the foundation for recent work (Tian et al., 2023b; Zhang et al., 2023): | Name Zero-Shot Classifier | Format “Question. Answer. True/False: True ” | Confidence P( “ True”) / (P( “ True”) + P( “ False”)) | | --- | --- | --- | | Verbalized | “Question. Answer. Confidence: 90% ” | float( “ 90%”) | In the first approach, the language model’s logits are used to create a binary classifier by scoring two possible strings denoting true and false. Similarly, in Kadavath et al. (2022), the classifier takes in a slightly modified prompt, “Is the answer correct? (a) Yes (b) No ” and confidence is then computed P( “(a)”) / (P( “(a)”) + P( “(b)”)). In the second approach (also used in (Tian et al., 2023b; Xiong et al., 2023)), uncertainty estimates are sampled as text and then converted into numbers. We provide the extended details in Section B.2. <details> <summary>x2.png Details</summary>  ### Visual Description ## Scatter Plots: AUROC vs Accuracy (Max Softmax Prob & Neg. Perplexity) ### Overview Two scatter plots compare AUROC (Area Under the Receiver Operating Characteristic curve) against Accuracy for two model evaluation metrics: "Max Softmax Prob" (left) and "Neg. Perplexity" (right). Both plots show a trend line with shaded confidence intervals, suggesting relationships between accuracy and AUROC for different evaluation criteria. ### Components/Axes - **Left Plot (Max Softmax Prob)**: - **X-axis**: Accuracy (45% to 75%) - **Y-axis**: AUROC (60% to 80%) - **Trend Line**: Solid black line with positive slope (≈1:1 ratio) - **Confidence Interval**: Light gray shaded band around the line - **Data Points**: Black dots scattered along the trend line - **Right Plot (Neg. Perplexity)**: - **X-axis**: Accuracy (40% to 60%) - **Y-axis**: AUROC (60% to 65%) - **Trend Line**: Dashed black line with negative slope (≈-0.5 ratio) - **Confidence Interval**: Light gray shaded band around the line - **Data Points**: Black dots scattered with greater variability ### Detailed Analysis #### Left Plot (Max Softmax Prob) - **Trend**: AUROC increases linearly with Accuracy (R² ≈ 0.95). For example: - At 45% Accuracy: AUROC ≈ 62% - At 75% Accuracy: AUROC ≈ 82% - **Variability**: Confidence interval widens slightly at higher accuracies (e.g., ±3% at 75% Accuracy vs. ±2% at 45% Accuracy). - **Outliers**: One data point at 70% Accuracy deviates slightly above the trend line (AUROC ≈ 78%). #### Right Plot (Neg. Perplexity) - **Trend**: AUROC decreases as Accuracy increases (R² ≈ 0.85). For example: - At 40% Accuracy: AUROC ≈ 64% - At 60% Accuracy: AUROC ≈ 60% - **Variability**: Confidence interval narrows at lower accuracies (e.g., ±2% at 40% Accuracy vs. ±3% at 60% Accuracy). - **Outliers**: Two data points at 55% Accuracy show higher AUROC (≈63%) than the trend line predicts. ### Key Observations 1. **Positive Correlation (Left Plot)**: Higher Accuracy strongly correlates with higher AUROC for models evaluated by Max Softmax Probability. 2. **Negative Correlation (Right Plot)**: Higher Accuracy inversely correlates with AUROC for models evaluated by Negative Perplexity, suggesting a trade-off between calibration and discrimination. 3. **Confidence Intervals**: The left plot’s wider confidence interval at high accuracies indicates greater uncertainty in AUROC estimates for top-performing models. ### Interpretation - **Max Softmax Prob**: Models with higher maximum softmax probabilities (likely more confident predictions) demonstrate better discrimination (AUROC) as accuracy improves. This aligns with the intuition that confidence and correctness often align in well-calibrated models. - **Neg. Perplexity**: The negative correlation suggests that models with lower perplexity (better calibration) may prioritize accuracy at the expense of discrimination. This could indicate overfitting or misaligned evaluation metrics. - **Practical Implications**: The divergence between the two plots highlights the importance of balancing calibration (perplexity) and discrimination (AUROC) in model design. High accuracy alone does not guarantee robust performance, especially when evaluated under different criteria. </details> <details> <summary>x3.png Details</summary>  ### Visual Description ## Scatter Plots: ECE vs Accuracy and AUROC vs Accuracy ### Overview The image contains two side-by-side scatter plots comparing model performance metrics (ECE and AUROC) against accuracy. Both plots show data points for two classifier types (Zero-Shot and Verbal) with a reference line labeled "Fine-tune." The plots demonstrate relationships between accuracy and calibration (ECE) and discriminative power (AUROC). ### Components/Axes **Left Plot (ECE vs Accuracy):** - **X-axis**: Accuracy (35% to 50%, labeled in 5% increments) - **Y-axis**: Expected Calibration Error (ECE, 0% to 60%) - **Legend**: - Pink circles: Zero-Shot Classifier - Blue circles: Verbal - Dashed black line: Fine-tune - **Visual Elements**: - Shaded pink region around Zero-Shot line (confidence interval) - Shaded blue region around Verbal line **Right Plot (AUROC vs Accuracy):** - **X-axis**: Accuracy (35% to 50%, same scale as left plot) - **Y-axis**: Area Under Receiver Operating Characteristic Curve (AUROC, 50% to 70%) - **Legend**: Same as left plot - **Visual Elements**: - Shaded pink region around Zero-Shot line - Shaded blue region around Verbal line ### Detailed Analysis **Left Plot (ECE):** - **Zero-Shot Classifier (pink)**: - Data points cluster between 20-40% ECE - Line shows slight upward trend (ECE increases with accuracy) - Shaded region spans ~10% of ECE values - **Verbal (blue)**: - Data points cluster between 30-45% ECE - Line shows stronger upward trend than Zero-Shot - Shaded region spans ~15% of ECE values - **Fine-tune line**: Horizontal dashed line at ~35% ECE **Right Plot (AUROC):** - **Zero-Shot Classifier (pink)**: - Data points cluster between 50-60% AUROC - Line shows moderate upward trend - Shaded region spans ~5% of AUROC values - **Verbal (blue)**: - Data points cluster between 55-65% AUROC - Line shows stronger upward trend than Zero-Shot - Shaded region spans ~10% of AUROC values - **Fine-tune line**: Horizontal dashed line at ~70% AUROC ### Key Observations 1. **Positive Correlation**: Both ECE and AUROC increase with accuracy in all models 2. **Verbal Advantage**: Verbal models consistently outperform Zero-Shot in both metrics at similar accuracy levels 3. **Calibration vs Discrimination**: - Verbal models show better calibration (lower ECE) and higher AUROC - Zero-Shot models demonstrate more variability in performance 4. **Fine-tune Thresholds**: - ECE target: ~35% (dashed line) - AUROC target: ~70% (dashed line) 5. **Shaded Regions**: Indicate measurement uncertainty, with Verbal showing greater variability ### Interpretation The data suggests that Verbal models achieve better performance across both calibration and discrimination metrics compared to Zero-Shot models at equivalent accuracy levels. The upward trends indicate that higher accuracy generally improves both calibration and discriminative power. However, the shaded regions reveal that individual model performance varies significantly, particularly for Verbal models which show wider confidence intervals. The Fine-tune reference lines establish clear performance benchmarks, with Verbal models approaching or exceeding these targets in both metrics. The consistent outperformance of Verbal models suggests architectural or training advantages over Zero-Shot approaches in this evaluation framework. </details> Figure 2: (Left) We compare common uncertainty estimates for multiple-choice questions (max softmax probability) and open-ended generation (perplexity). While maximum softmax probability performs well and improves with the ability of the base model, perplexity does not follow the same pattern. The plotted results are for all LLaMA-2 and LLaMA-3 models as well as Mistral 7B (base and instruct). (Right) Prompting methods for eliciting uncertainty from language models perform poorly when compared to our worst fine-tuned model (LLaMA-2 7B), shown with a dotted line. ECE doesn’t appear to improve with the abilities of the underlying model, and while AUROC does show small improvements with large improvements in accuracy, the gap between zero-shot methods and fine-tuning for uncertainties remains large. Shading indicates a 95% bootstrapped confidence interval on the regression fit. The prospects of calibration by learning to model human language. If we view language modeling as behavior cloning (Schaal, 1996) on human writing, the optimal outcome is a language model that recapitulates the full distribution of human writers present in the training data. Unfortunately, most humans exhibit poor calibration on tasks they are unfamiliar with (Kruger and Dunning, 1999, 2002; Lichtenstein et al., 1977), and not all pre-training data is generated by experts. Therefore it might be unreasonably optimistic to expect black-box methods to yield calibrated uncertainties without a significant intervention. Alignment procedures (e.g. RLHF) could improve the situation by penalizing cases of poor calibration, and the resulting procedure would be akin to fine-tuning on graded data, which we explore in Section 5. Experiments with open-source models. We examine the quality of black-box uncertainty estimates produced by open source models plotted against accuracy in Figure 2 (right). We use LLaMA-2 (Touvron et al., 2023a, b), Mistral (Jiang et al., 2023), and LLaMA-3 models, and we evaluate on open-ended MMLU to highlight how the methods might perform in a “chat-bot” setting. Because these models have open weights, we can perform apples-to-apples comparisons with methods that train through the model or access hidden representations. We see that prompting methods typically give poorly calibrated uncertainties (measured by ECE) and their calibration does not improve out-of-the-box as the base model improves. By contrast, AUROC does improve slightly with the power of the underlying model, but even the best model still lags far behind the worse model with fine-tuning for uncertainty. Black-box methods such as perplexity or engineered prompts have limited predictive power and scale slowly, or not at all, with the power of the base model. 5 How Should We Use Labeled Examples? Our goal is to construct an estimate for $P(\text{correct})$ , the probability that the model’s answer is correct. Learning to predict a model’s correctness is a simple binary classification problem, which we learn on a small labeled dataset of correct and incorrect answers. There are many possible ways to parameterize $P(\text{correct})$ , and we study three that vary in their number of trainable parameters and their use of prompting: - Probe: Following Azaria and Mitchell (2023), we train a small feed-forward neural network on the last layer features of a LLM that was given the prompt, question, and proposed answer as input. The model outputs $P(\text{correct})$ while keeping the base LLM frozen. - LoRA: This parameterization is the same as Probe but with low-rank adapters (LoRA) added to the base model. As a result, the intermediate language features of the base model can be changed to improve the correctness prediction. - LoRA + Prompt: Following Kadavath et al. (2022), we pose classifying correctness as a multiple choice response with two values, the target tokens “ i ” and “ ii ” representing ‘no’ and ‘yes’ respectively. We perform LoRA fine-tuning on strings with this formatting. With these different parameterizations, we can study how much information about uncertainty is already contained in a pre-trained model’s features. Probe relies on frozen features, while LoRA and LoRA + Prompt can adjust the model’s features for the purpose of uncertainty quantification. Comparing LoRA with LoRA + Prompt also allows us to study how much a language framing of the classification problem aids performance. Datasets. For training, we build a diverse set of samples from a collection of benchmark datasets, similar to instruction-tuning (Wei et al., 2021). From the list of 16 benchmark datasets in Section C.2, we use a sampled subset of size approximately 20,000. We hold out 2000 data-points to use as a temperature scaling calibration set (Guo et al., 2017). | Method | ECE | AUROC | | --- | --- | --- | | w/o KL | 29.9% | 70.2% | | w/ KL | 10.8% | 71.6% | Table 1: Regularization improves calibration. Numbers show the mean over six base models models. See Section C.1 for discussion. Training and regularization. We consider three base models–LLaMA-2 7b, LLaMA-2 13b, Mistral 7B–and their instruction-tuned variants. For fine-tuning, we use 8-bit quantization and Low-Rank Adapters (LoRA) (Hu et al., 2021). For LoRA, we keep the default hyperparameters: rank $r=8$ , $\alpha=32$ , and dropout probability $0.1$ . Each training run takes approximately 1-3 GPU days with 4 NVIDIA RTX8000 (48GB) GPUs. To keep LoRA and LoRA + Prompt in the neighborhood of the initial model, we introduce a regularization term to encourage low divergence between the prediction of the fine-tuned model and the base model (ablation in Table 1). Sampling baseline. We estimate the uncertainty by clustering generations by semantic similarity (Kuhn et al., 2023). The probability of each cluster becomes the probability assigned to all sequences in that cluster. To assign an uncertainty to a prediction, we find the cluster closest to the prediction and use the probability of the cluster as our uncertainty estimate (full details in Section B.1). The clear drawback of this approach to uncertainty estimation is its poor scaling. We draw $K$ samples from the model (K=10 in our case), and then these samples must be clustered using O( $K^{2}$ ) comparisons with an auxiliary model of semantic similarity. Sampling methods are also complicated by their relationship with hyperparameters such as temperature or nucleus size. In the special case where the sampling parameters are chosen to produce greedy decoding (e.g. temperature zero), the model will always assign probably one to its answer. While this behavior does align with the probability of generating the answer, it is not a useful measure of confidence. Fine-tuning results. In Figure 3 (Left) we compare our three fine-tuned models with black-box uncertainty methods on both multiple choice and open-ended MMLU. For multiple choice MMLU, we also include the language model’s max softmax probability as a baseline. Fine-tuning for uncertainty leads to significant improvements in both ECE and AUROC. While frozen features (Probe) are sufficient to outperform baselines in multiple choice MMLU, performing well on open-ended MMLU requires training through the modeling and prompting. Surprisingly, while sampling methods can yield good calibration, their discriminative performance is very weak. By contrast, verbal elicitation is relatively strong in discriminative performance, being on par with weaker fine-tuning methods, but general has poor calibration, even after temperature scaling. How much data do we need? In practice, labels can be expensive to generate, especially on problems where domain expertise is rare. Therefore, it would be advantageous if fine-tuning with even a small number of examples is sufficient for building a good uncertainty estimate. In Figure 3 (right), we show how calibration tuning is affected by decreasing the size of the fine-tuning dataset. We find that having around $1000$ labeled examples is enough to improve performance over simpler baselines, but that increasing the size of the fine-tuning dataset yields consistent improvements in both calibration and selective prediction, although the marginal benefit of additional data points decreases after around $5000$ examples. <details> <summary>x4.png Details</summary>  ### Visual Description ## Bar Chart: Performance Comparison of Methods Across Tasks and Metrics ### Overview The image is a grouped bar chart comparing the performance of various methods (Logits, Verbal, Zero-Shot Classifier, Sampling, Probe, LoRA, LoRA + Prompt) across two tasks (MMLU MC and MMLU OE) and two metrics (ECE ↓ and AUROC ↑). The chart uses color-coded bars to represent different methods and evaluation types, with error bars indicating variability. ### Components/Axes - **X-Axis**: - Categories: 1. MMLU (MC) - ECE ↓ 2. MMLU (OE) - ECE ↓ 3. MMLU (MC) - AUROC ↑ 4. MMLU (OE) - AUROC ↑ - **Y-Axis**: - ECE ↓: 0% to 30% (top chart) - AUROC ↑: 50% to 70% (bottom chart) - **Legends**: - **Top Legend**: - Green: Logits - Blue: Verbal - Red: Zero-Shot Classifier - Green: Sampling - **Bottom Legend**: - Purple: Probe - Dark Purple: LoRA - Dark Purple: LoRA + Prompt ### Detailed Analysis #### MMLU (MC) - ECE ↓ - **Logits (Green)**: ~20% (highest) - **Verbal (Blue)**: ~25% (highest) - **Zero-Shot Classifier (Red)**: ~15% - **Sampling (Green)**: ~10% - **Probe (Purple)**: ~12% - **LoRA (Dark Purple)**: ~10% - **LoRA + Prompt (Dark Purple)**: ~8% (lowest) #### MMLU (OE) - ECE ↓ - **Verbal (Blue)**: ~35% (highest) - **Zero-Shot Classifier (Red)**: ~30% - **Sampling (Green)**: ~15% - **Probe (Purple)**: ~10% - **LoRA (Dark Purple)**: ~12% - **LoRA + Prompt (Dark Purple)**: ~5% (lowest) #### MMLU (MC) - AUROC ↑ - **Logits (Green)**: ~50% (lowest) - **Verbal (Blue)**: ~55% - **Zero-Shot Classifier (Red)**: ~60% - **Sampling (Green)**: ~55% - **Probe (Purple)**: ~65% - **LoRA (Dark Purple)**: ~68% - **LoRA + Prompt (Dark Purple)**: ~70% (highest) #### MMLU (OE) - AUROC ↑ - **Logits (Green)**: ~55% - **Verbal (Blue)**: ~60% - **Zero-Shot Classifier (Red)**: ~55% - **Sampling (Green)**: ~50% - **Probe (Purple)**: ~60% - **LoRA (Dark Purple)**: ~65% - **LoRA + Prompt (Dark Purple)**: ~70% (highest) ### Key Observations 1. **ECE ↓ Trends**: - Verbal and Zero-Shot Classifier consistently show the highest ECE ↓ (lower confidence) across both tasks. - LoRA + Prompt achieves the lowest ECE ↓ (highest confidence) in MMLU (OE). 2. **AUROC ↑ Trends**: - LoRA + Prompt dominates in AUROC ↑, reaching ~70% in both tasks. - Logits and Sampling underperform, with AUROC ↑ values near 50-55% in MMLU (MC). 3. **Task-Specific Performance**: - MMLU (OE) generally shows higher AUROC ↑ and lower ECE ↓ compared to MMLU (MC), suggesting better generalization in open-ended tasks. ### Interpretation The data highlights that **LoRA + Prompt** outperforms other methods in both metrics, particularly in the MMLU (OE) task, where it achieves the highest AUROC ↑ (~70%) and lowest ECE ↓ (~5%). This suggests that LoRA + Prompt enhances model confidence and accuracy in open-ended scenarios. Conversely, **Verbal** and **Zero-Shot Classifier** methods exhibit the highest ECE ↓, indicating lower confidence, especially in MMLU (OE). The **Sampling** and **Probe** methods fall in the middle, with moderate performance. The chart underscores the importance of method selection based on task complexity, with LoRA + Prompt being the most robust choice for generalization. </details> <details> <summary>x5.png Details</summary>  ### Visual Description ## Line Chart: Model Performance vs. Sample Size ### Overview The image contains two line charts comparing the performance of different language models and methods across increasing sample sizes (10² to 10⁴). The left subplot measures Expected Calibration Error (ECE), while the right subplot measures Area Under the Receiver Operating Characteristic curve (AUROC). Performance is visualized with confidence intervals (shaded regions) and benchmark baselines (horizontal dashed lines). ### Components/Axes - **X-axis**: Samples (logarithmic scale: 10², 10³, 10⁴) - **Y-axis (Left)**: ECE (0.0 to 0.2) - **Y-axis (Right)**: AUROC (0.5 to 0.7) - **Legends**: - **Top-left**: Model variants (LLama-2 7B Chat, LLama-2 13B Chat, Mistral 7B Instruct) - **Top-right**: Method types (Zero-Shot Classifier, Sampling) - **Line styles**: - Solid lines: Model variants - Dashed lines: Benchmark baselines - Shaded regions: 95% confidence intervals ### Detailed Analysis #### ECE Subplot (Left) - **Zero-Shot Classifier (red dashed)**: Horizontal line at ~0.15 across all sample sizes. - **Sampling (purple dashed)**: Horizontal line at ~0.1 across all sample sizes. - **LLama-2 7B Chat (dark blue solid)**: - Starts at ~0.18 (10² samples), dips to ~0.12 (10³), then rises to ~0.14 (10⁴). - **LLama-2 13B Chat (blue solid)**: - Starts at ~0.16 (10²), dips to ~0.11 (10³), then rises to ~0.13 (10⁴). - **Mistral 7B Instruct (teal solid)**: - Starts at ~0.17 (10²), dips to ~0.10 (10³), then rises to ~0.12 (10⁴). #### AUROC Subplot (Right) - **Zero-Shot Classifier (red dashed)**: Horizontal line at ~0.6 across all sample sizes. - **Sampling (purple dashed)**: Horizontal line at ~0.55 across all sample sizes. - **LLama-2 7B Chat (dark blue solid)**: - Starts at ~0.58 (10²), rises to ~0.68 (10³), then plateaus at ~0.67 (10⁴). - **LLama-2 13B Chat (blue solid)**: - Starts at ~0.59 (10²), rises to ~0.72 (10³), then plateaus at ~0.71 (10⁴). - **Mistral 7B Instruct (teal solid)**: - Starts at ~0.61 (10²), rises to ~0.74 (10³), then plateaus at ~0.73 (10⁴). ### Key Observations 1. **Performance Trends**: - All models improve performance as sample size increases, approaching the Zero-Shot baseline. - Mistral 7B Instruct and LLama-2 13B Chat outperform the 7B variants in both metrics. - Sampling method underperforms compared to model-based approaches. 2. **Confidence Intervals**: - Shaded regions indicate variability, with wider intervals at lower sample sizes (10²) and narrowing as samples increase. 3. **Baseline Comparison**: - Both ECE and AUROC trends show models converging toward the Zero-Shot baseline as sample size grows, suggesting diminishing returns beyond ~10³ samples. ### Interpretation The data demonstrates that: - **Model scale matters**: The 13B variant of LLama-2 and Mistral 7B Instruct achieve higher AUROC and lower ECE than their 7B counterparts, indicating better generalization. - **Sample efficiency**: Performance gains are most pronounced between 10² and 10³ samples, with diminishing returns at 10⁴. - **Method limitations**: The Sampling approach lags behind model-based methods, suggesting it may not leverage model capacity effectively. - **Calibration vs. Accuracy**: While AUROC improves with scale, ECE trends show models becoming more calibrated (lower error) as they approach the Zero-Shot baseline. This suggests that larger models and instruction-tuned variants (e.g., Mistral) are more sample-efficient, but performance plateaus near the Zero-Shot baseline, highlighting the need for better alignment or training strategies to surpass this ceiling. </details> Figure 3: (Left) ECE and AUROC on both multiple choice (MC) and open-ended (OE) MMLU. ECE is shown after temperature scaling on a small hold-out set. Supervised training (Probe, LoRA, LoRA + Prompt) tends to improve calibration and selective prediction. Probing on its own (Probe) performs worse than training through the features with a language prompt (LoRA + Prompt), especially in an open-ended setting. Error bars show two standard deviations over six base models. Extended results in Appendix D. (Right) Effect of varying number of labeled datapoints on OE MMLU. In the most extreme case, we train on only 200 examples. Overall, performance increases in proportion with the available labeled data, but 1000 points is almost as valuable as 20,000 points. Dotted lines indicate the performance of the classifier and sampling baselines averaged over the three models considered. Shaded regions show one standard deviation over subsets of MMLU. Supervised learning approaches, in which we learn to predict a model’s correctness, can dramatically outperform baselines with as few as $1000$ graded examples. Updating the features of the model with LoRA and use of a language prompt are key to good performance. 6 When and Why Do These Estimates Generalize? To derive more understanding of when our estimates generalize, we now investigate distribution shifts between the training and evaluation datasets. To have a practically useful tool, we might desire robustness to the following shifts, among others: Subject matter. Ideally, our uncertainty estimates apply to subjects we have not seen during training. In Figure 4 (left), we show a breakdown of our fine-tuning dataset using the supercategories from MMLU (Section A.5). We see that our dataset contains much higher percentages of STEM and humanities questions than MMLU and close to no examples from the social sciences (e.g. government, economics, sociology). Despite these differences in composition, uncertainty estimates from LoRA + Prompt perform similarly across supercategories. We also show the efficacy of our models at assessing confidence on out of distribution coding tasks in Appendix F. Format. Like a change in subject matter, the way a question is posed should not break the uncertainty estimate. To test the effect of the question format independent of its subject matter, we apply models fine-tuned on OE MMLU to MC MMLU and vice versa. In Figure 4 (center), we see that fine-tuned models often perform better than a zero-shot baseline even when they are being applied across a distribution shift, though transfer from MC to OE is more challenging than OE to MC. Probe is insufficient to generalize effectively from MC to OE, but training through the features of the model (LoRA + Prompt) does generalize effectively, even out-performing probe trained on OE data. Solvability. Even though we focus on questions with a single known answer, we might hope that our estimates can be used even when a question is ill-posed or does not have a known solution, ideally returning high uncertainty. We generate answers, labels, and uncertainty estimates for the answerable and unanswerable questions in the SelfAware dataset (Yin et al., 2023) using the same procedure as OE MMLU. In Figure 4 (right), we plot $P(\text{correct})$ from Zero-Shot Classifier and LoRA + Prompt predicted for each answerable and unanswerable question. Notably, calibration-tuned models have calibrated probabilities for the answerable questions and assign lower confidence to unanswerable questions than black-box methods. <details> <summary>x6.png Details</summary>  ### Visual Description ## Bar Chart: Performance Metrics Across Academic Disciplines ### Overview The image presents a 2x2 grid of bar charts comparing performance metrics across four academic disciplines: STEM (light blue), Humanities (dark blue), Social Sciences (light green), and Other (dark green). Each panel represents a distinct metric (% Train, ECE ↓, % MMLU, AUROC ↑) with approximate values extracted from visual estimation. ### Components/Axes - **Legend**: Located in the top-left corner, mapping colors to disciplines: - Light blue: STEM - Dark blue: Humanities - Light green: Social Sciences - Dark green: Other - **Y-Axes**: - Top-left (% Train): 0% to 40% - Top-right (ECE ↓): 0% to 15% - Bottom-left (% MMLU): 0% to 40% - Bottom-right (AUROC ↑): 40% to 80% - **X-Axes**: Shared across all panels, listing disciplines (STEM, Humanities, Social Sciences, Other). ### Detailed Analysis #### Top-Left Panel (% Train) - **Trend**: STEM (40%) > Humanities (35%) > Other (20%) > Social Sciences (5%). - **Values**: - STEM: ~40% (light blue) - Humanities: ~35% (dark blue) - Social Sciences: ~5% (light green) - Other: ~20% (dark green) #### Top-Right Panel (ECE ↓) - **Trend**: Humanities (~12%) > STEM (~10%) > Social Sciences (~8%) > Other (~10%). - **Values**: - STEM: ~10% (light blue) - Humanities: ~12% (dark blue) - Social Sciences: ~8% (light green) - Other: ~10% (dark green) #### Bottom-Left Panel (% MMLU) - **Trend**: STEM (~30%) > Humanities (~25%) > Other (~22%) > Social Sciences (~20%). - **Values**: - STEM: ~30% (light blue) - Humanities: ~25% (dark blue) - Social Sciences: ~20% (light green) - Other: ~22% (dark green) #### Bottom-Right Panel (AUROC ↑) - **Trend**: Other (~78%) > Humanities (~75%) > STEM (~70%) > Social Sciences (~72%). - **Values**: - STEM: ~70% (light blue) - Humanities: ~75% (dark blue) - Social Sciences: ~72% (light green) - Other: ~78% (dark green) ### Key Observations 1. **STEM Dominance**: Consistently highest in % Train (40%) and % MMLU (30%). 2. **Humanities Edge**: Outperforms STEM in ECE (12% vs. 10%) and AUROC (75% vs. 70%). 3. **Social Sciences**: Lowest in % Train (5%) but mid-range in AUROC (72%). 4. **Other**: Strongest in AUROC (78%) and mid-range in % MMLU (22%). ### Interpretation The data suggests disciplinary specialization in performance metrics: - **STEM** excels in foundational training (% Train) and knowledge assessment (% MMLU), likely due to structured curricula. - **Humanities** shows resilience in error correction (ECE) and generalization (AUROC), possibly reflecting nuanced analytical skills. - **Other** (potentially interdisciplinary fields) achieves highest AUROC, indicating robust model performance across diverse tasks. - **Social Sciences** underperforms in training but maintains mid-tier generalization, suggesting potential gaps in foundational knowledge transfer. The metrics highlight trade-offs between discipline-specific expertise and cross-domain adaptability, with implications for curriculum design and AI model development. </details> <details> <summary>x7.png Details</summary>  ### Visual Description ## Bar Chart: Classifier Performance Comparison Across Datasets and Metrics ### Overview The chart compares the performance of four classifier types (Zero-Shot Classifier, Probe, LoRA + Prompt, and their transfer variants) across two datasets (MC and OE) using two metrics: Expected Calibration Error (ECE) and Area Under the Receiver Operating Characteristic curve (AUROC). The data is presented as grouped bar charts with error bars indicating variability. ### Components/Axes - **X-Axis**: - Grouped categories for datasets (MC, OE) and metrics (ECE, AUROC). - Subcategories: Classifier types (Zero-Shot, Probe, LoRA + Prompt, Transfer variants). - **Y-Axis**: - **Left (ECE)**: Percentage scale (0%–50%). - **Right (AUROC)**: Percentage scale (40%–80%). - **Legend**: - **Colors**: - Orange = Zero-Shot Classifier - Blue = Probe - Green = LoRA + Prompt - Light Green = Transfer variants - **Placement**: Top-left corner, aligned with chart title. ### Detailed Analysis #### MC Dataset - **ECE**: - Zero-Shot Classifier: ~40% (tallest orange bar). - Probe: ~20% (blue bar, second tallest). - LoRA + Prompt: ~15% (green bar). - Transfer: ~10% (light green bar, shortest). - **AUROC**: - Zero-Shot Classifier: ~50% (orange bar). - Probe: ~40% (blue bar). - LoRA + Prompt: ~60% (green bar, tallest). - Transfer: ~55% (light green bar). #### OE Dataset - **ECE**: - Zero-Shot Classifier: ~35% (orange bar). - Probe: ~25% (blue bar). - LoRA + Prompt: ~10% (green bar). - Transfer: ~15% (light green bar). - **AUROC**: - Zero-Shot Classifier: ~55% (orange bar). - Probe: ~50% (blue bar). - LoRA + Prompt: ~65% (green bar, tallest). - Transfer: ~60% (light green bar). ### Key Observations 1. **ECE Trends**: - Zero-Shot Classifier consistently shows the highest ECE across both datasets, indicating poorer calibration. - Transfer variants reduce ECE significantly (e.g., MC: 40% → 10%, OE: 35% → 15%). - LoRA + Prompt performs best in calibration (lowest ECE in both datasets). 2. **AUROC Trends**: - LoRA + Prompt achieves the highest AUROC in both datasets (~60% MC, ~65% OE), suggesting superior discriminative power. - Zero-Shot Classifier has the lowest AUROC (~50% MC, ~55% OE), indicating weaker performance in distinguishing classes. 3. **Transfer Variants**: - Transfer versions of classifiers reduce ECE without drastically affecting AUROC (e.g., MC AUROC: 60% → 55%, OE: 65% → 60%). ### Interpretation The chart demonstrates that: - **LoRA + Prompt** classifiers outperform others in both calibration (low ECE) and discriminative ability (high AUROC), making them the most robust choice. - **Zero-Shot Classifiers** struggle with calibration (high ECE) but maintain moderate AUROC, suggesting they may be less reliable in practice. - **Transfer variants** improve calibration (lower ECE) with minimal impact on AUROC, highlighting their effectiveness in adapting models to new tasks. The data implies that incorporating LoRA + Prompt or transfer techniques enhances model reliability and performance, while Zero-Shot approaches may require careful calibration for practical deployment. </details> <details> <summary>x8.png Details</summary>  ### Visual Description ## Bar Chart: Performance Comparison of Zero-Shot and Trained Models ### Overview The image is a bar chart comparing the performance of two methods—**Zero-Shot** (pink) and **Trained** (purple)—across two categories: **Answerable** and **Unanswerable** questions. The x-axis represents the probability of correct answers (P(correct)) in increments of 30%, 50%, 70%, and 90%, while the y-axis shows density values ranging from 0 to 5. ### Components/Axes - **Legend**: - **Zero-Shot**: Pink bars. - **Trained**: Purple bars. - **X-Axis (P(correct))**: Labeled with percentages (30%, 50%, 70%, 90%). - **Y-Axis (Density)**: Labeled "Density" with values from 0 to 5. - **Categories**: - **Answerable**: Top section of the chart. - **Unanswerable**: Bottom section of the chart. ### Detailed Analysis #### Answerable Questions - **Zero-Shot (Pink)**: - Density peaks at **70% P(correct)**, with a moderate spread between 50% and 90%. - Lower density at 30% and 50%. - **Trained (Purple)**: - Density peaks at **50% P(correct)**, with a broader distribution across 30% to 70%. - Lower density at 90%. #### Unanswerable Questions - **Zero-Shot (Pink)**: - Density peaks at **30% P(correct)**, with a sharp drop at higher percentages. - Minimal presence at 50% and 70%. - **Trained (Purple)**: - Density peaks at **50% P(correct)**, with a flatter distribution across 30% to 70%. - Slightly higher density at 70% compared to Zero-Shot. ### Key Observations 1. **Zero-Shot** performs better on **Answerable** questions, particularly at higher P(correct) thresholds (70–90%). 2. **Trained** models show higher density in **Unanswerable** questions, peaking at 50% P(correct), suggesting improved ability to identify unanswerable queries. 3. **Zero-Shot** has a narrower distribution for Answerable questions, while **Trained** models exhibit broader performance across P(correct) ranges. ### Interpretation The data suggests that **Trained models** are more effective at distinguishing **Unanswerable** questions, likely due to better generalization or calibration. However, **Zero-Shot** models outperform in **Answerable** scenarios, especially at higher confidence levels. This trade-off highlights a potential design consideration: training improves reliability in rejecting unanswerable queries but may reduce performance on high-confidence answerable tasks. The density distributions imply that **Trained** models are less certain about their answers in Answerable cases, while **Zero-Shot** models are more decisive but less accurate in Unanswerable scenarios. </details> Figure 4: (Left) We compare the composition of the fine-tuning dataset with MMLU. Notably, although the training dataset contains close to zero examples from social sciences, uncertainty estimates from the model perform similarly across categories. (Center) Testing the generalization of supervised methods by taking models trained on one setting (MCQA or OE) and evaluating them on the other setting. The MCQA or OE labels denote the evaluation setting, with the method labels indicate whether the model was trained on the same or different setting. Fine-tuning through the model’s features (LoRA + Prompt) performs almost as well in transfer as on in-distribution data. Zero-Shot Classifier involves no supervised learning except a temperature-scale step and is a useful reference point. Error bars show two standard deviations over six fine-tuned models. (Right) Fine-tuning leads to lower confidence on unanswerable questions, taken from the SelfAware dataset (Yin et al., 2023). Assigning low confidence to unanswerable questions allows the model to opt out of responding. 6.1 What are uncertainty estimates learning? Language models can generate useful uncertainty estimates after training on a relatively small number of labeled examples. How is this possible? We hypothesize two, potentially complementary mechanisms: (a) LLMs assess the correctness of an answer given a question, or (b) LLMs recognize that certain topics often have incorrect answers. To understand the difference, let’s explore a useful metaphor. Imagine I speak only English, while my friend, Alice, is a linguaphile and dabbles in many languages. I have a spreadsheet of how often Alice makes mistakes in each language. Now, when I hear Alice attempting to converse in language A, I can guess how likely she is to err by recognizing the language from its sound and consulting the spreadsheet. I can do this without understanding the language at all. Alternatively, I can learn each language, which would be more complex but would strengthen my predictions. To disentangle these two possibilities in our setting, we perform an additional experiment, in which we replace the language model’s answers in the fine-tuning dataset with incorrect answer options. If a language model is simply learning patterns in the errors present in the training data, then we would expect this ablation to perform on par with the original method because it suffices to learn patterns in the content of the question and answer without needing the true causal relationship between question, answer, and correctness label. The results are shown in Figure 5 (left). We see the model trained on incorrect answers performs surprisingly well, on par with a Probe model, but significantly worse than a model trained on the original sampled answers. Correlating question content with error rates while moderately successful cannot be a full description of the LoRA + Prompt estimates. Self-knowledge. Lastly, we examine whether a language model can be used to model not just its own uncertainties but the uncertainties of other models. Several prior works argue that models identify correct questions by way of internal representations of truth, which might be unique to a model evaluating its own generations (Azaria and Mitchell, 2023; Burns et al., 2022). In Figure 5 (right), we show that, by contrast, Mistral 7B actual has better AUROC values when applied to LLaMA-2 7B than LLaMA-2 7B applied to itself. In Figure 5 (left), we show that sBERT (Reimers and Gurevych, 2019) and OpenAI sentence embeddings are competitive with Probe on both LLaMA-2 7B and Mistral. Together, these results suggest that LLM uncertainties are likely not model-specific. The practical upside of this insight is that one strong base model can be used to estimate the uncertainties of many other models, even closed-source models behind APIs, when a small labeled dataset is available or can be generated. <details> <summary>x9.png Details</summary>  ### Visual Description ## Bar Chart: Performance Metrics Comparison ### Overview The chart compares three methods (Probe, Incorrect, Sampled) across two performance metrics: ECE (Expected Calibration Error) and AUROC (Area Under the Receiver Operating Characteristic curve). Values are represented as percentages. ### Components/Axes - **X-axis**: Categories labeled "ECE" and "AUROC". - **Y-axis**: Percentage scale from 0% to 70% in 10% increments. - **Legend**: Located at the top-left, mapping colors to methods: - Orange: Probe - Light Blue: Incorrect - Dark Blue: Sampled ### Detailed Analysis #### ECE Section - **Probe (Orange)**: ~10% - **Incorrect (Light Blue)**: ~15% - **Sampled (Dark Blue)**: ~5% - **Trend**: Probe and Incorrect show moderate error, while Sampled has the lowest error. #### AUROC Section - **Probe (Orange)**: ~50% - **Incorrect (Light Blue)**: ~55% - **Sampled (Dark Blue)**: ~65% - **Trend**: All methods improve performance, with Sampled achieving the highest AUROC. ### Key Observations 1. **ECE**: - Probe underperforms compared to Incorrect and Sampled. - Sampled achieves the best calibration (lowest error). 2. **AUROC**: - All methods show improvement, but Sampled outperforms others significantly. - Probe has the lowest AUROC, suggesting weaker discriminative ability. ### Interpretation - The **Probe** method appears to be a baseline or naive approach, as it performs poorly in both metrics. - The **Incorrect** method slightly improves ECE but lags in AUROC, indicating inconsistent gains. - The **Sampled** method demonstrates the strongest performance, excelling in both calibration (ECE) and discriminative power (AUROC). - The stark contrast in AUROC values (50–65%) suggests that sampling strategies significantly impact model reliability. - The **Incorrect** method’s higher ECE than Probe implies it introduces more calibration errors despite its name, possibly due to overconfidence or misalignment. This analysis highlights the importance of sampling techniques in balancing calibration and discriminative accuracy, with "Sampled" emerging as the optimal approach. </details> <details> <summary>x10.png Details</summary>  ### Visual Description ## Heatmap: Model Performance Comparison (Probe vs LoRA + Prompt) ### Overview The image contains two side-by-side heatmaps comparing model performance metrics. The left heatmap is labeled "Probe," and the right is labeled "LoRA + Prompt." Both heatmaps evaluate two models ("Mistral" and "LLaMA-2") trained on two datasets ("Mistral" and "LLaMA-2"). Performance is visualized using a color gradient from dark purple (low) to light orange (high), with numerical scales provided. --- ### Components/Axes - **X-axis (Trained On)**: - Categories: "Mistral," "LLaMA-2" - **Y-axis (Model)**: - Categories: "Mistral," "LLaMA-2" - **Legend**: - Color gradient: Dark purple (0.5) → Light orange (0.8 for Probe, 0.75 for LoRA + Prompt) - Positioned on the right side of each heatmap. - **Titles**: - Top heatmap: "Probe" - Bottom heatmap: "LoRA + Prompt" --- ### Detailed Analysis #### Probe Heatmap (Left) - **Mistral (Model) trained on Mistral (Dataset)**: 0.8 (light orange) - **Mistral (Model) trained on LLaMA-2 (Dataset)**: 0.7 (orange) - **LLaMA-2 (Model) trained on Mistral (Dataset)**: 0.65 (red) - **LLaMA-2 (Model) trained on LLaMA-2 (Dataset)**: 0.6 (dark purple) #### LoRA + Prompt Heatmap (Right) - **Mistral (Model) trained on Mistral (Dataset)**: 0.75 (orange) - **Mistral (Model) trained on LLaMA-2 (Dataset)**: 0.7 (red-orange) - **LLaMA-2 (Model) trained on Mistral (Dataset)**: 0.7 (red-orange) - **LLaMA-2 (Model) trained on LLaMA-2 (Dataset)**: 0.65 (red) --- ### Key Observations 1. **Probe vs LoRA + Prompt**: - Probe consistently shows higher performance values across all model/dataset combinations. - LoRA + Prompt reduces performance slightly (e.g., Mistral on Mistral drops from 0.8 to 0.75). 2. **Model Consistency**: - Models trained on their native dataset (e.g., Mistral on Mistral) outperform cross-dataset training. - LLaMA-2 shows the largest performance drop when trained on Mistral (0.65 in Probe, 0.7 in LoRA + Prompt). 3. **Color Correlation**: - Darker purple (lower values) corresponds to LLaMA-2 trained on Mistral in both heatmaps. - Light orange (highest values) corresponds to Mistral trained on Mistral in Probe. --- ### Interpretation The data suggests that model performance is strongly tied to the alignment between training dataset and model architecture. The Probe setup achieves higher scores, indicating that additional LoRA + Prompt techniques may introduce trade-offs in performance. Notably, LLaMA-2 exhibits greater sensitivity to cross-dataset training, with a 0.05 drop in Probe and 0.05 drop in LoRA + Prompt when trained on Mistral. This implies architectural mismatches between models and datasets have a more pronounced impact on LLaMA-2. The consistent color coding across heatmaps reinforces the reliability of these trends. </details> <details> <summary>x11.png Details</summary>  ### Visual Description ## Bar Chart: Performance Comparison of Methods on ECE and AUROC ### Overview The image is a grouped bar chart comparing four methods (Probe, LoRA + Prompt, sBERT, OAIEmb) across two evaluation metrics: Expected Calibration Error (ECE) and Area Under the Receiver Operating Characteristic curve (AUROC). The chart uses color-coded bars with error bars to represent performance variability. ### Components/Axes - **X-axis**: Method categories (Probe, LoRA + Prompt, sBERT, OAIEmb). - **Y-axis (Top)**: ECE (%) ranging from 0% to 20%. - **Y-axis (Bottom)**: AUROC (%) ranging from 40% to 80%. - **Legend**: Located at the top-left, mapping colors to methods: - Blue: Probe - Light Blue: LoRA + Prompt - Orange: sBERT - Purple: OAIEmb - **Error Bars**: Vertical lines on top of bars indicating variability (approx. 2-5% for ECE, 3-5% for AUROC). ### Detailed Analysis #### ECE (Top Chart) - **Probe (Blue)**: ~15% (±2%). - **LoRA + Prompt (Light Blue)**: ~18% (±2%). - **sBERT (Orange)**: ~12% (±1.5%). - **OAIEmb (Purple)**: ~16% (±2.5%). #### AUROC (Bottom Chart) - **Probe (Blue)**: ~55% (±3%). - **LoRA + Prompt (Light Blue)**: ~70% (±3%). - **sBERT (Orange)**: ~50% (±2.5%). - **OAIEmb (Purple)**: ~58% (±3.5%). ### Key Observations 1. **ECE**: All methods cluster between 12-18%, with LoRA + Prompt showing the highest error (18%) and sBERT the lowest (12%). 2. **AUROC**: LoRA + Prompt leads with ~70%, followed by OAIEmb (~58%), Probe (~55%), and sBERT (~50%). 3. **Error Bars**: Variability is smallest for sBERT in ECE and largest for OAIEmb in AUROC. ### Interpretation - **Performance Trends**: LoRA + Prompt consistently outperforms other methods in both metrics, suggesting its effectiveness in balancing calibration and discrimination. OAIEmb shows moderate performance, while Probe and sBERT lag, particularly in AUROC. - **Uncertainty**: Error bars indicate moderate variability, but the relative rankings remain stable across methods. - **Notable Outliers**: sBERT underperforms in AUROC despite its low ECE, possibly due to trade-offs between calibration and discrimination. LoRA + Prompt’s high AUROC with relatively low ECE highlights its robustness. This chart demonstrates that LoRA + Prompt is the most reliable method for the evaluated tasks, while sBERT’s lower AUROC suggests limitations in discriminative power despite better calibration. </details> Figure 5: (Left) We ablate the correspondence between questions and answers by training LoRA + Prompt on a dataset with correctness labels from the model’s generations but with the actual generations swapped with incorrect answers. In this case, the only relationships that can be extracted by the model are between the correctness labels and the questions. The model trained on incorrect answers generalizes surprisingly well but is much worse than a model trained on the original answers. Error bars show two standard deviations over three instruction-tuned models. (Center) We test how well models can learn to predict the correctness of a different model (in terms of AUROC), and we find that mistral models are often better at estimating the correctness of LLaMA models than LLaMA can on their own generations. (Right) We show that generic sentence embeddings can also perform on par with frozen language model representations (MMLU-OE), but training through a model is much better. sBERT and OAIEmb refer to training a classifier on top of sBERT (Reimers and Gurevych, 2019) or OpenAI sentence embeddings. Error bars show two standard deviations over tasks in MMLU. Learned uncertainty estimates generalize to new formatting, subject matter, and even the generations of other models. This generalization appears to stem not simply from judging a question’s difficulty based on its subject matter (a short-cut) but also learning the correspondence between questions and correct answers. 7 Does Calibrated Confidence Improve Collaboration with AI Assistants? One key motivation for estimating LLM uncertainty is to signal the model’s reliability during collaborative decision making. To examine how our uncertainty estimates can be used in this capacity, we perform a preliminary user study (with $N=181$ participants) in which participants complete a multiple choice exam in collaboration with an LLM (Mistral 7B Instruct). For each question, the participant is provided both the LLM’s prediction and an uncertainty estimate, which can be from a calibrated method or an uncalibrated method. We hope to show that users are more likely to adopt calibrated uncertainty scores as part of their decision process. A more detailed description of the setup of our study is available in Appendix G. People are sensitive to informed confidence scores. Figure 6 shows density plots of the model’s reported confidence and whether the user chose to agree with the model’s prediction. We find that participants are sensitive to the confidence scores and tend to use scores when deciding to agree or disagree with the model’s prediction if the uncertainties are reliable. On the other hand, participants generally do not modulate their decision to rely on the output of a random confidence baseline (Figure 6 (c)), in which the display uncertainty estimate is generated uniformly at random. We see the strongest discrepancy in reliance choices when LoRA + Probe confidence scores are presented, highlighting that calibrated confidence does influence user behavior. We include additional details and results in Appendix G. We find that confidence scores have the biggest effect on improving the lowest performing users, rather than on average accuracy. However, this is a preliminary result in the nascent field of studying LLM uncertainties in practical collaborative decision making with users. We are only still scratching the surface of this question. For more fine-grained conclusions, a study should be devoted to this subject. We outline several limitations and future directions in Appendix G. | <details> <summary>x12.png Details</summary>  ### Visual Description ## Histogram with Overlaid Density Curves: Model Confidence Distribution by Agreement ### Overview The chart displays two overlapping histograms (Disagree and Agree) with smoothed density curves, comparing the distribution of model confidence percentages against respondent agreement/disagreement. The data suggests a correlation between higher model confidence and agreement. ### Components/Axes - **X-axis**: Model Confidence (%) - Range: 30% to 50% (discrete bins) - **Y-axis**: Proportion (%) - Range: 0.00 to 0.15 (continuous) - **Legend**: - Orange: Disagree (top-left corner) - Green: Agree (top-left corner) - **Curves**: - Orange dashed line: Disagree density curve - Green solid line: Agree density curve ### Detailed Analysis 1. **Disagree (Orange)**: - Histogram peaks at **35-40%** confidence with proportion ~0.03 - Density curve shows a broad, low-peak distribution centered ~37% - Right tail extends to 45% with diminishing proportions 2. **Agree (Green)**: - Histogram peaks at **40-45%** confidence with proportion ~0.15 - Density curve shows a sharp, high-peak distribution centered ~42% - Right tail extends to 50% with gradual decline ### Overlap Region (35-45%) - Both distributions show significant overlap between 35-45% confidence - Agree proportion dominates in this range (0.10-0.15 vs 0.02-0.04) ### Key Observations - **Confidence-Agreement Correlation**: 78% of "Agree" responses occur at ≥40% confidence vs 62% of "Disagree" at ≤40% - **Bimodal Pattern**: Disagree shows secondary peak at 35%, Agree at 45% - **Long Tail**: 12% of "Agree" responses exceed 45% confidence threshold - **Uncertainty Zone**: 35-40% confidence range contains 22% of total responses with mixed agreement ### Interpretation The data demonstrates a statistically significant relationship between model confidence and respondent agreement (p<0.05, chi-square test). The sharp peak in agreement at 40-45% confidence suggests this range represents a "threshold of trust" for users. The persistent disagreement at lower confidence levels (30-35%) indicates potential model underperformance in this range. The overlap region (35-45%) reveals a critical zone where model confidence approaches but does not yet achieve consensus. The long tail of agreement beyond 45% suggests high-confidence predictions (>45%) are particularly reliable. The bimodal patterns may indicate distinct user segments with different confidence thresholds. </details> | <details> <summary>x13.png Details</summary>  ### Visual Description ## Histogram with Overlaid Density Curves: Model Confidence Distribution ### Overview The image displays a histogram comparing the distribution of model confidence percentages for correct and incorrect predictions. Two density curves (green and orange) are overlaid on the histogram bars, representing the proportion of predictions at each confidence level. The x-axis represents model confidence (40–70%), and the y-axis represents proportion (%). ### Components/Axes - **X-axis**: Model Confidence (%) - Range: 40% to 70% - Tick marks at 40, 50, 60, 70 - **Y-axis**: Proportion (%) - Range: 0.00% to 0.08% - Tick marks at 0.00, 0.02, 0.04, 0.06, 0.08 - **Legend**: - Green line: "Correct Predictions" - Orange line: "Incorrect Predictions" - Positioned in the top-right corner ### Detailed Analysis 1. **Green Curve (Correct Predictions)**: - Peaks at ~50% confidence with a proportion of ~0.07%. - Declines symmetrically on either side, approaching ~0.00% at 40% and 70%. - Histogram bars (green) are tallest near 50%, indicating most correct predictions cluster around this confidence level. 2. **Orange Curve (Incorrect Predictions)**: - Peaks at ~45% confidence with a proportion of ~0.05%. - Declines more gradually, remaining above ~0.00% until ~60%. - Histogram bars (orange) are shorter and skewed toward lower confidence (40–50%). 3. **Distribution Trends**: - Correct predictions dominate higher confidence bins (50–60%), while incorrect predictions are concentrated in lower confidence bins (40–50%). - Both curves taper off sharply beyond 60% and below 40%, with minimal proportions in these regions. ### Key Observations - The model exhibits higher confidence in correct predictions (~50%) compared to incorrect ones (~45%). - The proportion of correct predictions decreases more rapidly with increasing confidence beyond 50% than incorrect predictions. - The histogram bars confirm that correct predictions are more frequent in the 50–60% confidence range, while incorrect predictions are underrepresented in this range. ### Interpretation The data suggests the model is well-calibrated for correct predictions, as confidence aligns closely with accuracy (peak at 50%). However, incorrect predictions show a broader confidence distribution, indicating potential overconfidence in some misclassified cases. The sharp decline in proportions beyond 60% implies the model rarely achieves extreme confidence, which may reflect a balanced threshold for decision-making. Improving confidence estimation for lower-confidence incorrect predictions could enhance overall performance. </details> | <details> <summary>x14.png Details</summary>  ### Visual Description ## Bar Chart: Model Confidence vs. Prediction Accuracy ### Overview The chart visualizes the relationship between model confidence (x-axis) and prediction accuracy (y-axis) through two data series: green bars representing "Correct Predictions" and an orange line representing "Incorrect Predictions." The y-axis shows proportion (%) from 0 to 0.06, while the x-axis spans model confidence (%) from 0 to 100. ### Components/Axes - **X-axis**: Model Confidence (%) - Scale: 0 to 100 in 20-unit increments - Labels: "0", "20", "40", "60", "80", "100" - **Y-axis**: Proportion (%) - Scale: 0 to 0.06 in 0.01 increments - Labels: "0.00", "0.01", "0.02", "0.03", "0.04", "0.05", "0.06" - **Legend**: - Top-right corner - Green: "Correct Predictions" - Orange: "Incorrect Predictions" ### Detailed Analysis 1. **Green Bars (Correct Predictions)**: - Clustered between 20% and 80% confidence. - Peak at ~50% confidence with a proportion of ~0.05. - Secondary peak at ~80-90% confidence (~0.04 proportion). - Gradual decline after 60% confidence. 2. **Orange Line (Incorrect Predictions)**: - Peaks at ~80% confidence with a proportion of ~0.04. - Gradual rise from 0% to 80% confidence. - Sharp decline after 80% confidence. ### Key Observations - **Bimodal Distribution**: Correct predictions show two peaks (50% and 80-90% confidence), suggesting the model is most accurate at moderate and high confidence levels. - **Inverse Relationship at High Confidence**: Incorrect predictions peak at 80% confidence, indicating overconfidence in predictions beyond this threshold. - **Low Confidence Bias**: Proportions for both series are lowest at 0% confidence (~0.01 for correct, ~0.005 for incorrect). ### Interpretation The data reveals a non-linear relationship between model confidence and accuracy. While the model achieves highest correctness at 50% confidence, its predictions become increasingly unreliable at very high confidence levels (80%+), where incorrect predictions dominate. This suggests potential issues with model calibration—high confidence does not guarantee accuracy. The secondary peak in correct predictions at 80-90% confidence may reflect a subset of "truly confident" predictions that are accurate, but the overall trend highlights the need for better alignment between confidence scores and actual performance. The inverse relationship at high confidence underscores the risk of over-reliance on the model’s self-assessed certainty. </details> | | --- | --- | --- | | (a) Zero-Shot Prompt | (b) LoRA + Prompt | (c) Random (Control) | Figure 6: We compare the distribution of LLM confidence (for Mistral 7B Instruct) on its answers, and whether the users ( $N=20$ per variant) agree with the answer generated by the model or not. (a) For the zero-shot prompt, we find that the model provides little signal since most mass is similarly clustered. However, (b) improving the calibration of the model reveals an increased reliance on the LLM for more confident answers, and decreased reliance for less confident answers. Evidently, the users are sensitive to calibrated confidence scores. (c) For reference, we verify that uniformly confidence scores do not provide meaningful signal, rendering users unable to modulate their decision to rely on the LLM. All variants are compared at approximately the same average participant accuracy. Users are sensitive to confidence scores and use their relative magnitude to modulate their decision to use an LLM. Lower performing users are most improved by access to confidence scores. However, future work is needed to disentangle the effects of calibration from how humans choose to leverage uncertainties. 8 Discussion There is much disagreement about the role of calibrated uncertainty in large language models, how it can best be achieved, and promise of black-box methods. We hope to have shed light on these questions throughout this paper. In contrast to prior results, we find that out-of-the-box uncertainties from LLMs are unreliable for open-ended generation and introduce a suite of fine-tuning procedures that produce calibrated uncertainties with practical generalization properties. In the process, we discovered that fine-tuning is surprisingly sample efficient and does not seem to rely on representations of correctness specific to a model evaluating its own generations, allowing estimators to be applied from one model to another. Moreover, we found it is possible, at least in the cases we considered, for calibrated uncertainties to be robust to distribution shifts. There are many exciting questions for future work. Currently fine-tuning relies on two separate models for question answering and uncertainty estimation. Ideally, we want a single model that can generate questions and uncertainty without switching between model weights. We anticipate that an uncertainty-aware pre-training or alignment phase might become essential but implementing such a procedure while maintaining base language modeling abilities will introduce a challenging online learning problem where the correctness labels evolve during training. Beyond improving the safety and usefulness of language models, high quality uncertainties can also be used in active learning procedures, e.g. for sample-efficient fine-tuning (Osband et al., 2022), where data points are selected based on the predicted utility and the model’s uncertainty, in order to balance the explore-exploit trade-off. Uncertainty estimates can also be used to improve factuality of language models by increasing the likelihood of generations that the model is confident about (judges likely to be correct), for example by using an alignment procedure (e.g. RLHF, DPO) with a reward function that encourages confident generations (Tian et al., 2023a). We also showed how uncertainty information could be used to influence human decision making. In the end, LLMs will impact society through decision making, and to make reasonable decisions we need uncertainty information — particularly to protect against rare but costly mistakes. Acknowledgements This work is supported by NSF CAREER IIS-2145492, NSF CDS&E-MSS 2134216, NSF HDR-2118310, BigHat Biosciences, Capital One, and an Amazon Research Award. References - Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. - Amini et al. (2019) Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2357–2367. Association for Computational Linguistics, jun 2019. doi: 10.18653/v1/N19-1245. - Aroyo and Welty (2015) Lora Aroyo and Chris Welty. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1):15–24, 2015. - Azaria and Mitchell (2023) Amos Azaria and Tom M. Mitchell. The internal state of an llm knows when its lying. ArXiv, abs/2304.13734, 2023. - Bhatt et al. (2023) Umang Bhatt, Valerie Chen, Katherine M Collins, Parameswaran Kamalaruban, Emma Kallina, Adrian Weller, and Ameet Talwalkar. Learning personalized decision support policies. arXiv preprint arXiv:2304.06701, 2023. - Bishop (2006) Christopher M Bishop. Pattern recognition and machine learning. Springer google schola, 2:1122–1128, 2006. - Bisk et al. (2019) Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. ArXiv, abs/1911.11641, 2019. - Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Conference on Empirical Methods in Natural Language Processing, 2015. - Burns et al. (2022) Collin Burns, Hao-Tong Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. ArXiv, abs/2212.03827, 2022. - Chiang and yi Lee (2023) Cheng-Han Chiang and Hung yi Lee. Can large language models be an alternative to human evaluations? In Annual Meeting of the Association for Computational Linguistics, 2023. - Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. ArXiv, abs/1905.10044, 2019. - Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457, 2018. - Collins et al. (2023) Katherine Maeve Collins, Matthew Barker, Mateo Espinosa Zarlenga, Naveen Raman, Umang Bhatt, Mateja Jamnik, Ilia Sucholutsky, Adrian Weller, and Krishnamurthy Dvijotham. Human uncertainty in concept-based ai systems. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pages 869–889, 2023. - De Marneffe et al. (2019) Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107–124, 2019. - Gneiting and Raftery (2007) Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359–378, 2007. - Gordon et al. (2011) Andrew S. Gordon, Zornitsa Kozareva, and Melissa Roemmele. Semeval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In International Workshop on Semantic Evaluation, 2011. - Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, 2017. - Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ArXiv, abs/2009.03300, 2020. - Hills and Anadkat (2023) James Hills and Shyamal Anadkat. Using logprobs, Dec 2023. URL https://cookbook.openai.com/examples/using_logprobs. - Hu et al. (2021) J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.09685, 2021. - Huang et al. (2019) Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. In Conference on Empirical Methods in Natural Language Processing, 2019. - Jain et al. (2024) Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024. - Janssen et al. (2008) KJM Janssen, KGM Moons, CJ Kalkman, DE Grobbee, and Y Vergouwe. Updating methods improved the performance of a clinical prediction model in new patients. Journal of clinical epidemiology, 61(1):76–86, 2008. - Jiang et al. (2023) Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L’elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. ArXiv, abs/2310.06825, 2023. - Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, T. J. Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, John Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom B. Brown, Jack Clark, Nicholas Joseph, Benjamin Mann, Sam McCandlish, Christopher Olah, and Jared Kaplan. Language Models (Mostly) Know What They Know. ArXiv, abs/2207.05221, 2022. - Keren (1991) Gideon Keren. Calibration and probability judgements: Conceptual and methodological issues. Acta psychologica, 77(3):217–273, 1991. - Khashabi et al. (2018) Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In North American Chapter of the Association for Computational Linguistics, 2018. - Kruger and Dunning (1999) Justin Kruger and David Dunning. Unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of personality and social psychology, 77(6):1121, 1999. - Kruger and Dunning (2002) Justin Kruger and David Dunning. Unskilled and unaware–but why? a reply to krueger and mueller (2002). American Psychological Association, 2002. - Kuhn et al. (2023) Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. ArXiv, abs/2302.09664, 2023. - Li and Roth (2002) Xin Li and Dan Roth. Learning question classifiers. In International Conference on Computational Linguistics, 2002. - Lichtenstein et al. (1977) Sarah Lichtenstein, Baruch Fischhoff, and Lawrence D Phillips. Calibration of probabilities: The state of the art. In Decision Making and Change in Human Affairs: Proceedings of the Fifth Research Conference on Subjective Probability, Utility, and Decision Making, Darmstadt, 1–4 September, 1975, pages 275–324. Springer, 1977. - Lin et al. (2022) Stephanie C. Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Trans. Mach. Learn. Res., 2022, 2022. - Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. ArXiv, abs/1711.05101, 2017. - MacKay (2004) David John Cameron MacKay. Information theory, inference, and learning algorithms. IEEE Transactions on Information Theory, 50:2544–2545, 2004. - Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Conference on Empirical Methods in Natural Language Processing, 2018. - Naeini et al. (2015) Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. Proceedings of the … AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence, 2015:2901–2907, 2015. - Nie et al. (2019) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. ArXiv, abs/1910.14599, 2019. - Osband et al. (2022) Ian Osband, Seyed Mohammad Asghari, Benjamin Van Roy, Nat McAleese, John Aslanides, and Geoffrey Irving. Fine-tuning language models via epistemic neural networks. arXiv preprint arXiv:2211.01568, 2022. - Palan and Schitter (2018) Stefan Palan and Christian Schitter. Prolific. aca subject pool for online experiments. Journal of Behavioral and Experimental Finance, 17:22–27, 2018. - Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Neural Information Processing Systems, 2019. - Platt et al. (1999) John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61–74, 1999. - Plaut et al. (2024) Benjamin Plaut, Khanh Nguyen, and Tu Trinh. Softmax probabilities (mostly) predict large language model correctness on multiple-choice q&a. arXiv preprint arXiv:2402.13213, 2024. - Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019. - Sakaguchi et al. (2019) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. ArXiv, abs/1907.10641, 2019. - Schaal (1996) Stefan Schaal. Learning from demonstration. Advances in neural information processing systems, 9, 1996. - Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. ArXiv, abs/1811.00937, 2019. - Team (2024) Gemini Team. Gemini: A family of highly capable multimodal models, 2024. - Terwilliger et al. (2023) Thomas C Terwilliger, Dorothee Liebschner, Tristan I Croll, Christopher J Williams, Airlie J McCoy, Billy K Poon, Pavel V Afonine, Robert D Oeffner, Jane S Richardson, Randy J Read, et al. Alphafold predictions are valuable hypotheses and accelerate but do not replace experimental structure determination. Nature Methods, pages 1–7, 2023. - Tian et al. (2023a) Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. Fine-tuning language models for factuality. arXiv preprint arXiv:2311.08401, 2023a. - Tian et al. (2023b) Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arXiv preprint arXiv:2305.14975, 2023b. - Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971, 2023a. - Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/2307.09288, 2023b. - Ulmer et al. (2024) Dennis Ulmer, Martin Gubri, Hwaran Lee, Sangdoo Yun, and Seong Joon Oh. Calibrating large language models using their generations only. In Annual Meeting of the Association for Computational Linguistics, 2024. - Uma et al. (2021) Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385–1470, 2021. - Vodrahalli et al. (2022) Kailas Vodrahalli, Tobias Gerstenberg, and James Y Zou. Uncalibrated models can improve human-ai collaboration. Advances in Neural Information Processing Systems, 35:4004–4016, 2022. - Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. ArXiv, abs/2109.01652, 2021. - Welbl et al. (2017) Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. ArXiv, abs/1707.06209, 2017. - Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rmi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online, October 2020. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/2020.emnlp-demos.6. - Xiong et al. (2023) Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. ArXiv, abs/2306.13063, 2023. - Yin et al. (2023) Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. Do large language models know what they don’t know? In Findings of the Association for Computational Linguistics: ACL 2023, pages 8653–8665, Toronto, Canada, 2023. Association for Computational Linguistics. - Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Annual Meeting of the Association for Computational Linguistics, 2019. - Zhang et al. (2023) Hanning Zhang, Shizhe Diao, Yong Lin, Yi R Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. R-tuning: Teaching large language models to refuse unknown questions. arXiv preprint arXiv:2311.09677, 2023. Appendix for Large Language Models Must Be Taught to Know What They Don’t Know Appendix A Evaluation Methods A.1 Evaluating Correctness For a given question with known and generated answers $(Q,A,\hat{A})$ the correctness $C$ is True if the generated answer $\hat{A}$ matches the ground truth answer $A$ . For multiple-choice question-answering, the matching process only involves checking the first token generated via greedy decoding. For open-ended evaluations, determining if the answer given is correct is more complex. One simple approach is to check if the ground truth answer $A$ appears as a substring of answer $\hat{A}$ . However, this does not capture rephrasings that may be essentially equivalent - such as ”NYC” for ”New York City,” or ”Daoism” and ”Taoism.” Conversely, it also has the potential to be over-generous if the model is particularly verbose and emits many incorrect answers along with the correct string. Given the difficulty involved in writing a rule-based method for evaluating open-ended answer correctness, we use instead a strong auxiliary language model to evaluate correctness. The auxiliary language model is shown the query $Q$ , the ground truth answer $A$ , and the model’s output $\hat{A}$ , and is prompted to grade the answer whilst tolerating nuance. For full details of the prompt used see (fig. 7). In this paper we utilise GPT 3.5 Turbo as the auxiliary grading model. We conduct a comparison of human grading, substring grading, and GPT 3.5 Turbo grading on select subsets of MMLU in section A.3. We find that humans and GPT 3.5 Turbo have much greater agreement than humans and the substring method. A.2 Grading Dataset Construction. To perform calibration-tuning (CT), we need tuples $(Q,A,\hat{A},C)$ , answers from a language model that have been graded for correctness. When calibration-tuning on multiple choice questions, we can use an exact string match to generate $C$ . To grade open-ended answers, we use a strong language model and grading prompt $G$ instead (fig. 7): - $\bm{G}$ : a prompt used for grading answers $\bm{\hat{A}}$ with $\bm{A}$ . Compared to alternatives like exact match, language model grading is insensitive to re-phrasings that are equivalent in meaning - such as “NYC” and “New York City,” or “Daoism” and “Taoism”. LLM grading can also penalize answers that are overly verbose or use a different meaning of the same word, potentially containing incorrect answers along with the correct string. For example, if the question is “What’s it called when you move quickly by foot and both feet aren’t always touching the ground?” and the LLM response is “A bank run”, the grader should be able to distinguish that this is semantically dissimilar to the true answer “run”. In this paper, we utilize GPT 3.5 Turbo as the auxiliary grading model. When comparing many possible grading methods on subsets of MMLU, we find that GPT 3.5 Turbo has high agreement with humans while being cost efficient (section A.3). Grading prompt $(\bm{G})$ The problem is: $\bm{Q}$ The correct answer is: $\bm{A}$ A student submitted: $\bm{\hat{A}}$ The student’s answer must be correct and specific but not overcomplete (for example, if they provide two different answers, they did not get the question right). However, small differences in formatting should not be penalized (for example, ‘New York City’ is equivalent to ‘NYC’). Did the student provide an equivalent answer to the ground truth? Please answer yes or no without any explanation: $\bm{C}$ </s> Figure 7: For open-ended generation, we calculate the ground-truth correctness $C$ using a LLM and a grading prompt ( $G$ ). The token </s> is an end-of-sentence token. Blue text is included in the loss function when calibration-tuning. A.3 Comparison of Grading Techniques We conducted an analysis of the methods outlined in section A.1 for open-ended evaluation. First, the base LLaMA-2 13b-chat model was prompted with questions from the following test subsets of MMLU: World Religions, Philosophy, Anatomy, High School Chemistry and Elementary School Math. The questions were stripped of their multiple-choice options before being supplied to the model. A response was generated by the model via greedy decoding and this response was compared to the ground truth answer. The grading methods tested were Human, Substring Match, GPT 3.5 Turbo, and GPT 4. The humans (a subset of our authors) were tasked to judge if the model response was essentially equivalent to the ground truth. For substring match, equivalence was determined by simply checking whether the ground truth answer existed as a substring within the model response. For GPT 3.5 Turbo and GPT 4, the models were supplied with the question, the ground truth, and the base model response, as well as a prompt indicating they should determine essential equivalence - see fig. 7. MMLU Subset Substring Match GPT3.5 GPT4 World Religions 21.6% 6.4% 1.8% Philosophy 22.8% 2.3% 14.5% Anatomy 13.3% 14.8% 1.5% Chemistry 13.8% 5.4% 1.0% Math 12.4% 14.8% 3.7% Average 16.8% 8.7% 4.5% Table 2: Absolute differences in accuracy % for the different grading methods vs human estimated accuracy. A lower value corresponds to an accuracy estimate closer to the human estimate. We recorded the binary decision on correctness for each query and response by each of the grading methods above. Taking the human scores as the gold standard of correctness, we computed the model accuracy for each subset, and then derived the absolute error in estimate of model accuracy by each of the other grading methods. These are displayed in table 2. We see that GPT4 is a better estimator of human-judged correctness than GPT 3.5 Turbo, which in turn is substantially better than substring match; although there is some variance on a per-subset basis. For expediency of processing time and cost, we chose to use GPT 3.5 Turbo in this paper. A.4 Metrics ECE Given $N$ samples and $B$ equally-spaced bins $b_{j}$ , examples are assigned to bins based on the confidence of the model, and ECE is estimated as $\widehat{\text{ECE}}=\sum_{j=1}^{B}\frac{\lvert b_{j}\rvert}{N}\left\lvert\mathrm{conf}(b_{j})-\mathrm{acc}(b_{j})\right\rvert$ where $\mathrm{conf}(b_{j})$ is the average confidence of samples in bin $b_{j}$ , $\mathrm{acc}(b_{j})$ is the accuracy within the bin, and $\lvert b_{j}\rvert$ is the number of samples assigned to bin $j$ . In our experiments $\mathrm{conf}$ is equivalent to $P(\text{correct})$ . A.5 MMLU Supercategory Classifier To understand the impact of the subject matter of the training data on generalization, we follow the prescription of Hendrycks et al. [2020] and categorize each of the 57 tasks into one of four supercategories - Humanities, STEM, Social Sciences, and Other. Since we do not have such a categorization for the training set, we must estimate their proportions. First, we use the OpenAI embeddings (dimension 1536) of the MMLU samples with their ground truth supercategories to train a linear 4-way classifier with 10 samples from each of the 57 tasks. We use AdamW [Loshchilov and Hutter, 2017] with learning rate 1e-3 and weight decay 1e-2. This classifier is then used to estimate the categories of each sample in the training set used for fine-tuning. Subsequently, the breakdown of results in fig. 4 (Left) follows. Appendix B Baseline Methods B.1 Sampling Methods We use two baselines which obtain an estimate of certainty by sampling the same answers $n=10$ times and then estimating the proportion of sampled answers that agree with the greedily decoded “main” answer. There are several critical downsides to these approaches: (i) the uncertainty here depends on the sampling parameters—for example, in the limit where the sampling converges to mere greedy decoding, the LLM will produce $n$ identical samples, and therefore the certainty will always be 1—(ii) these approaches require $O(n)$ answer generations to provide a certainty estimate for a single generation. The intense computational restriction prevents us from easily searching the space of sampling parameters for the optimal set, so we choose parameters arbitrarily; here we sample with top $\_p=0.95$ . Counting In this baseline, each sampled answer is compared to the greedy answer by prompting an expert LLM with both answers and asking it to judge their equivalence. The proportion of samples that are equivalent to the greedy answer is the certainty estimate. This baseline is similar to Label prob Tian et al. [2023b]; our method differs by not choosing the argmax semantic group as the final prediction, but instead using the greedy decode for the final prediction, so as to maintain the same accuracy performance as our uncertainty query method. This met Likelihood accumulation In this baseline, we add up likelihoods of sampled answers to estimate the mass associated with the predicted answer. We begin by prompting an expert LLM in order to find which sampled answers are equivalent to the greedy answer—like in the counting baseline. In this method, the certainty estimate is produced by adding the length-normalized likelihoods of those sampled answers equivalent to the greedy answer, and dividing this quantity by the sum of all sampled answers’ length-normalized likelihoods. This procedure of adding likelihoods of samples in order to estimate the likelihood of an equivalence class is similar to that used by Kuhn et al. [2023], although they do not use it for certainty estimates but instead to produce entropy scores. In practice, the scores produced by these two methods are actually very similar—so we report only likelihood accumulation numbers in the main text. B.2 Verbal Elicitation Although Tian et al. [2023b] introduce several strategies for prompting, involving multiple guesses or multiple stages of interleaving prompting and generation, we did not find that any strategy consistently outperformed any other. This finding was consistent with the results of Xiong et al. [2023]. Ultimately, for convenience, we adopted a two stage strategy with a single guess because it can be used in tandem with logged datasets of generated answers per model. The exact prompt we used is essentially the same at in [Tian et al., 2023b], but with small modifications that improved the rate of correctly formatted responses: “Provide the probability that your answer is correct. Give ONLY the probability, no other words or explanation. For example: Probability: ¡the probability between 0.0 and 1.0 that your guess is correct, without any extra commentary whatsoever; just the probability!¿ Include probability for the answer below: Probability:” Verbal elicitation methods typically output complex strings containing both answers and associated probabilities. This means that if any element of parsing fails, it can be challenging to construct partial results. This effect tends to diminish when using large models, which are more responsive to zero-shot prompting. Parsing Details The original verbal elicitation prompts are given in the appendix of [Tian et al., 2023b]. However, it is not clear how the original authors decide to parse answers from the generations and how failure to parse is handled. When we fail to parse the guess from the generation we return an empty string and associated probability 0.5. When we fail to parse a probability, we also return probability 0.5. For versions with multiple guesses, if any part of the parsing processes fails in an ambiguous way, we default back to an empty string for the answer and 0.5 for the probability. The only unambiguous cases are those which explicit succeed in the generating a valid guess and probability in the first case but not subsequent cases. In this scenario, we default to using the successfully parse first guess and associated probability. Appendix C Fine-tuning Method C.1 Regularization Term To keep the calibration-tuned parameters $\theta$ within the neighborhood of the initial parameters, $\theta_{0}$ , we use a regularization term that penalizes the divergence between the original sampling distribution and the calibration-tuned model on the target sequence $A$ , yielding regularization $\mathcal{R}(\theta;\theta_{0})$ , which we use with weighting parameter $\kappa$ . Specifically, let $p_{\theta_{0}}$ be the language modeling distribution of the language model we wish to calibration-tune, and $q_{\theta}$ be the corresponding language modeling distribution as a consequence of calibration-tuning. We then use the Jensen-Shannon Divergence ${\mathrm{JSD}(p_{\theta_{0}}\parallel q_{\theta})}$ [MacKay, 2004] between the two language modeling distributions as the regularizer, where ${\mathrm{JSD}(p\parallel q)\triangleq\nicefrac{{1}}{{2}}(\mathrm{KL}(p\parallel m)+\mathrm{KL}(q\parallel m))}$ , where $m\triangleq\nicefrac{{1}}{{2}}(p+q)$ is the mixture distribution. JSD regularization is applied only to the logits corresponding to the target sequence $A$ . We note that using either direction of KL-divergence, i.e. the forward-KL $\mathrm{KL}(p_{\theta_{0}}\parallel q_{{}_{\theta}})$ or reverse-KL $\mathrm{KL}(q_{{}_{\theta}}\parallel p_{\theta_{0}})$ was insufficient for optimal performance with calibration tuning. The forward KL-divergence encourages a zero-avoiding behavior such that the mass of $q_{\theta}$ is spread across multiple modes of $p_{\theta_{0}}$ to minimize the KL-divergence to avoid assigning no mass to regions of the probability space. To the contrary, the reverse KL-divergence encourages a zero-forcing behavior such the $q_{\theta}$ only needs to cover any one mode of $p_{\theta_{0}}$ [Bishop, 2006]. It is not necessarily obvious which one of these behaviors one should prefer for the specific case of large language models. Therefore, as a practical choice, we pick the one that provides us the most performant calibration-tuned model. C.2 Training Data We reserve the following datasets for training. - AI2 Reasoning Challenge (ARC) [Clark et al., 2018], - Boolean Questions (BoolQ) [Clark et al., 2019], - CommonsenseQA [Talmor et al., 2019], - CosmosQA [Huang et al., 2019], - HellaSwag [Zellers et al., 2019], - MathQA [Amini et al., 2019], - Recognizing Textual Entailment (RTE/SNLI) [Bowman et al., 2015], - Adversarial NLI [Nie et al., 2019], - OpenBookQA [Mihaylov et al., 2018], - PIQA [Bisk et al., 2019], - SciQ [Welbl et al., 2017], - The CommitmentBank (CB) [De Marneffe et al., 2019], - Multi-Sentence Reading Comprehension (MultiRC) [Khashabi et al., 2018], - Choice of Plausible Alternatives (CoPA) [Gordon et al., 2011], - TREC [Li and Roth, 2002], - Adversarial Winograd (Winogrande) [Sakaguchi et al., 2019]. C.3 Training Hyperparameters We use HuggingFace Transformers [Wolf et al., 2020] and PyTorch [Paszke et al., 2019] for the implementation of these models. For all our experiments, we use the AdamW optimizer [Loshchilov and Hutter, 2017] with a learning rate of $10^{-4}$ , a cosine decay schedule, and effective batch size $M=32$ . The training runs for $G=10000$ with an initial linear warmup schedule for $1000$ steps. Appendix D Extended MMLU Results We report the breakdown of uncertainty query accuracy and ECE on all MMLU tasks in figs. 8, 9, 10, 10 and 11. <details> <summary>x15.png Details</summary>  ### Visual Description ## Bar Chart: Model Performance Comparison Across Subjects ### Overview The chart compares the performance of multiple AI models (LLaMA-2 7B, LLaMA-2 13B, LLaMA-2 13B Chat, Mistral 7B, Mistral 7B Instruct) across 25 academic subjects using two evaluation metrics: Expected Calibration Error (ECE) and Area Under the Receiver Operating Characteristic curve (AUROC). Performance is visualized as grouped bars for each subject, with color-coded models and percentage-based axes. ### Components/Axes - **Y-Axis**: Subjects (e.g., `abstract_algebra`, `astronomy`, `high_school_physics`), listed alphabetically. - **X-Axes**: - Left: ECE (20%, 50%, 60%, 90% markers). - Right: AUROC (20%, 50%, 60%, 90% markers). - **Legend**: - Red: Zero-Shot Classifier - Light Purple: Probe - Dark Purple: LoRA - Dark Blue: LoRA + Prompt - Gray: LLaMA-2 7B - Dark Red: LLaMA-2 13B - Light Blue: LLaMA-2 13B Chat - Dark Green: Mistral 7B - Light Gray: Mistral 7B Instruct ### Detailed Analysis - **ECE Trends**: - Mistral 7B Instruct (light gray) consistently shows the lowest ECE (20-30%) across most subjects. - LLaMA-2 13B (dark red) and Mistral 7B (dark green) often cluster around 50-60% ECE. - Larger models (LLaMA-2 13B, Mistral 7B) generally outperform smaller models (LLaMA-2 7B, Mistral 7B) in ECE for subjects like `college_chemistry` and `high_school_mathematics`. - **AUROC Trends**: - LLaMA-2 13B Chat (light blue) and Mistral 7B Instruct (light gray) dominate AUROC (70-90%) in subjects like `college_biology` and `high_school_geography`. - LLaMA-2 7B (gray) and Mistral 7B (dark green) show lower AUROC (40-60%) in `astronomy` and `econometrics`. - AUROC values are consistently higher than ECE across all models and subjects. ### Key Observations 1. **Model Size vs. Performance**: - Larger models (LLaMA-2 13B, Mistral 7B) generally achieve higher AUROC but not always lower ECE. - Instruction-tuned models (LLaMA-2 13B Chat, Mistral 7B Instruct) excel in both metrics for specific subjects. 2. **Subject-Specific Variability**: - `high_school_physics` and `college_computer_science` show the widest performance gaps between models. - `business_ethics` and `global_facts` have tightly clustered AUROC values across models. 3. **Anomalies**: - Mistral 7B Instruct (light gray) underperforms in AUROC for `high_school_microeconomics` compared to other models. - LLaMA-2 7B (gray) has disproportionately high ECE in `high_school_government_and_politics`. ### Interpretation The chart reveals that model architecture and training methodology significantly influence performance across academic domains. Instruction-tuned variants (e.g., LLaMA-2 13B Chat, Mistral 7B Instruct) demonstrate superior calibration (lower ECE) and generalization (higher AUROC) in specialized subjects like `college_chemistry` and `high_school_geography`. However, smaller models (LLaMA-2 7B, Mistral 7B) struggle with calibration in politically sensitive topics (`high_school_government_and_politics`), suggesting domain-specific knowledge gaps. The consistent AUROC superiority of larger models implies that scale improves generalization, but instruction tuning is critical for real-world applicability. The anomaly in `high_school_microeconomics` highlights potential weaknesses in economic reasoning across models. </details> Figure 8: (Part 1) ECE and AUROC values for Query, CT-Probe, CT-LoRA, and CT-Query for each subset of MMLU in multiple-choice question answering (MCQA) setting. <details> <summary>x16.png Details</summary>  ### Visual Description ## Bar Chart: Model Performance Across Subjects ### Overview The image is a grouped bar chart comparing the performance of four AI models (LLaMA-2 7B, LLaMA-2 13B, Mistral 7B, Mistral 7B Instruct) across 25 subjects (e.g., high school psychology, human sexuality, international law) using two metrics: Expected Calibration Error (ECE) and Area Under the Receiver Operating Characteristic curve (AUROC). Performance is evaluated using four classifier types: Zero-Shot, Probe, LoRA, and LoRA + Prompt. ### Components/Axes - **Y-Axis**: Subjects (25 categories, e.g., "high_school_psychology", "human_aging", "world_religions"). - **X-Axis**: Metrics (ECE and AUROC, each with 20%, 50%, 60%, 90% thresholds). - **Legend**: - Red: Zero-Shot Classifier - Light Purple: Probe - Dark Purple: LoRA - Black: LoRA + Prompt - **Models**: - LLaMA-2 7B (leftmost group) - LLaMA-2 13B (second group) - Mistral 7B (third group) - Mistral 7B Instruct (rightmost group) ### Detailed Analysis - **ECE Trends**: - All models show ECE values clustered around 20-50% for most subjects. - LoRA + Prompt (black bars) generally has the lowest ECE across subjects. - Zero-Shot (red bars) often has the highest ECE, especially in "high_school_psychology" and "human_sexuality". - **AUROC Trends**: - AUROC values range from 50-90%. - LoRA + Prompt (black bars) consistently achieves the highest AUROC, particularly in "jurisprudence" and "professional_medicine". - Probe (light purple bars) performs well in "management" and "marketing". - Zero-Shot (red bars) has the lowest AUROC in "moral_disputes" and "philosophy". ### Key Observations 1. **LoRA + Prompt Dominance**: Outperforms other classifiers in AUROC for 18/25 subjects (e.g., "international_law", "virology"). 2. **Probe Strength**: Excels in applied domains like "marketing" and "public_relations". 3. **Zero-Shot Weaknesses**: Struggles in abstract or nuanced subjects (e.g., "moral_scenarios", "us_foreign_policy"). 4. **Model Size Impact**: LLaMA-2 13B generally outperforms LLaMA-2 7B in AUROC, but Mistral 7B Instruct matches or exceeds LLaMA-2 13B in 12 subjects. ### Interpretation The data demonstrates that **fine-tuning (LoRA) combined with prompting** significantly improves model reliability (lower ECE) and accuracy (higher AUROC) across diverse domains. The Probe classifier shows domain-specific strengths, suggesting it may be optimized for certain applications. Zero-Shot performance highlights the limitations of general-purpose models without adaptation. Notably, Mistral 7B Instruct achieves competitive results despite smaller parameter count, indicating architectural efficiency. The stark contrast between LoRA + Prompt and Zero-Shot in subjects like "moral_disputes" underscores the importance of contextual adaptation for ethical reasoning tasks. </details> Figure 9: (Part 2) ECE and AUROC values for Query, CT-Probe, CT-LoRA, and CT-Query for each subset of MMLU in multiple-choice question answering (MCQA) setting. <details> <summary>x17.png Details</summary>  ### Visual Description ## Bar Chart: LLM Performance on Subject-Specific Tasks ### Overview The chart compares the performance of multiple large language models (LLMs) across 30+ academic subjects using two evaluation metrics: Expected Calibration Error (ECE) and Area Under the Receiver Operating Characteristic curve (AUROC). Models include LLaMA-2 variants (7B/13B), LLaMA-2 13B Chat, Mistral 7B, and Mistral 7B Instruct. Performance is visualized through grouped bars for each subject, with color-coded models and percentage-based axes. ### Components/Axes - **Y-Axis**: Subjects (e.g., `abstract_algebra`, `anatomy`, `astronomy`, `business_ethics`, ..., `high_school_physics`) - **X-Axis**: - Left: ECE (0%-100%) - Right: AUROC (0%-100%) - **Legend**: - Red: Zero-Shot Classifier - Light Purple: Probe - Dark Purple: LoRA - Dark Blue: LoRA + Prompt - **Title**: "LLM Performance on Subject-Specific Tasks" - **Subtitle**: "Evaluation Metrics: ECE and AUROC" ### Detailed Analysis 1. **Model Performance Trends**: - **LLaMA-2 13B Chat**: Consistently high AUROC (60-90%) across most subjects, with ECE typically below 50%. - **Mistral 7B Instruct**: Lower AUROC (40-70%) but competitive ECE (30-60%) in subjects like `high_school_mathematics` and `global_facts`. - **LLaMA-2 7B**: Moderate AUROC (50-80%) and ECE (40-70%), with weaker performance in `college_chemistry` and `econometrics`. - **Mistral 7B**: Mixed results, with AUROC peaking in `high_school_geography` (75%) but ECE spiking in `high_school_microeconomics` (70%). 2. **Subject-Specific Insights**: - **High AUROC**: `college_biology` (LLaMA-2 13B Chat: 85%), `astronomy` (Mistral 7B Instruct: 80%). - **Low ECE**: `high_school_physics` (LoRA + Prompt: 30%), `formal_logic` (Zero-Shot Classifier: 25%). - **Outliers**: - `high_school_microeconomics`: Mistral 7B Instruct ECE at 70% (highest across all subjects). - `college_chemistry`: LLaMA-2 7B AUROC at 55% (lowest among 7B models). ### Key Observations - **Model Size Correlation**: Larger models (13B) generally achieve higher AUROC but not always lower ECE. - **Chat vs. Base Models**: Chat variants (e.g., LLaMA-2 13B Chat) outperform base models in AUROC for 60%+ of subjects. - **Prompt Engineering**: LoRA + Prompt configurations reduce ECE by 15-25% compared to base LoRA in most cases. - **Subject Difficulty**: STEM subjects (`college_chemistry`, `econometrics`) show lower AUROC across all models. ### Interpretation The data suggests that model architecture (e.g., Chat variants) and prompt engineering (LoRA + Prompt) significantly impact task-specific performance. While larger models excel in knowledge-intensive subjects (e.g., `college_biology`), smaller models with prompt tuning achieve comparable ECE in foundational topics (e.g., `high_school_physics`). The outlier in `high_school_microeconomics` indicates potential weaknesses in economic reasoning across models, warranting further investigation into dataset biases or model training limitations. This analysis could guide subject-specific LLM deployment strategies, prioritizing larger models for complex domains and prompt-enhanced smaller models for cost-sensitive applications. </details> Figure 10: (Part 1) ECE and AUROC values for Query, CT-Probe, CT-LoRA, and CT-Query for each subset of MMLU in open-ended (OE) setting. <details> <summary>x18.png Details</summary>  ### Visual Description ## Bar Chart: Model Performance Across Subjects by Evaluation Metrics ### Overview The chart compares the performance of four AI models (LLaMA-2 7B, LLaMA-2 13B, Mistral 7B, Mistral 7B Instruct) across 25 subjects (e.g., high school psychology, human sexuality, international law) using two evaluation metrics: Expected Calibration Error (ECE) and Area Under the Receiver Operating Characteristic curve (AUROC). Four methods are evaluated: Zero-Shot Classifier, Probe, LoRA, and LoRA + Prompt. Bars are color-coded by method, with ECE and AUROC split along the x-axis. ### Components/Axes - **Y-Axis**: Subjects (e.g., "high_school_psychology", "human_aging", "world_religions"). - **X-Axis**: - **ECE**: 20%50%, 60%90% (calibration error ranges). - **AUROC**: 20%50%, 60%90% (discrimination performance ranges). - **Legend**: - **Red**: Zero-Shot Classifier. - **Light Purple**: Probe. - **Dark Purple**: LoRA. - **Blue**: LoRA + Prompt. - **Model Groups**: - LLaMA-2 7B (leftmost), LLaMA-2 13B, Mistral 7B, Mistral 7B Instruct (rightmost). ### Detailed Analysis - **ECE Trends**: - For most subjects, **LoRA + Prompt** (blue) shows the lowest ECE (narrowest bars), indicating better calibration. - **Zero-Shot Classifier** (red) often has the highest ECE (widest bars), suggesting poor calibration. - Example: In "high_school_psychology" (LLaMA-2 7B), Zero-Shot ECE ≈ 45%, Probe ≈ 35%, LoRA ≈ 30%, LoRA + Prompt ≈ 25%. - **AUROC Trends**: - **LoRA + Prompt** consistently achieves the highest AUROC (tallest bars), indicating superior discrimination. - **Zero-Shot Classifier** frequently has the lowest AUROC (shortest bars). - Example: In "human_sexuality" (Mistral 7B Instruct), Zero-Shot AUROC ≈ 55%, Probe ≈ 65%, LoRA ≈ 70%, LoRA + Prompt ≈ 75%. ### Key Observations 1. **LoRA + Prompt Dominance**: Outperforms other methods in both ECE and AUROC across nearly all subjects and models. 2. **Model Size Impact**: LLaMA-2 13B generally shows better performance than LLaMA-2 7B, particularly in AUROC. 3. **Subject Variability**: - **High AUROC**: "prehistory" (LLaMA-2 13B, LoRA + Prompt ≈ 85%). - **Low AUROC**: "virology" (Mistral 7B, Zero-Shot ≈ 45%). 4. **ECE Anomalies**: - "moral_scenarios" (Mistral 7B Instruct) shows unusually high ECE for LoRA + Prompt (≈ 40%). ### Interpretation The data demonstrates that **LoRA + Prompt** significantly enhances model performance across diverse subjects, likely by refining parameter efficiency and task-specific adaptation. The **Zero-Shot Classifier** struggles with calibration and discrimination, highlighting the need for fine-tuning. Larger models (e.g., LLaMA-2 13B) outperform smaller counterparts, but method choice (LoRA + Prompt) remains the strongest predictor of success. Outliers like "moral_scenarios" suggest domain-specific challenges requiring further investigation. </details> Figure 11: (Part 2) ECE and AUROC values for Query, CT-Probe, CT-LoRA, and CT-Query for each subset of MMLU in open-ended (OE) setting. Appendix E Confidence as a Function of Target Length As we noted when motivating calibration tuning, one limitation of sequence-level probabilities is their intrinsic connection to sequence length. The probability of a sequence decreases with increasing length, regardless of the correctness of the response. By contrast, we wouldn’t expect concept-level probabilities to have any discernible relationship with sequence length. In appendix E, we show there is no consistent relationship between the confidence estimated by the calibration-tuned model and target sequence length on MMLU tasks. A key limitation of using token likelihoods is that they necessarily decay with the length of the generation. In figs. 12, 13 and 14, we confirm over all subsets of MMLU that the length of the target does not strongly correlate with the confidence associated with the targets. This behavior is an essential ingredient towards an effective confidence estimation in practice, such that longer sequences are not penalized in confidence despite being correct. | <details> <summary>x19.png Details</summary>  ### Visual Description ## Scatter Plot: abstract_algebra ### Overview The image is a scatter plot titled "abstract_algebra" with a trend line and shaded confidence interval. It includes histograms on the top and right axes, showing distributions of "Target Length" and "Confidence" respectively. The plot visualizes the relationship between "Target Length" (x-axis) and "Confidence" (y-axis), with data points and a regression line. ### Components/Axes - **Title**: "abstract_algebra" (top center). - **X-axis**: "Target Length" (horizontal axis), labeled with ticks at 0, 25, 50. Scale ranges from 0 to 50. - **Y-axis**: "Confidence" (vertical axis), labeled with ticks at 0, 0.2, 0.4, 0.6. Scale ranges from 0 to 0.6. - **Legend**: Not explicitly labeled, but the trend line is represented by a solid purple line with a shaded purple confidence interval. - **Histograms**: - **Top histogram**: Bar chart for "Target Length" (x-axis distribution). - **Right histogram**: Bar chart for "Confidence" (y-axis distribution). ### Detailed Analysis - **Data Points**: Purple dots scattered across the plot. Most points cluster near the lower-left (low Target Length, low Confidence), with some spread toward higher values. - **Trend Line**: A solid purple line slopes upward from ~0.15 at Target Length 0 to ~0.45 at Target Length 50. The shaded area around the line (confidence interval) widens slightly as Target Length increases. - **Histograms**: - **Target Length**: Bars show a bimodal distribution, with peaks near 0 and 25, and a smaller peak near 50. - **Confidence**: Bars show a right-skewed distribution, with most values concentrated between 0.2 and 0.4. ### Key Observations 1. **Positive Correlation**: The trend line indicates a general increase in Confidence with Target Length, though the relationship is not perfectly linear. 2. **Confidence Interval**: The shaded area suggests uncertainty in the trend line, with wider variability at higher Target Length values. 3. **Distribution Patterns**: - Target Length has a bimodal distribution, suggesting two common ranges (0–25 and 25–50). - Confidence values are more concentrated in the 0.2–0.4 range, with fewer extreme values. ### Interpretation The data suggests that **Target Length** and **Confidence** are positively correlated, but the relationship is not deterministic. The upward trend line implies that longer Target Lengths generally correspond to higher Confidence, though the shaded confidence interval indicates variability in this relationship. The bimodal distribution of Target Length may reflect distinct subgroups or categories within the data, while the right-skewed Confidence distribution highlights that most values cluster in the mid-range. Outliers (e.g., high Confidence at low Target Length) suggest exceptions to the general trend, possibly indicating anomalies or special cases in the dataset. This analysis could inform models where Target Length influences Confidence, but the uncertainty in the trend line emphasizes the need for further validation. </details> | <details> <summary>x20.png Details</summary>  ### Visual Description ## Scatter Plot: Anatomy of Target Length vs. Confidence ### Overview The image presents a scatter plot titled "anatomy" with a downward-sloping trend line overlay. The plot visualizes the relationship between "Target Length" (x-axis) and "Confidence" (y-axis), with histograms in the margins showing distributions. Data points are purple, and the trend line is dark purple. The chart includes axis labels, a legend, and marginal histograms. --- ### Components/Axes - **X-axis (Target Length)**: - Label: "Target Length" - Scale: 0 to 100 (increments of 50) - Position: Bottom of the plot - **Y-axis (Confidence)**: - Label: "Confidence" - Scale: 0.0 to 0.6 (increments of 0.2) - Position: Left side of the plot - **Legend**: - Located in the **top-left** corner - Label: "Trend Line" (dark purple) - **Histograms**: - **Top histogram**: Distribution of "Target Length" (x-axis values) - **Right histogram**: Distribution of "Confidence" (y-axis values) --- ### Detailed Analysis 1. **Scatter Plot**: - **Data Points**: - Approximately 50 purple dots scattered across the plot. - Confidence values range from ~0.0 to ~0.6. - Target Length values range from ~0 to ~100. - **Trend Line**: - Dark purple line slopes **downward** from left to right. - Equation: Approximately `Confidence = -0.005 × Target Length + 0.5` (estimated from intercept and slope). - Confidence decreases by ~0.005 per unit increase in Target Length. 2. **Histograms**: - **Top Histogram (Target Length)**: - Peak at ~50 (most frequent Target Length). - Bimodal distribution with secondary peaks near 0 and 100. - **Right Histogram (Confidence)**: - Peak at ~0.2 (most frequent Confidence). - Long tail extending to ~0.6. --- ### Key Observations 1. **Negative Correlation**: - Confidence decreases as Target Length increases (R² ≈ 0.6 based on trend line fit). 2. **Outliers**: - A few data points at high Confidence (~0.4–0.6) with low Target Length (~10–30). - A cluster of low Confidence (~0.0–0.2) at high Target Length (~70–100). 3. **Distribution Patterns**: - Most data points cluster around Target Length = 50 and Confidence = 0.2. - Histograms confirm bimodal distributions for both axes. --- ### Interpretation The plot suggests an inverse relationship between Target Length and Confidence: longer targets correlate with lower confidence. However, the trend line’s moderate slope (-0.005) indicates the relationship is not strictly deterministic. The histograms reveal that most data points cluster around mid-range values (Target Length = 50, Confidence = 0.2), but outliers at extremes (e.g., high Confidence with short Target Length) suggest contextual factors may influence the relationship. The marginal histograms highlight the need to consider data distribution when interpreting the trend line. This could reflect a scenario where moderate Target Lengths are optimal for Confidence, with diminishing returns at extremes. </details> | <details> <summary>x21.png Details</summary>  ### Visual Description ## Scatter Plot: Astronomy - Confidence vs. Target Length ### Overview The image is a scatter plot titled "astronomy" with a line of best fit and a shaded confidence interval. The x-axis represents "Target Length" (0–200), and the y-axis represents "Confidence" (0.25–0.75). Purple data points are scattered across the plot, with a blue line of best fit and a light blue shaded confidence interval. ### Components/Axes - **Title**: "astronomy" (top-center). - **X-axis**: "Target Length" (0–200, labeled in increments of 100). - **Y-axis**: "Confidence" (0.25–0.75, labeled in increments of 0.25). - **Legend**: - **Line of Best Fit**: Blue (top-right corner). - **Confidence Interval**: Light blue (top-right corner). - **Data Points**: Purple dots distributed across the plot. ### Detailed Analysis - **Line of Best Fit**: - Slope: Approximately 0.003 (positive trend). - Equation: Confidence ≈ 0.003 × Target Length + 0.25 (estimated from intercept and slope). - Position: Passes through the center of the data distribution. - **Confidence Interval**: - Width: Narrower at higher Target Length values (e.g., ~0.05 at Target Length 200 vs. ~0.15 at Target Length 0). - Position: Centered on the line of best fit. - **Data Points**: - Distribution: Scattered but clustered around the line of best fit. - Outliers: A few points deviate significantly (e.g., Confidence ~0.75 at Target Length ~50). ### Key Observations 1. **Positive Correlation**: Confidence increases with Target Length (line of best fit slope > 0). 2. **Confidence Interval Shape**: Narrower at higher Target Length values, suggesting more precise estimates. 3. **Data Spread**: Most points lie below the line of best fit, indicating variability or potential outliers. ### Interpretation The plot demonstrates a **positive relationship** between Target Length and Confidence, with Confidence increasing by ~0.003 per unit increase in Target Length. The narrowing confidence interval at higher Target Length values suggests improved model reliability for larger targets. However, the spread of data points below the line (e.g., Confidence ~0.25–0.35 at Target Length ~50) highlights potential variability or unaccounted factors affecting Confidence. The shaded interval’s asymmetry (wider at lower Target Lengths) implies greater uncertainty in predictions for smaller targets. This could reflect challenges in data collection or model limitations for low-Target Length scenarios. </details> | <details> <summary>x22.png Details</summary>  ### Visual Description ## Scatter Plot: Confidence vs Target Length in Clinical Knowledge ### Overview The image displays a scatter plot titled "clinical_knowledge" analyzing the relationship between "Target Length" (x-axis) and "Confidence" (y-axis). Purple data points are distributed across the plot, with a horizontal reference line at approximately 0.25 confidence. Marginal density plots (top) and histograms (right) provide additional distributional context. ### Components/Axes - **Title**: `clinical_knowledge` - **X-axis**: "Target Length" (scale: 0 to 100, linear) - **Y-axis**: "Confidence" (scale: 0.00 to 0.75, linear) - **Reference Line**: Horizontal dashed line at y ≈ 0.25 - **Marginal Plots**: - Top: Density plot of confidence values (peaks near 0.25) - Right: Histogram of target lengths (peaks at lower values) ### Detailed Analysis - **Data Points**: - Most points cluster below y = 0.25, with a few outliers reaching up to y ≈ 0.75. - Confidence decreases as target length increases, though variability exists (e.g., some high-confidence points at low target lengths). - **Marginal Plots**: - Confidence density peaks sharply at ~0.25, with a long tail toward lower values. - Target length histogram shows a right-skewed distribution, with most values concentrated below 50. ### Key Observations 1. **Negative Correlation**: Higher target lengths generally correspond to lower confidence, though exceptions exist. 2. **Threshold Effect**: The horizontal line at 0.25 may represent a critical confidence threshold (e.g., minimum acceptable performance). 3. **Distribution Skew**: Both confidence and target length distributions are skewed toward lower values, suggesting a focus on shorter targets or lower-confidence scenarios. ### Interpretation The data implies that longer clinical knowledge targets are associated with reduced confidence, potentially indicating challenges in handling complex or extended information. The 0.25 confidence threshold could signify a performance benchmark, with most data points failing to meet this standard. The marginal plots reinforce this, showing a concentration of low-confidence, short-target scenarios. This might reflect limitations in clinical knowledge systems when processing extended or nuanced information, highlighting a need for optimization in handling longer targets. </details> | | --- | --- | --- | --- | | <details> <summary>x23.png Details</summary>  ### Visual Description ## Scatter Plot: college_biology ### Overview The image is a scatter plot titled "college_biology" with a linear regression trend line and a shaded confidence interval region. The plot visualizes the relationship between "Target Length" (x-axis) and "Confidence" (y-axis), with additional histograms for marginal distributions. ### Components/Axes - **Title**: "college_biology" (top-center) - **X-axis**: - Label: "Target Length" - Range: 0 to 200 (linear scale) - Grid lines: Present - **Y-axis**: - Label: "Confidence" - Range: 0 to 0.6 (linear scale) - Grid lines: Present - **Legend**: - Position: Top-left - Label: "Confidence Interval" (purple) - **Trend Line**: - Color: Purple - Type: Linear regression - Shaded Region: 95% confidence interval (lighter purple) - **Data Points**: - Color: Purple dots - Distribution: Scattered across the plot - **Histograms**: - Top: Distribution of "Target Length" (x-axis values) - Right: Distribution of "Confidence" (y-axis values) ### Detailed Analysis - **Trend Line**: - Slope: Positive (increasing trend) - Equation: Not explicitly labeled, but visually approximated as `y = 0.002x + 0.1` (based on intercept ~0.1 and slope ~0.002). - **Confidence Interval**: - Width: ~±0.05 around the trend line (e.g., at x=100, y≈0.3 ± 0.05). - **Data Points**: - Most points cluster below the trend line (e.g., x=50, y≈0.2; x=150, y≈0.4). - Outliers: - One point at (200, 0.6) (top-right corner). - A few points near (0, 0.1) (bottom-left). - **Histograms**: - Top Histogram: - Peaks near x=50–100 (right-skewed distribution). - Right Histogram: - Peaks near y=0.3–0.4 (approximately normal distribution). ### Key Observations 1. **Positive Correlation**: Confidence increases with target length, but with significant variability. 2. **Confidence Interval**: The shaded region indicates uncertainty in the trend line, widening slightly at higher x-values. 3. **Outliers**: The point at (200, 0.6) deviates significantly from the trend, suggesting an anomaly or exceptional case. 4. **Distribution Skew**: Target lengths are concentrated in the 50–150 range, while confidence values are more evenly distributed. ### Interpretation The plot suggests that in the "college_biology" dataset, longer target lengths generally correlate with higher confidence levels. However, the wide scatter of data points and the confidence interval indicate that this relationship is not deterministic. The outlier at (200, 0.6) may represent a unique case or measurement error. The histograms reveal that most target lengths fall within a moderate range (50–150), while confidence values are moderately concentrated around 0.3–0.4. The shaded confidence interval implies that predictions for confidence at a given target length have a margin of error, emphasizing the need for caution in interpreting the trend line as absolute. This could reflect biological variability or measurement limitations in the dataset. </details> | <details> <summary>x24.png Details</summary>  ### Visual Description ## Scatter Plot: college_chemistry ### Overview The image is a scatter plot titled "college_chemistry" with a trend line and shaded confidence interval. It visualizes the relationship between "Target Length" (x-axis) and "Confidence" (y-axis). Two histograms are embedded: one on the top (x-axis distribution) and one on the right (y-axis distribution). Data points are represented as purple dots, with a central trend line and a shaded region indicating uncertainty. ### Components/Axes - **Title**: "college_chemistry" (top-center). - **X-axis**: "Target Length" (0 to 100, linear scale). - **Y-axis**: "Confidence" (0.25 to 0.75, linear scale). - **Legend**: Not explicitly labeled, but the trend line and shaded region are implied as the primary data series. - **Histograms**: - Top histogram: Distribution of "Target Length" (x-axis values). - Right histogram: Distribution of "Confidence" (y-axis values). ### Detailed Analysis - **Data Points**: - Purple dots scattered across the plot, with a concentration in the lower-left quadrant (low target length, low confidence). - A few points extend toward higher target lengths (up to ~100) and confidence levels (up to ~0.75). - **Trend Line**: - A straight line slopes upward from the lower-left to upper-right, indicating a positive correlation between target length and confidence. - The line passes through the center of the data cluster, with a slope suggesting moderate linear association. - **Confidence Interval**: - Shaded region around the trend line (approximately ±0.15 in confidence units). - The interval widens slightly at higher target lengths, suggesting increased uncertainty in predictions for longer targets. - **Histograms**: - Top histogram: Peaks near 0–20 (low target lengths), with a long tail extending to 100. - Right histogram: Peaks near 0.3–0.5 (moderate confidence), with a gradual decline toward higher confidence levels. ### Key Observations 1. **Positive Correlation**: The upward trend line confirms that longer target lengths generally correspond to higher confidence. 2. **Data Clustering**: Most data points cluster in the lower-left quadrant, indicating that shorter targets are associated with lower confidence. 3. **Outliers**: A few points in the upper-right quadrant (e.g., target length ~50–70, confidence ~0.6–0.7) deviate from the trend, suggesting exceptions where longer targets achieved higher confidence. 4. **Uncertainty**: The widening confidence interval at higher target lengths implies reduced precision in predictions for longer targets. ### Interpretation The plot demonstrates a statistically significant positive relationship between target length and confidence in the "college_chemistry" dataset. While longer targets tend to yield higher confidence, the variability in data points (e.g., outliers and the widening confidence interval) highlights that this relationship is not deterministic. The histograms reveal that most targets are short (0–20), with confidence levels predominantly in the 0.3–0.5 range. The shaded confidence interval suggests that predictions for longer targets are less reliable, possibly due to limited data or inherent variability in the system. This could imply that while increasing target length improves confidence on average, other factors (e.g., data quality, model complexity) may influence outcomes. The presence of outliers warrants further investigation to identify contextual factors driving these exceptions. </details> | <details> <summary>x25.png Details</summary>  ### Visual Description ## Scatter Plot: college_computer_science ### Overview The image is a scatter plot titled "college_computer_science" with a linear regression trend line and shaded confidence interval. It includes marginal histograms on the top and right. The plot visualizes the relationship between "Target Length" (x-axis) and "Confidence" (y-axis), with data points represented as purple dots. ### Components/Axes - **Title**: "college_computer_science" (top-center). - **X-axis**: "Target Length" (0 to 100, labeled at bottom). - **Y-axis**: "Confidence" (0.2 to 0.8, labeled at left). - **Trend Line**: A purple line with a shaded confidence interval (purple gradient). - **Marginal Histograms**: - Top histogram: Distribution of "Target Length" (purple bars). - Right histogram: Distribution of "Confidence" (purple bars). ### Detailed Analysis - **Data Points**: - Purple dots scattered across the plot, with higher density near the trend line. - Confidence values range from ~0.2 to ~0.8, with most points clustered between 0.4 and 0.6. - **Trend Line**: - Slope: Positive (increasing Confidence with Target Length). - Confidence Interval: Shaded area narrows as Target Length increases, indicating reduced variability at higher lengths. - **Histograms**: - **Target Length**: Skewed right, with most values between 0 and 50. - **Confidence**: Peaks near 0.5–0.6, with a gradual decline toward 0.2 and 0.8. ### Key Observations 1. **Positive Correlation**: Confidence increases with Target Length (e.g., at Target Length = 50, Confidence ≈ 0.5; at Target Length = 100, Confidence ≈ 0.7). 2. **Confidence Interval Narrowing**: The shaded area becomes tighter at higher Target Lengths, suggesting more consistent relationships. 3. **Distribution Patterns**: - Target Length is more variable (wide spread in the top histogram). - Confidence is concentrated around the trend line (narrower distribution in the right histogram). ### Interpretation The data suggests a **positive relationship** between Target Length and Confidence in college computer science contexts. Longer Target Lengths are associated with higher Confidence, and the narrowing confidence interval at higher lengths implies greater reliability in this trend. The histograms reveal that while Target Length varies widely, Confidence is more clustered, particularly around the trend line. This could indicate that longer projects or courses in computer science may correlate with increased student confidence, though variability persists, especially for shorter lengths. The marginal histograms highlight the need to consider both central tendency and distribution when interpreting the relationship. </details> | <details> <summary>x26.png Details</summary>  ### Visual Description ## Scatter Plot: college_mathematics ### Overview The image is a scatter plot titled "college_mathematics" with a trend line and marginal histograms. It visualizes the relationship between "Target Length" (x-axis) and "Confidence" (y-axis), with data points clustered in the lower-left quadrant. A shaded confidence interval surrounds the trend line, and histograms on the right and top edges show distributions of the variables. ### Components/Axes - **Title**: college_mathematics - **X-axis (Horizontal)**: - Label: Target Length - Scale: 0 to 100 (linear) - **Y-axis (Vertical)**: - Label: Confidence - Scale: 0.2 to 0.6 (linear) - **Legend**: - Position: Top-left corner - Color: Purple (matches data points and trend line) - **Marginal Histograms**: - Right histogram: Distribution of Confidence values (y-axis) - Top histogram: Distribution of Target Length values (x-axis) ### Detailed Analysis - **Data Points**: - Purple dots scattered across the plot, with higher density in the lower-left region (Target Length < 50, Confidence < 0.4). - Fewer points in the upper-right quadrant (Target Length > 50, Confidence > 0.4). - **Trend Line**: - Solid purple line slopes upward from ~(0, 0.2) to ~(100, 0.55), indicating a positive correlation between Target Length and Confidence. - Shaded area around the line represents a 95% confidence interval, widening slightly toward higher Target Length values. - **Histograms**: - Top histogram: Peaks near Target Length = 0–20, with a long tail extending to 100. - Right histogram: Peaks near Confidence = 0.3–0.4, with a secondary peak near 0.5. ### Key Observations 1. **Positive Correlation**: The upward trend line suggests that longer Target Lengths are associated with higher Confidence, though the relationship is not perfectly linear. 2. **Data Clustering**: Most data points (70–80%) are concentrated in the lower-left quadrant, indicating lower Confidence for shorter Target Lengths. 3. **Confidence Interval Width**: The shaded area widens as Target Length increases, implying greater uncertainty in the trend at higher values. 4. **Histogram Distributions**: - Target Length values are skewed left (most data < 50). - Confidence values are bimodal, with peaks near 0.3–0.4 and 0.5. ### Interpretation The plot demonstrates a weak but statistically significant positive relationship between Target Length and Confidence in college mathematics contexts. The clustering of data points in the lower-left suggests that shorter Target Lengths are more common and associated with lower Confidence. The widening confidence interval at higher Target Lengths may reflect increased variability in outcomes or measurement noise. The bimodal Confidence distribution implies two distinct groups: one with moderate Confidence (0.3–0.4) and another with higher Confidence (0.5+), potentially corresponding to different subgroups (e.g., student performance levels). The histograms highlight that the dataset is skewed toward shorter Target Lengths, which could indicate a focus on foundational mathematics problems in this context. </details> | | <details> <summary>x27.png Details</summary>  ### Visual Description ## Scatter Plot: college_medicine ### Overview The image is a scatter plot titled "college_medicine" with a downward-sloping trend line. It includes histograms on the top and right edges, visualizing distributions of the x-axis ("Target Length") and y-axis ("Confidence"). Data points are represented by purple dots, with a shaded confidence interval around the trend line. ### Components/Axes - **Title**: "college_medicine" (top center). - **X-axis**: "Target Length" (horizontal axis, range: 0–100, labeled in increments of 25). - **Y-axis**: "Confidence" (vertical axis, range: 0.00–0.75, labeled in increments of 0.25). - **Legend**: Not explicitly visible, but data points are purple. - **Histograms**: - Top histogram: Distributes purple bars across "Target Length" (0–100), peaking near 0–50. - Right histogram: Distributes purple bars across "Confidence" (0.00–0.75), peaking near 0.25–0.50. - **Trend Line**: A solid purple line with a shaded confidence interval (light purple band) sloping downward from ~0.75 (left) to ~0.25 (right). ### Detailed Analysis - **Data Points**: - Approximately 50–60 purple dots scattered across the plot. - Highest density of points in the lower-left quadrant (Target Length: 0–50, Confidence: 0.25–0.50). - Fewer points in the upper-right quadrant (Target Length: 50–100, Confidence: 0.00–0.25). - **Trend Line**: - Slope: Negative (decreasing Confidence with increasing Target Length). - Confidence Interval: Shaded band widens slightly toward the right, indicating increased uncertainty at higher Target Lengths. - **Histograms**: - Top histogram: Majority of Target Lengths cluster between 0–50 (peak at ~25). - Right histogram: Majority of Confidence values cluster between 0.25–0.50 (peak at ~0.4). ### Key Observations 1. **Negative Correlation**: Confidence decreases as Target Length increases, with a clear trend line slope of ~-0.005 per unit increase in Target Length. 2. **Distribution Peaks**: - Target Length: ~25 (most frequent). - Confidence: ~0.4 (most frequent). 3. **Outliers**: - A few data points deviate from the trend, e.g., high Confidence (~0.75) at low Target Length (~10) and low Confidence (~0.1) at high Target Length (~90). ### Interpretation The plot suggests that in the context of "college_medicine," shorter Target Lengths are associated with higher Confidence. The downward trend implies that longer Target Lengths introduce uncertainty or complexity, reducing Confidence. The histograms confirm that most data points cluster in the lower-left quadrant, reinforcing the trend. The widening confidence interval at higher Target Lengths indicates growing variability in outcomes, possibly due to increased difficulty or ambiguity in longer tasks. This could reflect challenges in medical education or research where extended Target Lengths (e.g., complex procedures or studies) correlate with lower predictability or reliability. </details> | <details> <summary>x28.png Details</summary>  ### Visual Description ## Scatter Plot: computer_security ### Overview The image is a scatter plot titled "computer_security" with a main plot and two marginal distribution plots. The main plot shows the relationship between "Target Length" (x-axis) and "Confidence" (y-axis), while the top and right marginal plots display the distributions of these variables. The data points are purple, and a trend line with a shaded region is overlaid on the main plot. ### Components/Axes - **Main Plot**: - **X-axis (Target Length)**: Ranges from 0 to 200, labeled "Target Length". - **Y-axis (Confidence)**: Ranges from 0 to 0.8, labeled "Confidence". - **Data Points**: Purple dots scattered across the plot. - **Trend Line**: A solid purple line with a shaded region (likely representing confidence intervals or uncertainty). - **Marginal Plots**: - **Top Plot**: Histogram of "Target Length" with a purple line indicating the distribution. - **Right Plot**: Histogram of "Confidence" with a purple line indicating the distribution. - **Legend**: No explicit legend is present in the image. ### Detailed Analysis - **Data Points**: - Approximately 50-60 purple dots are distributed across the plot. - Most points cluster around the middle ranges of "Target Length" (50-150) and "Confidence" (0.3-0.6). - A few outliers are visible at the extremes (e.g., low "Target Length" with high "Confidence" and vice versa). - **Trend Line**: - The trend line is nearly flat, suggesting minimal correlation between "Target Length" and "Confidence". - The shaded region around the line is narrow, indicating low uncertainty in the trend estimation. - **Marginal Distributions**: - **Target Length**: The histogram peaks around 100-150, with a long tail toward higher values (up to 200). - **Confidence**: The histogram peaks around 0.4-0.5, with a slight skew toward lower values (0.2-0.3). ### Key Observations 1. **Flat Trend Line**: The lack of a clear upward or downward slope suggests that "Confidence" does not significantly vary with "Target Length". 2. **Distribution Peaks**: Both variables show central tendencies, with most data points concentrated in mid-range values. 3. **Outliers**: A few data points deviate from the trend, but they are sparse and do not strongly influence the overall pattern. ### Interpretation The data suggests that "Confidence" levels are relatively stable across different "Target Lengths", with a slight tendency to increase as "Target Length" grows. However, the flat trend line indicates that this relationship is weak or non-existent. The marginal distributions reveal that both variables have a central tendency, which might imply that the model or system being analyzed performs consistently for typical "Target Lengths". The shaded region around the trend line could represent uncertainty, but its narrowness suggests that the model's predictions are relatively reliable. The absence of a legend or explicit labels for the trend line leaves some ambiguity about its exact meaning (e.g., confidence interval, prediction interval). Overall, the plot highlights the need for further analysis to determine the causal or correlational relationship between the variables. </details> | <details> <summary>x29.png Details</summary>  ### Visual Description ## Scatter Plot: Econometrics Analysis ### Overview The image presents a scatter plot titled "econometrics" with a secondary title "Confidence Interval." It visualizes the relationship between "Target Length" (x-axis) and "Confidence" (y-axis), with a linear trend line and shaded confidence interval. Histograms on the top and right edges show marginal distributions of the data. ### Components/Axes - **X-axis (Target Length)**: - Label: "Target Length" - Scale: 0 to 100 (discrete ticks at 0, 50, 100) - Units: Not explicitly stated, but implied as a numerical measure. - **Y-axis (Confidence)**: - Label: "Confidence" - Scale: 0.4 to 0.8 (discrete ticks at 0.4, 0.6, 0.8) - Units: Likely a probability or normalized metric (0–1 range). - **Legend**: - Located in the top-right corner. - Label: "Confidence Interval" (light purple shading). - **Histograms**: - Top histogram: Marginal distribution of "Target Length" (x-axis data). - Right histogram: Marginal distribution of "Confidence" (y-axis data). ### Detailed Analysis - **Scatter Plot**: - Data points: ~50–100 purple dots scattered across the plot. - Trend line: Dark purple line with a slight positive slope, indicating a weak positive correlation between "Target Length" and "Confidence." - Confidence interval: Light purple shaded band around the trend line, suggesting uncertainty in the regression estimate. - **Histograms**: - Top histogram: Bimodal distribution with peaks near 0 and 50–75. - Right histogram: Unimodal distribution peaking near 0.6–0.7. ### Key Observations 1. **Positive Correlation**: The trend line slopes upward, suggesting higher "Target Length" values are associated with marginally higher "Confidence." 2. **Confidence Interval Width**: The shaded band widens slightly at higher "Target Length" values, indicating increased uncertainty in predictions. 3. **Outliers**: A few data points deviate significantly from the trend line (e.g., low "Confidence" at high "Target Length"). 4. **Distribution Skew**: The "Target Length" histogram shows a left-skewed distribution, while "Confidence" is more symmetric. ### Interpretation The plot demonstrates a weak but statistically significant positive relationship between "Target Length" and "Confidence" in an econometric model. The confidence interval’s widening at higher "Target Length" values implies diminishing reliability of predictions as the independent variable increases. The bimodal distribution of "Target Length" suggests two distinct subgroups in the data, which may require further investigation. The marginal histograms highlight that most data points cluster around moderate values for both variables, with fewer extreme cases. This could indicate a need for model refinement to address outliers or subgroup heterogeneity. </details> | <details> <summary>x30.png Details</summary>  ### Visual Description ## Scatter Plot: Confidence vs Target Length in Electrical Engineering ### Overview The image displays a scatter plot analyzing the relationship between "Target Length" and "Confidence" in an electrical engineering context. A linear trend line is overlaid on the data points, with histograms showing distributions of both variables. The plot uses purple data points and a blue trend line. ### Components/Axes - **X-axis (Target Length)**: - Label: "Target Length" - Scale: 0 to 60 (increments of 10) - Position: Bottom - **Y-axis (Confidence)**: - Label: "Confidence" - Scale: 0 to 0.6 (increments of 0.2) - Position: Left - **Legend**: - Located in the top-left corner - Text: "Confidence vs Target Length" (purple color) - **Histograms**: - Top histogram: Horizontal distribution of "Target Length" - Right histogram: Vertical distribution of "Confidence" - Both histograms use purple bars ### Detailed Analysis - **Scatter Plot**: - Data points: ~50 purple dots scattered across the plot - Trend line equation: **y = 0.01x + 0.15** (approximate) - Key data points: - (0, 0.15): Intercept at x=0 - (50, 0.65): Extrapolated point (outside observed range) - **Histograms**: - Target Length: Peaks between 0–30, with a long tail to 60 - Confidence: Peaks between 0.2–0.4, with a sharp drop above 0.5 ### Key Observations 1. **Positive Correlation**: Confidence increases linearly with Target Length (slope ≈ 0.01). 2. **Data Spread**: - 70% of points cluster between Target Length 0–40 and Confidence 0.2–0.4. - Outliers: 5 points exceed Confidence > 0.5 (Target Length > 50). 3. **Distribution Skew**: - Target Length: Right-skewed (longer lengths less frequent). - Confidence: Bimodal distribution (peaks at 0.2–0.4 and 0.5–0.6). ### Interpretation The data suggests a weak but statistically significant relationship between Target Length and Confidence in electrical engineering tasks. The linear trend (y = 0.01x + 0.15) implies that for every 1-unit increase in Target Length, Confidence rises by ~1%. However, the low slope (0.01) indicates the effect is modest. The histograms reveal that most tasks involve shorter Target Lengths (<30) and moderate Confidence levels (0.2–0.4), with fewer high-confidence, long-target tasks. The outlier points (Confidence > 0.5) may represent specialized or high-stakes scenarios. The right-skewed Target Length distribution aligns with real-world engineering constraints where shorter tasks dominate. </details> | | <details> <summary>x31.png Details</summary>  ### Visual Description ## Scatter Plot: elementary_mathematics ### Overview The image is a scatter plot titled "elementary_mathematics" with a marginal histogram on the right. It visualizes the relationship between "Target Length" (x-axis) and "Confidence" (y-axis). A trend line is overlaid on the scatter plot, and the histogram shows the distribution of confidence values. ### Components/Axes - **Title**: "elementary_mathematics" (top of the plot). - **X-axis**: Labeled "Target Length" with a scale from 0 to 100. - **Y-axis**: Labeled "Confidence" with a scale from 0.25 to 0.75. - **Legend**: Not explicitly labeled, but the trend line is visually distinct (dark purple with a shaded area). - **Marginal Histogram**: Located on the right side of the plot, showing the distribution of confidence values. ### Detailed Analysis - **Scatter Plot Data Points**: - Purple dots are distributed across the plot, with a slight upward trend. - The trend line (dark purple) slopes upward, indicating a weak positive correlation between Target Length and Confidence. - The shaded area around the trend line suggests a confidence interval, though the exact bounds are not labeled. - **Marginal Histogram**: - The histogram on the right shows a unimodal distribution of confidence values, peaking around 0.5. - The distribution appears approximately normal, with most values clustered between 0.4 and 0.6. ### Key Observations - The trend line indicates a **weak positive relationship** between Target Length and Confidence, but the spread of data points suggests significant variability. - The histogram reveals that **most confidence values are centered around 0.5**, with fewer extreme values (e.g., near 0.25 or 0.75). - No clear outliers are visible in the scatter plot, though the data points are spread across the entire range of Target Length (0–100). ### Interpretation The data suggests that while there is a **slight tendency for confidence to increase with target length**, the relationship is not strong. The confidence values are predominantly moderate (around 0.5), implying that factors other than target length may play a more significant role in determining confidence. The weak positive trend could indicate that longer targets are associated with marginally higher confidence, but this effect is not dominant. The histogram’s normal distribution of confidence values further supports the idea that confidence levels are relatively stable across different target lengths. </details> | <details> <summary>x32.png Details</summary>  ### Visual Description ## Scatter Plot with Histogram: Confidence vs. Target Length ### Overview The image displays a scatter plot titled "formal_logic" with a secondary histogram on the right. The scatter plot visualizes the relationship between "Target Length" (x-axis) and "Confidence" (y-axis), while the histogram shows the distribution of Confidence values. A line of best fit is overlaid on the scatter plot, and a legend identifies the data series as "formal_logic" in purple. --- ### Components/Axes - **Main Chart (Scatter Plot)**: - **X-axis**: "Target Length" (range: 0 to 200, linear scale). - **Y-axis**: "Confidence" (range: 0 to 0.8, linear scale). - **Data Points**: Purple dots representing individual observations. - **Line of Best Fit**: A dashed purple line with a negative slope, indicating a downward trend. - **Legend**: Positioned at the top, labeled "formal_logic" in purple. - **Secondary Chart (Histogram)**: - **X-axis**: "Confidence" (range: 0.2 to 0.6, linear scale). - **Y-axis**: "Frequency" (approximate count of observations, no explicit scale). - **Bars**: Purple, matching the scatter plot's color scheme. --- ### Detailed Analysis - **Scatter Plot Trends**: - The line of best fit slopes downward, suggesting a negative correlation between Target Length and Confidence. For example: - At Target Length ≈ 0, Confidence ≈ 0.6. - At Target Length ≈ 100, Confidence ≈ 0.4. - At Target Length ≈ 200, Confidence ≈ 0.2. - Data points are scattered but cluster around the line, with some variability (e.g., Confidence values between 0.2 and 0.6 for Target Lengths between 50 and 150). - **Histogram Distribution**: - The Confidence values are most concentrated around 0.4 (peak frequency). - The distribution tapers off toward 0.2 and 0.6, indicating fewer observations at the extremes. --- ### Key Observations 1. **Negative Correlation**: As Target Length increases, Confidence decreases, as evidenced by the downward slope of the line of best fit. 2. **Confidence Distribution**: Most observations cluster around Confidence ≈ 0.4, with a bimodal spread toward lower and higher values. 3. **Outliers**: No extreme outliers are visible, but some points deviate slightly from the trend line (e.g., higher Confidence at mid-Target Lengths). --- ### Interpretation - **What the Data Suggests**: The negative correlation implies that longer Target Lengths are associated with lower confidence levels. This could reflect a trade-off between complexity (Target Length) and certainty (Confidence) in a formal logical or analytical context. - **Relationships**: The line of best fit quantifies the trend, while the histogram highlights the variability in Confidence. The formal_logic label suggests the data may relate to formal systems, reasoning, or model performance. - **Anomalies**: The spread of Confidence values at mid-Target Lengths (e.g., 0.3–0.5) indicates inconsistency, possibly due to contextual factors or measurement noise. - **Significance**: The visualization underscores the importance of balancing Target Length and Confidence in decision-making or model design, particularly in formal logic applications where precision is critical. </details> | <details> <summary>x33.png Details</summary>  ### Visual Description ## Scatter Plot: global_facts ### Overview The image is a scatter plot titled "global_facts" showing the relationship between "Target Length" (x-axis) and "Confidence" (y-axis). A line of best fit with a shaded confidence interval is overlaid on the data points. The plot uses a purple color scheme for data points, line, and shaded region. ### Components/Axes - **Title**: "global_facts" (top center) - **X-axis**: "Target Length" (0 to 100, linear scale) - **Y-axis**: "Confidence" (0.25 to 0.75, linear scale) - **Data Points**: Purple dots scattered across the plot - **Line of Best Fit**: Solid purple line with a shaded confidence interval (lighter purple) - **Legend**: Not explicitly visible, but inferred from color coding (purple = data series) ### Detailed Analysis - **Data Points**: - Clustered primarily in the lower-left quadrant (Target Length: 0–50, Confidence: 0.25–0.5) - Sparse distribution in the upper-right quadrant (Target Length: 50–100, Confidence: 0.5–0.75) - Notable outlier: A single data point at (Target Length: 100, Confidence: 0.7) - **Line of Best Fit**: - Slope: Positive (increasing Confidence with Target Length) - Equation: Approximately `Confidence = 0.005 * Target Length + 0.25` (estimated from endpoints) - Endpoints: - Left: (0, 0.25) - Right: (100, 0.75) - **Shaded Region**: - Represents a 95% confidence interval around the line of best fit - Width increases slightly toward the right, indicating greater variability at higher Target Lengths ### Key Observations 1. **Positive Correlation**: Confidence increases with Target Length (R² ≈ 0.85 based on visual estimation). 2. **Outlier**: The point at (100, 0.7) deviates slightly below the line of best fit. 3. **Confidence Interval**: The shaded area suggests uncertainty in predictions, especially at higher Target Lengths. 4. **Data Distribution**: Most data points are concentrated in the lower range of Target Length (0–50), with fewer observations at higher lengths. ### Interpretation The plot demonstrates a strong positive relationship between Target Length and Confidence, suggesting that longer target lengths are associated with higher confidence levels. However, the slight deviation of the rightmost data point and the widening confidence interval at higher Target Lengths indicate potential limitations or variability in the relationship. The shaded region highlights that while the trend is clear, predictions for extreme values (e.g., Target Length = 100) carry more uncertainty. This could imply diminishing returns or external factors influencing Confidence at longer lengths. The data distribution’s skew toward shorter Target Lengths may reflect sampling bias or practical constraints in the dataset. </details> | <details> <summary>x34.png Details</summary>  ### Visual Description ## Scatter Plot: high_school_biology ### Overview The image is a scatter plot titled "high_school_biology" with a horizontal line at approximately 0.5 confidence. The plot visualizes the relationship between "Target Length" (x-axis) and "Confidence" (y-axis), with data points distributed across the plot. Marginal histograms on the top and right edges show the distribution of target lengths and confidence levels, respectively. ### Components/Axes - **Title**: "high_school_biology" (top of the plot). - **X-axis**: "Target Length" (ranges from 0 to 100, with gridlines at 0, 50, 100). - **Y-axis**: "Confidence" (ranges from 0.0 to 0.6, with gridlines at 0.0, 0.2, 0.4, 0.6). - **Marginal Plots**: - **Top**: Histogram of "Target Length" (x-axis distribution). - **Right**: Histogram of "Confidence" (y-axis distribution). - **Horizontal Line**: A dashed line at y = 0.5 (confidence level), spanning the entire x-axis range. - **Data Points**: Purple dots scattered across the plot, with no visible legend or color key. ### Detailed Analysis - **Data Points**: - Approximately 50-100 purple dots are distributed across the plot. - Most points cluster in the lower-left quadrant (low target length, high confidence). - Fewer points appear in the upper-right quadrant (high target length, low confidence). - **Horizontal Line**: - Positioned at y = 0.5, suggesting a threshold or reference value for confidence. - Approximately 30-40% of data points lie above this line, while the majority fall below. - **Marginal Histograms**: - **Target Length**: Peaks around 0-50, with a gradual decline toward 100. - **Confidence**: Peaks near 0.5, with a bimodal distribution (two smaller peaks near 0.3 and 0.7). ### Key Observations 1. **Negative Correlation**: As target length increases, confidence generally decreases, though the relationship is not strictly linear. 2. **Threshold at 0.5**: The horizontal line at 0.5 confidence may represent a critical benchmark, with most data points falling below it. 3. **Distribution Patterns**: - Target lengths are more concentrated in the lower range (0-50). - Confidence levels are more evenly distributed but show a slight preference for mid-range values (0.3-0.7). ### Interpretation The plot suggests that in high school biology, longer target lengths are associated with lower confidence levels. The horizontal line at 0.5 confidence could indicate a performance threshold, where half the data points fall below this level. The marginal histograms reveal that target lengths are more concentrated in shorter ranges, while confidence levels are more evenly spread. The scatter plot's lack of a clear trend implies variability in how target length affects confidence, possibly due to factors like student ability, question difficulty, or assessment design. The absence of a legend or color key limits the ability to categorize data points further, but the purple color consistently represents all observations. </details> | | <details> <summary>x35.png Details</summary>  ### Visual Description ## Scatter Plot with Trend Line and Histograms: High School Chemistry Confidence vs. Target Length ### Overview The image displays a scatter plot analyzing the relationship between "Target Length" (x-axis) and "Confidence" (y-axis) in a high school chemistry context. A linear trend line with a shaded confidence interval is overlaid on the data points, accompanied by histograms on the top and right axes to show distributions. --- ### Components/Axes - **X-axis (Target Length)**: Labeled "Target Length" with a scale from 0 to 100. - **Y-axis (Confidence)**: Labeled "Confidence" with a scale from 0.25 to 0.75. - **Data Points**: Purple dots scattered across the plot. - **Trend Line**: A solid purple line with a shaded confidence interval (light purple) around it. - **Histograms**: - **Top Histogram**: Distribution of "Target Length" (x-axis values). - **Right Histogram**: Distribution of "Confidence" (y-axis values). - **Legend**: Located in the top-left corner, indicating the color of the data points and trend line (purple). --- ### Detailed Analysis - **Data Points**: - Approximately 50–60 purple dots are distributed across the plot. - Most points cluster between **Target Length = 0–100** and **Confidence = 0.25–0.75**. - Notable outliers: A few points near **Target Length = 0** with **Confidence ≈ 0.25** and **Target Length ≈ 100** with **Confidence ≈ 0.75**. - **Trend Line**: - The line slopes **upward** from left to right, indicating a **positive correlation** between Target Length and Confidence. - The equation of the line is not explicitly provided, but the slope appears moderate. - The shaded confidence interval (light purple) spans roughly **±0.15** around the trend line, suggesting moderate uncertainty in the relationship. - **Histograms**: - **Top Histogram (Target Length)**: - Peaks around **Target Length = 50**, with a roughly symmetric distribution. - Most values fall between **0–100**, with a slight skew toward lower values. - **Right Histogram (Confidence)**: - Peaks around **Confidence = 0.5**, with a bimodal distribution (two smaller peaks near 0.3 and 0.7). - Most values cluster between **0.25–0.75**, with fewer extremes. --- ### Key Observations 1. **Positive Correlation**: The upward trend line confirms that longer Target Lengths are associated with higher Confidence levels. 2. **Variability**: The shaded confidence interval and scattered data points indicate that the relationship is not perfectly linear. 3. **Distribution Patterns**: - Target Lengths are more evenly distributed, while Confidence values show a central tendency around 0.5. - The bimodal distribution in Confidence suggests two distinct subgroups (e.g., low and high confidence). 4. **Outliers**: Points at the extremes (e.g., Target Length = 0, Confidence = 0.25) may represent edge cases or measurement errors. --- ### Interpretation The data suggests that in high school chemistry, **longer Target Lengths (e.g., experiments, tasks)** are generally associated with **higher Confidence levels**. However, the variability in the data (as shown by the shaded confidence interval and scattered points) implies that other factors (e.g., student ability, resource availability) may influence this relationship. The bimodal Confidence distribution could indicate that students either struggle significantly (low confidence) or perform well (high confidence), with a middle group of moderate confidence. The histograms further highlight that most data points fall within the middle ranges, reinforcing the central tendency of the trend. This analysis could inform curriculum design or assessment strategies by emphasizing the importance of balancing Target Length with student preparedness to optimize Confidence outcomes. </details> | <details> <summary>x36.png Details</summary>  ### Visual Description ## Scatter Plot: High School Computer Science Confidence vs. Target Length ### Overview The image is a scatter plot titled "high_school_computer_science" showing the relationship between "Target Length" (x-axis) and "Confidence" (y-axis). A linear trend line with a shaded confidence interval is overlaid on the data points. Box plots are embedded in the top and right margins to show distributions of the variables. --- ### Components/Axes - **Title**: "high_school_computer_science" (top-left, bold text). - **X-axis**: - Label: "Target Length" (bottom, horizontal). - Scale: 0 to 200 (linear, with ticks at 0, 100, 200). - **Y-axis**: - Label: "Confidence" (left, vertical). - Scale: 0.25 to 0.75 (linear, with ticks at 0.25, 0.50, 0.75). - **Legend**: No explicit legend, but the trend line and shaded area are visually distinct. - **Box Plots**: - **Top-left**: Horizontal box plot for "Target Length" (median ~100, range 0–200). - **Top-right**: Vertical box plot for "Confidence" (median ~0.5, range 0.3–0.7). --- ### Detailed Analysis - **Scatter Plot**: - **Data Points**: Purple dots distributed across the plot. Most points cluster between x=50–150 and y=0.4–0.6. - **Trend Line**: A solid purple line slopes upward from ~0.3 at x=0 to ~0.75 at x=200. The line equation appears linear (y = mx + b, with m > 0). - **Shaded Area**: A light purple band around the trend line, likely representing a 95% confidence interval (uncertainty range). - **Box Plots**: - **Target Length (Top-left)**: - Median: ~100. - Interquartile Range (IQR): ~50–150. - Whiskers: Extend to 0 and 200 (outliers not visible). - **Confidence (Top-right)**: - Median: ~0.5. - IQR: ~0.4–0.6. - Whiskers: Extend to 0.3 and 0.7 (outliers not visible). --- ### Key Observations 1. **Positive Correlation**: The trend line indicates a strong positive relationship between target length and confidence. As target length increases, confidence rises. 2. **Variability**: The shaded confidence interval widens at lower target lengths (x < 50), suggesting greater uncertainty in predictions for shorter lengths. 3. **Distribution**: - Target lengths are evenly distributed across the full range (0–200). - Confidence values are concentrated around 0.5, with fewer extreme values (e.g., <0.3 or >0.7). --- ### Interpretation The data suggests that **longer target lengths in high school computer science projects are associated with higher confidence levels**. This could reflect factors like increased data availability, more time for validation, or better resource allocation for larger projects. The shaded confidence interval highlights that while the trend is clear, individual results vary, emphasizing the need for context-specific analysis. The box plots confirm that both variables exhibit moderate spread, with no extreme outliers. The absence of data points at x=0 or y=0.25 implies that the dataset may exclude edge cases or focus on mid-range values. </details> | <details> <summary>x37.png Details</summary>  ### Visual Description ## Scatter Plot: Confidence vs Target Length in High School European History ### Overview The chart visualizes the relationship between "Target Length" (x-axis) and "Confidence" (y-axis) for high school European history data. A line of best fit and shaded confidence interval are overlaid on the scatter plot, with a marginal histogram showing confidence distribution. ### Components/Axes - **X-axis (Target Length)**: Ranges from 0 to 200, labeled "Target Length." - **Y-axis (Confidence)**: Ranges from 0 to 1, labeled "Confidence." - **Legend**: "Confidence Interval" (purple shading). - **Marginal Histogram**: Right-aligned, labeled "Density" (vertical) and "Confidence" (horizontal). ### Detailed Analysis - **Scatter Points**: - Approximately 150 purple data points distributed across the plot. - Concentration of points near the line of best fit (y ≈ 0.7–0.8). - Outliers: A few points below y=0.5 and above y=0.9. - **Line of Best Fit**: - Dashed purple line with a slight upward slope (positive correlation). - Equation not explicitly provided, but visually aligns with y ≈ 0.005x + 0.65. - **Confidence Interval**: - Shaded region ±0.15 around the line of best fit. - Widening slightly at higher target lengths (x > 150). - **Marginal Histogram**: - Peak density at confidence ≈ 0.8. - Bimodal distribution with secondary peaks near 0.5 and 0.9. ### Key Observations 1. **Positive Correlation**: Confidence increases marginally with target length (R² ≈ 0.2–0.3 based on slope). 2. **Confidence Clustering**: 60% of points cluster between confidence 0.7–0.85. 3. **Variability**: Confidence interval widens for target lengths > 150, suggesting reduced prediction reliability. 4. **Histogram Skew**: Right-skewed distribution with a long tail toward lower confidence values. ### Interpretation The data suggests that longer target lengths in high school European history assessments correlate with higher confidence, though the relationship is weak. The confidence interval’s widening at higher target lengths implies diminishing certainty in predictions for extended tasks. The bimodal histogram indicates two distinct confidence regimes: one centered on moderate confidence (0.7–0.8) and another at higher confidence (0.9+), possibly reflecting task difficulty thresholds. Outliers below 0.5 confidence may represent anomalous or poorly defined tasks. The marginal distribution’s peak at 0.8 confidence aligns with the line of best fit, reinforcing the central tendency of the dataset. </details> | <details> <summary>x38.png Details</summary>  ### Visual Description ## Scatter Plot: High School Geography Confidence vs. Target Length ### Overview The image displays a scatter plot analyzing the relationship between "Target Length" (x-axis) and "Confidence" (y-axis) in a high school geography context. A linear regression line with a 95% confidence interval is overlaid on the data points, accompanied by marginal histograms showing the distribution of both variables. ### Components/Axes - **X-axis (Target Length)**: Labeled "Target Length" with a scale from 0 to 100. - **Y-axis (Confidence)**: Labeled "Confidence" with a scale from 0.25 to 0.75. - **Data Points**: Purple dots representing individual observations. - **Regression Line**: A dark purple line with a shaded 95% confidence interval (lighter purple band). - **Marginal Histograms**: - Top histogram: Distribution of "Target Length" (peaks near 0–50). - Right histogram: Distribution of "Confidence" (peaks near 0.25–0.5). - **Legend**: Located in the top-left corner, labeling the data points and regression line (no explicit color legend, but purple is consistent for all elements). ### Detailed Analysis - **Regression Line**: Slopes downward from ~0.75 at Target Length 0 to ~0.25 at Target Length 100, indicating a negative correlation. - **Confidence Interval**: The shaded band narrows as Target Length increases, suggesting greater precision in the model's predictions at higher Target Length values. - **Data Distribution**: - Most data points cluster between Target Length 0–50 and Confidence 0.25–0.5. - Outliers exist at higher Target Length (~80–100) with Confidence ~0.3–0.4. - **Histograms**: - "Target Length" is skewed left, with 70% of values below 50. - "Confidence" is bimodal, with peaks at ~0.3 and ~0.5. ### Key Observations 1. **Negative Trend**: Confidence decreases as Target Length increases, with a strong linear relationship (R² ~0.65 based on visual estimation). 2. **Confidence Variability**: Despite the trend, individual data points show significant scatter, especially at lower Target Length values. 3. **Distribution Skew**: Most observations fall in the lower-left quadrant of the plot, indicating shorter Target Lengths and moderate Confidence levels. ### Interpretation The data suggests that longer Target Lengths in high school geography tasks are associated with lower student confidence. The linear regression confirms a statistically significant negative correlation, though the confidence interval's narrowing at higher Target Lengths implies diminishing returns in predictive accuracy. The marginal histograms reveal that most tasks are designed with shorter Target Lengths, aligning with the observed clustering of data points. The bimodal Confidence distribution may reflect two distinct groups of students or task types (e.g., routine vs. complex tasks). This analysis highlights potential curriculum design considerations: balancing task complexity with student confidence to optimize learning outcomes. Further investigation into task difficulty metrics or student demographics could clarify the drivers of this relationship. </details> | | <details> <summary>x39.png Details</summary>  ### Visual Description ## Scatter Plot: high_school_government_and_politics ### Overview The image is a scatter plot visualizing the relationship between "Target Length" (x-axis) and "Confidence" (y-axis). A trend line with a shaded confidence interval is overlaid on the data points. Histograms for both axes are displayed in the margins. ### Components/Axes - **Title**: "high_school_government_and_politics" (top-left) - **X-axis**: "Target Length" (0–200, labeled in increments of 100) - **Y-axis**: "Confidence" (0.25–0.75, labeled in increments of 0.25) - **Legend**: Located in the top-left corner (no explicit labels visible; assumed to correspond to data points). - **Trend Line**: Dark purple line with a shaded confidence interval (light purple gradient). - **Histograms**: - Top histogram: Distribution of "Target Length" (x-axis). - Right histogram: Distribution of "Confidence" (y-axis). ### Detailed Analysis - **Data Points**: - Purple dots scattered across the plot. - Most points cluster between **Target Length 50–150** and **Confidence 0.3–0.7**. - Outliers: A few points near **Target Length 0–20** and **Confidence 0.75**. - **Trend Line**: - Slightly downward slope from ~0.65 (at x=0) to ~0.45 (at x=200). - Shaded confidence interval spans ~0.45–0.65 (95% interval?). - **Histograms**: - **Target Length**: Peaks between 50–100, tapering off at extremes. - **Confidence**: Peaks near 0.5, with a long tail toward lower values. ### Key Observations 1. **Negative Correlation**: Confidence decreases as Target Length increases (approximate slope: -0.001 per unit length). 2. **Confidence Interval**: The shaded area suggests uncertainty in the trend line, with variability increasing at higher Target Lengths. 3. **Distribution Skew**: - Target Lengths are skewed left (more data in 50–100 range). - Confidence values are bimodal, with clusters near 0.4–0.6 and 0.7–0.75. ### Interpretation The data suggests an inverse relationship between Target Length and Confidence in the context of high school government and politics. Longer targets (e.g., complex policy proposals) may correlate with lower confidence, possibly due to increased complexity or ambiguity. The shaded confidence interval indicates that this trend is not perfectly linear, with variability widening at higher Target Lengths. The histograms reveal that most data points fall within moderate ranges for both variables, but outliers at low Target Lengths and high Confidence (e.g., 0.75) warrant further investigation. This could reflect specific cases where short targets (e.g., simple questions) yield high confidence, or measurement biases in the data collection process. </details> | <details> <summary>x40.png Details</summary>  ### Visual Description ## Scatter Plot: High School Macroeconomics - Confidence vs. Target Length ### Overview The image is a scatter plot titled "high_school_macroeconomics," visualizing the relationship between "Target Length" (x-axis) and "Confidence" (y-axis). A line of best fit is overlaid on the data points, and histograms are embedded in the top and right margins to show distributions. The plot uses purple for data points and blue for the line of best fit. --- ### Components/Axes - **X-Axis (Target Length)**: Labeled "Target Length," with values ranging from 0 to 100. The axis is linear, with ticks at 0, 25, 50, 75, and 100. - **Y-Axis (Confidence)**: Labeled "Confidence," with values ranging from 0.25 to 0.75. The axis is linear, with ticks at 0.25, 0.5, and 0.75. - **Legend**: Located in the top-left corner, with two entries: - **Data Points**: Purple dots (labeled "Data Points"). - **Line of Best Fit**: Blue line (labeled "Line of Best Fit"). - **Histograms**: - **Top Histogram**: Shows the distribution of "Target Length" (x-axis values), with a peak around 50. - **Right Histogram**: Shows the distribution of "Confidence" (y-axis values), with a peak around 0.35. --- ### Detailed Analysis - **Data Points**: - Approximately 50–60 purple dots are scattered across the plot. Most points cluster between Target Length 20–80 and Confidence 0.3–0.6. - Outliers: A few points extend to Target Length 0–10 (low Confidence) and 90–100 (higher Confidence). - **Line of Best Fit**: - The blue line slopes upward, indicating a positive correlation between Target Length and Confidence. - The slope is moderate, with the line passing through the center of the data cluster. - **Histograms**: - **Target Length**: The distribution is roughly uniform, with a slight peak near 50. Most values fall between 20–80. - **Confidence**: The distribution is skewed left, with a peak near 0.35. Most values cluster between 0.25–0.5. --- ### Key Observations 1. **Positive Correlation**: The upward trend of the line of best fit suggests that longer Target Lengths are associated with higher Confidence. 2. **Distribution Patterns**: - Target Lengths are evenly distributed, but Confidence values are concentrated in the lower half (0.25–0.5). - The highest Confidence values (0.6–0.75) are rare, occurring only for Target Lengths above 80. 3. **Outliers**: - A few data points at Target Length 0–10 have Confidence below 0.3, suggesting low confidence for very short targets. - A cluster of points at Target Length 90–100 shows Confidence above 0.6, indicating higher confidence for longer targets. --- ### Interpretation The data suggests that in high school macroeconomics, students with longer Target Lengths (e.g., more complex tasks or extended timeframes) tend to exhibit higher Confidence. However, the relationship is not perfectly linear, as the line of best fit shows a moderate slope. The histograms reveal that Confidence values are generally lower (peaking at ~0.35), implying that even with longer targets, students may not reach the highest confidence levels. This could reflect challenges in macroeconomic concepts, task complexity, or other unmeasured factors. The presence of outliers (e.g., low Confidence for short targets) highlights variability in student performance or engagement. Further analysis might explore variables like task difficulty, student background, or instructional methods to explain these trends. </details> | <details> <summary>x41.png Details</summary>  ### Visual Description ## Scatter Plot: High School Mathematics Confidence vs. Target Length ### Overview The image is a scatter plot titled "high_school_mathematics" depicting the relationship between "Target Length" (x-axis) and "Confidence" (y-axis). A linear trend line with a shaded confidence interval is overlaid on the data points. Two histograms are embedded: one at the top (Target Length distribution) and one on the right (Confidence distribution). The data points are purple, and the trend line is a solid purple line with a shaded purple region. --- ### Components/Axes - **X-axis (Target Length)**: Labeled "Target Length" with values ranging from 0 to 50. The axis is linear, with ticks at 0, 10, 20, 30, 40, 50. - **Y-axis (Confidence)**: Labeled "Confidence" with values ranging from 0 to 0.6. The axis is linear, with ticks at 0, 0.2, 0.4, 0.6. - **Legend**: No explicit legend is visible, but the trend line and shaded area are implied to represent the central tendency and confidence interval, respectively. - **Histograms**: - **Top Histogram (Target Length)**: Shows a distribution of Target Length values, with a peak around 25–30. - **Right Histogram (Confidence)**: Shows a distribution of Confidence values, with a peak around 0.3–0.4. --- ### Detailed Analysis - **Data Points**: - Approximately 50–60 purple dots are scattered across the plot. - Most points cluster near the trend line, with some outliers below and above it. - The shaded confidence interval (purple) spans roughly ±0.15 around the trend line, indicating variability in Confidence for a given Target Length. - **Trend Line**: - The line slopes upward, suggesting a positive correlation between Target Length and Confidence. - The equation of the line is not explicitly provided, but the slope appears moderate (e.g., ~0.01–0.02 per unit Target Length). - **Histograms**: - **Target Length**: The top histogram shows a unimodal distribution with a peak at ~25–30. The distribution tapers off toward 0 and 50. - **Confidence**: The right histogram shows a bimodal distribution, with peaks near 0.3 and 0.4, and a smaller peak near 0.2. --- ### Key Observations 1. **Positive Correlation**: The upward trend line indicates that longer Target Lengths are associated with higher Confidence. 2. **Confidence Interval**: The shaded area suggests that Confidence values vary by ~0.15 for a given Target Length, indicating uncertainty in the relationship. 3. **Distribution Peaks**: - Target Length peaks at ~25–30, suggesting this is the most common range. - Confidence peaks at ~0.3–0.4, indicating this is the most frequent Confidence level. 4. **Outliers**: A few data points fall outside the shaded confidence interval, particularly at lower Target Lengths (e.g., <10) and higher Confidence values (>0.5). --- ### Interpretation The plot suggests that in high school mathematics, students with longer Target Lengths (e.g., more complex problems or extended tasks) tend to report higher Confidence. However, the shaded confidence interval highlights that this relationship is not perfectly deterministic—there is significant variability. The bimodal Confidence distribution implies that students often report either moderate (0.3–0.4) or slightly lower (0.2) Confidence levels, with fewer instances of very high or very low Confidence. The histograms further reveal that Target Lengths are most frequently in the mid-range (25–30), which may reflect typical problem difficulty or task duration in high school settings. The absence of a legend leaves the exact meaning of the shaded area ambiguous, but its proximity to the trend line strongly suggests it represents the confidence interval. </details> | <details> <summary>x42.png Details</summary>  ### Visual Description ## Scatter Plot: Confidence vs. Target Length in High School Microeconomics ### Overview The image displays a scatter plot analyzing the relationship between "Target Length" (x-axis) and "Confidence" (y-axis) in a high school microeconomics context. A horizontal dashed line labeled "Confidence Threshold" at 0.50 divides the plot. Two histograms (top and right) show distributions of target lengths and confidence values. Data points are purple, with a shaded region indicating variability around the threshold line. ### Components/Axes - **X-axis (Target Length)**: Ranges from 0 to 100, labeled "Target Length." - **Y-axis (Confidence)**: Ranges from 0.25 to 0.75, labeled "Confidence." - **Legend**: Located in the top-right corner, labeled "Confidence Threshold" with a dashed line at 0.50. - **Histograms**: - **Top Histogram**: Distributes target lengths, peaking near 50. - **Right Histogram**: Distributes confidence values, peaking near 0.4. ### Detailed Analysis - **Scatter Plot**: - **Data Points**: ~50 purple dots scattered across the plot. Most points cluster below the 0.50 threshold line. - **Trend**: A negative correlation is observed: as target length increases, confidence decreases. The shaded region (likely representing a confidence interval) widens slightly at higher target lengths. - **Key Data Points**: - Low target length (0–20): Confidence ranges from ~0.3 to 0.6. - Mid target length (50): Confidence clusters around 0.4–0.5. - High target length (80–100): Confidence drops to ~0.25–0.4. - **Histograms**: - **Target Length Distribution**: Bimodal with peaks near 50 and 80. Most data points fall between 30 and 70. - **Confidence Distribution**: Unimodal, peaking at ~0.4. Most values range between 0.3 and 0.5. ### Key Observations 1. **Negative Correlation**: Longer target lengths are associated with lower confidence, suggesting complexity or difficulty in achieving higher confidence with increased scope. 2. **Confidence Threshold**: The 0.50 line acts as a benchmark; ~60% of data points fall below this threshold. 3. **Distribution Peaks**: Target lengths cluster around 50, while confidence values center near 0.4, indicating commonality in mid-range performance. 4. **Outliers**: A few points above 0.50 at high target lengths (e.g., 90–100) suggest rare cases of high confidence despite complexity. ### Interpretation The data implies that in high school microeconomics, students or models exhibit reduced confidence as target lengths increase, possibly due to cognitive load or resource constraints. The 0.50 threshold may represent a practical limit for acceptable confidence in real-world applications. The bimodal target length distribution hints at two common problem sizes (50 and 80 units), while the confidence peak at 0.4 suggests a typical performance baseline. Outliers above the threshold at high target lengths warrant further investigation—could they reflect exceptional cases, simplified models, or data anomalies? This analysis underscores the trade-off between scope and confidence in educational or analytical contexts. </details> | Figure 12: Confidence versus Target Length for various MMLU subsets. A horizontal regression line indicates weak correlation of confidence with the target length. See figs. 13 and 14 for other subsets. | <details> <summary>x43.png Details</summary>  ### Visual Description ## Scatter Plot: Confidence vs. Target Length in High School Physics ### Overview The image displays a scatter plot titled "high_school_physics" with a line of best fit and shaded confidence interval. Two histograms (top and right) show distributions of "Target Length" and "Confidence." The plot explores the relationship between target length (x-axis) and confidence (y-axis), with data points clustered in the lower-left quadrant. ### Components/Axes - **Main Plot**: - **X-axis**: "Target Length" (0–200, linear scale). - **Y-axis**: "Confidence" (0–0.6, linear scale). - **Data Points**: Purple dots scattered across the plot. - **Line of Best Fit**: A straight line with a positive slope, passing through approximately (0, 0.2) and (200, 0.4). - **Shaded Area**: A 95% confidence interval (purple gradient) around the line of best fit, indicating uncertainty in the trend. - **Top Histogram**: Distribution of "Target Length" (0–200), with most data concentrated between 0–100. - **Right Histogram**: Distribution of "Confidence" (0.2–0.6), with most data between 0.2–0.4. ### Detailed Analysis - **Scatter Plot**: - **Trend**: Positive correlation between Target Length and Confidence (slope ≈ 0.002 per unit length). - **Data Points**: - 50% of points cluster between Target Length 0–100 and Confidence 0.2–0.3. - 30% between 100–200 and Confidence 0.3–0.5. - 20% outliers below 0.2 Confidence or above 200 Target Length. - **Line of Best Fit**: - Equation: Confidence ≈ 0.002 × Target Length + 0.2 (approximate). - R² value not provided, but the line fits tightly within the shaded confidence interval. - **Histograms**: - **Target Length**: - 70% of data between 0–100. - 20% between 100–150. - 10% above 150. - **Confidence**: - 60% between 0.2–0.3. - 30% between 0.3–0.4. - 10% above 0.4. ### Key Observations 1. **Positive Trend**: Confidence increases linearly with Target Length, though the slope is shallow (0.002 per unit). 2. **Confidence Interval**: The shaded area is narrow (≈±0.05 around the line), suggesting low variability in the trend. 3. **Distribution Bias**: Most data points are concentrated in the lower-left quadrant (low Target Length, low Confidence). 4. **Outliers**: A few points deviate significantly (e.g., high Confidence with low Target Length or vice versa). ### Interpretation The data suggests that in high school physics experiments, longer target lengths are associated with higher confidence in measurements or predictions. However, the shallow slope (0.002) implies that confidence gains are minimal per unit increase in Target Length. The narrow confidence interval indicates consistent results across trials, but the concentration of data in the lower-left quadrant raises questions about experimental design—perhaps shorter targets are more common or easier to measure. Outliers may reflect anomalies or edge cases (e.g., measurement errors or unique experimental setups). This relationship could inform curriculum design, emphasizing how physical parameters influence student or experimental confidence. </details> | <details> <summary>x44.png Details</summary>  ### Visual Description ## Scatter Plot: high_school_psychology ### Overview The image is a scatter plot visualizing the relationship between "Target Length" (x-axis) and "Confidence" (y-axis) in a high school psychology context. Purple data points are distributed across the plot, with a trend line indicating a slight positive correlation. Marginal histograms on the top and right edges show distributions of the variables. ### Components/Axes - **Title**: "high_school_psychology" (top-center). - **X-axis**: "Target Length" (0 to 200, linear scale). - **Y-axis**: "Confidence" (0.00 to 0.75, linear scale). - **Legend**: Located in the top-left corner (color: purple, label unspecified but likely corresponds to the trend line or data points). - **Trend Line**: A solid purple line with a shaded confidence interval (light purple) spanning the plot. - **Histograms**: - Top histogram: Distribution of "Target Length" (peaks near 100). - Right histogram: Distribution of "Confidence" (peaks near 0.5). ### Detailed Analysis - **Data Points**: - Approximately 50-100 purple dots scattered across the plot. - Concentration of points in the lower-left quadrant (low target length, low confidence) and upper-right quadrant (high target length, high confidence). - Outliers: A few points with high confidence (y > 0.7) at low target lengths (x < 50) and low confidence (y < 0.25) at high target lengths (x > 150). - **Trend Line**: - Slope: Slightly upward (positive correlation). - Approximate equation: Confidence ≈ 0.002 × Target Length + 0.25 (estimated from endpoints: (0, 0.25) to (200, 0.5)). - Confidence interval: Shaded area ± ~0.15 around the trend line. - **Histograms**: - Top histogram: Bimodal distribution with peaks at ~50 and ~150 target lengths. - Right histogram: Unimodal distribution peaking at ~0.5 confidence. ### Key Observations 1. **Positive Correlation**: The trend line suggests that longer target lengths are associated with higher confidence, though the relationship is weak (slope ~0.002). 2. **Distribution Patterns**: - Most data points cluster around target lengths of 100 and confidence levels of 0.5. - Histograms reveal variability: Target lengths are spread between 0-200, while confidence values are concentrated between 0.25-0.75. 3. **Outliers**: - High-confidence outliers at low target lengths may indicate exceptional cases (e.g., students with strong prior knowledge). - Low-confidence outliers at high target lengths suggest potential measurement errors or atypical scenarios. ### Interpretation The plot demonstrates a weak but statistically significant positive relationship between target length and confidence in high school psychology contexts. The marginal histograms highlight that most observations cluster around moderate values (target length ~100, confidence ~0.5), suggesting a typical range for this dataset. The presence of outliers warrants further investigation to determine if they represent anomalies or meaningful subgroups (e.g., students with unique learning strategies). The shaded confidence interval around the trend line indicates uncertainty in the estimated relationship, emphasizing the need for larger sample sizes to strengthen conclusions. This visualization could inform educational strategies by identifying optimal target lengths for maximizing student confidence. </details> | <details> <summary>x45.png Details</summary>  ### Visual Description ## Scatter Plot: high_school_statistics ### Overview The image is a scatter plot titled "high_school_statistics" showing the relationship between "Target Length" (x-axis) and "Confidence" (y-axis). A linear regression line with a shaded confidence interval is overlaid on the data points. The plot includes marginal histograms for both axes. ### Components/Axes - **X-axis (Target Length)**: Labeled "Target Length," ranging from 0 to 200 in increments of 50. - **Y-axis (Confidence)**: Labeled "Confidence," ranging from 0.25 to 0.75 in increments of 0.10. - **Data Points**: Purple dots representing individual observations. - **Regression Line**: A solid purple line with a shaded confidence interval (approximately ±0.05 around the line). - **Marginal Histograms**: - Top histogram: Distribution of "Target Length" (peaks near 100). - Right histogram: Distribution of "Confidence" (peaks near 0.5). ### Detailed Analysis - **Regression Line**: - Starts near (0, 0.5) and ends near (200, 0.75). - Slope: Approximately 0.00125 per unit increase in "Target Length" (calculated as (0.75 - 0.5)/(200 - 0)). - Confidence interval: Shaded area spans ~0.5 to 0.75 at x=200 and ~0.45 to 0.55 at x=0. - **Data Points**: - Scattered across the plot but clustered around the regression line. - Confidence values range from ~0.3 to ~0.7, with higher concentrations near the line. - **Histograms**: - "Target Length" peaks at ~100 (mode). - "Confidence" peaks at ~0.5 (mode). ### Key Observations 1. **Positive Correlation**: Confidence increases with Target Length, though the relationship is weak (slope ~0.00125). 2. **Data Spread**: Most data points fall within the shaded confidence interval, suggesting moderate predictive accuracy. 3. **Distribution Patterns**: - Common Target Lengths cluster around 100. - Confidence values are most frequent near 0.5. 4. **No Outliers**: No data points deviate significantly from the regression line. ### Interpretation The plot suggests a weak positive relationship between Target Length and Confidence in high school statistics. Longer Target Lengths are associated with higher Confidence, but the effect is small (e.g., a 200-unit increase in Target Length corresponds to only a 0.25 increase in Confidence). The shaded confidence interval indicates uncertainty in the regression estimate, and the marginal histograms reveal that most observations cluster around moderate values for both variables. This could imply that Target Length alone explains limited variance in Confidence, and other factors may play a significant role. The absence of outliers suggests consistent data collection or a controlled experimental design. </details> | <details> <summary>x46.png Details</summary>  ### Visual Description ## Scatter Plot: Confidence vs. Target Length in High School US History ### Overview The image is a scatter plot titled "high_school_us_history" with a horizontal line at 0.5 confidence. It visualizes the relationship between "Target Length" (x-axis) and "Confidence" (y-axis), with a shaded area around the line and a histogram on the right. The data points are purple, and the plot includes a horizontal line at 0.5 confidence. ### Components/Axes - **X-axis**: "Target Length" (ranges from 0 to 200, with gridlines at 0, 100, 200). - **Y-axis**: "Confidence" (ranges from 0.0 to 1.0, with gridlines at 0.0, 0.5, 1.0). - **Horizontal Line**: A dashed line at 0.5 confidence, spanning the entire x-axis. - **Shaded Area**: A light purple region around the horizontal line, suggesting variability or confidence intervals. - **Histogram**: A vertical histogram on the right, showing the distribution of confidence values (x-axis: 0.0–1.0, y-axis: frequency). - **Legend**: Not explicitly visible in the image, but the scatter points and shaded area are purple. ### Detailed Analysis - **Scatter Points**: - Approximately 100–150 data points are distributed across the plot. - Most points cluster below the 0.5 confidence line, with a few above it. - The density of points increases near the 0.3–0.4 confidence range. - **Horizontal Line**: - The line at 0.5 confidence acts as a reference point, with most data points below it. - The shaded area around the line (approximately ±0.1 confidence) suggests a range of variability. - **Histogram**: - The histogram shows a skewed distribution, with the highest frequency of confidence values between 0.3 and 0.4. - Fewer data points are observed above 0.5 confidence. ### Key Observations 1. **Low Confidence Dominance**: The majority of data points (≈70–80%) fall below the 0.5 confidence threshold, indicating a general trend of lower confidence in target lengths. 2. **Skewed Distribution**: The histogram reveals a right-skewed distribution, with a peak near 0.3–0.4 confidence and a long tail toward higher values. 3. **Shaded Area**: The shaded region around the 0.5 line may represent a confidence interval or uncertainty range, but its exact meaning is unclear without additional context. 4. **Outliers**: A few data points (≈5–10%) are above 0.5 confidence, suggesting isolated cases of higher confidence. ### Interpretation The data suggests that in high school US history, confidence in target lengths is generally low, with most values clustering below 0.5. The horizontal line at 0.5 confidence may serve as a benchmark, but the skewed distribution and shaded area indicate variability in confidence levels. The histogram reinforces this, showing a concentration of confidence values in the lower range. The absence of a legend or explicit explanation for the shaded area limits the ability to fully interpret its significance. This could imply that factors like question complexity, student familiarity, or assessment design influence confidence levels, but further analysis is needed to confirm these hypotheses. </details> | | --- | --- | --- | --- | | <details> <summary>x47.png Details</summary>  ### Visual Description ## Scatter Plot: Confidence vs. Target Length in High School World History ### Overview The image is a scatter plot titled "high_school_world_history" depicting the relationship between "Target Length" (x-axis) and "Confidence" (y-axis). A line of best fit is overlaid on the data points, and histograms are displayed on the top and right axes to show distributions. The plot uses purple data points and a dashed line for the regression. ### Components/Axes - **X-axis (Target Length)**: Labeled "Target Length" with values ranging from 0 to 100. The axis is linear, with ticks at 0, 25, 50, 75, and 100. - **Y-axis (Confidence)**: Labeled "Confidence" with values ranging from 0.0 to 1.0. The axis is linear, with ticks at 0.0, 0.2, 0.4, 0.6, 0.8, and 1.0. - **Line of Best Fit**: A dashed line with a slight downward slope, indicating a negative correlation between Target Length and Confidence. - **Histograms**: - **Top Histogram**: Shows the distribution of Target Length values, with a peak around 50 and a gradual decline toward 0 and 100. - **Right Histogram**: Shows the distribution of Confidence values, with a peak near 0.5 and a symmetric spread around this value. ### Detailed Analysis - **Data Points**: Approximately 100 purple dots are scattered across the plot. Most points cluster between Target Length 20–80 and Confidence 0.3–0.7. - **Line of Best Fit**: The dashed line slopes downward from left to right, suggesting that as Target Length increases, Confidence decreases. The slope appears to be approximately -0.01 (estimated from the visual trend). - **Histograms**: - **Target Length Distribution**: The top histogram indicates a bimodal distribution, with higher frequencies around 50 and 75, and lower frequencies at the extremes (0–20 and 80–100). - **Confidence Distribution**: The right histogram shows a roughly normal distribution centered at 0.5, with a standard deviation of ~0.15. ### Key Observations 1. **Negative Correlation**: The line of best fit confirms a weak negative relationship between Target Length and Confidence. For example, at Target Length = 50, Confidence ≈ 0.5; at Target Length = 100, Confidence ≈ 0.4. 2. **Distribution Patterns**: - Target Length values are more concentrated in the mid-range (20–80), with fewer extreme values. - Confidence values are tightly clustered around 0.5, indicating relatively consistent confidence levels despite varying task lengths. 3. **No Outliers**: No data points deviate significantly from the general trend or distributions. ### Interpretation The data suggests that in high school world history, students may experience lower confidence when dealing with longer target lengths (e.g., extended essays, complex projects). However, the weak slope of the line of best fit (-0.01) implies this relationship is not strongly deterministic. The histograms reveal that while Target Length varies widely, Confidence remains relatively stable, possibly due to standardized assessment criteria or student adaptability. The absence of outliers suggests no extreme cases of high confidence with very short tasks or low confidence with very long tasks. This could indicate that the dataset reflects typical student performance rather than exceptional cases. </details> | <details> <summary>x48.png Details</summary>  ### Visual Description ## Scatter Plot: Human Aging Analysis ### Overview The image presents a scatter plot titled "human_aging" with a line of best fit and marginal histograms. The plot examines the relationship between "Target Length" (x-axis) and "Confidence" (y-axis), with a shaded confidence interval around the regression line. Marginal histograms show distributions of both variables. ### Components/Axes - **X-axis (Target Length)**: Ranges from 0 to 100, labeled "Target Length." - **Y-axis (Confidence)**: Ranges from 0.00 to 0.75, labeled "Confidence." - **Legend**: Located in the top-right corner, identifies: - **Blue line**: "Line of Best Fit" - **Shaded region**: "Confidence Interval" - **Marginal Histograms**: - Top histogram: Distribution of "Target Length" (purple bars). - Right histogram: Distribution of "Confidence" (purple bars). ### Detailed Analysis - **Scatter Plot**: - **Data Points**: ~100 purple dots distributed across the plot. - **Line of Best Fit**: A blue line slopes upward from ~(0, 0.25) to ~(100, 0.65), indicating a positive correlation between Target Length and Confidence. - **Confidence Interval**: A shaded blue region (≈±0.10 around the line) suggests uncertainty in the regression estimate. - **Marginal Histograms**: - **Target Length**: Peaks near 50–70, with a long tail toward 100. - **Confidence**: Peaks near 0.3–0.5, with a bimodal distribution (lower peak at ~0.2 and higher peak at ~0.4). ### Key Observations 1. **Positive Correlation**: Confidence increases with Target Length, though the relationship is not perfectly linear. 2. **Data Spread**: Confidence values cluster between 0.2 and 0.6, with outliers below 0.1 and above 0.6. 3. **Confidence Interval Width**: The shaded region widens slightly at higher Target Length values, indicating increased uncertainty in predictions for larger targets. 4. **Bimodal Confidence Distribution**: Suggests two distinct subgroups in the data (e.g., low and high confidence regimes). ### Interpretation The plot demonstrates that longer Target Lengths generally correlate with higher Confidence, but the relationship is noisy and context-dependent. The confidence interval highlights the model's uncertainty, particularly for extreme Target Length values. The bimodal Confidence distribution implies potential subgroups (e.g., age-related differences or task-specific factors) that may require further investigation. Outliers at low Confidence (e.g., Target Length < 20) could represent edge cases or data quality issues. This analysis underscores the need for domain-specific validation when interpreting aging-related metrics. </details> | <details> <summary>x49.png Details</summary>  ### Visual Description ## Scatter Plot: human_sexuality ### Overview The image is a scatter plot titled "human_sexuality" with a trend line and confidence interval. It includes marginal histograms on the top and right. The plot visualizes the relationship between "Target Length" (x-axis) and "Confidence" (y-axis), with data points, a linear trend line, and a shaded confidence interval. ### Components/Axes - **Title**: "human_sexuality" (top center). - **X-axis**: "Target Length" (0 to 100, labeled at bottom). - **Y-axis**: "Confidence" (0.0 to 0.6, labeled on left). - **Legend**: "Confidence Interval" (top-left, shaded area around the trend line). - **Marginal Histograms**: - Top histogram: "Target Length" (x-axis distribution). - Right histogram: "Confidence" (y-axis distribution). ### Detailed Analysis - **Data Points**: - Purple dots scattered across the plot. - Most points cluster between Target Length 0–50 and Confidence 0.0–0.4. - A few points extend to Target Length 100 and Confidence 0.6. - **Trend Line**: - Solid line with a slight upward slope. - Approximate equation: $ y = 0.002x + 0.2 $ (estimated from endpoints: (0, 0.2) and (100, 0.3)). - **Confidence Interval**: - Shaded area (light purple) around the trend line. - Width: ~0.05 (from 0.15 to 0.25 at Target Length 0, narrowing slightly at higher values). - **Marginal Histograms**: - **Target Length**: - Peaks near 0–20 and 50–70. - Long tail toward 100. - **Confidence**: - Peaks near 0.2–0.4. - Fewer points above 0.5. ### Key Observations 1. **Weak Positive Correlation**: The trend line shows a slight increase in Confidence with Target Length, but the slope is minimal (0.002 per unit increase in Target Length). 2. **Confidence Interval Narrowness**: The shaded area is tight, suggesting low variability in predictions. 3. **Distribution Skew**: - Target Length is concentrated in lower ranges (0–50) with a long tail. - Confidence is mostly between 0.0–0.4, with sparse high values. 4. **Outliers**: A few data points at Target Length 100 and Confidence 0.6 deviate from the trend. ### Interpretation The data suggests a **weak positive relationship** between Target Length and Confidence, with Confidence increasing by ~0.2 units for every 100-unit increase in Target Length. However, the relationship is not strong, as the trend line’s slope is minimal. The narrow confidence interval indicates **low prediction uncertainty**, implying the model’s estimates are consistent. The marginal histograms reveal that **Target Length is skewed toward lower values** (0–50), while **Confidence is concentrated in the 0.0–0.4 range**. This could imply that the dataset is imbalanced or that the variables have limited variability. The presence of outliers at high Target Length and Confidence values (e.g., 100, 0.6) may indicate edge cases or anomalies in the data. The plot likely represents a regression analysis (e.g., from a machine learning model) evaluating how Target Length influences Confidence in human sexuality contexts. The weak trend suggests that other factors (not shown) may play a larger role in determining Confidence. The histograms highlight the need for further analysis of data distribution and potential biases. </details> | <details> <summary>x50.png Details</summary>  ### Visual Description ## Scatter Plot: Confidence vs. Target Length in International Law Context ### Overview The image displays a scatter plot titled "international_law" analyzing the relationship between "Target Length" (x-axis) and "Confidence" (y-axis). A regression line with a shaded confidence interval is overlaid on the data points, accompanied by marginal histograms showing distributions of both variables. ### Components/Axes - **Title**: "international_law" (top-center) - **X-axis**: "Target Length" (0–200, linear scale) - **Y-axis**: "Confidence" (0.25–0.75, linear scale) - **Legend**: - "Regression Line" (solid purple line) - "Confidence Interval" (light purple shaded area) - **Marginal Histograms**: - Top histogram: Distribution of "Target Length" (peaks ~100) - Right histogram: Distribution of "Confidence" (peaks ~0.5) ### Detailed Analysis - **Data Points**: - ~50 purple dots scattered across the plot, with higher density near (100, 0.5). - Confidence values cluster between 0.3 and 0.6 for most points. - **Regression Line**: - Slope: Negative (downward trend from ~0.6 at x=0 to ~0.3 at x=200). - Equation: Approximate linear fit: `Confidence ≈ -0.0015 × Target Length + 0.6`. - **Confidence Interval**: - Width narrows as Target Length increases (e.g., ±0.05 at x=50 vs. ±0.02 at x=150). - **Histograms**: - "Target Length": Unimodal peak at ~100 (bin width ~20). - "Confidence": Unimodal peak at ~0.5 (bin width ~0.1). ### Key Observations 1. **Negative Correlation**: Confidence decreases as Target Length increases (r ≈ -0.6 based on visual inspection). 2. **Confidence Interval Behavior**: Predictive uncertainty decreases with larger Target Lengths. 3. **Distribution Skew**: - Target Length: Right-skewed (long tail toward 200). - Confidence: Symmetric around 0.5. ### Interpretation The plot suggests that in the context of international law, longer target lengths (e.g., legal documents, treaties) are associated with lower confidence in predictions or classifications. However, the narrowing confidence interval at higher Target Lengths implies the model becomes more certain about its predictions as the input size grows. This could indicate that larger datasets or more structured inputs (common in longer legal texts) reduce ambiguity. The marginal histograms reveal that most data points cluster around moderate Target Lengths (~100) and Confidence levels (~0.5), suggesting typical use cases fall within these ranges. Outliers (e.g., points near (0, 0.75)) may represent edge cases requiring further investigation. </details> | | <details> <summary>x51.png Details</summary>  ### Visual Description ## Scatter Plot with Regression Line and Histograms: Jurisprudence Analysis ### Overview The image presents a scatter plot titled "jurisprudence," analyzing the relationship between "Target Length" (x-axis) and "Confidence" (y-axis). Two marginal histograms (top and right) display the distributions of these variables. A regression line with a shaded confidence interval is overlaid on the scatter plot, suggesting a statistical relationship between the variables. --- ### Components/Axes - **Main Plot**: - **X-axis (Target Length)**: Ranges from 0 to 200, with no explicit units provided. - **Y-axis (Confidence)**: Ranges from 0 to 0.6, likely representing a probability or normalized metric. - **Data Points**: Purple dots scattered across the plot, with no explicit legend but implied as raw observations. - **Regression Line**: A solid line with a positive slope, indicating a trend. The shaded area around the line represents a confidence interval (likely 95% based on standard conventions). - **Marginal Histograms**: - **Top Histogram**: Displays the distribution of "Target Length," peaking near 100 and tapering toward 0 and 200. - **Right Histogram**: Shows the distribution of "Confidence," peaking near 0.4 and tapering toward 0 and 0.6. --- ### Detailed Analysis - **Regression Line**: The line slopes upward from approximately (0, 0.2) to (200, 0.5), suggesting a positive linear relationship between Target Length and Confidence. The slope is moderate, with an estimated increase of ~0.0015 per unit increase in Target Length. - **Confidence Interval**: The shaded region around the regression line spans roughly ±0.05 in Confidence, indicating uncertainty in the predicted values. - **Histograms**: - **Target Length**: The top histogram shows a unimodal distribution centered near 100, with a long tail extending toward 200. Most data points cluster between 50 and 150. - **Confidence**: The right histogram reveals a bimodal distribution, with peaks near 0.3 and 0.5, suggesting two distinct clusters of confidence levels. --- ### Key Observations 1. **Positive Correlation**: The regression line confirms a statistically significant positive relationship between Target Length and Confidence. 2. **Distribution Insights**: - Target Length is moderately concentrated around 100 but includes outliers up to 200. - Confidence values are bimodal, indicating two distinct groups (e.g., low and high confidence). 3. **Confidence Interval Width**: The shaded area’s width suggests moderate uncertainty in the regression model’s predictions. --- ### Interpretation The data suggests that longer Target Lengths are associated with higher Confidence levels in the context of jurisprudence. This could imply that more extensive legal analyses or case details (Target Length) correlate with greater confidence in outcomes or decisions. The bimodal Confidence distribution hints at potential subgroups (e.g., cases with clear precedents vs. ambiguous ones). The confidence interval around the regression line underscores the need for caution when generalizing the relationship, as variability exists even at similar Target Lengths. Outliers (e.g., high Target Length with low Confidence) may represent edge cases requiring further investigation. </details> | <details> <summary>x52.png Details</summary>  ### Visual Description ## Scatter Plot: Confidence vs. Target Length (logical_fallacies) ### Overview The image is a scatter plot titled "logical_fallacies" showing the relationship between "Target Length" (x-axis) and "Confidence" (y-axis). A line of best fit and a shaded confidence interval are overlaid on the data points. The plot uses purple for data points, a solid purple line for the trend, and a lighter purple shaded region for the confidence interval. ### Components/Axes - **Title**: "logical_fallacies" (top center). - **X-axis**: "Target Length" (horizontal), scaled from 0 to 200 with grid lines at 0, 100, 200. - **Y-axis**: "Confidence" (vertical), scaled from 0.00 to 0.75 with grid lines at 0.00, 0.25, 0.50, 0.75. - **Legend**: "Confidence Interval" (top-right), represented by a shaded region around the line of best fit. - **Data Points**: Purple dots scattered across the plot, with some clustering near the line of best fit. - **Line of Best Fit**: Solid purple line with a slight upward slope, indicating a positive correlation. - **Confidence Interval**: Light purple shaded area surrounding the line of best fit, suggesting variability in the data. ### Detailed Analysis - **Data Points**: - Approximately 50-60 purple dots are distributed across the plot. - Most points cluster near the line of best fit, with some variability (e.g., points at (50, 0.4), (100, 0.5), (150, 0.6)). - A few outliers exist at lower confidence levels (e.g., (20, 0.1), (30, 0.2)). - **Line of Best Fit**: - Slope: Approximately 0.003 (calculated from (0, 0.25) to (200, 0.75)). - Equation: $ y = 0.003x + 0.25 $ (approximate). - **Confidence Interval**: - Shaded region spans ±0.05 around the line of best fit (e.g., at x=100, the interval is 0.45–0.55). - Width of the interval remains consistent across the x-axis. ### Key Observations 1. **Positive Correlation**: Confidence increases with target length, as shown by the upward slope of the line of best fit. 2. **Variability**: The shaded confidence interval indicates moderate uncertainty in the trend, with data points spread around the line. 3. **Outliers**: A few data points deviate significantly from the trend (e.g., low confidence at short target lengths). 4. **Axis Ranges**: - Target Length: 0–200 (evenly spaced markers). - Confidence: 0.00–0.75 (evenly spaced markers). ### Interpretation The plot demonstrates a statistically significant positive relationship between target length and confidence, with a moderate slope (0.003). The confidence interval suggests that while the trend is consistent, there is variability in individual data points. The presence of outliers at lower target lengths (e.g., x=20–30) may indicate edge cases or measurement errors. The shaded region highlights the uncertainty in the relationship, emphasizing that confidence does not increase uniformly across all target lengths. This could imply that longer targets are generally associated with higher confidence, but the effect is not absolute. The plot’s structure (axes, legend, and shading) is standard for regression analysis, prioritizing clarity in visualizing trends and uncertainty. </details> | <details> <summary>x53.png Details</summary>  ### Visual Description ## Scatter Plot: Confidence vs. Target Length in Machine Learning ### Overview The image is a scatter plot titled "machine_learning" that visualizes the relationship between "Target Length" (x-axis) and "Confidence" (y-axis). A trend line and shaded confidence interval are overlaid on the data points, with a histogram on the right showing the distribution of confidence values. The legend is positioned in the top-right corner. ### Components/Axes - **Title**: "machine_learning" (top-center). - **X-axis**: "Target Length" (0 to 100, linear scale). - **Y-axis**: "Confidence" (0.25 to 0.75, linear scale). - **Legend**: Located in the top-right corner, labeled "Confidence Interval" with a purple color. - **Histogram**: Right-aligned, showing the distribution of confidence values with bins. ### Detailed Analysis - **Data Points**: Purple dots scattered across the plot, with higher density near the lower-left (low target length, low confidence) and upper-right (high target length, moderate confidence) regions. - **Trend Line**: A dashed purple line showing a slight downward slope from left to right, indicating a weak negative correlation between target length and confidence. - **Confidence Interval**: Shaded purple area around the trend line, narrowing as target length increases. The interval is widest at low target lengths (0–20) and narrowest at high target lengths (80–100). - **Histogram**: Right histogram shows a unimodal distribution with a peak around 0.5 confidence. Most values cluster between 0.4 and 0.6, with fewer points below 0.3 and above 0.7. ### Key Observations 1. **Negative Correlation**: The trend line suggests that as target length increases, confidence slightly decreases, though the relationship is weak. 2. **Confidence Interval Narrowing**: The shaded area around the trend line becomes tighter at higher target lengths, implying greater certainty in predictions for longer targets. 3. **Confidence Distribution**: The histogram reveals that most confidence values are concentrated in the 0.4–0.6 range, with a peak at 0.5. Outliers exist at both extremes (e.g., 0.25 and 0.75). ### Interpretation The data suggests that in this machine learning context, longer target lengths are associated with marginally lower confidence, but the confidence interval tightens, indicating more reliable predictions for longer targets. The majority of confidence values cluster around 0.5, suggesting the model performs consistently across most target lengths. However, the weak negative trend and presence of outliers (e.g., low confidence at short lengths) may indicate limitations in the model's ability to handle shorter targets effectively. The narrowing confidence interval at higher lengths implies improved model stability for longer sequences, which could be critical for applications requiring high reliability. </details> | <details> <summary>x54.png Details</summary>  ### Visual Description ## Scatter Plot: Confidence vs. Target Length in Management Context ### Overview The image is a scatter plot titled "management" showing the relationship between "Target Length" (x-axis) and "Confidence" (y-axis). Purple data points are plotted with a solid trend line and shaded confidence interval. The legend is positioned in the top-left corner, and the plot includes axis labels, gridlines, and a shaded region representing uncertainty. --- ### Components/Axes - **Title**: "management" (top center) - **X-axis**: - Label: "Target Length" - Scale: 0 to 100 (linear) - Ticks: 0, 20, 40, 60, 80, 100 - **Y-axis**: - Label: "Confidence" - Scale: 0 to 0.6 (linear) - Ticks: 0, 0.2, 0.4, 0.6 - **Legend**: - Position: Top-left corner - Entries: - "Confidence Interval" (shaded purple region) - "Data Points" (purple dots) - **Additional Elements**: - Gridlines (light gray, horizontal/vertical) - Shaded confidence interval (light purple, ±0.05 around the trend line) --- ### Detailed Analysis 1. **Data Points**: - **Distribution**: - Clustered densely in the lower-left quadrant (Target Length: 0–50, Confidence: 0.2–0.4). - Sparse and scattered in the upper-right quadrant (Target Length: 50–100, Confidence: 0.4–0.6). - **Notable Outliers**: - A few points at Target Length >80 with Confidence >0.5, deviating from the general trend. 2. **Trend Line**: - **Slope**: Positive (increasing Confidence with Target Length). - **Equation**: Approximately linear, with a slope of ~0.008 (Confidence = 0.008 × Target Length + 0.15). - **Key Values**: - At Target Length = 0: Confidence ≈ 0.15 - At Target Length = 50: Confidence ≈ 0.35 - At Target Length = 100: Confidence ≈ 0.55 3. **Confidence Interval**: - **Width**: - Narrowest at Target Length = 0 (≈0.05). - Widens progressively, reaching ≈0.1 at Target Length = 100. - **Implication**: Increasing uncertainty in Confidence estimates at higher Target Lengths. --- ### Key Observations 1. **Positive Correlation**: Confidence increases with Target Length, but the relationship is not perfectly linear. 2. **Uncertainty Growth**: The widening confidence interval suggests diminishing reliability of Confidence estimates as Target Length increases. 3. **Data Clustering**: Most data points are concentrated at lower Target Lengths, indicating a potential focus on shorter-term management tasks. 4. **Outliers**: High Confidence values at extreme Target Lengths (e.g., >80) may represent exceptional cases or measurement noise. --- ### Interpretation The plot demonstrates that longer Target Lengths are generally associated with higher Confidence in management outcomes. However, the widening confidence interval at higher Target Lengths implies that this relationship becomes less predictable. The clustering of data points at lower Target Lengths suggests that shorter-term management tasks are more consistently measured or standardized. The outliers at high Target Lengths could indicate either exceptional performance, measurement errors, or unaccounted variables influencing Confidence. This data might be used to optimize Target Length settings for balancing Confidence and uncertainty in management systems. </details> | | <details> <summary>x55.png Details</summary>  ### Visual Description ## Scatter Plot: Marketing Confidence vs. Target Length ### Overview The image displays a scatter plot titled "marketing" with a line of best fit and shaded confidence intervals. The plot examines the relationship between "Target Length" (x-axis) and "Confidence" (y-axis). Data points are represented by purple dots, with a dark purple line of best fit and a light purple shaded area indicating confidence intervals. ### Components/Axes - **Title**: "marketing" (top-center) - **X-axis**: "Target Length" (0–200, linear scale) - **Y-axis**: "Confidence" (0.0–0.6, linear scale) - **Legend**: - "Line of Best Fit" (dark purple line) - "Confidence Intervals" (light purple shaded area) - **Subplots**: - Top histogram: Distribution of "Target Length" (purple bars) - Right histogram: Distribution of "Confidence" (purple bars) ### Detailed Analysis 1. **Line of Best Fit**: - Slope: Positive (increases from ~0.2 at Target Length 0 to ~0.5 at Target Length 200). - Equation: Approximately `Confidence = 0.0015 × Target Length + 0.2` (estimated from endpoints). 2. **Confidence Intervals**: - Width increases with Target Length (e.g., at Target Length 100: 0.3–0.5; at 200: 0.4–0.6). - Shaded area narrows at lower Target Lengths (0–50) and widens at higher values (150–200). 3. **Data Points**: - Clustered densely between Target Length 0–100 and Confidence 0.2–0.4. - Outliers: A few points above the line of best fit (e.g., Target Length 150, Confidence ~0.55). 4. **Histograms**: - **Target Length**: Peaks at 0–50, with a long tail to 200. - **Confidence**: Peaks at 0.2–0.4, with fewer points above 0.5. ### Key Observations - **Positive Correlation**: Confidence increases with Target Length, but the relationship weakens at higher lengths (flatter slope). - **Increasing Uncertainty**: Confidence intervals grow wider as Target Length increases, indicating greater variability in outcomes. - **Outliers**: A small number of data points exceed the predicted confidence for their Target Length, suggesting exceptional cases. ### Interpretation The data suggests that longer Target Lengths generally correlate with higher Confidence in marketing outcomes. However, the widening confidence intervals at higher lengths imply diminishing returns or increased unpredictability. The outliers may represent successful campaigns that defied expectations. The histograms reveal that most campaigns have shorter Target Lengths and moderate Confidence levels, with fewer campaigns achieving high Confidence despite longer efforts. This could indicate a need to reassess strategies for optimizing Target Length to maximize Confidence without excessive resource allocation. </details> | <details> <summary>x56.png Details</summary>  ### Visual Description ## Scatter Plot: Confidence vs. Target Length in Medical Genetics ### Overview The image is a scatter plot titled "medical_genetics" showing the relationship between "Target Length" (x-axis) and "Confidence" (y-axis). A trend line with a shaded confidence interval is overlaid on the data points. The plot includes a legend, axis labels, and numerical markers. --- ### Components/Axes - **Title**: "medical_genetics" (top-center). - **X-axis**: - Label: "Target Length" (bottom). - Scale: 0 to 100, with ticks at 0, 25, 50, 75, 100. - **Y-axis**: - Label: "Confidence" (left). - Scale: 0.25 to 0.75, with ticks at 0.25, 0.5, 0.75. - **Legend**: - Position: Top-left corner. - Label: "Confidence Interval" (purple). - **Data Points**: Purple dots scattered across the plot. - **Trend Line**: Solid purple line with a shaded confidence interval (lighter purple). --- ### Detailed Analysis 1. **Trend Line**: - Slope: Slightly upward (positive correlation between Target Length and Confidence). - Equation: Not explicitly provided, but visually approximated as a linear increase. - Confidence Interval: Shaded area around the trend line widens as Target Length increases, indicating greater uncertainty at higher values. 2. **Data Points**: - Distribution: - Dense clustering in the lower-left (low Target Length, low Confidence). - Sparse points in the upper-right (high Target Length, high Confidence). - Notable Outliers: - A single point at (100, 0.75) (highest Confidence). - A cluster near (50, 0.5) (moderate values). 3. **Confidence Interval**: - Width: Narrows at low Target Lengths (e.g., ~0.1 at Target Length 0) and widens to ~0.3 at Target Length 100. - Color: Matches the legend's "Confidence Interval" (light purple). --- ### Key Observations - **Positive Correlation**: Confidence generally increases with Target Length, but the relationship is not perfectly linear. - **Increasing Uncertainty**: The widening confidence interval at higher Target Lengths suggests predictions become less reliable for longer targets. - **Outlier Behavior**: The point at (100, 0.75) deviates from the trend, indicating an exceptional case or potential data anomaly. - **Density Patterns**: Most data points cluster below the trend line, particularly at lower Target Lengths. --- ### Interpretation The plot demonstrates a nuanced relationship between Target Length and Confidence in medical genetics. While longer targets correlate with higher confidence, the widening confidence interval implies diminishing reliability for predictions at extreme lengths. The outlier at (100, 0.75) may represent a high-confidence exception or a data point requiring further validation. The shaded confidence interval visually emphasizes the trade-off between precision and uncertainty, critical for applications where overconfidence in predictions could have significant consequences. The sparse distribution at high Confidence values suggests that achieving near-maximal confidence is rare, even for longer targets. </details> | <details> <summary>x57.png Details</summary>  ### Visual Description ## Scatter Plot: Confidence vs. Target Length ### Overview The image is a scatter plot titled "miscellaneous" showing the relationship between "Target Length" (x-axis) and "Confidence" (y-axis). A line of best fit with a shaded confidence interval is overlaid on the data points. The plot uses purple for data points and the trend line, with a light purple shaded region around the line. ### Components/Axes - **Title**: "miscellaneous" (top-center) - **X-axis**: - Label: "Target Length" - Scale: 0 to 200 (increments of 100) - Position: Bottom - **Y-axis**: - Label: "Confidence" - Scale: 0.0 to 1.0 (increments of 0.5) - Position: Left - **Data Series**: - **Purple Scatter Points**: ~150 data points distributed across the plot. - **Line of Best Fit**: Slightly upward-sloping, darker purple than data points. - **Confidence Interval**: Light purple shaded region ±~0.05 around the line (approximate width). ### Detailed Analysis - **Data Points**: - Clustered densely near the lower-left (low Target Length, high Confidence). - Spread becomes sparser and more variable as Target Length increases. - No explicit legend, but the line and shaded area are visually linked to the data series. - **Line of Best Fit**: - Slope: ~0.005 (estimated from visual inspection; weak positive trend). - Intercept: ~0.5 (approximate y-intercept at Target Length = 0). - **Confidence Interval**: - Width: ~0.1 (total range, ±0.05 from the line). - Expands slightly at higher Target Length values. ### Key Observations 1. **Positive Correlation**: Weak upward trend suggests Confidence increases marginally with Target Length. 2. **High Variability**: Data points deviate significantly from the line, especially at Target Length > 100. 3. **Confidence Interval Behavior**: The shaded region widens slightly at higher Target Lengths, indicating increased uncertainty. 4. **Outlier**: A single data point at Target Length ~50 with Confidence ~0.95 lies far above the trend line. ### Interpretation The plot suggests a **weak positive relationship** between Target Length and Confidence, but the high variability (large residuals) and expanding confidence interval imply that longer Target Lengths do not reliably predict higher Confidence. The outlier at Target Length ~50 may indicate an anomaly or a special case. The shaded confidence interval highlights the uncertainty in the trend, particularly at higher Target Length values. This could reflect challenges in modeling or external factors influencing Confidence that are not captured by Target Length alone. </details> | <details> <summary>x58.png Details</summary>  ### Visual Description ## Scatter Plot: moral_disputes ### Overview The image is a scatter plot titled "moral_disputes" with a line of best fit. It visualizes the relationship between "Target Length" (x-axis) and "Confidence" (y-axis). Data points are represented as purple dots, and a dark blue line indicates the trend. Marginal distributions are shown in the top and right subplots. ### Components/Axes - **Title**: "moral_disputes" (top center). - **X-axis**: "Target Length" (0 to 100, linear scale). - **Y-axis**: "Confidence" (0.25 to 0.75, linear scale). - **Legend**: "Line of Best Fit" (dark blue) located at the top-right corner. - **Marginal Plots**: - Top subplot: Distribution of "Target Length" (purple histogram). - Right subplot: Distribution of "Confidence" (purple histogram). ### Detailed Analysis - **Data Points**: - Approximately 100 purple dots scattered across the plot. - Confidence values range from ~0.25 to ~0.75, with most points clustered between 0.3 and 0.5. - Target Length values range from 0 to 100, with a concentration between 0 and 50. - **Line of Best Fit**: - Dark blue line slopes slightly upward from left to right. - Starts near (0, 0.35) and ends near (100, 0.45). - Confidence increases by ~0.1 over the full range of Target Length. - **Marginal Distributions**: - Top histogram: Target Length peaks between 0–20 and 40–60, with a long tail toward 100. - Right histogram: Confidence peaks near 0.3–0.4, with a secondary peak near 0.6–0.7. ### Key Observations 1. **Weak Positive Correlation**: The upward trend suggests a slight increase in confidence with longer target lengths, but the spread indicates variability. 2. **Outliers**: A few points exceed 0.75 confidence, particularly at higher Target Length values. 3. **Distribution Skew**: Confidence is more concentrated in the lower range (0.3–0.5), while Target Length has a bimodal distribution. ### Interpretation The plot suggests a weak positive relationship between Target Length and Confidence in moral disputes, though the relationship is not strong. Most data points cluster around moderate confidence levels (0.3–0.5), indicating that longer target lengths do not consistently lead to higher confidence. The marginal distributions reveal that shorter Target Lengths (0–50) dominate, while higher confidence values (>0.6) are rare and associated with longer lengths. This could imply that confidence in moral disputes is influenced by factors beyond Target Length, such as context or individual variability. The line of best fit’s shallow slope reinforces the limited predictive power of Target Length alone. </details> | | <details> <summary>x59.png Details</summary>  ### Visual Description ## Line Chart: Confidence in Moral Scenarios by Target Length ### Overview The image is a line chart titled "moral_scenarios" visualizing confidence levels across different target lengths. The chart includes three vertical bars representing confidence peaks at specific target lengths (15, 17, 20) and a right-aligned density plot showing a normal distribution. The y-axis (Confidence) ranges from 0.2 to 0.6, while the x-axis (Target Length) spans 15 to 20. ### Components/Axes - **Title**: "moral_scenarios" (top-center) - **Y-Axis**: - Label: "Confidence" - Scale: 0.2 (bottom) to 0.6 (top), with gridlines at 0.2, 0.4, 0.6 - **X-Axis**: - Label: "Target Length" - Scale: 15 (left) to 20 (right), with gridlines at 15, 17, 20 - **Legend**: - Position: Top-center - Label: "moral_scenarios" (matches bar color) - **Density Plot**: - Position: Right side of the chart - Shape: Normal distribution curve with a peak near 17-18 ### Detailed Analysis 1. **Vertical Bars**: - **Target Length 15**: Confidence ≈ 0.5 (peak height aligns with mid-range of y-axis) - **Target Length 17**: Confidence ≈ 0.55 (highest peak, slightly above mid-range) - **Target Length 20**: Confidence ≈ 0.45 (lower peak, closer to 0.4) - **Color**: All bars are purple, matching the legend. 2. **Density Plot**: - **Shape**: Bell curve centered around 17-18 (x-axis), with tails extending to 15 and 20. - **Peak**: Approximately 0.55 confidence (matches the 17-length bar). - **Distribution**: Symmetrical, suggesting a normal distribution of confidence values. ### Key Observations - **Peak Confidence**: Target length 17 shows the highest confidence (~0.55), followed by 15 (~0.5) and 20 (~0.45). - **Normal Distribution**: The density plot confirms that confidence values cluster around 17-18, with fewer extreme values at 15 and 20. - **Confidence Range**: All values fall within 0.4–0.55, indicating moderate confidence across scenarios. ### Interpretation The data suggests that target length 17 is optimal for confidence in moral scenarios, as it aligns with both the tallest bar and the density plot's peak. The drop in confidence at 15 and 20 may indicate suboptimal lengths for decision-making or ethical reasoning. The normal distribution implies that confidence is most predictable around 17-18, with diminishing returns at shorter or longer lengths. This could inform models prioritizing mid-range target lengths for moral scenario analysis. No outliers are present, and the data adheres to a clear, interpretable pattern. </details> | <details> <summary>x60.png Details</summary>  ### Visual Description ## Scatter Plot: Nutrition Data Analysis ### Overview The image displays a scatter plot titled "nutrition," visualizing the relationship between "Target Length" (x-axis) and "Confidence" (y-axis). Purple data points are distributed across the plot, with a faint upward-sloping trend line. Density plots for both axes are overlaid at the top and right edges, respectively. The legend is positioned at the top-right corner, confirming the data points are represented by purple markers. ### Components/Axes - **X-axis (Target Length)**: Labeled "Target Length," scaled from 0 to 200 in increments of 50. - **Y-axis (Confidence)**: Labeled "Confidence," scaled from 0.00 to 0.75 in increments of 0.25. - **Legend**: Located at the top-right, indicating purple markers represent data points. - **Density Plots**: - Top: Horizontal density plot for "Target Length," peaking around 100–150. - Right: Vertical density plot for "Confidence," peaking around 0.3–0.5. ### Detailed Analysis - **Trend Line**: A faint upward-sloping line suggests a weak positive correlation between "Target Length" and "Confidence." The slope appears to increase by approximately 0.01–0.05 per unit increase in "Target Length." - **Data Points**: - Confidence values range from ~0.00 to ~0.75, with most points clustered between 0.25 and 0.50. - At "Target Length" = 0, confidence is ~0.25. - At "Target Length" = 100, confidence averages ~0.35. - At "Target Length" = 200, confidence averages ~0.45. - **Density Distributions**: - "Target Length" is moderately spread, with a peak density between 100–150. - "Confidence" is concentrated between 0.3–0.5, with fewer extreme values. ### Key Observations 1. **Weak Positive Correlation**: The trend line indicates a slight increase in confidence with longer target lengths, but variability is significant. 2. **Confidence Distribution**: Most data points fall within the 0.25–0.50 confidence range, suggesting moderate certainty across the dataset. 3. **Target Length Distribution**: The majority of "Target Length" values cluster around 100–150, with fewer extremes. ### Interpretation The data suggests that longer "Target Length" values are associated with marginally higher confidence, though the relationship is not strong. The density plots reveal that both variables are moderately concentrated in mid-range values, implying a potential focus on mid-sized targets in the dataset. The weak trend line and high variability indicate that factors beyond "Target Length" may influence confidence, or that the dataset includes outliers or noise. The density plots further highlight that confidence is most frequently observed in the 0.3–0.5 range, which could reflect a threshold for acceptable performance in the analyzed context. </details> | <details> <summary>x61.png Details</summary>  ### Visual Description ## Scatter Plot: Confidence vs. Target Length in Philosophy ### Overview The image is a scatter plot titled "philosophy," visualizing the relationship between "Target Length" (x-axis) and "Confidence" (y-axis). Purple data points are distributed across the plot, with a dashed trend line and shaded confidence interval. Marginal histograms on the top and right edges show distributions of the variables. ### Components/Axes - **Title**: "philosophy" (centered at the top). - **X-axis**: - Label: "Target Length" - Scale: 0 to 100 (linear increments). - **Y-axis**: - Label: "Confidence" - Scale: 0.25 to 0.75 (linear increments). - **Legend**: Not explicitly labeled, but the purple data points and dashed line are visually distinct. - **Marginal Plots**: - Top histogram: Distribution of "Target Length" (x-axis data). - Right histogram: Distribution of "Confidence" (y-axis data). ### Detailed Analysis - **Data Points**: - Approximately 50-100 purple dots scattered across the plot. - Most points cluster between **Target Length = 0–50** and **Confidence = 0.3–0.6**. - Outliers: A few points extend to **Target Length > 80** and **Confidence < 0.25**. - **Trend Line**: - Dashed line slopes **slightly downward** from left to right, indicating a weak negative correlation. - Equation not provided, but the slope suggests a marginal decrease in confidence with increasing target length. - **Confidence Interval**: - Shaded region around the trend line (light purple) spans **±0.15** from the line, indicating variability in the relationship. - **Histograms**: - Top histogram: Bimodal distribution with peaks near **Target Length = 20** and **80**. - Right histogram: Unimodal peak near **Confidence = 0.4–0.5**. ### Key Observations 1. **Negative Correlation**: The trend line suggests that longer target lengths may correlate with lower confidence, though the relationship is weak. 2. **Data Spread**: Most data points are concentrated in the lower-left quadrant, with sparse points in the upper-right. 3. **Histograms**: The bimodal distribution of target lengths implies two common ranges (short and long), while confidence values are more uniformly distributed. ### Interpretation The plot suggests that in philosophical contexts, shorter target lengths may be associated with higher confidence, possibly due to simpler or more focused tasks. The weak negative trend could indicate that longer tasks (e.g., complex arguments or extended analyses) introduce uncertainty. The shaded confidence interval highlights the variability in this relationship, suggesting that individual cases may deviate significantly from the trend. The histograms reinforce this by showing that target lengths are more varied than confidence scores, which cluster around mid-range values. This could reflect a dataset where confidence is relatively stable despite differences in task complexity. </details> | <details> <summary>x62.png Details</summary>  ### Visual Description ## Scatter Plot: Confidence vs. Target Length in Prehistory Context ### Overview The image presents a scatter plot titled "prehistory" with a secondary histogram overlay. The main plot visualizes the relationship between "Target Length" (x-axis) and "Confidence" (y-axis), with a shaded confidence interval and a vertical reference line. Two marginal histograms summarize distributions of the primary variables. ### Components/Axes - **Primary Plot**: - **X-axis**: "Target Length" (0–100, linear scale) - **Y-axis**: "Confidence" (0.00–0.75, linear scale) - **Data Points**: Purple dots (n ≈ 200) representing individual observations - **Shaded Area**: Light purple band between y=0.25 and y=0.35 (confidence interval) - **Vertical Line**: Dashed gray line at x=50 (target length threshold) - **Marginal Histograms**: - **Top Histogram**: Distribution of "Target Length" (peaks at 0–20 and 80–100) - **Right Histogram**: Distribution of "Confidence" (peaks at 0.25–0.35) ### Detailed Analysis 1. **Scatter Plot Trends**: - **Negative Correlation**: Confidence decreases as Target Length increases (R² ≈ 0.65 based on visual inspection). - **Clustered Data**: 60% of points cluster between x=0–50 and y=0.25–0.35 (within shaded area). - **Outliers**: 15% of points fall outside the shaded confidence interval, particularly at x>70 (low confidence) and x<10 (high confidence). 2. **Histogram Insights**: - **Target Length**: Bimodal distribution with peaks at short (<20) and long (>80) lengths. - **Confidence**: Unimodal distribution centered at 0.30, with 80% of values between 0.20–0.40. 3. **Vertical Line Significance**: - The x=50 line divides data into two groups: - **Left (x<50)**: 55% of points, median confidence ≈ 0.32 - **Right (x>50)**: 45% of points, median confidence ≈ 0.28 ### Key Observations - **Confidence Threshold**: 70% of observations have confidence <0.35, suggesting difficulty in high-confidence predictions for longer targets. - **Length-Confidence Tradeoff**: Longer targets (>70) correlate with confidence dropping below 0.25 in 40% of cases. - **Bimodal Length Distribution**: Indicates two distinct target length regimes (short vs. long) with differing confidence profiles. ### Interpretation The data suggests a fundamental tradeoff between target complexity (length) and predictive confidence in prehistoric datasets. The bimodal length distribution implies two operational regimes: 1. **Short Targets** (<20): High confidence (0.30–0.40) but limited scope 2. **Long Targets** (>80): Low confidence (<0.25) but broader coverage The shaded confidence interval (0.25–0.35) represents the "sweet spot" where 60% of predictions occur, likely corresponding to mid-range target lengths (20–70). The vertical threshold at x=50 may represent an operational boundary where prediction strategies shift. Notably, the 15% of outliers at x>70 with confidence <0.20 highlight extreme cases requiring special handling, possibly due to data scarcity or measurement noise in prehistoric records. </details> | | <details> <summary>x63.png Details</summary>  ### Visual Description ## Scatter Plot: professional_accounting ### Overview The image displays a scatter plot titled "professional_accounting" with a secondary horizontal line and marginal histograms. The plot visualizes the relationship between "Target Length" (x-axis) and "Confidence" (y-axis), with data points distributed across the plot. Histograms on the top and right edges show distributions of the respective variables. ### Components/Axes - **Title**: "professional_accounting" (top-center, bold text). - **X-axis**: - Label: "Target Length" (bottom, black text). - Scale: 0 to 100 (linear, with ticks at 0, 10, 20, ..., 100). - **Y-axis**: - Label: "Confidence" (left, black text). - Scale: 0 to 0.6 (linear, with ticks at 0, 0.2, 0.4, 0.6). - **Data Points**: - Color: Purple (circular markers). - Distribution: Scattered across the plot, with a horizontal line at ~0.2 (dark purple). - **Histograms**: - Top histogram: Represents "Target Length" distribution (purple bars). - Right histogram: Represents "Confidence" distribution (purple bars). ### Detailed Analysis - **Data Points**: - Most points cluster below the 0.2 confidence threshold, with a few outliers reaching up to ~0.4. - Target lengths range from ~0 to ~80, with sparse data beyond 80. - **Horizontal Line**: - Positioned at y ≈ 0.2, spanning the entire x-axis. - Likely represents a reference or threshold value. - **Histograms**: - **Target Length**: Peaks at lower values (0–20), with a long tail extending to 100. - **Confidence**: Peaks near 0.2, with a sharp decline above 0.4. ### Key Observations 1. **Low Confidence Dominance**: Over 70% of data points fall below the 0.2 confidence line, suggesting poor performance or uncertainty in most cases. 2. **Target Length Distribution**: Shorter target lengths (0–20) are most frequent, with fewer instances of longer tasks. 3. **Confidence Distribution**: Confidence values are heavily concentrated around 0.2, indicating a bimodal or threshold-dependent behavior. 4. **Outliers**: A small cluster of points near (60, 0.4) suggests rare high-confidence, longer tasks. ### Interpretation The data implies that in professional accounting tasks, shorter target lengths are more common, and confidence levels are generally low. The horizontal line at 0.2 may represent a performance benchmark, with most data points failing to meet it. The histograms confirm that longer target lengths are rare, and confidence diminishes sharply for values above 0.4. This could indicate challenges in handling complex or extended tasks, or a model optimized for shorter, simpler problems. The sparse high-confidence outliers might reflect edge cases or specialized scenarios. </details> | <details> <summary>x64.png Details</summary>  ### Visual Description ## Scatter Plot: Professional Psychology Confidence vs. Target Length ### Overview The image displays a scatter plot analyzing the relationship between "Target Length" (x-axis) and "Confidence" (y-axis) in a professional psychology context. A linear fit line is overlaid on the data points, with marginal histograms showing distributions of both variables. The plot uses purple data points and a dashed line for the regression model. ### Components/Axes - **X-axis (Target Length)**: Labeled "Target Length," scaled from 0 to 200 in increments of 50. - **Y-axis (Confidence)**: Labeled "Confidence," scaled from 0.0 to 0.6 in increments of 0.1. - **Legend**: Located at the top-right corner, labeled "Linear Fit" with a dashed line symbol. - **Marginal Histograms**: - Top histogram: Distribution of "Target Length" (purple bars). - Right histogram: Distribution of "Confidence" (purple bars). ### Detailed Analysis - **Scatter Plot**: - Data points (purple) show a positive correlation between Target Length and Confidence. - The linear fit line slopes upward, indicating a statistically significant relationship (R² ≈ 0.45 based on visual estimation). - Key data points: - At Target Length = 50, Confidence ≈ 0.25. - At Target Length = 150, Confidence ≈ 0.45. - At Target Length = 200, Confidence ≈ 0.55. - **Marginal Distributions**: - **Target Length**: Peaks at ~100 (mode), with a right-skewed distribution (median ≈ 120). - **Confidence**: Peaks at ~0.3 (mode), with a bimodal distribution (secondary peak near 0.5). ### Key Observations 1. **Positive Correlation**: Confidence increases with Target Length, though the relationship is moderate (R² ~0.45). 2. **Distribution Patterns**: - Most Target Length values cluster between 50–150. - Confidence values are concentrated between 0.2–0.5, with fewer extreme values. 3. **Outliers**: No significant outliers detected; data points follow the trend line closely. ### Interpretation The data suggests that in professional psychology contexts, longer target lengths are associated with higher confidence levels. The linear fit indicates a moderate positive relationship, implying that increasing Target Length may systematically improve Confidence. However, the scatter plot's spread and bimodal Confidence distribution suggest other factors (e.g., task complexity, individual differences) also influence outcomes. The marginal histograms highlight that most observations fall within mid-range values for both variables, emphasizing the need for further analysis to isolate confounding variables. </details> | <details> <summary>x65.png Details</summary>  ### Visual Description ## Scatter Plot: public_relations ### Overview The image is a scatter plot titled "public_relations" showing the relationship between "Target Length" (x-axis) and "Confidence" (y-axis). A trend line with a shaded confidence interval is overlaid on the data points. The plot includes a legend in the top-left corner and grid lines for reference. ### Components/Axes - **Title**: "public_relations" (top center) - **X-axis**: "Target Length" (0 to 100, with ticks at 0, 50, 100) - **Y-axis**: "Confidence" (0.00 to 0.75, with ticks at 0.25, 0.50, 0.75) - **Legend**: Located in the top-left corner, labeled "Confidence Interval" with a shaded area. - **Data Points**: Purple dots scattered across the plot. - **Trend Line**: A dark purple line with a positive slope, indicating a relationship between Target Length and Confidence. - **Shaded Area**: A lighter purple region above the trend line, representing the confidence interval. ### Detailed Analysis - **Data Points**: Approximately 50-60 purple dots are distributed across the plot. Most points cluster near the trend line, with some variability. - **Trend Line**: The line starts near (0, 0.25) and ends near (100, 0.75), suggesting a linear increase in Confidence with Target Length. The slope is approximately 0.005 per unit of Target Length (calculated as (0.75 - 0.25)/100). - **Confidence Interval**: The shaded area above the trend line indicates the upper bound of the confidence interval. The width of the shaded region suggests a 95% confidence interval (common default, though not explicitly stated). - **Legend**: The legend is positioned in the top-left corner, with the "Confidence Interval" label and shaded area matching the plot's visual elements. ### Key Observations - **Positive Correlation**: The trend line shows a clear upward slope, indicating that higher Target Lengths are associated with higher Confidence levels. - **Data Variability**: While most points align with the trend line, there is noticeable scatter, suggesting some outliers or noise in the data. - **Confidence Interval**: The shaded area above the trend line implies uncertainty in the relationship, with the interval widening as Target Length increases. ### Interpretation The data suggests a positive relationship between Target Length and Confidence in the context of public relations. As Target Length increases, Confidence tends to rise, with the trend line indicating a linear relationship. The shaded confidence interval highlights the uncertainty around this relationship, suggesting that while the general trend is upward, individual data points may vary. The legend's placement and color coding ensure clarity, but the absence of explicit confidence level (e.g., 95%) leaves room for interpretation. The plot emphasizes the importance of Target Length in influencing Confidence, though the variability in data points underscores the need for further analysis to account for outliers or external factors. </details> | <details> <summary>x66.png Details</summary>  ### Visual Description ## Scatter Plot with Marginal Histograms: security_studies ### Overview The image displays a scatter plot titled "security_studies" with marginal histograms on the top and right axes. The plot visualizes the relationship between "Target Length" (x-axis) and "Confidence" (y-axis), with data points represented as purple dots. The marginal histograms show the distribution of values along each axis. ### Components/Axes - **Title**: "security_studies" (top-center, bold text). - **X-axis (Target Length)**: - Label: "Target Length" (bottom-left, horizontal axis). - Scale: 0 to 500 (linear increments of ~25). - **Y-axis (Confidence)**: - Label: "Confidence" (left side, vertical axis). - Scale: 0 to 0.6 (linear increments of ~0.1). - **Marginal Histograms**: - Top histogram: Distribution of "Target Length" (purple bars). - Right histogram: Distribution of "Confidence" (purple bars). - **Legend**: No explicit legend labels are present. The title "security_studies" may implicitly describe the dataset. ### Detailed Analysis - **Scatter Plot**: - Data points are clustered primarily between **Target Length = 100–300** and **Confidence = 0.2–0.4**. - A few outliers exist, including one at **(500, 0.1)** (far right, low confidence). - The marginal histograms confirm that: - **Target Length** is skewed toward lower values (peak at ~200). - **Confidence** is concentrated around **0.2–0.4**, with a long tail toward lower values. ### Key Observations 1. **Negative Correlation**: As "Target Length" increases, "Confidence" generally decreases (e.g., points near (500, 0.1) vs. (100, 0.5)). 2. **Outlier**: The point at **(500, 0.1)** deviates significantly from the main cluster, suggesting an anomaly or edge case. 3. **Distribution Skew**: Most data points fall within the lower ranges of both axes, indicating a focus on shorter targets and moderate confidence levels. ### Interpretation The data suggests that in the context of "security_studies," longer target lengths are associated with lower confidence levels. This could imply that more complex or extended targets (e.g., security protocols, threat scenarios) are harder to achieve high confidence in. The outlier at (500, 0.1) may represent an exceptional case where a very long target resulted in minimal confidence, warranting further investigation. The marginal histograms reinforce that the dataset is dominated by shorter targets and moderate confidence, highlighting a potential bias or limitation in the studied scope. </details> | Figure 13: Continuing from fig. 12. See also fig. 14. | <details> <summary>x67.png Details</summary>  ### Visual Description ## Scatter Plot: Sociology Data Analysis ### Overview The image displays a scatter plot titled "sociology," visualizing the relationship between "Target Length" (x-axis) and "Confidence" (y-axis). Purple data points are distributed across the plot, with marginal histograms on the top and right axes. A horizontal reference line at y=0.3 is present. ### Components/Axes - **Title**: "sociology" (top center). - **X-axis**: "Target Length" (0 to 100, linear scale). - **Y-axis**: "Confidence" (0.25 to 0.75, linear scale). - **Marginal Histograms**: - Top histogram: Distribution of "Target Length" (peaks near 50-70). - Right histogram: Distribution of "Confidence" (peaks near 0.3-0.4). - **Reference Line**: Horizontal line at y=0.3 (dashed, spans full x-axis). - **Data Points**: Purple dots (no explicit legend, but color is consistent). ### Detailed Analysis - **Data Trends**: - Scatter points show a **negative correlation**: higher "Target Length" generally corresponds to lower "Confidence." - Points cluster densely around **Target Length = 50-70** and **Confidence = 0.3-0.4**. - Outliers exist at both extremes (e.g., high confidence at low target lengths and low confidence at high target lengths). - **Marginal Histograms**: - "Target Length" is skewed right, with a mode near 60. - "Confidence" is bimodal, with peaks near 0.3 and 0.4. - **Reference Line**: The line at y=0.3 aligns with the lower confidence peak in the histogram, suggesting a threshold or baseline. ### Key Observations 1. **Negative Correlation**: As "Target Length" increases, "Confidence" decreases (R² ≈ 0.6 based on visual inspection). 2. **Confidence Threshold**: The horizontal line at y=0.3 may represent a critical confidence level, with ~60% of data points falling below it. 3. **Distribution Skew**: "Target Length" values are more variable at higher ranges, while "Confidence" stabilizes below 0.5. ### Interpretation The data suggests that **longer target lengths are associated with reduced confidence**, potentially indicating task complexity or resource limitations. The bimodal confidence distribution implies two distinct groups: one with moderate confidence (0.3-0.4) and another with higher confidence (0.4-0.5). The horizontal line at 0.3 could signify a benchmark for "acceptable" confidence, with most data points falling short. Outliers at low target lengths with high confidence may reflect exceptional cases or measurement errors. This pattern could inform strategies for setting realistic goals in sociological studies or resource allocation. </details> | <details> <summary>x68.png Details</summary>  ### Visual Description ## Scatter Plot with Line of Best Fit: us_foreign_policy ### Overview The image displays a scatter plot titled "us_foreign_policy" with a line of best fit and marginal histograms. The plot examines the relationship between "Target Length" (x-axis) and "Confidence" (y-axis), with additional distributions visualized in histograms. The data points are represented by purple dots, and the line of best fit is shaded with a confidence interval. ### Components/Axes - **X-axis (Target Length)**: Labeled "Target Length," with ticks at 0, 50, and 100. The scale ranges from 0 to 100. - **Y-axis (Confidence)**: Labeled "Confidence," with ticks at 0.00, 0.25, 0.50, and 0.75. The scale ranges from 0.00 to 0.75. - **Line of Best Fit**: A dark purple line with a shaded confidence interval (light purple band) around it. - **Histograms**: - **Top Histogram**: Distribution of "Target Length" (purple bars). - **Right Histogram**: Distribution of "Confidence" (purple bars). ### Detailed Analysis - **Scatter Plot**: - Data points are clustered primarily between Target Length 0–100 and Confidence 0.00–0.75. - The line of best fit is calculated as **y = 0.005x + 0.3**, with a shaded confidence interval (likely ±0.05 based on visual width). - The slope of the line is shallow, indicating a weak positive correlation between Target Length and Confidence. - **Histograms**: - **Target Length**: Most values cluster between 0–100, with a slight skew toward lower values (mode near 50). - **Confidence**: Concentrated between 0.3 and 0.5, with fewer points near the extremes (0.00 and 0.75). ### Key Observations 1. **Weak Positive Trend**: The line of best fit shows a slight upward slope, suggesting that longer Target Lengths marginally increase Confidence. 2. **Confidence Interval**: The shaded area around the line indicates uncertainty in the trend, with the interval widening slightly at higher Target Lengths. 3. **Distribution Skew**: The histograms reveal that both variables are not uniformly distributed, with Confidence peaking in the mid-range (0.3–0.5). ### Interpretation The data suggests a minimal relationship between Target Length and Confidence in U.S. foreign policy contexts. While the line of best fit implies a weak positive correlation, the shallow slope (0.005) and wide confidence interval indicate that Target Length alone does not strongly predict Confidence. The histograms further highlight that Confidence values are concentrated in the mid-range, implying other factors may dominate this metric. The marginal histograms provide context for the scatter plot, showing that the relationship is not driven by extreme values in either variable. This analysis could inform policy modeling by emphasizing the need to consider additional variables beyond Target Length. </details> | <details> <summary>x69.png Details</summary>  ### Visual Description ## Scatter Plot with Histograms: Confidence vs. Target Length in Virology ### Overview The image presents a scatter plot titled "virology" with a main chart and two histograms. The main chart visualizes the relationship between "Confidence" (y-axis) and "Target Length" (x-axis), with a horizontal reference line at y=0.25. Two histograms are overlaid: one at the top (x-axis distribution) and one on the right (y-axis distribution). Data points are purple, with a shaded area around the reference line. --- ### Components/Axes - **Main Chart**: - **X-axis (Target Length)**: Labeled "Target Length," scaled from 0 to 100. - **Y-axis (Confidence)**: Labeled "Confidence," scaled from 0 to 0.75. - **Data Points**: Purple dots scattered across the plot. - **Reference Line**: Horizontal line at y=0.25 (confidence threshold). - **Shaded Area**: Light purple region between y=0.2 and y=0.3, indicating variability around the reference line. - **Top Histogram**: - **X-axis**: Labeled "Target Length," scaled from 0 to 100. - **Y-axis**: Unlabeled, showing frequency distribution. - **Peak**: Concentration of data points between 20 and 30. - **Right Histogram**: - **X-axis**: Unlabeled, showing frequency distribution. - **Y-axis**: Labeled "Confidence," scaled from 0 to 0.75. - **Peak**: Concentration of data points around 0.25. --- ### Detailed Analysis - **Main Chart**: - **Data Points**: Approximately 50-60 purple dots. Most cluster below y=0.25, with a few outliers above (e.g., y=0.5–0.75 at x=50–70). - **Reference Line**: y=0.25 (exact value). The shaded area (0.2–0.3) suggests a confidence interval or variability range. - **Trend**: No clear linear correlation; data points are dispersed but slightly denser at lower target lengths (x=0–50). - **Top Histogram**: - **Distribution**: Bimodal with peaks at x=20–30 and x=70–80. Most data points fall between x=0 and x=50. - **Right Histogram**: - **Distribution**: Unimodal peak at y=0.25, with a long tail extending to y=0.75. Most data points cluster between y=0.2 and y=0.3. --- ### Key Observations 1. **Confidence Threshold**: The horizontal line at y=0.25 acts as a benchmark, with most data points below this value. 2. **Outliers**: A small cluster of high-confidence points (y=0.5–0.75) occurs at target lengths of 50–70. 3. **Distribution Skew**: The top histogram shows a left-skewed distribution for target lengths, while the right histogram is right-skewed for confidence. 4. **Shaded Area**: The light purple region around y=0.25 indicates variability, possibly representing measurement uncertainty or model confidence intervals. --- ### Interpretation - **Confidence vs. Target Length**: The data suggests that confidence generally decreases as target length increases, though this relationship is not strictly linear. The majority of data points (60–70%) fall below the 0.25 confidence threshold, indicating lower reliability for most cases. - **Outliers**: The high-confidence outliers (y=0.5–0.75) at x=50–70 may represent exceptional cases or specific virological markers with stronger predictive power. - **Histograms**: The bimodal distribution in target lengths (20–30 and 70–80) could reflect common biological ranges (e.g., viral genome sizes) or experimental constraints. The confidence histogram’s peak at 0.25 aligns with the reference line, suggesting this is a common or expected value. - **Shaded Area**: The variability around y=0.25 implies that confidence estimates are not absolute, with a ±0.05 range indicating measurement noise or model uncertainty. --- ### Conclusion The chart highlights a critical threshold (0.25 confidence) in virological data, with most cases falling below this value. The presence of outliers and bimodal distributions suggests variability in both target lengths and confidence levels, potentially pointing to underlying biological or methodological factors. Further analysis could explore why certain target lengths correlate with higher confidence or investigate the significance of the shaded variability range. </details> | <details> <summary>x70.png Details</summary>  ### Visual Description ## Scatter Plot: world_religions ### Overview The image is a scatter plot titled "world_religions" with a line of best fit and a shaded confidence interval. The x-axis represents "Target Length" (0–50), and the y-axis represents "Confidence" (0.25–0.75). Purple data points are scattered across the plot, with a downward-sloping line of best fit and a shaded region around it. ### Components/Axes - **Title**: "world_religions" (top center). - **X-axis**: "Target Length" (0–50, with ticks at 0, 10, 20, 30, 40, 50). - **Y-axis**: "Confidence" (0.25–0.75, with ticks at 0.25, 0.5, 0.75). - **Data Points**: Purple dots scattered across the plot. - **Line of Best Fit**: A solid line sloping downward from (0, ~0.5) to (50, ~0.25). - **Shaded Area**: A light purple region around the line of best fit, representing a confidence interval (exact bounds unspecified). ### Detailed Analysis - **Data Points**: Approximately 20–30 purple dots are distributed across the plot. Most points cluster near the line of best fit, with some outliers below and above it. - **Line of Best Fit**: The line starts at (0, ~0.5) and ends at (50, ~0.25), indicating a negative correlation between "Target Length" and "Confidence." - **Shaded Area**: The shaded region spans the width of the plot, with its width varying slightly. It appears to represent a 95% confidence interval, though the exact calculation method is not specified. ### Key Observations 1. **Negative Correlation**: As "Target Length" increases, "Confidence" decreases. The line of best fit shows a consistent downward trend. 2. **Confidence Interval**: The shaded area widens slightly toward the ends of the plot (near 0 and 50), suggesting greater uncertainty at extreme target lengths. 3. **Data Distribution**: Data points are more concentrated near the line of best fit, with some variability. A few points deviate significantly (e.g., one point near (10, 0.75) and another near (40, 0.3)). 4. **Axis Ranges**: The x-axis spans 0–50, and the y-axis spans 0.25–0.75. No data points fall outside these ranges. ### Interpretation The plot suggests that longer target lengths are associated with lower confidence in the "world_religions" dataset. The line of best fit quantifies this relationship, while the shaded confidence interval highlights the uncertainty in the trend. The widening of the shaded area at the extremes implies that predictions or measurements for very short or very long target lengths are less reliable. The scatter of data points indicates variability in the relationship, which could stem from factors like data quality, measurement methods, or inherent complexity in the "world_religions" dataset. The absence of a legend for the line of best fit or shaded area limits interpretability, but the visual cues (color, slope, and shading) are consistent with standard statistical practices. </details> | | --- | --- | --- | --- | Figure 14: Continuing from figs. 12 and 13. Appendix F Generalization to Coding Tasks Because there are no coding tasks in our training dataset, we can use a coding competition task introduced in LiveCodeBench [Jain et al., 2024] to assess how well finetuned uncertainty estimation methods perform on completely out of distribution tasks. To conduct the analysis in table 3, we evaluate several base models on the 62 LeetCode easy questions from the livecodebench_generation_lite task. We asking for the model to write a Python solution and grade the solution using test cases (marking it as correct iff it passes all test cases). We then apply Lora + Prompt and Zero-Shot Classifier uncertainty estimation methods—with these methods only using training/temperature scaling data from our main dataset mixture which notably does not include any coding tasks section C.2. Accuracy is shown to contextualize the model’s overall level of performance on the task. On Mistral-7B, the best performing model on the coding task, the supervised Lora + Prompt approach dramatically improves calibration and selective prediction as compared to Zero-Shot Classifier; on the worse-performing Mistral-7B-Instruct and LLaMa-2-7B, selective prediction improves but calibration slightly degrades. | Model | Method | Acc | ECE | AUROC | | --- | --- | --- | --- | --- | | LLaMa-2-7B | Zero-Shot Classifier | 3.2% | 41.0% | 56.9% | | Lora + Prompt | 3.2% | 46.4% | 80.0% | | | Mistral-7B | Zero-Shot Classifier | 27.4% | 70.2% | 66.2% | | Lora + Prompt | 27.4% | 21.4% | 85.1% | | | Mistral-7B-Instruct | Zero-Shot Classifier | 21.0% | 52.7% | 47.1% | | Lora + Prompt | 21.0% | 56.1% | 70.2% | | Table 3: ECE and AUROC on livecodebench_generation_lite (LeetCode easy subset). ECE is shown after temperature scaling on a small hold-out set of the original dataset mixture section C.2. Acc is task accuracy (proportion of coding solutions that are correct). Supervised training (LoRA + Prompt) seems to always improve selective prediction, although supervised training only heavily improves calibration for Mistral-7B and in fact slightly degrades calibration for the two other models. Appendix G User Studies G.1 Additional Details on Setup Stimuli and Participant Selection We closely followed the setup of [Bhatt et al., 2023]. We used the same 180 MMLU questions from which were pre-batched into three sets of 60 MMLU questions. Within each variant, we randomly assigned participants to one of the three batches. In total, we recruit $181$ participants (20 per variant With the exception of one extra participant due to random batching allocation effects.). All participants were recruited through the crowdsourcing platform Prolific [Palan and Schitter, 2018]; we restrict our participant pool to those based in the United States who speak English as a first language. Compensation Participants were told that the study would take approximately 30 minutes and were paid at a base rate of $9/hr and informed that they would receive an optional bonus up to $10 for answering questions correctly. We applied the bonus to all participants. LLM Answers and Uncertainty Elicitation Bhatt et al. originally used GPT-3.5 as their LLM. While at first, we explored user performance when provided with confidence scores modulated over the original GPT-3.5 responses that the authors had collected, the authors had filtered LLM performance to ensure the LLM achieved high performance on biology, computer science, and foreign policy and poor performance on mathematics. As such, we noticed that participants overwhelmingly uptook the LLM’s answer (which was rational behaviour, given the model’s high performance). To explore a more nuanced performance profile, we regenerated LLM answers using Mistral 7B Instruct via greedy decoding. We then generated confidence scores on top of the LLM responses. For our random baseline, we sample a confidence score uniformly between 0 and 100% for each question. G.2 Important considerations There are many reasons to heed caution in interpreting our results as definitive indications of the utility of displaying confidence to users in LLM assistive settings. In particular: (i) users are presented with feedback after each trial as in [Bhatt et al., 2023] – as such, they can determine (potentially rapidly) whether or not a model is reliable, even without confidence scores. However, in practical settings users may not know whether or not the model was truly correct and therefore confidence scores could have an even larger impact; (ii) MMLU questions can be challenging for non-experts – we see the biggest differences in performance for the no-LLM vs. any-LLM-assistance condition. We may see a wider range of reliance behaviors in settings wherein people have more confidence in their own abilities; (iii) we present users with numeric confidence; however, humans are not always able to reliably process confidence estimates nor appropriately calibrate uncertainty estimates themselves [Keren, 1991, Vodrahalli et al., 2022, Collins et al., 2023, Lichtenstein et al., 1977]. It may be that alternate modes of communicating confidence improve users’ ability to appropriately leverage the confidence scores in their decision making process. We see targeted exploration of each component through interdisciplinary collaboration across AI, behavioral science, and human-computer interaction as ripe for future work. G.3 Extended Results Task Accuracy and Reliance Sensibility We depict average user task accuracy and reliance sensibility across variants in Figure 15. We follow Bhatt et al. in computing reliance sensibility as the proportion of times the user appropriately sided with the model prediction when the model was correct and did not respond with the model’s prediction when the model was incorrect. <details> <summary>x71.png Details</summary>  ### Visual Description ## Violin Plot: Accuracy Comparison Across Model Configurations ### Overview The image displays a violin plot comparing the accuracy distributions of five different model configurations. Each violin plot represents the spread and central tendency of accuracy values for a specific setup, with horizontal lines indicating median or mean values. The x-axis categorizes configurations, while the y-axis measures accuracy (0.0–1.0). ### Components/Axes - **X-axis (Categories)**: - "No LLM" (purple) - "LLM" (red) - "LLM + Conf (Rand)" (green) - "LLM + Conf (Query)" (gray) - "LLM + Conf (CT)" (blue) - **Y-axis (Accuracy)**: Ranges from 0.0 to 1.0, labeled "Accuracy." - **Legend**: Embedded in x-axis labels, with colors directly mapping to categories. ### Detailed Analysis 1. **No LLM (Purple)**: - Median accuracy ~0.5 (horizontal line). - Wide distribution (violin width), indicating high variability. - Lower quartile ~0.3, upper quartile ~0.7. 2. **LLM (Red)**: - Median accuracy ~0.7. - Narrower distribution than "No LLM," suggesting reduced variability. - Lower quartile ~0.6, upper quartile ~0.8. 3. **LLM + Conf (Rand) (Green)**: - Median accuracy ~0.65. - Slightly wider than "LLM," with lower quartile ~0.55 and upper quartile ~0.75. 4. **LLM + Conf (Query) (Gray)**: - Median accuracy ~0.75. - Narrower than "LLM + Conf (Rand)," with lower quartile ~0.7 and upper quartile ~0.85. 5. **LLM + Conf (CT) (Blue)**: - Median accuracy ~0.8. - Narrowest distribution, indicating the least variability. - Lower quartile ~0.75, upper quartile ~0.85. ### Key Observations - **Trend**: Accuracy improves as configurations evolve from "No LLM" to "LLM + Conf (CT)." - **Outliers**: "No LLM" shows the widest spread, with some values dipping below 0.4. - **Notable Pattern**: The "CT" configuration achieves the highest median accuracy (~0.8) and the smallest spread, suggesting it is the most stable and effective setup. ### Interpretation The data demonstrates that incorporating a language model (LLM) and confidence mechanisms significantly enhances accuracy. The "CT" configuration (likely a specific confidence thresholding method) outperforms others, achieving the highest median accuracy and lowest variability. The "No LLM" group’s lower accuracy and wider distribution highlight the importance of model integration for consistent performance. The incremental improvements across configurations suggest that each added component (e.g., confidence strategies) contributes to better outcomes, with "CT" being the most effective. </details> <details> <summary>x72.png Details</summary>  ### Visual Description ## Violin Plot: Reliance Sensitivity Comparison Across Model Configurations ### Overview The image presents a comparative analysis of Reliance Sensitivity across four model configurations using violin plots. Each plot visualizes the distribution of sensitivity scores, with horizontal lines indicating median and quartile values. The configurations compared are: 1. **LLM** (baseline) 2. **LLM + Conf (Rand)** 3. **LLM + Conf (Query)** 4. **LLM + Conf (CT)** ### Components/Axes - **X-axis**: Model configurations (LLM, LLM + Conf (Rand), LLM + Conf (Query), LLM + Conf (CT)), labeled in sequence from left to right. - **Y-axis**: Reliance Sensitivity, scaled from 0.3 to 1.0 in increments of 0.1. - **Legend**: Located at the bottom, mapping colors to configurations: - Red: LLM - Teal: LLM + Conf (Rand) - Gray: LLM + Conf (Query) - Blue: LLM + Conf (CT) ### Detailed Analysis 1. **LLM (Red)**: - Median: ~0.7 (horizontal line). - Interquartile range (IQR): ~0.65–0.75. - Full range: ~0.4–0.8. - Distribution: Symmetrical with a slight skew toward higher values. 2. **LLM + Conf (Rand) (Teal)**: - Median: ~0.65. - IQR: ~0.6–0.7. - Full range: ~0.5–0.8. - Distribution: Narrower spread compared to LLM, with a peak near the median. 3. **LLM + Conf (Query) (Gray)**: - Median: ~0.7. - IQR: ~0.65–0.75. - Full range: ~0.55–0.85. - Distribution: Broader than LLM + Conf (Rand), with a slight upward skew. 4. **LLM + Conf (CT) (Blue)**: - Median: ~0.75. - IQR: ~0.7–0.8. - Full range: ~0.6–0.9. - Distribution: Widest spread, with a pronounced peak near the median and a long tail toward higher values. ### Key Observations - **Highest Median**: LLM + Conf (CT) achieves the highest median Reliance Sensitivity (~0.75), outperforming all other configurations. - **Lowest Median**: LLM + Conf (Rand) has the lowest median (~0.65), indicating poorer performance compared to the baseline LLM. - **Spread Variability**: - LLM + Conf (CT) exhibits the widest distribution, suggesting greater variability in sensitivity scores. - LLM + Conf (Rand) has the narrowest spread, indicating more consistent (but lower) performance. - **Baseline Comparison**: The baseline LLM (red) performs better than LLM + Conf (Rand) but worse than LLM + Conf (Query) and CT. ### Interpretation The data suggests that augmenting the LLM with configuration-specific enhancements (Conf) generally improves Reliance Sensitivity, with the **CT (Contextual Tuning)** configuration yielding the most significant gains. The **Query** configuration matches the baseline LLM in median performance but shows slightly better upper-bound performance. The **LLM + Conf (Rand)** configuration underperforms the baseline, raising questions about the efficacy of random configuration additions. The **CT** configuration’s wider spread implies that while it achieves higher sensitivity on average, its performance is more variable across different use cases or datasets. This analysis highlights the importance of targeted configuration tuning (e.g., CT) over generic or random enhancements for optimizing Reliance Sensitivity in LLM-based systems. </details> Figure 15: (Left) User accuracy on 60 MMLU questions per variant ( $N=20$ users per variant); violin plots show quartiles as dashed lines (Right) Average reliance sensibility (proportion of instances where the user sided with the model when the model was correct, and overrode the model’s prediction when the model was incorrect); higher indicates better reliance calibration. We depict per-topic accuracy, with the LLM’s average performance in Figure 16. <details> <summary>x73.png Details</summary>  ### Visual Description ## Violin Plot: High School Biology Model Accuracy Comparison ### Overview The image presents a comparative analysis of model accuracy distributions across five configurations in a high school biology context. Five colored violin plots represent different model setups, with a red dashed threshold line at 0.65 accuracy. The visualization emphasizes distributional characteristics (medians, quartiles, and spread) rather than just mean values. ### Components/Axes - **X-axis**: Model configurations (categorical) - No LLM (blue) - LLM (orange) - LLM + Conf (Rand) (green) - LLM + Conf (Query) (red) - LLM + Conf (CT) (purple) - **Y-axis**: Accuracy metric (continuous, 0.0-1.0 scale) - **Legend**: Color-coded model configurations (implicitly mapped) - **Threshold line**: Red dashed horizontal line at 0.65 accuracy ### Detailed Analysis 1. **No LLM (blue)**: - Wide distribution with heavy lower tail - Median accuracy ~0.4 (black line) - Interquartile range (IQR) spans 0.3-0.5 - Long tail extends to 0.8 but with low density 2. **LLM (orange)**: - Symmetrical distribution centered ~0.6 - Median ~0.6 with IQR 0.5-0.7 - Moderate spread with slight right skew 3. **LLM + Conf (Rand) (green)**: - Similar median to LLM (~0.6) - Narrower IQR (0.55-0.65) - More concentrated distribution than base LLM 4. **LLM + Conf (Query) (red)**: - Highest median (~0.65) - Tightest distribution (IQR 0.6-0.7) - Significant portion above threshold line 5. **LLM + Conf (CT) (purple)**: - Lowest median (~0.55) - Bimodal distribution with peaks at 0.5 and 0.6 - Wide spread with heavy lower tail ### Key Observations - **Threshold performance**: Only LLM + Conf (Query) shows substantial representation above the 0.65 threshold - **Distribution characteristics**: - No LLM exhibits highest variability (widest spread) - LLM + Conf (CT) shows unusual bimodality - Confidence mechanisms generally reduce spread but with mixed median effects - **Color-legend alignment**: All colors match their respective categories without ambiguity ### Interpretation The data suggests that: 1. **Confidence mechanisms improve consistency**: LLM + Conf variants show narrower distributions than base LLM, indicating more stable performance 2. **Query method optimization**: The Query configuration achieves highest median accuracy and best threshold penetration 3. **CT method limitations**: The CT configuration underperforms other variants despite confidence mechanisms 4. **Baseline performance**: No LLM shows poorest performance with highest variability The visualization reveals that while confidence mechanisms generally improve model consistency, their effectiveness depends on implementation method. The Query approach demonstrates optimal balance between accuracy and reliability, while the CT method introduces unexpected performance degradation despite confidence integration. The red threshold line serves as a critical benchmark, showing that only one configuration consistently meets the 0.65 accuracy standard. </details> <details> <summary>x74.png Details</summary>  ### Visual Description ## Violin Plot: High School CS Model Accuracy Comparison ### Overview The image presents a comparative analysis of model accuracy distributions across five configurations in a high school computer science context. Five violin plots are arranged horizontally, each representing a different model variant, with a red dashed reference line at 0.7 accuracy. ### Components/Axes - **X-axis**: Model configurations (categorical) - "No LLM" (blue) - "LLM" (orange) - "LLM + Conf (Rand)" (green) - "LLM + Conf (Query)" (red) - "LLM + Conf (CT)" (purple) - **Y-axis**: Accuracy (continuous scale from 0.0 to 1.0) - **Legend**: Right-aligned color-coded labels matching x-axis categories - **Reference Line**: Red dashed horizontal line at y=0.7 ### Detailed Analysis 1. **No LLM (Blue)**: - Distributed between 0.3-0.6 accuracy - Median ~0.5 (horizontal line within violin) - Wide distribution indicates high variability 2. **LLM (Orange)**: - Concentrated around 0.6-0.75 - Median ~0.65 - Narrower distribution than "No LLM" 3. **LLM + Conf (Rand) (Green)**: - Peaks at 0.7-0.8 - Median ~0.75 - Symmetrical distribution 4. **LLM + Conf (Query) (Red)**: - Distributed 0.7-0.85 - Median ~0.78 - Slightly skewed toward higher values 5. **LLM + Conf (CT) (Purple)**: - Peaks at 0.8-0.9 - Median ~0.85 - Most concentrated distribution ### Key Observations - All "LLM + Conf" variants exceed the 0.7 reference line - "LLM + Conf (CT)" achieves highest median accuracy (~0.85) - "No LLM" shows lowest and most variable performance - Confidence augmentation improves accuracy by 0.15-0.25 compared to base LLM - "LLM + Conf (Rand)" and "(Query)" show similar performance (~0.75 median) ### Interpretation The data demonstrates that confidence augmentation consistently improves model accuracy in high school CS applications. The "CT" (likely "Contextual Tuning") method yields the most significant gains, suggesting specialized confidence mechanisms enhance performance. The red reference line at 0.7 appears to represent a performance threshold, with all confidence-augmented models exceeding this benchmark. The progressive improvement from "LLM" to "LLM + Conf (CT)" indicates that confidence augmentation strategies are additive and context-dependent, with CT providing the most substantial benefits. The variability in "No LLM" suggests baseline models struggle with consistency in this domain. </details> <details> <summary>x75.png Details</summary>  ### Visual Description ## Violin Plot: US Foreign Policy Accuracy Comparison ### Overview The image is a violin plot comparing the accuracy distributions of different configurations of a US Foreign Policy model. The x-axis represents five model configurations, while the y-axis shows accuracy values ranging from 0 to 1. A red dashed line at 0.8 serves as a reference threshold. ### Components/Axes - **X-axis (Categories)**: - No LLM (blue) - LLM (orange) - LLM + Conf (Rand) (green) - LLM + Conf (Query) (red) - LLM + Conf (CT) (purple) - **Y-axis (Accuracy)**: Labeled "Accuracy" with a scale from 0 to 1. - **Legend**: Implicitly defined by color coding (no explicit legend box). - **Reference Line**: Red dashed line at 0.8 accuracy. ### Detailed Analysis 1. **No LLM (Blue)**: - Distribution spans 0.2–0.6, with a median ~0.4. - Narrowest distribution, indicating low variability. 2. **LLM (Orange)**: - Distribution spans 0.4–0.8, with a median ~0.65. - Wider spread than No LLM, suggesting increased variability. 3. **LLM + Conf (Rand) (Green)**: - Distribution spans 0.5–0.75, with a median ~0.65. - Similar median to LLM but slightly narrower spread. 4. **LLM + Conf (Query) (Red)**: - Distribution spans 0.6–0.85, with a median ~0.75. - Highest median among non-CT configurations, with moderate spread. 5. **LLM + Conf (CT) (Purple)**: - Distribution spans 0.7–0.9, with a median ~0.82. - Tightest distribution around the 0.8 threshold, indicating consistent high performance. ### Key Observations - **CT Configuration (Purple)**: - Median accuracy (~0.82) exceeds the 0.8 reference line. - Narrowest distribution, suggesting stable performance. - **Query Configuration (Red)**: - Second-highest median (~0.75) but still below the 0.8 threshold. - **No LLM (Blue)**: - Lowest median (~0.4) and narrowest distribution, indicating poor performance. - **Rand Configuration (Green)**: - Similar median to LLM but slightly better spread. ### Interpretation The data demonstrates that the **LLM + Conf (CT)** configuration achieves the highest and most consistent accuracy, surpassing the 0.8 benchmark. This suggests that the CT (likely "Contextual Tuning" or similar) enhancement is critical for optimal performance. The **LLM + Conf (Query)** configuration shows promise but falls short of the target. The **No LLM** baseline performs poorly, highlighting the necessity of language model integration. The **Rand** configuration offers minimal improvement over the base LLM, indicating that random confidence adjustments may not be effective. The red dashed line at 0.8 acts as a performance threshold, with only the CT configuration meeting or exceeding it. This analysis underscores the importance of targeted confidence adjustments (e.g., CT) in improving model reliability for US Foreign Policy tasks. </details> <details> <summary>x76.png Details</summary>  ### Visual Description ## Violin Plot: Elementary Math Accuracy Comparison ### Overview The image displays a violin plot comparing the accuracy distributions of five different configurations in an Elementary Math task. The plot uses color-coded distributions to visualize performance across configurations, with a red dashed line at 0.3 accuracy serving as a reference threshold. ### Components/Axes - **X-axis**: Categorical axis with five configurations: 1. No LLM (blue) 2. LLM (orange) 3. LLM + Conf (Rand) (green) 4. LLM + Conf (Query) (red) 5. LLM + Conf (CT) (purple) - **Y-axis**: Accuracy metric ranging from 0.0 to 1.0 - **Legend**: Right-aligned color key matching configurations to colors - **Reference Line**: Red dashed horizontal line at y=0.3 ### Detailed Analysis 1. **No LLM (Blue)**: - Distribution peaks between 0.7-0.8 accuracy - Narrow spread indicates consistent performance - Median accuracy ~0.75 2. **LLM (Orange)**: - Lower median (~0.55) than No LLM - Wider spread (0.4-0.7 range) - Bimodal distribution with peaks at 0.5 and 0.6 3. **LLM + Conf (Rand) (Green)**: - Highest median (~0.8) - Broadest distribution (0.6-0.9 range) - Multiple peaks suggesting varied performance 4. **LLM + Conf (Query) (Red)**: - Median ~0.75 - Narrower spread than LLM + Conf (Rand) - Single peak at 0.7-0.8 range 5. **LLM + Conf (CT) (Purple)**: - Highest median (~0.85) - Tightest distribution (0.7-0.9 range) - Most consistent performance ### Key Observations - All configurations except "No LLM" exceed the 0.3 accuracy threshold - LLM + Conf (CT) shows the highest and most consistent performance - LLM + Conf (Rand) has the widest spread, indicating highest variability - "LLM" configuration underperforms compared to all LLM + Conf variants - Red dashed line at 0.3 serves as a clear performance benchmark ### Interpretation The data demonstrates that incorporating LLM improves accuracy over baseline (No LLM), with configuration enhancements (Conf) further boosting performance. The "CT" configuration achieves the highest median accuracy (0.85) with the tightest distribution, suggesting it's the most reliable method. The "Rand" configuration, while having the highest peak, shows significant variability, indicating potential instability. The consistent performance above the 0.3 threshold across all LLM-enhanced methods suggests this benchmark is easily achievable, but the differences between configurations highlight the importance of careful configuration design for optimal performance. </details> Figure 16: User accuracies per topic for the Mistral variants. Red line indicates the model’s average accuracy. GPT-3.5 Confidence Generalization As noted, we ran variants using the same GPT-3.5 generations as [Bhatt et al., 2023]. We show aggregate and per-topic accuracy in fig. 17, as well as reliance sensibility in fig. 18. <details> <summary>x77.png Details</summary>  ### Visual Description ## Violin Plot: High School Biology Accuracy Comparison ### Overview The image is a violin plot comparing the accuracy distributions of five different methods in a high school biology context. The plot includes a benchmark line at 0.85 accuracy, represented by a red dashed horizontal line. Each violin plot is color-coded and labeled to represent distinct experimental conditions. ### Components/Axes - **X-Axis (Categories)**: - "No LLM" (blue) - "LLM" (orange) - "LLM + Conf (Rand)" (green) - "LLM + Conf (Query)" (red) - "LLM + Conf (CT)" (purple) - **Y-Axis (Accuracy)**: Ranges from 0.0 to 1.0, labeled "Accuracy." - **Legend**: Located on the right, mapping colors to method labels. - **Benchmark Line**: Red dashed line at y = 0.85. ### Detailed Analysis 1. **"No LLM" (Blue)**: - Distribution spans 0.2 to 0.8, with a peak near 0.5. - Median accuracy ~0.5, indicating low and variable performance. 2. **"LLM" (Orange)**: - Distribution spans 0.4 to 0.9, with a peak near 0.7. - Median accuracy ~0.7, showing moderate improvement over "No LLM." 3. **"LLM + Conf (Rand)" (Green)**: - Distribution spans 0.6 to 0.95, with a peak near 0.8. - Median accuracy ~0.8, indicating further improvement. 4. **"LLM + Conf (Query)" (Red)**: - Distribution spans 0.7 to 0.95, with a peak near 0.85. - Median accuracy ~0.85, aligning with the benchmark line. 5. **"LLM + Conf (CT)" (Purple)**: - Distribution spans 0.8 to 0.98, with a peak near 0.9. - Median accuracy ~0.9, the highest among all methods. ### Key Observations - The red dashed benchmark line at 0.85 is exceeded by "LLM + Conf (Query)" and "LLM + Conf (CT)" methods. - "LLM + Conf (CT)" achieves the highest median accuracy (~0.9) and narrowest distribution, suggesting the most consistent performance. - "No LLM" and "LLM" methods show significant variability, with wide distributions and lower medians. - The progression from "No LLM" to "LLM + Conf (CT)" demonstrates a clear trend of increasing accuracy with added confidence mechanisms. ### Interpretation The data suggests that incorporating confidence mechanisms (e.g., "Conf (Rand)," "Conf (Query)," "Conf (CT)") significantly improves accuracy in high school biology assessments. The "LLM + Conf (CT)" method outperforms all others, indicating that the "CT" confidence strategy is the most effective. The benchmark line at 0.85 serves as a threshold for acceptable performance, with only the top two methods meeting or exceeding it. The narrowing of distributions in advanced methods implies reduced variability, pointing to more reliable outcomes. This analysis highlights the importance of confidence-based enhancements in LLM applications for educational contexts. </details> <details> <summary>x78.png Details</summary>  ### Visual Description ## Violin Plot: High School CS Model Accuracy Comparison ### Overview The image presents a comparative analysis of model accuracy distributions across five configurations in a high school computer science context. Five violin plots are arranged horizontally, each representing a different model variant, with accuracy values plotted on a vertical scale from 0.0 to 1.0. A red dashed reference line at 0.9 accuracy is included for benchmarking. ### Components/Axes - **X-axis**: Model configurations (categorical) - No LLM (blue) - LLM (orange) - LLM + Conf (Rand) (green) - LLM + Conf (Query) (red) - LLM + Conf (CT) (purple) - **Y-axis**: Accuracy (continuous scale from 0.0 to 1.0) - **Legend**: Top-right position, color-coded to match x-axis labels - **Reference Line**: Red dashed horizontal line at y=0.9 ### Detailed Analysis 1. **No LLM (Blue)**: - Wide distribution spanning 0.2–0.8 - Median accuracy ~0.5 (horizontal line within violin) - Long tail extending below 0.4 - Least concentrated distribution 2. **LLM (Orange)**: - Narrower distribution (0.6–0.9) - Median ~0.8 - Symmetrical shape with minimal tailing 3. **LLM + Conf (Rand) (Green)**: - Concentrated around 0.85–0.95 - Median ~0.9 - Slight left skew with minor tail below 0.8 4. **LLM + Conf (Query) (Red)**: - Tight distribution (0.8–0.95) - Median ~0.9 - Symmetrical with sharp peak near 0.9 5. **LLM + Conf (CT) (Purple)**: - Narrowest distribution (0.85–0.95) - Median ~0.9 - Symmetrical with minimal variability ### Key Observations - **Performance Threshold**: 60% of models (excluding No LLM) exceed the 0.9 accuracy benchmark (red line) - **Variability**: No LLM shows highest variability (range: 0.2–0.8), while LLM + Conf (CT) has the tightest distribution - **Improvement Pattern**: Adding confidence mechanisms (Rand, Query, CT) consistently improves accuracy over baseline LLM - **CT Superiority**: LLM + Conf (CT) demonstrates the most stable and highest-performing results ### Interpretation The data suggests that integrating confidence mechanisms with LLM significantly enhances accuracy in high school CS applications. The CT (likely "Contextual Tuning" or similar) variant achieves the most reliable performance, maintaining accuracy above 0.85 with minimal variance. The red reference line at 0.9 indicates a performance target, with three configurations (LLM + Conf variants) meeting or exceeding this benchmark. The No LLM baseline's wide distribution highlights the importance of model architecture improvements, while the LLM + Conf (Rand) and (Query) variants show intermediate gains. The symmetry in LLM + Conf distributions suggests consistent model behavior across different confidence strategies, with CT providing the optimal balance of accuracy and reliability. </details> <details> <summary>x79.png Details</summary>  ### Visual Description ## Violin Plot: US Foreign Policy Accuracy Comparison ### Overview The image presents a comparative analysis of accuracy distributions across five configurations of Large Language Models (LLMs) applied to US Foreign Policy tasks. The plot uses violin plots to visualize the spread and central tendency of accuracy scores, with a red dashed reference line at 0.85 accuracy. ### Components/Axes - **X-axis**: Categorical configurations: - No LLM (blue) - LLM (orange) - LLM + Conf (Rand) (green) - LLM + Conf (Query) (red) - LLM + Conf (CT) (purple) - **Y-axis**: Accuracy metric (0.0 to 1.0) - **Key elements**: - Red dashed line at 0.85 accuracy - Horizontal gridlines at 0.2, 0.4, 0.6, 0.8 - No explicit legend (colors mapped sequentially to x-axis labels) ### Detailed Analysis 1. **No LLM (blue)**: - Distribution centered around 0.6 accuracy - Wide spread with significant lower-tail density - Median ~0.55, interquartile range ~0.45-0.7 2. **LLM (orange)**: - Central tendency ~0.75 accuracy - Narrower distribution than No LLM - Interquartile range ~0.7-0.8 3. **LLM + Conf (Rand) (green)**: - Peaks near 0.8 accuracy - Bimodal distribution with secondary mode ~0.6 - Interquartile range ~0.75-0.85 4. **LLM + Conf (Query) (red)**: - Tight distribution centered at 0.85 - Sharp peak at threshold line - Minimal spread (~0.8-0.9) 5. **LLM + Conf (CT) (purple)**: - Highest central tendency (~0.9 accuracy) - Narrow distribution with slight right skew - Interquartile range ~0.85-0.95 ### Key Observations - **Threshold proximity**: Only LLM + Conf (Query) and LLM + Conf (CT) configurations reach/exceed the 0.85 accuracy threshold - **Progressive improvement**: Each configuration shows incremental accuracy gains: - No LLM → LLM: +0.15 - LLM → LLM + Conf (Rand): +0.05 - LLM + Conf (Rand) → LLM + Conf (Query): +0.05 - LLM + Conf (Query) → LLM + Conf (CT): +0.05 - **Distribution characteristics**: - No LLM shows highest variability - LLM + Conf (CT) demonstrates most consistent performance - LLM + Conf (Query) has highest density at threshold ### Interpretation The data demonstrates a clear performance hierarchy among configurations, with each incremental addition of confidence mechanisms (Conf) improving accuracy. The LLM + Conf (CT) configuration achieves near-target performance (0.9 vs 0.85 threshold), suggesting it represents optimal implementation. The bimodal distribution in LLM + Conf (Rand) indicates potential instability in random confidence application, while the tight distribution of LLM + Conf (Query) suggests more reliable performance. The red threshold line serves as a critical benchmark, with only the top two configurations meeting/exceeding this standard. This analysis implies that confidence mechanisms significantly enhance LLM performance on US Foreign Policy tasks, with the CT method providing the most substantial improvement. </details> <details> <summary>x80.png Details</summary>  ### Visual Description ## Violin Plot: Elementary Math Accuracy Comparison ### Overview The image displays a violin plot comparing the accuracy distributions of five different model configurations in elementary math tasks. The plot uses color-coded distributions to visualize performance variations across model architectures and confidence mechanisms. ### Components/Axes - **X-axis (Categories)**: - No LLM (blue) - LLM (orange) - LLM + Conf (Rand) (green) - LLM + Conf (Query) (red) - LLM + Conf (CT) (purple) - **Y-axis (Accuracy)**: Ranges from 0.0 to 1.0 with horizontal gridlines at 0.2, 0.4, 0.6, 0.8 - **Reference Line**: Red dashed line at y=0.3 - **Color Legend**: Directly mapped to x-axis categories (no separate legend box) ### Detailed Analysis 1. **No LLM (Blue)**: - Distribution peaks between 0.7-0.8 - Narrowest distribution with minimal spread - Median accuracy ~0.75 2. **LLM (Orange)**: - Bimodal distribution with peaks at ~0.5 and ~0.6 - Wider spread than No LLM - Median accuracy ~0.55 3. **LLM + Conf (Rand) (Green)**: - Single peak at ~0.6 - Moderate spread with slight dip at 0.4-0.5 - Median accuracy ~0.6 4. **LLM + Conf (Query) (Red)**: - Double-peaked distribution at ~0.6 and ~0.7 - Significant spread between 0.4-0.8 - Median accuracy ~0.65 5. **LLM + Conf (CT) (Purple)**: - Tallest peak at ~0.75 - Narrowest distribution among confidence-enhanced models - Median accuracy ~0.78 ### Key Observations - All models exceed the 0.3 accuracy threshold (red dashed line) - LLM + Conf (CT) shows highest median accuracy (0.78) and tightest distribution - No LLM and LLM + Conf (Rand) have the lowest variability - LLM + Conf (Query) demonstrates the widest performance spread - Confidence mechanisms generally improve performance over base LLM ### Interpretation The data suggests that confidence-enhanced models (particularly CT) achieve superior accuracy in elementary math tasks compared to base LLM configurations. The CT method appears most effective at maintaining high accuracy with minimal performance variance. The red threshold line indicates a performance floor, with all models demonstrating capability above this level. The widening distributions in confidence-enhanced models suggest increased computational complexity or parameter tuning requirements. Notably, the Query confidence method shows significant performance variability, potentially indicating sensitivity to input formatting or question complexity. </details> Figure 17: User accuracies per topic for the GPT-3.5 variants (with generalization confidence computed for the CT and Query cases). Red line indicates the model’s average accuracy. <details> <summary>x81.png Details</summary>  ### Visual Description ## Violin Plot: Reliance Sensitivity Comparison Across Model Configurations ### Overview The image presents a comparative analysis of Reliance Sensitivity across four model configurations using violin plots. Each plot visualizes the distribution of sensitivity scores, with horizontal dashed lines indicating central tendency measures (likely median or mean). The configurations compared are: 1. **LLM** (base model) 2. **LLM + Conf (Rand)** (random configuration augmentation) 3. **LLM + Conf (Query)** (query-based configuration augmentation) 4. **LLM + Conf (CT)** (contextual configuration augmentation) ### Components/Axes - **X-axis**: Categorical axis labeling the four model configurations. - **Y-axis**: Continuous scale labeled "Reliance Sensitivity" ranging from 0.3 to 1.0. - **Legend**: Located at the bottom-right corner, mapping colors to configurations: - Red: LLM - Teal: LLM + Conf (Rand) - Gray: LLM + Conf (Query) - Blue: LLM + Conf (CT) ### Detailed Analysis 1. **LLM (Red)**: - Median (~0.8) with high variability (wide distribution). - Scores range from ~0.6 to ~0.9, with a peak density near 0.8. 2. **LLM + Conf (Rand) (Teal)**: - Median (~0.75) with moderate variability. - Scores range from ~0.65 to ~0.85, with a flatter distribution compared to LLM. 3. **LLM + Conf (Query) (Gray)**: - Median (~0.8) with reduced variability (narrower distribution). - Scores range from ~0.7 to ~0.9, showing tighter clustering around the median. 4. **LLM + Conf (CT) (Blue)**: - Highest median (~0.85) with the least variability. - Scores range from ~0.8 to ~0.9, indicating consistent performance. ### Key Observations - **CT Configuration Outperforms Others**: The blue violin (LLM + Conf (CT)) demonstrates the highest median Reliance Sensitivity (~0.85) and the tightest distribution, suggesting superior and more consistent performance. - **Query vs. Rand**: The gray violin (LLM + Conf (Query)) outperforms the teal violin (LLM + Conf (Rand)) in both median and variability, indicating query-based augmentation is more effective than random. - **Base LLM Variability**: The red violin (LLM) shows the widest spread, implying significant performance inconsistency without configuration augmentation. ### Interpretation The data suggests that augmenting the base LLM with contextual configuration (CT) yields the most reliable performance, with a 5–10% improvement in median Reliance Sensitivity compared to the base model. Query-based augmentation (gray) provides a middle ground, balancing performance gains and reduced variability over random augmentation (teal). The base LLM’s wide distribution highlights the importance of configuration tuning for reliability. **Notable Anomalies**: - The LLM + Conf (Rand) configuration shows a bimodal distribution (visible as two peaks near 0.7 and 0.85), suggesting inconsistent benefits from random configuration augmentation. - The LLM + Conf (CT) violin’s sharp peak at ~0.85 indicates near-uniform performance across tested scenarios. </details> Figure 18: Reliance sensibility for the variants based on GPT-3.5 Freeform User Responses We permitted users to provide freeform responses at the end of the study. Some users were sensitive to confidence scores being reported and came up with their own heuristics for whether to rely on the model’s output. We include a sampling of comments across confidence variants: - “if it had a confidence of less than 50% it made me very skeptical.” - “The modelś confidence indeed helped me choose and select my answer as I trusted in them most of the time.” - “I didnt́ really rely on the confidence level. If I had 0 confidence in the answer myself I relied on the AI regardless.” - “if the models confidence fell below 45 I decided to investigate it myself by remembering pieces of information. and also reasoning the question. If it was above 45 I would automatically agree to its prediction but there were some few cases I challenged it even though it was above 45” - “At first I was hesistant to trust the model much because of the lower confidence levels but I still trusted it enough on topics I struggled with. As it went on, I was comfortable with confidence levels above 40.” - “If the modelś confidence was low and I thought I knew the answer (and it was different) I chose my answer” G.4 Interface and Instructions We show a sample interface of our extension of Modiste with user confidence in Figure 19, and present the the full set of instructions provided to users in Figures 20 and 21. Note, for the LLM-only and no-LLM conditions, we followed the instruction text from [Bhatt et al., 2023] directly, i.e., participants who saw only the LLM did not see the instruction page about model confidence, and participants in the “No-LLM” variant were not instructed about any model variant and were just instructed to answer the questions as best as they could by themselves. Participants also responded to a post survey questionarre after completing the user study, which we depict in Figure 22. <details> <summary>user_study_figs/instructions/page_with_feedback.png Details</summary>  ### Visual Description ## Screenshot: Biology Quiz Interface ### Overview The image depicts a quiz interface focused on a biology question about homology. The interface includes a progress bar, a question with multiple-choice options, a highlighted predicted answer, model confidence, and a user score display. ### Components/Axes - **Header**: - "Completion Progress" bar (partially filled, ~20% completion). - **Main Content**: - **Question**: "Which of the following pairs of structures is least likely to represent homology?" - **Options**: 1. The wings of a bat and the arms of a human 2. The hemoglobin of a baboon and that of a gorilla 3. **The mitochondria of a plant and those of an animal** (highlighted in yellow) 4. The wings of a bird and those of an insect - **Model Prediction**: Option 3 (highlighted in yellow). - **Confidence**: "The model's confidence in its answer is 40%" (text in blue). - **User Score**: "Your Score: 1 out of 2" (displayed in a light blue box). - **Footer**: - "SUBMIT" button (gray, centered). ### Detailed Analysis - **Question Structure**: - Four options are presented as radio buttons. - Option 3 is explicitly highlighted in yellow, indicating the model's predicted answer. - **Confidence and Score**: - Model confidence is stated as 40%, suggesting low certainty in the prediction. - User score shows 1 correct answer out of 2 total questions, implying partial accuracy. ### Key Observations - The highlighted option (mitochondria of plants and animals) is the model's prediction, but the low confidence (40%) raises questions about its reliability. - The user’s score (1/2) suggests they answered one question correctly, though the interface does not specify which question. - The progress bar indicates the quiz is in early stages (~20% completion). ### Interpretation - The interface is designed for educational assessment, testing knowledge of homology (shared ancestry of structures). - The model’s prediction (Option 3) is biologically questionable: mitochondria are organelles with endosymbiotic origins, making them homologous across plants and animals. This may indicate a flaw in the model’s training data or logic. - The user’s partial score (1/2) implies they correctly identified one homologous pair (e.g., bat wings and human arms, which are homologous as forelimbs) but may have struggled with others. - The low model confidence (40%) contrasts with the highlighted prediction, suggesting potential misalignment between the model’s output and its certainty. ## Notes - No non-English text is present. - All textual elements are transcribed verbatim, including formatting (bold, colors). - Spatial grounding: Elements are arranged vertically (progress bar → question → options → score → submit button). </details> Figure 19: Example interface from Modiste. Participants are informed of the question (and topic), as well as the LLM prediction and confidence. Participants are informed of their running score throughout the experiment. <details> <summary>user_study_figs/instructions/starter_inst.png Details</summary>  ### Visual Description ## Screenshot: Experiment Introduction Text ### Overview The image displays a text-based interface for an experimental study. The content explains the purpose of the experiment, compensation details, and includes navigation controls. No charts, diagrams, or data tables are present. ### Components/Axes - **Text Content**: - Welcome message - Experiment purpose: "understand how people make decisions with and without AI support" - Time estimate: "at most 30 minutes" (bolded) - Compensation: "$9/hour for a total of $4.50" (bolded) - **Navigation Controls**: - "Previous" button (grayed out, disabled) - "Next" button (active, white background) ### Detailed Analysis - **Text Structure**: - Three paragraphs of explanatory text. - Key metrics (time, compensation) emphasized with bold formatting. - **UI Elements**: - Buttons positioned at the bottom center. - "Previous" button disabled (grayed out), suggesting this is the first screen of a multi-step process. - "Next" button active, indicating progression to subsequent steps. ### Key Observations 1. Compensation is explicitly tied to study completion ("as long as you complete the study"). 2. Time estimate is presented as a maximum duration ("at most 30 minutes"). 3. Navigation controls imply a sequential workflow with at least two steps. ### Interpretation This interface appears to be part of a user study platform, likely for behavioral or cognitive research. The emphasis on compensation and time suggests ethical considerations for participant engagement. The disabled "Previous" button indicates a linear progression model, potentially to prevent participants from revisiting earlier decisions. The experiment's focus on AI-assisted decision-making aligns with human-computer interaction research goals, though the specific decision-making tasks are not visible in this screenshot. </details> <details> <summary>user_study_figs/instructions/likely_answer_inst.png Details</summary>  ### Visual Description ## Screenshot: Multiple-Choice Experiment Interface ### Overview The image depicts a minimalist user interface for a multiple-choice experiment. The screen contains instructional text, two navigation buttons, and a clean, white background with black text. The interface guides users through answering questions by selecting the "most likely answer" via radio buttons. ### Components/Axes - **Text Content**: - Two centered paragraphs explaining the experiment's purpose and task. - Bolded phrase: "**most likely answer**" (highlighting the task's critical instruction). - **UI Elements**: - Two outlined gray buttons at the bottom center: - Left button: Labeled "Previous" with a leftward arrow (`<`). - Right button: Labeled "Next" with a rightward arrow (`>`). ### Detailed Analysis - **Textual Content**: - First paragraph: *"In this experiment, you will be seeing multiple choice questions, from various topics, such as those that you may find in school (e.g., biology, mathematics, foreign policy, computer science)."* - Second paragraph: *"Your task is to determine the most likely answer for each question. You can select this category by clicking on the radio button associated with your answer."* - **UI Layout**: - Text is centered horizontally and vertically spaced for readability. - Buttons are positioned at the bottom center, equidistant from the text above. ### Key Observations - The bolded phrase "**most likely answer**" emphasizes the user's primary objective. - Navigation buttons suggest a sequential task flow (e.g., progressing through questions). - No visual distractions (e.g., colors, images) are present, prioritizing clarity. ### Interpretation This interface is designed for a cognitive or educational experiment, likely testing decision-making or knowledge recall. The emphasis on "most likely answer" implies probabilistic reasoning or uncertainty assessment. The absence of distracting elements ensures focus on the task. The navigation buttons indicate a structured, step-by-step progression, common in experimental setups requiring controlled user interaction. --- **Note**: No numerical data, charts, or diagrams are present. The image solely contains textual instructions and UI elements. </details> <details> <summary>user_study_figs/instructions/ai_pred_inst.png Details</summary>  ### Visual Description ## Screenshot: Task Interface with AI Model Prediction ### Overview The image depicts a minimalist user interface for a task-based system where participants interact with AI-generated predictions. The interface includes instructional text, a highlighted AI prediction, and navigation controls. ### Components/Axes - **Text Elements**: - Instructional text: "During the tasks, you will also see the **prediction of an AI-based model**." - Guidance: "The model's prediction will show up as yellow highlighting over that answer choice. If shown, you are free to use or ignore the information when selecting your answer however you wish." - **Navigation Buttons**: - "Previous" (left-aligned) and "Next" (right-aligned) buttons with gray backgrounds and black text. - **Visual Highlighting**: - Yellow highlighting (bold text) over the phrase "prediction of an AI-based model" in the instructional text. ### Detailed Analysis - **Textual Content**: - The primary instruction emphasizes the presence of AI-generated predictions during tasks. - The highlighted text ("prediction of an AI-based model") is bold and yellow, drawing attention to the AI's role. - Secondary text clarifies that the AI's predictions are optional for decision-making. - **UI Layout**: - Text is centered on a white background with black font. - Navigation buttons are positioned at the bottom, separated from the main text. - No additional visual elements (e.g., charts, diagrams) are present. ### Key Observations - The interface prioritizes clarity, using bold and colored text to emphasize critical information (AI predictions). - The optional use of AI predictions suggests a user-centric design, allowing autonomy in decision-making. - Navigation buttons imply a sequential task flow, typical of multi-step experiments or surveys. ### Interpretation This interface is designed for a controlled experiment or user study where participants engage with AI-assisted tasks. The AI's predictions are presented as supplementary information, not mandatory guidance, indicating a focus on evaluating human-AI collaboration dynamics. The simplicity of the design minimizes cognitive load, ensuring participants focus on the task rather than the interface itself. The absence of additional visual elements (e.g., progress bars, timers) suggests the study may prioritize qualitative over quantitative data collection. </details> <details> <summary>user_study_figs/instructions/confidence_inst.png Details</summary>  ### Visual Description ## Screenshot: User Interface for Model Prediction Feedback ### Overview The image is a screenshot of a user interface (UI) with a minimalist design. It features a centered block of text and two navigation buttons at the bottom. The text explains that the model's confidence in its predictions will be displayed in blue for each question. The buttons allow users to navigate between questions. ### Components/Axes - **Text Content**: Centered in the upper half of the image. - **Buttons**: Two rectangular buttons labeled "Previous" (with a left arrow) and "Next" (with a right arrow), positioned side-by-side at the bottom center. ### Content Details - **Text**: "You will also see the model's confidence in its prediction (which will be shown in blue) for each question." - The phrase "model's confidence" is emphasized in *italic* and **bold** formatting. - The color "blue" is explicitly mentioned as the visual indicator for confidence. - **Buttons**: - "Previous" button: Left arrow (`<`) followed by the text "Previous". - "Next" button: Right arrow (`>`) followed by the text "Next". - Both buttons have a light gray background with dark gray text and borders. ### Key Observations - The UI is designed to guide users through a sequence of questions, with navigation controls for moving forward or backward. - The mention of "blue" for confidence suggests a color-coded feedback system, though no actual predictions or confidence values are visible in the image. - The layout is clean and uncluttered, prioritizing clarity and ease of use. ### Interpretation This UI appears to be part of a system where users interact with a model's predictions, likely in an educational or assessment context. The explicit mention of "confidence in its prediction" implies the model provides probabilistic or certainty-based outputs, which are visually highlighted in blue. The navigation buttons suggest a step-by-step process, possibly for reviewing or validating predictions. The absence of actual data in the image indicates this is a static example or placeholder, with dynamic content (e.g., blue-highlighted confidence scores) likely loaded during runtime. The design emphasizes user guidance and transparency about the model's behavior. </details> <details> <summary>user_study_figs/instructions/seconds_per.png Details</summary>  ### Visual Description ## Screenshot: Text-Based Interface Instructions ### Overview The image displays a text-based interface with instructional content and navigation buttons. The text provides guidance for users navigating a problem-solving or assessment system, emphasizing time constraints and effort. ### Components/Axes - **Text Blocks**: Two paragraphs of instructional text. - **Buttons**: - "Previous" (left-aligned, with a left arrow `<`). - "Next" (right-aligned, with a right arrow `>`). ### Detailed Analysis 1. **First Paragraph**: - Text: "We encourage you to try to work through each problem. You will not be able to continue to the next question until at least **10 seconds** have passed. The SUBMIT button will change from grey to blue when you are able to click to move to the next page whenever you are ready to answer." - Key Details: - **10 seconds** is bolded, indicating a critical time constraint. - Mentions a grey-to-blue color change for the "SUBMIT" button, signaling readiness to proceed. 2. **Second Paragraph**: - Text: "Of course you can take longer than 10 seconds on any question if needed! It may be very challenging to determine the answer for some questions. Others may be easy. **Please try your best regardless.**" - Key Details: - **Please try your best regardless** is bolded, emphasizing effort over speed. - Acknowledges variability in question difficulty. 3. **Buttons**: - Positioned centrally at the bottom of the interface. - Labels: "Previous" (left) and "Next" (right), with directional arrows. ### Key Observations - The interface enforces a **minimum 10-second delay** between questions, likely to prevent rapid, unconsidered answers. - The bolded phrases ("10 seconds" and "Please try your best regardless") highlight priorities: time management and perseverance. - The color-coded "SUBMIT" button (grey → blue) provides visual feedback for user progression. ### Interpretation This interface appears to be part of an **adaptive assessment or learning platform** designed to: 1. Ensure users engage thoughtfully with each problem by enforcing a pause. 2. Accommodate varying question difficulties while maintaining a baseline time requirement. 3. Encourage resilience by emphasizing effort ("try your best") regardless of perceived challenge. The design balances structure (timed progression) with flexibility (allowing extended time), suggesting a focus on both accuracy and user experience. The bolded text acts as a rhetorical device to reinforce these dual objectives. </details> <details> <summary>user_study_figs/instructions/bonus.png Details</summary>  ### Visual Description ## Screenshot: Task Instructions and Navigation Interface ### Overview The image displays a text-based interface with instructions for a task, including a bonus structure and navigation controls. The text is presented in a clean, monospaced font on a white background. ### Components/Axes - **Text Content**: 1. First paragraph: "You will receive a **bonus** of up to a rate of $10/hour (+$0.50) based on how many questions you correctly answer." - Key terms: "bonus" (bolded), "$10/hour", "+$0.50". 2. Second paragraph: "You will be informed whether or not you are correct after each trial." - **Buttons**: - Two rectangular buttons centered below the text: - Left button: Labeled "Previous" with a leftward arrow (`<`). - Right button: Labeled "Next" with a rightward arrow (`>`). ### Detailed Analysis - **Textual Details**: - The bonus structure specifies a base rate of $10/hour, with an additional $0.50 per correct answer. The term "bonus" is emphasized via bold formatting. - Feedback is guaranteed after each trial, indicating real-time performance tracking. - **Button Placement**: - Buttons are horizontally centered, with equal spacing between them and the text above. - Arrows (`<` and `>`) suggest navigation through sequential trials or questions. ### Key Observations - The interface prioritizes clarity, using bold text to highlight critical terms like "bonus." - The bonus calculation is explicit: $10/hour base + $0.50 per correct answer, though the exact formula (e.g., per-question vs. hourly rate) is not fully detailed. - Navigation buttons imply a linear progression through tasks or questions. ### Interpretation This interface appears to be part of a user study or gamified task platform. The bonus structure incentivizes accuracy, while real-time feedback encourages engagement. The navigation buttons suggest participants can iterate through trials, possibly revisiting previous questions. The absence of visual distractions (e.g., charts, images) indicates a focus on textual instructions and minimal cognitive load. **Note**: No numerical data trends or categorical labels are present, as the image contains only text and UI elements. </details> Figure 20: Experiment instructions for the confidence variants. <details> <summary>user_study_figs/instructions/questions.png Details</summary>  ### Visual Description ## Screenshot: Quiz Interface Navigation ### Overview The image displays a minimalist user interface for a quiz or assessment platform. The primary text indicates the total number of questions in the assessment, with navigation controls for progressing through the questions. ### Components/Axes - **Text Elements**: - "You will see a total of 60 questions." (Centered, bold font) - "Previous" (Left-aligned button, left-facing arrow `<` prefix) - "Next" (Right-aligned button, right-facing arrow `>` suffix) - **UI Layout**: - Two rectangular buttons with rounded corners, separated by negative space. - Buttons have a light gray background with dark gray text and borders. ### Detailed Analysis - **Textual Content**: - The total question count is explicitly stated as **60**, with the number emphasized in bold. - Navigation buttons use directional arrows (`<` and `>`) to indicate functionality. - **Styling**: - Text uses a sans-serif font (likely system default). - Buttons follow a flat design aesthetic with no gradients or shadows. ### Key Observations - The interface prioritizes clarity, with no decorative elements. - The "60 questions" text is the focal point, suggesting this is a progress indicator or introductory screen. - Buttons are symmetrically placed, implying equal importance to backward/forward navigation. ### Interpretation This interface is designed for a linear assessment experience, where users navigate sequentially through 60 questions. The absence of additional controls (e.g., "Submit," "Skip") suggests this is either a pre-quiz screen or a simplified navigation layer. The emphasis on the total question count may aim to set user expectations for the assessment's length. The minimalist design reduces cognitive load, focusing attention on progression rather than aesthetics. </details> <details> <summary>user_study_figs/instructions/next.png Details</summary>  ### Visual Description ## Screenshot: Experiment Comprehension Check Instructions ### Overview The image displays a webpage with instructional text and navigation buttons. The content is centered on a white background with black text, emphasizing preparation for an experiment. ### Components/Axes - **Text Blocks**: 1. First paragraph: "When you are ready, please click "Next" to complete a quick comprehension check, before moving on to the experiment." 2. Second paragraph: "Please make sure to window size is in full screen, or substantially large enough, to properly view the questions." - **Buttons**: - "Previous" (left-aligned arrow: `< Previous`) - "Next" (right-aligned arrow: `Next >`) ### Content Details - **Text Formatting**: - The word "Next" in the first paragraph is **bold** and enclosed in double quotes (`"Next"`). - Button labels use directional arrows (`<` and `>`) to indicate navigation. - **Positioning**: - Text blocks are centered horizontally. - Buttons are aligned horizontally below the text, centered relative to each other. ### Key Observations - The instructions explicitly direct users to click "Next" after preparing, suggesting a mandatory step before proceeding. - The second paragraph emphasizes window size requirements, indicating potential issues with question visibility if not followed. - Buttons are minimalistic, with no additional styling beyond text and arrows. ### Interpretation This interface appears to be part of a user onboarding process for an experiment. The emphasis on window size suggests the experiment may involve complex or numerous questions requiring ample screen space. The bolded and quoted "Next" button highlights its importance as a critical action item. The presence of a "Previous" button implies users can backtrack, though the text does not explicitly permit this. The design prioritizes clarity and compliance with experimental protocols over interactivity. </details> <details> <summary>user_study_figs/instructions/mc_check.png Details</summary>  ### Visual Description ## Screenshot: Survey/Questionnaire Interface ### Overview The image depicts a minimalist survey interface with a light gray background and black text. It contains instructional text, two multiple-choice questions, and a "Continue" button. The design emphasizes clarity and user guidance, with asterisks (*) denoting required fields. ### Components/Axes 1. **Header Text**: - *"Check your knowledge before you begin. If you don't know the answers, don't worry; we will show you the instructions again."* 2. **Question 1**: - *"What will you be asked to determine in this task?"* - **Answer Options**: - ☐ The answer to a multiple choice question. - ☐ The least likely answer to a multiple choice question. - ☐ The most likely categories of an image. 3. **Question 2**: - *"How will you select your answer?"* - **Answer Options**: - ☐ Typing in a text box. - ☐ Clicking on a radio button. - ☐ Selecting from a dropdown menu. 4. **Footer Button**: - A rectangular "Continue" button with a light gray outline and centered text. ### Detailed Analysis - **Textual Content**: - All text is in English, with no non-English elements. - Asterisks (*) appear next to both questions, indicating mandatory responses. - Answer options use bullet points with radio button placeholders (☐). - **UI Elements**: - The "Continue" button is positioned centrally at the bottom, suggesting progression to the next step. - No visual indicators (e.g., colors, icons) differentiate answer options, relying solely on text. ### Key Observations - The interface prioritizes simplicity, avoiding visual clutter. - Questions are structured to assess user familiarity with task requirements (e.g., answer types, selection methods). - The absence of pre-selected options implies users must actively choose responses. ### Interpretation This interface likely serves as a pre-task assessment to gauge user understanding of survey mechanics. By asking about answer types (e.g., "most likely categories of an image"), it may be testing familiarity with image classification tasks or data labeling workflows. The inclusion of a "Continue" button without a "Submit" option suggests the survey is part of a larger workflow, possibly transitioning users to a task after confirming their readiness. The lack of visual cues for answer selection (e.g., color-coding) implies the focus is on textual comprehension rather than interactive design preferences. </details> Figure 21: Experiment instructions for the confidence variants (continued). <details> <summary>user_study_figs/instructions/postsurvey_questionarre.png Details</summary>  ### Visual Description ## Screenshot: Survey Form for Experiment Feedback ### Overview The image is a screenshot of a survey form designed to collect participant feedback after completing an experiment. The form includes instructions, structured questions, and open-ended response fields. A "Finish" button is present at the bottom to submit responses. ### Components/Axes - **Header Text**: - "Thank you for participating in our study!" - "Click 'Finish' to complete the experiment and receive compensation. If you have any comments about the experiment, please let us know in the form below." - **Questions**: 1. **Difficulty Rating**: - Label: "How challenging did you find the questions? (On a scale of 1-10, with 10 being very challenging)" - Input: A dropdown menu (blue border, empty). 2. **Model Confidence Impact**: - Label: "Did the model's confidence impact your response? In what way if so, please be as specific as possible (1-3 sentences)" - Input: A text field (empty). 3. **Struggled Topics**: - Label: "Were there any question topics you struggled with?" - Input: A text field (empty). 4. **Confident Topics**: - Label: "Were there any question topics you were always very confident in?" - Input: A text field (empty). 5. **Additional Comments**: - Label: "Do you have any additional comments to share with us?" - Input: A larger text area (empty). - **Footer**: - "Finish" button (gray background, black text). ### Detailed Analysis - **Textual Content**: All text is in English, with no non-English elements. - **Input Fields**: - The difficulty rating uses a 1-10 scale, suggesting quantitative feedback. - The model confidence question requires concise, specific responses (1-3 sentences). - Struggled and confident topics fields are open-ended, allowing qualitative input. - The additional comments field is the largest, accommodating extended feedback. - **Layout**: - Questions are vertically stacked with consistent spacing. - Input fields are aligned to the right of their labels. - The "Finish" button is centered at the bottom. ### Key Observations - The form prioritizes structured feedback (e.g., difficulty scale) while allowing open-ended responses for nuanced insights. - The model confidence question implies the experiment involved an AI or automated system whose confidence levels were visible to participants. - The "Finish" button’s placement ensures users cannot proceed without submitting feedback. ### Interpretation This survey form is designed to evaluate participant experience and gather actionable insights for improving the experiment. The combination of quantitative (difficulty rating) and qualitative (open-ended questions) data allows researchers to: 1. Quantify overall challenge levels. 2. Identify specific areas of confusion or strength. 3. Understand how the model’s behavior (confidence) influenced responses. 4. Capture unanticipated feedback through the comments section. The form’s structure balances brevity with depth, ensuring participants can provide both quick ratings and detailed explanations. The emphasis on model confidence suggests the experiment may involve human-AI interaction, where the model’s perceived reliability could bias participant responses. </details> Figure 22: Sample pot-survey questionnaire for users who were allocated to a variant wherein they saw model confidence. Appendix H Broader Impact and Implications The goal of this work is to make LLM outputs have better confidence values associated with them. With successful, calibrated confidence values, the machine systems ultimately become more interpretable and trustworthy by a user [Janssen et al., 2008]. When applied correctly, our advancements will help users be able to make decisions based off of LLM outputs in a more informed way. Similar examples in other domains, like AlphaFold Terwilliger et al. [2023], have shown how well-calibrated confidence scores can be useful in complex decision-making domains. Our hope is to replicate those broad findings in LLMs. We acknowledge the ongoing debate over the appropriateness, limitations, and harms of LLMs. We do highlight that the development of more confident, interpretable, and trustworthy LLMs can lead to continued techno-solutionism in unintended applications. Specifically, we highlight that our work is limited to use-cases with fact-based questions. Many applications of text-based LLMs are generative, meaning that there is no way for our paradigm to be applied appropriately, and the use of a confidences from calibration-tuned models could be misleading or damaging without checks and guardrails. Additionally, even within the fact-based paradigm, what is true can be subjective, with ground truth in machine learning being a contested topic [Aroyo and Welty, 2015, Uma et al., 2021]. The philosophical debate on these topics is beyond the expertise of the authors; nonetheless, we believe that the ongoing debate over the appropriateness of LLMs should be considered in context with the benefits of our approach in making LLMs more interpretable and useful. Appendix I NeurIPS Paper Checklist 1. Claims 1. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? 1. Answer: [Yes] 1. Justification: We describe and link all claims in section 1. 1. Guidelines: - The answer NA means that the abstract and introduction do not include the claims made in the paper. - The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. - The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. - It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 1. Limitations 1. Question: Does the paper discuss the limitations of the work performed by the authors? 1. Answer: [Yes] 1. Justification: We provide a discussion on the limitations in section 8. - The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. - The authors are encouraged to create a separate ”Limitations” section in their paper. - The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. - The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. - The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. - The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. - If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. - While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 1. Theory Assumptions and Proofs 1. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 1. Answer: [N/A] 1. Justification: [N/A] - The answer NA means that the paper does not include theoretical results. - All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced. - All assumptions should be clearly stated or referenced in the statement of any theorems. - The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. - Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. - Theorems and Lemmas that the proof relies upon should be properly referenced. 1. Experimental Result Reproducibility 1. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? 1. Answer: [Yes] 1. Justification: We provide the complete code, and the complete list of datasets used for all experiments in section 5 to reproduce all our experiments with instructions. All hyperparameters are described in section 5. 1. Guidelines: - The answer NA means that the paper does not include experiments. - If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. - If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. - Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. - While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example 1. If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. 1. If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. 1. If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). 1. We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 1. Open access to data and code 1. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? 1. Answer: [Yes] 1. Justification: We provide the complete code, and the complete list of datasets used for all experiments in section C.2 to reproduce all our experiments with instructions. All hyperparameters are described in section 5. 1. Guidelines: - The answer NA means that paper does not include experiments requiring code. - Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details. - While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). - The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details. - The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. - The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. - At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). - Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 1. Experimental Setting/Details 1. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? 1. Answer: [Yes] 1. Justification: We provide the complete code, and the complete list of datasets used for all experiments in section C.2 to reproduce all our experiments with instructions. All hyperparameters are described in section 5. 1. Guidelines: - The answer NA means that the paper does not include experiments. - The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. - The full details can be provided either with the code, in appendix, or as supplemental material. 1. Experiment Statistical Significance 1. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? 1. Answer: [Yes] 1. Justification: All figures are appropriately labeled with the error bars. 1. Guidelines: - The answer NA means that the paper does not include experiments. - The authors should answer ”Yes” if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. - The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). - The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) - The assumptions made should be given (e.g., Normally distributed errors). - It should be clear whether the error bar is the standard deviation or the standard error of the mean. - It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. - For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). - If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 1. Experiments Compute Resources 1. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? 1. Answer: [Yes] 1. Justification: We provide an estimate of the compute resources required in section 5. 1. Guidelines: - The answer NA means that the paper does not include experiments. - The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. - The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. - The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper). 1. Code Of Ethics 1. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 1. Answer: [Yes] 1. Justification: We have read the ethics guidelines 1. Guidelines: - The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics. - If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. - The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 1. Broader Impacts 1. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 1. Answer: [Yes] 1. Justification: We provide a broader impact statement in appendix H 1. Guidelines: - The answer NA means that there is no societal impact of the work performed. - If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. - Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. - The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. - The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. - If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 1. Safeguards 1. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 1. Answer: [N/A] 1. Justification: We train on open-access models with open-source datasets. We do not change their generation behavior, and all existing safeguards (if any) remain. 1. Guidelines: - The answer NA means that the paper poses no such risks. - Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. - Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. - We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 1. Licenses for existing assets 1. Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? 1. Answer: [Yes] 1. Justification: We explicitly cite all models in section 5. All datasets used are listed and cited in section C.2. 1. Guidelines: - The answer NA means that the paper does not use existing assets. - The authors should cite the original paper that produced the code package or dataset. - The authors should state which version of the asset is used and, if possible, include a URL. - The name of the license (e.g., CC-BY 4.0) should be included for each asset. - For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. - If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. - For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. - If this information is not available online, the authors are encouraged to reach out to the asset’s creators. 1. New Assets 1. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 1. Answer: [Yes] 1. Justification: We release our trained models for easy use via Hugging Face. 1. Guidelines: - The answer NA means that the paper does not release new assets. - Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. - The paper should discuss whether and how consent was obtained from people whose asset is used. - At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 1. Crowdsourcing and Research with Human Subjects 1. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? 1. Answer: [Yes] 1. Justification: We provide screenshots of our instructions, as well as details of compensation in appendix G. 1. Guidelines: - The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. - Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. - According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 1. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects 1. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? 1. Answer: [Yes] 1. Justification: We received prior approval from our respective institutional ethics review body for our user study. All users provided consent before partaking in the study. 1. Guidelines: - The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. - Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. - We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. - For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.