## Chart Type: Stacked Histogram ### Overview The image presents two stacked histograms, one for "Answerable" questions and one for "Unanswerable" questions. Each histogram shows the distribution of P(correct) values for two models: "Zero-Shot" (pink) and "Trained" (purple). The histograms are stacked, meaning the bars for each model are added on top of each other. ### Components/Axes * **Y-axis (Density):** Ranges from 1 to 5, with tick marks at 1, 3, and 5. * **X-axis (P(correct)):** Ranges from 30% to 90%, with tick marks at 30%, 50%, 70%, and 90%. * **Titles:** "Answerable" (top histogram) and "Unanswerable" (bottom histogram). * **Legend:** Located at the top of the image. "Zero-Shot" is represented by pink, and "Trained" is represented by purple. ### Detailed Analysis **Answerable Histogram:** * **Zero-Shot (Pink):** The distribution is skewed towards higher P(correct) values. The density increases from 30% to a peak around 70%-80%, then decreases slightly towards 90%. * **Trained (Purple):** The distribution is more uniform across the range of P(correct) values, with a slight increase in density between 50% and 70%. **Unanswerable Histogram:** * **Zero-Shot (Pink):** The distribution is centered around 50%-60% P(correct), with a lower density at both ends of the range. * **Trained (Purple):** The distribution is skewed towards lower P(correct) values, with a peak around 30%-40%. ### Key Observations * For "Answerable" questions, the "Zero-Shot" model tends to have higher P(correct) values compared to the "Trained" model. * For "Unanswerable" questions, the "Trained" model tends to have lower P(correct) values compared to the "Zero-Shot" model. * The "Trained" model shows a clear distinction between "Answerable" and "Unanswerable" questions, with higher P(correct) for "Answerable" and lower P(correct) for "Unanswerable". ### Interpretation The data suggests that the "Zero-Shot" model performs better on "Answerable" questions, while the "Trained" model is better at distinguishing between "Answerable" and "Unanswerable" questions. The "Trained" model seems to have learned to assign lower probabilities to "Unanswerable" questions, indicating a better understanding of the task. The "Zero-Shot" model, on the other hand, seems to assign similar probabilities to both types of questions. This could indicate that the "Zero-Shot" model is less sensitive to the nuances of the questions and answers.

\n ## Histograms: Probability of Correctness for Answerable and Unanswerable Questions ### Overview The image presents two histograms, stacked vertically. The top histogram represents the distribution of the probability of correctness (P(correct)) for "Answerable" questions, while the bottom histogram shows the distribution for "Unanswerable" questions. Each histogram displays two data series: "Zero-Shot" (pink) and "Trained" (purple), representing the performance of a model in these two scenarios. The x-axis represents P(correct) ranging from 30% to 90%, and the y-axis represents Density, ranging from 1 to 5. ### Components/Axes * **Title (Top):** Answerable * **Title (Bottom):** Unanswerable * **X-axis Label:** P(correct) * **X-axis Scale:** 30%, 50%, 70%, 90% * **Y-axis Label:** Density * **Y-axis Scale:** 1, 2, 3, 4, 5 * **Legend:** * Zero-Shot: Pink color * Trained: Purple color ### Detailed Analysis or Content Details **Top Histogram (Answerable):** * **Trained (Purple):** The distribution is unimodal, peaking around 70-80% P(correct). The density rises from approximately 1.5 at 30% to a maximum of approximately 4.8 at around 75%, then declines to approximately 1.5 at 90%. * **Zero-Shot (Pink):** The distribution is also unimodal, but is more spread out and peaks at a lower P(correct) value, around 80-85%. The density rises from approximately 0.5 at 30% to a maximum of approximately 3.5 at around 85%, then declines to approximately 0.5 at 90%. **Bottom Histogram (Unanswerable):** * **Trained (Purple):** The distribution is unimodal, peaking around 30-40% P(correct). The density rises from approximately 0 at 30% to a maximum of approximately 5 at around 35%, then declines to approximately 0.5 at 90%. * **Zero-Shot (Pink):** The distribution is unimodal, peaking around 50-60% P(correct). The density rises from approximately 0 at 30% to a maximum of approximately 3 at around 55%, then declines to approximately 0.5 at 90%. ### Key Observations * For "Answerable" questions, the "Trained" model consistently outperforms the "Zero-Shot" model, achieving higher probabilities of correctness. * For "Unanswerable" questions, the "Trained" model has a lower peak probability of correctness compared to the "Zero-Shot" model. * The distributions for both models are skewed towards higher P(correct) values for "Answerable" questions and lower P(correct) values for "Unanswerable" questions. * The "Trained" model shows a sharper peak in the "Answerable" histogram, indicating a more concentrated performance around a specific P(correct) value. ### Interpretation The data suggests that training the model significantly improves its performance on answerable questions, leading to a higher probability of correctness. However, when faced with unanswerable questions, the trained model appears to be less confident in its incorrect answers, resulting in a lower peak probability of correctness compared to the zero-shot model. This could indicate that the training process has taught the model to recognize when a question is unanswerable and to avoid making confident, but incorrect, predictions. The difference in distribution shapes between answerable and unanswerable questions highlights the model's ability to differentiate between the two types of questions, with the trained model demonstrating a stronger ability to do so. The zero-shot model, lacking this training, appears to attempt to answer all questions, even those that are unanswerable, leading to a broader, but less accurate, distribution of probabilities.

## Density Plot: Model Confidence by Question Answerability ### Overview The image displays two vertically stacked density plots comparing the confidence distributions of two models ("Zero-Shot" and "Trained") on questions categorized as "Answerable" and "Unanswerable." The plots visualize the probability of a correct answer, P(correct), on the x-axis against the density of predictions on the y-axis. ### Components/Axes * **Legend:** Positioned at the top center. Contains two entries: * **Zero-Shot:** Represented by pink/magenta bars. * **Trained:** Represented by purple/violet bars. * **Top Plot Title:** "Answerable" * **Bottom Plot Title:** "Unanswerable" * **Shared X-Axis Label:** "P(correct)" * **Axis Markers/Ticks:** 30%, 50%, 70%, 90%. * **Shared Y-Axis Label:** "Density" * **Axis Markers/Ticks:** 1, 3, 5. ### Detailed Analysis The analysis is segmented by plot region. **1. Top Plot: "Answerable" Questions** * **Trend Verification:** Both distributions are skewed toward higher probabilities, indicating higher confidence for answerable questions. * **Zero-Shot (Pink) Series:** The distribution is relatively narrow and peaks sharply in the high-confidence region. The highest density bars are located approximately between 70% and 80% P(correct). The density falls off rapidly below 60% and above 85%. * **Trained (Purple) Series:** The distribution is broader and more spread out than the Zero-Shot series. It also peaks in the high-confidence region (around 70-80%), but with a lower maximum density. It shows a more gradual slope, with significant density extending down to the 50-60% range. **2. Bottom Plot: "Unanswerable" Questions** * **Trend Verification:** The distributions shift leftward toward lower probabilities compared to the "Answerable" plot, indicating lower confidence for unanswerable questions. * **Zero-Shot (Pink) Series:** The distribution shows a clear peak in the low-to-mid confidence range. The highest density bars are located approximately between 40% and 50% P(correct). There is a long tail extending into higher probabilities, but density diminishes significantly above 70%. * **Trained (Purple) Series:** The distribution is flatter and more uniform compared to its counterpart in the "Answerable" plot. It does not have a single sharp peak. Density is relatively consistent across the 30% to 60% range, with a slight concentration around 40-50%. It shows less density in the very high confidence regions (>70%) compared to the Zero-Shot model on unanswerable questions. ### Key Observations 1. **Confidence Calibration by Category:** Both models exhibit higher confidence (higher P(correct)) for "Answerable" questions and lower confidence for "Unanswerable" questions, which is a desirable trait. 2. **Model Behavior Difference:** The "Zero-Shot" model displays more extreme confidence distributions—sharper peaks at high confidence for answerable questions and at lower confidence for unanswerable questions. The "Trained" model's distributions are more spread out and moderate. 3. **Overconfidence on Unanswerable:** The "Zero-Shot" model retains a notable tail of high-confidence predictions (60-80% P(correct)) even for "Unanswerable" questions, suggesting potential overconfidence. The "Trained" model shows a more subdued tail in this region. 4. **Clarity of Signal:** The separation between the "Answerable" and "Unanswerable" distributions appears more distinct for the "Zero-Shot" model. ### Interpretation This data suggests that the training process calibrates the model's confidence estimates. While the Zero-Shot model is more decisive (assigning very high or low probabilities), it may be more prone to overconfidence, particularly on difficult (unanswerable) questions. The Trained model, while less decisive, demonstrates more nuanced and potentially more reliable confidence scores across both question types. The plots visually argue that training improves a model's ability to express appropriate uncertainty, which is critical for trustworthy AI systems. The clear shift in distributions between "Answerable" and "Unanswerable" categories for both models indicates that the underlying model architecture is capable of distinguishing between these question types based on its internal representations.

## Bar Chart: Performance Comparison of Zero-Shot and Trained Models ### Overview The image is a bar chart comparing the performance of two methods—**Zero-Shot** (pink) and **Trained** (purple)—across two categories: **Answerable** and **Unanswerable** questions. The x-axis represents the probability of correct answers (P(correct)) in increments of 30%, 50%, 70%, and 90%, while the y-axis shows density values ranging from 0 to 5. ### Components/Axes - **Legend**: - **Zero-Shot**: Pink bars. - **Trained**: Purple bars. - **X-Axis (P(correct))**: Labeled with percentages (30%, 50%, 70%, 90%). - **Y-Axis (Density)**: Labeled "Density" with values from 0 to 5. - **Categories**: - **Answerable**: Top section of the chart. - **Unanswerable**: Bottom section of the chart. ### Detailed Analysis #### Answerable Questions - **Zero-Shot (Pink)**: - Density peaks at **70% P(correct)**, with a moderate spread between 50% and 90%. - Lower density at 30% and 50%. - **Trained (Purple)**: - Density peaks at **50% P(correct)**, with a broader distribution across 30% to 70%. - Lower density at 90%. #### Unanswerable Questions - **Zero-Shot (Pink)**: - Density peaks at **30% P(correct)**, with a sharp drop at higher percentages. - Minimal presence at 50% and 70%. - **Trained (Purple)**: - Density peaks at **50% P(correct)**, with a flatter distribution across 30% to 70%. - Slightly higher density at 70% compared to Zero-Shot. ### Key Observations 1. **Zero-Shot** performs better on **Answerable** questions, particularly at higher P(correct) thresholds (70–90%). 2. **Trained** models show higher density in **Unanswerable** questions, peaking at 50% P(correct), suggesting improved ability to identify unanswerable queries. 3. **Zero-Shot** has a narrower distribution for Answerable questions, while **Trained** models exhibit broader performance across P(correct) ranges. ### Interpretation The data suggests that **Trained models** are more effective at distinguishing **Unanswerable** questions, likely due to better generalization or calibration. However, **Zero-Shot** models outperform in **Answerable** scenarios, especially at higher confidence levels. This trade-off highlights a potential design consideration: training improves reliability in rejecting unanswerable queries but may reduce performance on high-confidence answerable tasks. The density distributions imply that **Trained** models are less certain about their answers in Answerable cases, while **Zero-Shot** models are more decisive but less accurate in Unanswerable scenarios.