## Line Graphs: "I-Don't-Know" Rate vs. Layer for Mistral-7B-v0.1 and Mistral-7B-v0.3 ### Overview The image presents two line graphs comparing the "I-Don't-Know" rate across different layers of the Mistral-7B model, specifically versions v0.1 and v0.3. Each graph plots the "I-Don't-Know" rate (y-axis) against the layer number (x-axis) for various question-answering (QA) tasks, distinguished by whether the question (Q) or answer (A) is anchored and the specific dataset used (PopQA, TriviaQA, HotpotQA, NQ). The graphs aim to illustrate how the model's uncertainty varies across layers and between the two versions. ### Components/Axes * **Titles:** * Left Graph: "Mistral-7B-v0.1" * Right Graph: "Mistral-7B-v0.3" * **Y-Axis:** * Label: "I-Don't-Know Rate" * Scale: 0 to 100, with tick marks at 0, 20, 40, 60, 80, and 100. * **X-Axis:** * Label: "Layer" * Scale: 0 to 30, with tick marks at intervals of 5 (0, 10, 20, 30). * **Legend:** Located at the bottom of the image, it identifies the different QA tasks represented by different colored lines: * Blue: Q-Anchored (PopQA) * Brown Dashed: A-Anchored (PopQA) * Green Dotted: Q-Anchored (TriviaQA) * Purple Dashed: A-Anchored (TriviaQA) * Gray Dashed: Q-Anchored (HotpotQA) * Orange Dotted: A-Anchored (HotpotQA) * Red Dashed: Q-Anchored (NQ) * Black Dotted: A-Anchored (NQ) ### Detailed Analysis #### Mistral-7B-v0.1 (Left Graph) * **Q-Anchored (PopQA) - Blue:** Starts at 100% at layer 0, drops sharply to near 0% by layer 10, then fluctuates between 0% and 40% for the remaining layers. * Values: ~100% at layer 0, ~0% at layer 10, fluctuates between ~0-40% from layer 10-30. * **A-Anchored (PopQA) - Brown Dashed:** Starts around 45% and remains relatively stable between 40% and 60% across all layers. * Values: Stays between ~40-60% from layer 0-30. * **Q-Anchored (TriviaQA) - Green Dotted:** Starts at 0% at layer 0, rises sharply to 100% at layer 5, then fluctuates between 0% and 60% for the remaining layers. * Values: ~0% at layer 0, ~100% at layer 5, fluctuates between ~0-60% from layer 5-30. * **A-Anchored (TriviaQA) - Purple Dashed:** Starts around 40% at layer 0, drops to 20% at layer 10, then fluctuates between 20% and 60% for the remaining layers. * Values: ~40% at layer 0, ~20% at layer 10, fluctuates between ~20-60% from layer 10-30. * **Q-Anchored (HotpotQA) - Gray Dashed:** Starts around 50% and remains relatively stable between 40% and 80% across all layers. * Values: Stays between ~40-80% from layer 0-30. * **A-Anchored (HotpotQA) - Orange Dotted:** Starts around 50% and remains relatively stable between 50% and 70% across all layers. * Values: Stays between ~50-70% from layer 0-30. * **Q-Anchored (NQ) - Red Dashed:** Starts around 40% and fluctuates between 40% and 90% across all layers. * Values: Fluctuates between ~40-90% from layer 0-30. * **A-Anchored (NQ) - Black Dotted:** Starts around 50% and fluctuates between 40% and 70% across all layers. * Values: Fluctuates between ~40-70% from layer 0-30. #### Mistral-7B-v0.3 (Right Graph) * **Q-Anchored (PopQA) - Blue:** Starts at 0% at layer 0, rises to 40% at layer 5, then fluctuates between 10% and 40% for the remaining layers. * Values: ~0% at layer 0, ~40% at layer 5, fluctuates between ~10-40% from layer 5-30. * **A-Anchored (PopQA) - Brown Dashed:** Starts around 65% and remains relatively stable between 60% and 80% across all layers. * Values: Stays between ~60-80% from layer 0-30. * **Q-Anchored (TriviaQA) - Green Dotted:** Starts at 60% at layer 0, rises to 100% at layer 5, then fluctuates between 40% and 60% for the remaining layers. * Values: ~60% at layer 0, ~100% at layer 5, fluctuates between ~40-60% from layer 5-30. * **A-Anchored (TriviaQA) - Purple Dashed:** Starts around 70% at layer 0, drops to 40% at layer 10, then fluctuates between 40% and 60% for the remaining layers. * Values: ~70% at layer 0, ~40% at layer 10, fluctuates between ~40-60% from layer 10-30. * **Q-Anchored (HotpotQA) - Gray Dashed:** Starts around 70% and remains relatively stable between 70% and 90% across all layers. * Values: Stays between ~70-90% from layer 0-30. * **A-Anchored (HotpotQA) - Orange Dotted:** Starts around 70% and remains relatively stable between 70% and 80% across all layers. * Values: Stays between ~70-80% from layer 0-30. * **Q-Anchored (NQ) - Red Dashed:** Starts around 70% and fluctuates between 70% and 90% across all layers. * Values: Fluctuates between ~70-90% from layer 0-30. * **A-Anchored (NQ) - Black Dotted:** Starts around 60% and fluctuates between 60% and 80% across all layers. * Values: Fluctuates between ~60-80% from layer 0-30. ### Key Observations * **Q-Anchored (PopQA):** Shows a significant drop in "I-Don't-Know" rate in v0.1, starting high and decreasing rapidly, while in v0.3, it starts low and remains relatively low. * **Overall Stability:** Most of the other QA tasks show relatively stable "I-Don't-Know" rates across layers in both versions, with some fluctuations. * **Version Comparison:** The "I-Don't-Know" rates for most tasks are generally higher in v0.3 compared to v0.1, suggesting a potential increase in uncertainty or a change in the model's confidence calibration. ### Interpretation The graphs provide insights into how the Mistral-7B model's uncertainty, as measured by the "I-Don't-Know" rate, varies across different layers and QA tasks. The most notable difference between versions v0.1 and v0.3 is the behavior of the Q-Anchored (PopQA) task, where v0.1 shows a significant decrease in the "I-Don't-Know" rate as the layers progress, while v0.3 maintains a low rate throughout. This suggests that the model's ability to handle PopQA questions has improved or changed significantly between the two versions. The relatively stable "I-Don't-Know" rates for other tasks indicate that the model's uncertainty is consistent across layers for those specific QA scenarios. The generally higher rates in v0.3 might reflect a deliberate recalibration of the model's confidence, potentially trading off some accuracy for increased awareness of its limitations. These findings are valuable for understanding the model's strengths and weaknesses in different QA tasks and for guiding further development and refinement of the Mistral-7B model. The differences between the two versions highlight the impact of specific training or architectural changes on the model's uncertainty and performance.

## Line Chart: I-Don't-Know Rate vs. Layer for Mistral Models ### Overview The image presents two line charts, side-by-side, comparing the "I-Don't-Know Rate" across different layers of two Mistral language models: Mistral-7B-v0.1 and Mistral-7B-v0.3. The x-axis represents the "Layer" (ranging from 0 to 30), and the y-axis represents the "I-Don't-Know Rate" (ranging from 0 to 100). Each chart displays multiple lines, each representing a different question-answering dataset and anchoring method. ### Components/Axes * **X-axis:** Layer (0 to 30) * **Y-axis:** I-Don't-Know Rate (0 to 100) * **Left Chart Title:** Mistral-7B-v0.1 * **Right Chart Title:** Mistral-7B-v0.3 * **Legend (Bottom):** * Q-Anchored (PopQA) - Blue solid line * A-Anchored (PopQA) - Orange dashed line * Q-Anchored (TriviaQA) - Purple solid line * A-Anchored (TriviaQA) - Green dashed line * Q-Anchored (HotpotQA) - Brown dashed line * A-Anchored (HotpotQA) - Red dashed line * Q-Anchored (NQ) - Light Blue solid line * A-Anchored (NQ) - Grey solid line ### Detailed Analysis or Content Details **Mistral-7B-v0.1 (Left Chart):** * **Q-Anchored (PopQA):** Starts at approximately 95, dips to around 20 at layer 8, then fluctuates between 40 and 80 until layer 30, ending around 60. * **A-Anchored (PopQA):** Starts at approximately 85, gradually decreases to around 50 at layer 10, then fluctuates between 50 and 75 until layer 30, ending around 65. * **Q-Anchored (TriviaQA):** Starts at approximately 90, decreases to around 50 at layer 10, then fluctuates between 50 and 80 until layer 30, ending around 70. * **A-Anchored (TriviaQA):** Starts at approximately 80, decreases to around 40 at layer 10, then fluctuates between 40 and 60 until layer 30, ending around 55. * **Q-Anchored (HotpotQA):** Starts at approximately 95, decreases to around 40 at layer 10, then fluctuates between 40 and 70 until layer 30, ending around 60. * **A-Anchored (HotpotQA):** Starts at approximately 90, decreases to around 50 at layer 10, then fluctuates between 50 and 80 until layer 30, ending around 75. * **Q-Anchored (NQ):** Starts at approximately 95, dips to around 20 at layer 8, then fluctuates between 40 and 70 until layer 30, ending around 60. * **A-Anchored (NQ):** Starts at approximately 85, decreases to around 40 at layer 10, then fluctuates between 40 and 60 until layer 30, ending around 50. **Mistral-7B-v0.3 (Right Chart):** * **Q-Anchored (PopQA):** Starts at approximately 95, dips to around 30 at layer 8, then fluctuates between 30 and 60 until layer 30, ending around 50. * **A-Anchored (PopQA):** Starts at approximately 85, gradually decreases to around 40 at layer 10, then fluctuates between 40 and 60 until layer 30, ending around 55. * **Q-Anchored (TriviaQA):** Starts at approximately 90, decreases to around 40 at layer 10, then fluctuates between 40 and 60 until layer 30, ending around 50. * **A-Anchored (TriviaQA):** Starts at approximately 80, decreases to around 30 at layer 10, then fluctuates between 30 and 50 until layer 30, ending around 45. * **Q-Anchored (HotpotQA):** Starts at approximately 95, decreases to around 40 at layer 10, then fluctuates between 40 and 60 until layer 30, ending around 50. * **A-Anchored (HotpotQA):** Starts at approximately 90, decreases to around 50 at layer 10, then fluctuates between 50 and 70 until layer 30, ending around 65. * **Q-Anchored (NQ):** Starts at approximately 95, dips to around 30 at layer 8, then fluctuates between 30 and 50 until layer 30, ending around 40. * **A-Anchored (NQ):** Starts at approximately 85, decreases to around 40 at layer 10, then fluctuates between 40 and 50 until layer 30, ending around 45. ### Key Observations * All lines in both charts start with high "I-Don't-Know Rates" (around 80-95) at layer 0. * There's a general decreasing trend in "I-Don't-Know Rate" up to around layer 10 for most datasets and anchoring methods. * After layer 10, the rates fluctuate, but generally remain between 40 and 80. * The Mistral-7B-v0.3 model consistently exhibits lower "I-Don't-Know Rates" compared to the Mistral-7B-v0.1 model across most datasets and anchoring methods. * Q-Anchored methods generally have higher "I-Don't-Know Rates" than A-Anchored methods for the same dataset. ### Interpretation The charts demonstrate how the "I-Don't-Know Rate" changes as information propagates through the layers of the Mistral language models. The initial high rate suggests the model has limited initial knowledge. The decrease up to layer 10 indicates that the model learns and gains confidence as it processes information. The subsequent fluctuations likely represent the model encountering more complex or ambiguous information. The consistent lower "I-Don't-Know Rates" in Mistral-7B-v0.3 suggest that this version of the model is more robust and has a better understanding of the datasets used for evaluation. The difference between Q-Anchored and A-Anchored methods suggests that the way questions are anchored (using the question itself vs. the answer) impacts the model's confidence. The fact that Q-Anchored methods generally have higher rates could indicate that the model finds it more challenging to reason directly from the question. The data suggests that model improvements (v0.3 over v0.1) lead to a reduction in uncertainty (lower I-Don't-Know Rate) across different knowledge domains (PopQA, TriviaQA, HotpotQA, NQ). This is a positive indicator of model performance and generalization ability.

## Line Charts: I-Don't-Know Rate Across Model Layers ### Overview The image displays two side-by-side line charts comparing the "I-Don't-Know Rate" across the layers (0-30) of two versions of the Mistral-7B language model: v0.1 (left) and v0.3 (right). Each chart plots eight data series, representing two different prompting methods ("Q-Anchored" and "A-Anchored") applied to four distinct question-answering datasets (PopQA, TriviaQA, HotpotQA, NQ). The charts illustrate how the model's expressed uncertainty (its rate of producing an "I don't know" response) changes as information propagates through its internal layers. ### Components/Axes * **Chart Titles:** "Mistral-7B-v0.1" (left chart), "Mistral-7B-v0.3" (right chart). * **Y-Axis (Both Charts):** Label: "I-Don't-Know Rate". Scale: 0 to 100, with major tick marks at intervals of 20 (0, 20, 40, 60, 80, 100). * **X-Axis (Both Charts):** Label: "Layer". Scale: 0 to 30, with major tick marks at intervals of 10 (0, 10, 20, 30). * **Legend (Bottom Center, spanning both charts):** Contains eight entries, each with a unique line color and style. * **Q-Anchored Series (Solid Lines):** * `Q-Anchored (PopQA)`: Solid blue line. * `Q-Anchored (TriviaQA)`: Solid green line. * `Q-Anchored (HotpotQA)`: Solid purple line. * `Q-Anchored (NQ)`: Solid pink/red line. * **A-Anchored Series (Dashed Lines):** * `A-Anchored (PopQA)`: Dashed orange line. * `A-Anchored (TriviaQA)`: Dashed red line. * `A-Anchored (HotpotQA)`: Dashed gray line. * `A-Anchored (NQ)`: Dashed brown line. * **Grid:** Light gray grid lines are present in the background of both charts. ### Detailed Analysis **Mistral-7B-v0.1 (Left Chart):** * **General Trend:** All series show high variability and fluctuation across layers. There is no single, smooth monotonic trend for any series. * **Q-Anchored Series (Solid Lines):** These lines generally start at a high rate (between ~60-100) in the early layers (0-5). They exhibit a sharp dip or valley between layers 5-10, often dropping below 40. After layer 10, they enter a phase of high-amplitude oscillation, with values swinging between approximately 10 and 90 through layer 30. The blue line (PopQA) and green line (TriviaQA) show particularly deep troughs near layer 10. * **A-Anchored Series (Dashed Lines):** These lines start at a moderate level (between ~40-60) in the early layers. They show a more gradual, undulating pattern compared to the Q-Anchored lines. They generally rise to a peak between layers 15-25, with values often reaching 70-90, before showing a slight decline or stabilization towards layer 30. The dashed lines are generally less volatile than the solid lines in the later layers (20-30). **Mistral-7B-v0.3 (Right Chart):** * **General Trend:** The patterns are distinctly different from v0.1, showing more separation between the two method types (Q-Anchored vs. A-Anchored). * **Q-Anchored Series (Solid Lines):** These lines start very high (near 100) in the earliest layers (0-3). They then experience a dramatic and sustained decline. The blue line (PopQA) plummets to near 0 by layer 10 and remains very low (mostly below 20) for the rest of the layers. The other solid lines (green, purple, pink) also decline significantly but stabilize at a higher plateau, fluctuating roughly between 20 and 50 from layer 10 to 30. * **A-Anchored Series (Dashed Lines):** These lines start at a moderate level (~50-70) and show a general upward trend, peaking in the middle-to-late layers (15-25). They maintain high values (mostly between 60 and 90) throughout the second half of the network, showing less decline than their v0.1 counterparts. They are consistently higher than the Q-Anchored lines after approximately layer 8. ### Key Observations 1. **Version Comparison:** The most striking difference is the behavior of the Q-Anchored (solid) lines. In v0.3, they show a strong, sustained decrease in "I-Don't-Know Rate" after the initial layers, which is not present in v0.1. This is especially extreme for the PopQA dataset. 2. **Method Divergence:** In v0.3, a clear gap opens up between the two methods after layer ~8. The A-Anchored method maintains a high uncertainty rate, while the Q-Anchored method's uncertainty drops significantly. This separation is much less pronounced in v0.1. 3. **Dataset Sensitivity:** The PopQA dataset (blue/orange lines) shows the most extreme behavior in both charts, particularly the near-zero rate for Q-Anchored in v0.3. The other three datasets (TriviaQA, HotpotQA, NQ) follow more similar, grouped patterns within each method. 4. **Early Layer Behavior:** Both model versions show very high uncertainty (near 100) for Q-Anchored methods in the first few layers, suggesting the model initially lacks confidence regardless of version. ### Interpretation The data suggests a significant evolution in the internal processing of the Mistral-7B model between versions v0.1 and v0.3, specifically regarding how it handles uncertainty when prompted with different formats. * **Model Maturation:** The dramatic drop in "I-Don't-Know Rate" for Q-Anchored prompts in v0.3's deeper layers indicates that the updated model has become much more confident in its internal representations when the question is directly anchored. It appears to resolve uncertainty earlier in its processing stream (by layer 10) for this prompting style. * **Anchoring Method Impact:** The persistent high uncertainty for A-Anchored prompts in v0.3 suggests that anchoring the answer format may prevent the model from consolidating confidence in the same way. The model seems to retain a higher degree of expressed uncertainty throughout its layers when the answer is pre-specified. * **Dataset Characteristics:** The outlier behavior of PopQA, especially in v0.3, implies that the nature of the questions or answers in this dataset interacts uniquely with the model's knowledge and the anchoring mechanism, leading to near-complete elimination of "I don't know" responses for Q-Anchored prompts in later layers. * **Architectural Insight:** The charts provide a window into the "confidence calibration" across the model's depth. The transition from high early-layer uncertainty to lower later-layer uncertainty (for Q-Anchored in v0.3) mirrors the expected flow of information processing, where raw inputs are transformed into more confident internal states. The lack of this trend in v0.1 suggests a less refined internal confidence mechanism. **Language:** All text in the image is in English.

## Line Chart: I-Don't-Know Rate Across Layers for Mistral-7B Models (v0.1 and v0.3) ### Overview The image contains two line charts comparing the "I-Don't-Know Rate" (y-axis) across 30 layers (x-axis) for different question-answering models and anchoring methods in two versions of Mistral-7B (v0.1 and v0.3). Each chart includes multiple data series with distinct line styles and colors, representing combinations of anchoring types (Q-Anchored/A-Anchored) and datasets (PopQA, TriviaQA, HotpotQA, NQ). Confidence intervals are visualized as shaded regions around the lines. --- ### Components/Axes - **X-Axis**: Layer (0–30, integer increments) - **Y-Axis**: I-Don't-Know Rate (0–100%, integer increments) - **Legends**: - **Left Chart (v0.1)**: - Q-Anchored (PopQA): Solid blue - A-Anchored (PopQA): Dashed orange - Q-Anchored (TriviaQA): Dotted green - A-Anchored (TriviaQA): Dash-dot red - Q-Anchored (HotpotQA): Solid purple - A-Anchored (HotpotQA): Dashed gray - **Right Chart (v0.3)**: - Q-Anchored (PopQA): Solid blue - A-Anchored (PopQA): Dashed orange - Q-Anchored (TriviaQA): Dotted green - A-Anchored (TriviaQA): Dash-dot red - Q-Anchored (HotpotQA): Solid purple - A-Anchored (HotpotQA): Dashed gray - Q-Anchored (NQ): Dotted pink - A-Anchored (NQ): Dash-dot gray --- ### Detailed Analysis #### Left Chart (Mistral-7B-v0.1) - **Q-Anchored (PopQA)** (blue solid): Peaks at ~90% at layer 5, drops to ~40% at layer 15, then fluctuates between 50–70%. - **A-Anchored (PopQA)** (orange dashed): Stable between 40–60%, with minor dips at layers 10 and 25. - **Q-Anchored (TriviaQA)** (green dotted): Sharp spike to ~80% at layer 10, then declines to ~30% by layer 30. - **A-Anchored (TriviaQA)** (red dash-dot): Gradual decline from ~70% to ~40%, with a plateau at layer 20. - **Q-Anchored (HotpotQA)** (purple solid): Oscillates between 50–70%, with a peak at layer 25 (~80%). - **A-Anchored (HotpotQA)** (gray dashed): Relatively flat (~50–60%), with a dip to ~40% at layer 15. #### Right Chart (Mistral-7B-v0.3) - **Q-Anchored (PopQA)** (blue solid): Peaks at ~80% at layer 10, then declines to ~50% by layer 30. - **A-Anchored (PopQA)** (orange dashed): Stable between 50–70%, with a minor dip at layer 20. - **Q-Anchored (TriviaQA)** (green dotted): Peaks at ~70% at layer 5, declines to ~40% by layer 30. - **A-Anchored (TriviaQA)** (red dash-dot): Gradual decline from ~60% to ~30%, with a plateau at layer 15. - **Q-Anchored (HotpotQA)** (purple solid): Peaks at ~75% at layer 20, then declines to ~50%. - **A-Anchored (HotpotQA)** (gray dashed): Stable between 50–60%, with a dip to ~40% at layer 10. - **Q-Anchored (NQ)** (pink dotted): Peaks at ~85% at layer 5, declines to ~40% by layer 30. - **A-Anchored (NQ)** (gray dash-dot): Stable between 50–70%, with a peak at layer 25 (~80%). --- ### Key Observations 1. **Version Comparison**: - v0.3 shows reduced variability in I-Don't-Know rates compared to v0.1 (narrower shaded confidence intervals). - v0.1 exhibits sharper spikes (e.g., Q-Anchored TriviaQA at layer 10), while v0.3 trends are smoother. 2. **Anchoring Impact**: - Q-Anchored models generally show higher I-Don't-Know rates than A-Anchored counterparts in both versions. - Exceptions: A-Anchored (NQ) in v0.3 matches Q-Anchored (NQ) in variability. 3. **Dataset-Specific Trends**: - **PopQA**: Q-Anchored models dominate in v0.1 but stabilize in v0.3. - **TriviaQA**: Q-Anchored models exhibit extreme fluctuations in v0.1, mitigated in v0.3. - **HotpotQA**: Q-Anchored models show late-layer spikes in v0.1 (layer 25) and v0.3 (layer 20). 4. **Outliers**: - Q-Anchored (TriviaQA) in v0.1 has an anomalous spike at layer 10 (~80%), far exceeding other series. - A-Anchored (NQ) in v0.3 peaks at layer 25 (~80%), matching Q-Anchored (NQ) in v0.3. --- ### Interpretation The data suggests that anchoring methods (Q vs. A) and dataset types significantly influence model uncertainty. Q-Anchored models (e.g., PopQA, TriviaQA) exhibit higher I-Don't-Know rates, particularly in earlier layers, indicating potential over-reliance on specific training data. The reduction in variability in v0.3 implies architectural improvements or better generalization. The late-layer spikes in HotpotQA (v0.1/v0.3) may reflect domain-specific challenges. Notably, A-Anchored (NQ) in v0.3 performs comparably to Q-Anchored models, suggesting that anchoring strategy may be less critical for NQ datasets. These trends highlight trade-offs between specialization and robustness in model design.