## Chart: I-Don't-Know Rate vs. Layer for Llama-3 Models ### Overview The image presents two line charts comparing the "I-Don't-Know Rate" across different layers of two Llama-3 models (8B and 70B). Each chart displays the rate for various question-answering datasets (PopQA, TriviaQA, HotpotQA, and NQ) using both question-anchored (Q-Anchored) and answer-anchored (A-Anchored) approaches. The x-axis represents the layer number, and the y-axis represents the I-Don't-Know Rate. ### Components/Axes * **Titles:** * Left Chart: "Llama-3-8B" * Right Chart: "Llama-3-70B" * **Y-Axis:** * Label: "I-Don't-Know Rate" * Scale: 0 to 100, with tick marks at 0, 20, 40, 60, 80, and 100. * **X-Axis:** * Label: "Layer" * Left Chart Scale: 0 to 30, with tick marks at 0, 10, 20, and 30. * Right Chart Scale: 0 to 80, with tick marks at 0, 20, 40, 60, and 80. * **Legend:** Located at the bottom of the image. * Q-Anchored (PopQA): Solid blue line * A-Anchored (PopQA): Dashed brown line * Q-Anchored (TriviaQA): Dotted green line * A-Anchored (TriviaQA): Dash-dot gray line * Q-Anchored (HotpotQA): Dash-dot-dot red line * A-Anchored (HotpotQA): Dotted orange line * Q-Anchored (NQ): Dashed pink line * A-Anchored (NQ): Dash-dot black line ### Detailed Analysis **Llama-3-8B (Left Chart):** * **Q-Anchored (PopQA):** (Solid Blue) Starts at approximately 0, rises sharply to around 90 by layer 5, then fluctuates between 10 and 40 for the remaining layers. * **A-Anchored (PopQA):** (Dashed Brown) Starts at approximately 40, rises to around 60 by layer 10, and then fluctuates between 50 and 70 for the remaining layers. * **Q-Anchored (TriviaQA):** (Dotted Green) Starts at approximately 50, drops to around 10 by layer 10, and then fluctuates between 10 and 30 for the remaining layers. * **A-Anchored (TriviaQA):** (Dash-dot Gray) Starts at approximately 50, rises to around 60 by layer 10, and then fluctuates between 50 and 60 for the remaining layers. * **Q-Anchored (HotpotQA):** (Dash-dot-dot Red) Starts at approximately 40, rises to around 90 by layer 10, and then fluctuates between 70 and 90 for the remaining layers. * **A-Anchored (HotpotQA):** (Dotted Orange) Starts at approximately 40, rises to around 70 by layer 10, and then fluctuates between 60 and 70 for the remaining layers. * **Q-Anchored (NQ):** (Dashed Pink) Starts at approximately 40, rises to around 60 by layer 10, and then fluctuates between 20 and 40 for the remaining layers. * **A-Anchored (NQ):** (Dash-dot Black) Starts at approximately 50, rises to around 60 by layer 10, and then fluctuates between 50 and 60 for the remaining layers. **Llama-3-70B (Right Chart):** * **Q-Anchored (PopQA):** (Solid Blue) Starts at approximately 20, fluctuates between 10 and 40 across all layers. * **A-Anchored (PopQA):** (Dashed Brown) Starts at approximately 60, fluctuates between 70 and 90 across all layers. * **Q-Anchored (TriviaQA):** (Dotted Green) Starts at approximately 40, fluctuates between 10 and 30 across all layers. * **A-Anchored (TriviaQA):** (Dash-dot Gray) Starts at approximately 60, fluctuates between 60 and 80 across all layers. * **Q-Anchored (HotpotQA):** (Dash-dot-dot Red) Starts at approximately 60, fluctuates between 70 and 90 across all layers. * **A-Anchored (HotpotQA):** (Dotted Orange) Starts at approximately 60, fluctuates between 70 and 90 across all layers. * **Q-Anchored (NQ):** (Dashed Pink) Starts at approximately 40, fluctuates between 20 and 50 across all layers. * **A-Anchored (NQ):** (Dash-dot Black) Starts at approximately 60, fluctuates between 60 and 80 across all layers. ### Key Observations * The I-Don't-Know Rate varies significantly depending on the dataset and anchoring method (Q-Anchored vs. A-Anchored). * The Llama-3-70B model shows more consistent I-Don't-Know Rates across layers compared to the Llama-3-8B model, which exhibits more pronounced initial changes in the first 10 layers. * For both models, A-Anchored approaches generally result in higher I-Don't-Know Rates than Q-Anchored approaches for PopQA, TriviaQA, and NQ datasets. * HotpotQA shows high I-Don't-Know Rates for both Q-Anchored and A-Anchored approaches in both models. ### Interpretation The charts illustrate how the "I-Don't-Know Rate" changes across different layers of the Llama-3 models when processing various question-answering datasets. The differences between Q-Anchored and A-Anchored approaches suggest that the way questions and answers are processed significantly impacts the model's confidence in its responses. The higher I-Don't-Know Rates for HotpotQA may indicate that this dataset poses a greater challenge for the models, possibly due to its complexity or the type of reasoning required. The more stable rates in the Llama-3-70B model suggest that larger models might have more consistent performance across layers. The initial fluctuations in the Llama-3-8B model could indicate that the earlier layers are more critical for learning and adapting to the specific dataset.

## Chart: I-Don't-Know Rate vs. Layer for Llama-3 Models ### Overview The image presents two line charts comparing the "I-Don't-Know Rate" across different layers of two Llama-3 models: Llama-3-8B and Llama-3-70B. The charts display the rate for different question-answering datasets (PopQA, TriviaQA, HotpotQA, and NQ) and anchoring methods (Q-Anchored and A-Anchored). ### Components/Axes * **X-axis:** "Layer" - ranging from 0 to 30 for Llama-3-8B and 0 to 80 for Llama-3-70B. * **Y-axis:** "I-Don't-Know Rate" - ranging from 0 to 100. * **Models:** Two separate charts, one for "Llama-3-8B" (left) and one for "Llama-3-70B" (right). * **Datasets/Anchoring:** The legend at the bottom identifies eight data series: * Q-Anchored (PopQA) - Blue line * A-Anchored (PopQA) - Orange line * Q-Anchored (TriviaQA) - Purple line * A-Anchored (TriviaQA) - Brown line * Q-Anchored (HotpotQA) - Light Blue dashed line * A-Anchored (HotpotQA) - Grey dashed line * Q-Anchored (NQ) - Green line * A-Anchored (NQ) - Red line ### Detailed Analysis or Content Details **Llama-3-8B (Left Chart)** * **Q-Anchored (PopQA):** Starts at approximately 95, rapidly decreases to around 15 by layer 10, then fluctuates between 10 and 25 until layer 30. * **A-Anchored (PopQA):** Starts at approximately 90, decreases to around 60 by layer 10, and remains relatively stable between 55 and 70 until layer 30. * **Q-Anchored (TriviaQA):** Starts at approximately 90, decreases to around 20 by layer 10, then fluctuates between 15 and 30 until layer 30. * **A-Anchored (TriviaQA):** Starts at approximately 85, decreases to around 65 by layer 10, and remains relatively stable between 60 and 75 until layer 30. * **Q-Anchored (HotpotQA):** Starts at approximately 95, decreases to around 30 by layer 10, then fluctuates between 20 and 40 until layer 30. * **A-Anchored (HotpotQA):** Starts at approximately 90, decreases to around 50 by layer 10, and remains relatively stable between 45 and 60 until layer 30. * **Q-Anchored (NQ):** Starts at approximately 95, decreases to around 10 by layer 10, then fluctuates between 5 and 20 until layer 30. * **A-Anchored (NQ):** Starts at approximately 90, decreases to around 60 by layer 10, and remains relatively stable between 55 and 70 until layer 30. **Llama-3-70B (Right Chart)** * **Q-Anchored (PopQA):** Starts at approximately 95, decreases to around 20 by layer 20, then fluctuates between 20 and 40 until layer 80. * **A-Anchored (PopQA):** Starts at approximately 90, decreases to around 60 by layer 20, and remains relatively stable between 55 and 70 until layer 80. * **Q-Anchored (TriviaQA):** Starts at approximately 90, decreases to around 25 by layer 20, then fluctuates between 25 and 40 until layer 80. * **A-Anchored (TriviaQA):** Starts at approximately 85, decreases to around 65 by layer 20, and remains relatively stable between 60 and 75 until layer 80. * **Q-Anchored (HotpotQA):** Starts at approximately 95, decreases to around 35 by layer 20, then fluctuates between 30 and 50 until layer 80. * **A-Anchored (HotpotQA):** Starts at approximately 90, decreases to around 55 by layer 20, and remains relatively stable between 50 and 65 until layer 80. * **Q-Anchored (NQ):** Starts at approximately 95, decreases to around 15 by layer 20, then fluctuates between 10 and 30 until layer 80. * **A-Anchored (NQ):** Starts at approximately 90, decreases to around 60 by layer 20, and remains relatively stable between 55 and 70 until layer 80. ### Key Observations * All data series exhibit a significant decrease in "I-Don't-Know Rate" in the initial layers (0-10). * The Q-Anchored series generally have lower "I-Don't-Know Rates" than the A-Anchored series across all datasets. * The "I-Don't-Know Rate" stabilizes after approximately layer 20 for Llama-3-8B and layer 20 for Llama-3-70B. * The 70B model generally exhibits a lower "I-Don't-Know Rate" than the 8B model, especially after the initial decrease. * PopQA and NQ datasets show the most significant reduction in "I-Don't-Know Rate" with increasing layers. ### Interpretation The charts demonstrate that as the model layers increase, the "I-Don't-Know Rate" decreases, indicating that the model becomes more confident in its answers. The difference between Q-Anchored and A-Anchored suggests that using question-based anchoring leads to lower uncertainty compared to answer-based anchoring. The larger model (70B) consistently outperforms the smaller model (8B), suggesting that increased model size improves knowledge retention and reduces uncertainty. The stabilization of the "I-Don't-Know Rate" after a certain number of layers indicates that adding more layers beyond that point may not significantly improve performance. The varying rates across datasets suggest that the model's performance is influenced by the complexity and nature of the questions in each dataset. The initial high rates suggest the model starts with limited knowledge and learns as it processes more layers. The consistent trend across both models suggests a general pattern in how these Llama-3 models learn and respond to different types of questions.

## Line Charts: Llama-3 Model "I-Don't-Know Rate" Across Layers ### Overview The image displays two side-by-side line charts comparing the "I-Don't-Know Rate" across the layers of two different-sized language models: Llama-3-8B (left) and Llama-3-70B (right). Each chart plots the performance of eight different experimental configurations, distinguished by anchoring method (Q-Anchored vs. A-Anchored) and evaluation dataset (PopQA, TriviaQA, HotpotQA, NQ). The charts visualize how the model's propensity to output an "I don't know" response changes as information propagates through its internal layers. ### Components/Axes * **Chart Titles:** * Left Chart: `Llama-3-8B` * Right Chart: `Llama-3-70B` * **Y-Axis (Both Charts):** * Label: `I-Don't-Know Rate` * Scale: 0 to 100, with major tick marks at 0, 20, 40, 60, 80, 100. * **X-Axis:** * Label: `Layer` * Left Chart Scale: 0 to 30, with major tick marks at 0, 10, 20, 30. * Right Chart Scale: 0 to 80, with major tick marks at 0, 20, 40, 60, 80. * **Legend (Bottom of Image, spanning both charts):** * Contains 8 entries, each with a line sample and text label. * **Q-Anchored (Solid Lines):** * Blue solid line: `Q-Anchored (PopQA)` * Green solid line: `Q-Anchored (TriviaQA)` * Purple solid line: `Q-Anchored (HotpotQA)` * Pink solid line: `Q-Anchored (NQ)` * **A-Anchored (Dashed Lines):** * Orange dashed line: `A-Anchored (PopQA)` * Red dashed line: `A-Anchored (TriviaQA)` * Brown dashed line: `A-Anchored (HotpotQA)` * Gray dashed line: `A-Anchored (NQ)` ### Detailed Analysis **Llama-3-8B Chart (Left):** * **Q-Anchored Lines (Solid):** These lines generally show a **downward trend** in the early layers (0-10), indicating a decreasing "I-Don't-Know Rate." After layer 10, they exhibit significant volatility but tend to stabilize at lower values (mostly between 10-40) compared to their starting points. * `Q-Anchored (PopQA)` (Blue): Starts very high (~90), drops sharply to ~10 by layer 10, then fluctuates between ~10-40. * `Q-Anchored (TriviaQA)` (Green): Starts high (~80), drops to near 0 by layer 10, then fluctuates at a very low level (0-20). * `Q-Anchored (HotpotQA)` (Purple): Starts high (~85), drops to ~20 by layer 10, then shows high volatility between ~10-50. * `Q-Anchored (NQ)` (Pink): Starts moderately high (~60), drops to ~20 by layer 10, then fluctuates between ~10-40. * **A-Anchored Lines (Dashed):** These lines show a general **upward trend** across layers, indicating an increasing "I-Don't-Know Rate." * `A-Anchored (PopQA)` (Orange): Starts around 40, rises steadily to ~70 by layer 30. * `A-Anchored (TriviaQA)` (Red): Starts around 40, rises to the highest level among all lines, reaching ~80 by layer 30. * `A-Anchored (HotpotQA)` (Brown): Starts around 40, rises to ~60 by layer 30. * `A-Anchored (NQ)` (Gray): Starts around 40, rises to ~60 by layer 30. **Llama-3-70B Chart (Right):** * **Q-Anchored Lines (Solid):** Similar to the 8B model, these lines show an initial **downward trend** but with more pronounced and sustained volatility across the deeper layers (0-80). * `Q-Anchored (PopQA)` (Blue): Starts very high (~95), drops steeply to ~10 by layer 20, then fluctuates widely between ~5-40. * `Q-Anchored (TriviaQA)` (Green): Starts high (~85), drops to near 0 by layer 20, then remains very low (0-10) with minor fluctuations. * `Q-Anchored (HotpotQA)` (Purple): Starts high (~90), drops to ~20 by layer 20, then exhibits extreme volatility between ~5-60. * `Q-Anchored (NQ)` (Pink): Starts moderately high (~70), drops to ~20 by layer 20, then fluctuates between ~10-50. * **A-Anchored Lines (Dashed):** These lines also show a general **upward trend**, but they reach higher peaks and exhibit more noise compared to the 8B model. * `A-Anchored (PopQA)` (Orange): Starts around 40, rises with high volatility to a peak near 90 around layer 60. * `A-Anchored (TriviaQA)` (Red): Starts around 40, rises to the highest sustained levels, fluctuating between 70-90 from layer 40 onward. * `A-Anchored (HotpotQA)` (Brown): Starts around 40, rises to fluctuate between 60-80 from layer 40 onward. * `A-Anchored (NQ)` (Gray): Starts around 40, rises to fluctuate between 50-70 from layer 40 onward. ### Key Observations 1. **Divergent Anchoring Effects:** There is a stark and consistent contrast between anchoring methods. **Q-Anchored** methods lead to a *decrease* in the "I-Don't-Know Rate" through the layers, while **A-Anchored** methods lead to an *increase*. 2. **Model Size Impact:** The larger Llama-3-70B model shows more extreme values (both higher peaks for A-Anchored and lower troughs for Q-Anchored) and significantly greater volatility in its layer-wise responses compared to the 8B model. 3. **Dataset Sensitivity:** The effect magnitude varies by dataset. For Q-Anchored methods, `TriviaQA` (green) consistently results in the lowest "I-Don't-Know Rate." For A-Anchored methods, `TriviaQA` (red) and `PopQA` (orange) often result in the highest rates. 4. **Early Layer Convergence:** For Q-Anchored methods, the most dramatic change occurs in the first 10-20 layers, after which the rate stabilizes or fluctuates around a new, lower baseline. ### Interpretation This data suggests a fundamental difference in how information is processed under the two anchoring paradigms. The **Q-Anchored** approach appears to progressively build confidence or extract answer-related information as data moves through the network layers, reducing uncertainty. Conversely, the **A-Anchored** approach seems to amplify uncertainty or "forget" initial priors, leading to a higher likelihood of a non-answer response in deeper layers. The increased volatility in the 70B model indicates that larger models may have more specialized or unstable internal representations across layers for these tasks. The consistent dataset-specific patterns (e.g., TriviaQA being easiest for Q-Anchored) imply that the underlying nature of the knowledge or question format in each dataset interacts predictably with the model's architecture and the anchoring method. From a technical document perspective, these charts provide strong evidence that the choice of anchoring method (Q vs. A) is a critical hyperparameter that dramatically influences model behavior and calibration (as measured by the "I-Don't-Know" rate) across its depth, with effects that scale with model size.

## Line Graph: I-Don't-Know Rate Across Llama-3 Model Sizes and Anchoring Methods ### Overview The image contains two line graphs comparing the "I-Don't-Know Rate" (percentage of unanswered questions) across layers of two Llama-3 language models: Llama-3-8B (left) and Llama-3-70B (right). Each graph shows six data series representing different question datasets (PopQA, TriviaQA, HotpotQA, NQ) and anchoring methods (Q-Anchored vs. A-Anchored). The graphs reveal layer-dependent performance variations, with notable fluctuations in higher layers for the 70B model. ### Components/Axes - **X-axis**: Layer (0–30 for Llama-3-8B, 0–80 for Llama-3-70B) - **Y-axis**: I-Don't-Know Rate (%) (0–100) - **Legend**: - Solid lines: Q-Anchored (PopQA, TriviaQA, HotpotQA, NQ) - Dashed lines: A-Anchored (PopQA, TriviaQA, HotpotQA, NQ) - **Color coding**: - Blue: PopQA - Green: TriviaQA - Purple: HotpotQA - Red: NQ ### Detailed Analysis #### Llama-3-8B (Left Chart) - **Q-Anchored (PopQA)**: Starts at ~90% at layer 0, drops sharply to ~40% by layer 10, then fluctuates between 30–60%. - **A-Anchored (PopQA)**: Begins at ~40%, rises to ~60% by layer 10, then stabilizes near 50–70%. - **Q-Anchored (TriviaQA)**: Peaks at ~80% at layer 0, declines to ~30% by layer 20, with erratic mid-range fluctuations. - **A-Anchored (TriviaQA)**: Starts at ~50%, dips to ~20% by layer 10, then rises to ~60% by layer 30. - **Q-Anchored (HotpotQA)**: Begins at ~70%, drops to ~20% by layer 10, then oscillates between 10–50%. - **A-Anchored (HotpotQA)**: Starts at ~30%, rises to ~50% by layer 10, then stabilizes near 40–60%. - **Q-Anchored (NQ)**: Peaks at ~85% at layer 0, declines to ~30% by layer 20, with sharp mid-layer dips. - **A-Anchored (NQ)**: Starts at ~40%, rises to ~70% by layer 10, then fluctuates between 50–80%. #### Llama-3-70B (Right Chart) - **Q-Anchored (PopQA)**: Starts at ~80%, drops to ~30% by layer 20, then fluctuates between 20–60%. - **A-Anchored (PopQA)**: Begins at ~50%, rises to ~70% by layer 40, then stabilizes near 60–80%. - **Q-Anchored (TriviaQA)**: Peaks at ~90% at layer 0, declines to ~20% by layer 60, with erratic mid-range fluctuations. - **A-Anchored (TriviaQA)**: Starts at ~40%, dips to ~10% by layer 20, then rises to ~70% by layer 80. - **Q-Anchored (HotpotQA)**: Begins at ~60%, drops to ~10% by layer 40, then oscillates between 5–50%. - **A-Anchored (HotpotQA)**: Starts at ~20%, rises to ~50% by layer 40, then stabilizes near 40–60%. - **Q-Anchored (NQ)**: Peaks at ~95% at layer 0, declines to ~20% by layer 80, with sharp mid-layer dips. - **A-Anchored (NQ)**: Starts at ~30%, rises to ~80% by layer 60, then fluctuates between 60–90%. ### Key Observations 1. **Model Size Impact**: The 70B model exhibits more pronounced fluctuations in higher layers (e.g., layer 60–80) compared to the 8B model. 2. **Anchoring Method Differences**: - Q-Anchored methods generally show higher initial I-Don't-Know rates but sharper declines. - A-Anchored methods maintain more stable or increasing rates in later layers. 3. **Dataset Variability**: - NQ (Natural Questions) consistently shows the highest initial I-Don't-Know rates. - HotpotQA (HotpotQA) demonstrates the most erratic behavior in the 70B model. 4. **Layer-Specific Trends**: - In Llama-3-8B, layer 10–20 shows critical performance shifts for most datasets. - In Llama-3-70B, layer 40–60 exhibits significant divergence between anchoring methods. ### Interpretation The data suggests that anchoring methods (Q vs. A) differentially affect model performance across layers and model sizes. Q-Anchored methods may prioritize early-layer accuracy at the cost of later-layer robustness, while A-Anchored methods appear more consistent in higher layers. The 70B model’s increased volatility in later layers could indicate greater sensitivity to architectural complexity or dataset-specific challenges. Notably, the NQ dataset’s extreme initial I-Don't-Know rates (up to 95%) highlight its role as a particularly challenging benchmark. These trends may reflect trade-offs between model capacity, question complexity, and anchoring strategy design.