## Line Charts: Performance Comparison of Llama Models ### Overview The image presents two line charts comparing the performance of Llama models (Llama-3-8B and Llama-3-70B) across different layers. The y-axis represents ΔP (change in performance), and the x-axis represents the layer number. Each chart displays multiple data series, distinguished by line style and color, representing different question-answering datasets and anchoring methods (Q-Anchored and A-Anchored). ### Components/Axes * **Titles:** * Left Chart: "Llama-3-8B" * Right Chart: "Llama-3-70B" * **X-Axis:** * Label: "Layer" * Left Chart: Scale ranges from 0 to 30, with tick marks at approximately 0, 10, 20, and 30. * Right Chart: Scale ranges from 0 to 80, with tick marks at approximately 0, 20, 40, 60, and 80. * **Y-Axis:** * Label: "ΔP" * Scale ranges from -80 to 0, with tick marks at -80, -60, -40, -20, and 0. * **Legend:** Located at the bottom of the image. * Q-Anchored (PopQA): Solid blue line * A-Anchored (PopQA): Dashed brown line * Q-Anchored (TriviaQA): Dotted green line * A-Anchored (TriviaQA): Dash-dot brown line * Q-Anchored (HotpotQA): Dash-dot-dot red line * A-Anchored (HotpotQA): Dotted brown line * Q-Anchored (NQ): Dotted pink line * A-Anchored (NQ): Dotted gray line ### Detailed Analysis **Left Chart (Llama-3-8B):** * **Q-Anchored (PopQA):** (Solid blue line) Starts near 0 and decreases to approximately -75 by layer 30. * **A-Anchored (PopQA):** (Dashed brown line) Remains relatively stable around 0 throughout all layers. * **Q-Anchored (TriviaQA):** (Dotted green line) Starts near 0 and decreases to approximately -65 by layer 30. * **A-Anchored (TriviaQA):** (Dash-dot brown line) Remains relatively stable around 0 throughout all layers. * **Q-Anchored (HotpotQA):** (Dash-dot-dot red line) Remains relatively stable around 0 throughout all layers. * **A-Anchored (HotpotQA):** (Dotted brown line) Remains relatively stable around 0 throughout all layers. * **Q-Anchored (NQ):** (Dotted pink line) Starts near 0 and decreases to approximately -30 by layer 30. * **A-Anchored (NQ):** (Dotted gray line) Remains relatively stable around 0 throughout all layers. **Right Chart (Llama-3-70B):** * **Q-Anchored (PopQA):** (Solid blue line) Starts near 0 and decreases to approximately -80 by layer 80. * **A-Anchored (PopQA):** (Dashed brown line) Remains relatively stable around 0 throughout all layers. * **Q-Anchored (TriviaQA):** (Dotted green line) Starts near 0 and decreases to approximately -70 by layer 80. * **A-Anchored (TriviaQA):** (Dash-dot brown line) Remains relatively stable around 0 throughout all layers. * **Q-Anchored (HotpotQA):** (Dash-dot-dot red line) Remains relatively stable around 0 throughout all layers. * **A-Anchored (HotpotQA):** (Dotted brown line) Remains relatively stable around 0 throughout all layers. * **Q-Anchored (NQ):** (Dotted pink line) Starts near 0 and decreases to approximately -30 by layer 80. * **A-Anchored (NQ):** (Dotted gray line) Remains relatively stable around 0 throughout all layers. ### Key Observations * The "Q-Anchored" series (PopQA, TriviaQA, and NQ) show a significant decrease in ΔP as the layer number increases for both Llama models. * The "A-Anchored" series (PopQA, TriviaQA, HotpotQA, and NQ) remain relatively stable around 0 across all layers for both Llama models. * The Llama-3-70B model has a larger layer range (0-80) compared to the Llama-3-8B model (0-30). * The Q-Anchored (HotpotQA) series remains stable around 0 for both models. ### Interpretation The data suggests that anchoring the question (Q-Anchored) leads to a decrease in performance (ΔP) as the model processes deeper layers, particularly for the PopQA, TriviaQA, and NQ datasets. This could indicate that the model's ability to answer questions from these datasets degrades with increasing layer depth when the question is anchored. Conversely, anchoring the answer (A-Anchored) results in stable performance across all layers, suggesting that the model maintains its ability to answer questions when the answer is anchored. The HotpotQA dataset shows stable performance regardless of whether the question or answer is anchored. The difference in layer range between the two models (8B vs 70B) highlights the larger processing capacity of the 70B model.

\n ## Line Chart: ΔP vs. Layer for Llama-3 Models ### Overview The image presents two line charts comparing the change in performance (ΔP) across different layers of two Llama-3 models: Llama-3-8B and Llama-3-70B. The charts display ΔP as a function of layer number, with different lines representing different question-answering datasets and anchoring methods. ### Components/Axes * **X-axis:** Layer (ranging from 0 to 30 for Llama-3-8B and 0 to 80 for Llama-3-70B). * **Y-axis:** ΔP (ranging from approximately -80 to 0). * **Models:** Llama-3-8B (left chart), Llama-3-70B (right chart). * **Datasets/Anchoring Methods (Legend):** * Q-Anchored (PopQA) - Blue line * A-Anchored (PopQA) - Light Orange dashed line * Q-Anchored (TriviaQA) - Green line * A-Anchored (TriviaQA) - Purple dashed line * Q-Anchored (HotpotQA) - Light Blue line * A-Anchored (HotpotQA) - Yellow dashed line * Q-Anchored (NQ) - Teal line * A-Anchored (NQ) - Red dashed line * **Legend Position:** Bottom-center of each chart. ### Detailed Analysis or Content Details **Llama-3-8B (Left Chart):** * **Q-Anchored (PopQA):** Starts at approximately 0, rapidly declines to around -60 by layer 10, then plateaus around -60 to -70 from layer 15 to 30. * **A-Anchored (PopQA):** Starts at approximately 0, declines more gradually to around -20 by layer 10, then plateaus around -20 to -30 from layer 15 to 30. * **Q-Anchored (TriviaQA):** Starts at approximately 0, declines rapidly to around -50 by layer 10, then plateaus around -50 to -60 from layer 15 to 30. * **A-Anchored (TriviaQA):** Starts at approximately 0, declines more gradually to around -30 by layer 10, then plateaus around -30 to -40 from layer 15 to 30. * **Q-Anchored (HotpotQA):** Starts at approximately 0, declines rapidly to around -60 by layer 10, then plateaus around -60 to -70 from layer 15 to 30. * **A-Anchored (HotpotQA):** Starts at approximately 0, declines more gradually to around -20 by layer 10, then plateaus around -20 to -30 from layer 15 to 30. * **Q-Anchored (NQ):** Starts at approximately 0, declines rapidly to around -50 by layer 10, then plateaus around -50 to -60 from layer 15 to 30. * **A-Anchored (NQ):** Starts at approximately 0, declines more gradually to around -30 by layer 10, then plateaus around -30 to -40 from layer 15 to 30. **Llama-3-70B (Right Chart):** * **Q-Anchored (PopQA):** Starts at approximately 0, rapidly declines to around -60 by layer 20, then plateaus around -60 to -70 from layer 40 to 80. * **A-Anchored (PopQA):** Starts at approximately 0, declines more gradually to around -20 by layer 20, then plateaus around -20 to -30 from layer 40 to 80. * **Q-Anchored (TriviaQA):** Starts at approximately 0, declines rapidly to around -50 by layer 20, then plateaus around -50 to -60 from layer 40 to 80. * **A-Anchored (TriviaQA):** Starts at approximately 0, declines more gradually to around -30 by layer 20, then plateaus around -30 to -40 from layer 40 to 80. * **Q-Anchored (HotpotQA):** Starts at approximately 0, declines rapidly to around -60 by layer 20, then plateaus around -60 to -70 from layer 40 to 80. * **A-Anchored (HotpotQA):** Starts at approximately 0, declines more gradually to around -20 by layer 20, then plateaus around -20 to -30 from layer 40 to 80. * **Q-Anchored (NQ):** Starts at approximately 0, declines rapidly to around -50 by layer 20, then plateaus around -50 to -60 from layer 40 to 80. * **A-Anchored (NQ):** Starts at approximately 0, declines more gradually to around -30 by layer 20, then plateaus around -30 to -40 from layer 40 to 80. ### Key Observations * In both models, Q-Anchored methods consistently show a larger drop in ΔP compared to A-Anchored methods. * The decline in ΔP appears to stabilize after a certain layer number (around 15-20 for the 8B model and 40 for the 70B model). * The 70B model exhibits a slower initial decline in ΔP compared to the 8B model, but the overall magnitude of the decline is similar. * PopQA and HotpotQA datasets show the most significant drops in ΔP for Q-Anchored methods. ### Interpretation The charts demonstrate how performance changes across layers in the Llama-3 models when evaluated on different question-answering datasets using different anchoring methods. The negative ΔP values indicate a decrease in performance as the layer number increases. The consistent difference between Q-Anchored and A-Anchored methods suggests that the method used to anchor the questions or answers significantly impacts performance, with Q-Anchoring generally leading to a more substantial performance drop. The stabilization of ΔP after a certain layer suggests that the models reach a point where adding more layers does not significantly improve (or even degrades) performance on these datasets. The slower decline in the 70B model might indicate that larger models are more robust to the performance degradation associated with increasing layer depth. The differences in performance across datasets (PopQA, TriviaQA, HotpotQA, NQ) highlight the sensitivity of the models to the specific characteristics of each dataset. The larger drops observed for PopQA and HotpotQA could indicate that these datasets are more challenging for the models, or that the models are more prone to overfitting on these datasets. The data suggests that the models' ability to generalize decreases with depth, and that the anchoring method plays a crucial role in mitigating this effect.

## Line Charts: Llama-3 Model Layer-wise ΔP Analysis ### Overview The image contains two side-by-side line charts comparing the performance metric "ΔP" across the layers of two different Large Language Models: Llama-3-8B (left chart) and Llama-3-70B (right chart). Each chart plots multiple data series representing different experimental conditions, defined by an anchoring method (Q-Anchored or A-Anchored) applied to four distinct question-answering datasets. ### Components/Axes * **Chart Titles:** * Left Chart: `Llama-3-8B` * Right Chart: `Llama-3-70B` * **Y-Axis (Both Charts):** * Label: `ΔP` (Delta P) * Scale: Linear, ranging from approximately -80 to 0. * Major Ticks: 0, -20, -40, -60, -80. * **X-Axis (Both Charts):** * Label: `Layer` * Scale: Linear. * Left Chart (8B) Range: 0 to 30. Major ticks appear at 0, 10, 20, 30. * Right Chart (70B) Range: 0 to 80. Major ticks appear at 0, 20, 40, 60, 80. * **Legend (Bottom Center, spanning both charts):** * Contains 8 entries, differentiating lines by color and line style (solid vs. dashed). * **Solid Lines (Q-Anchored):** * Blue: `Q-Anchored (PopQA)` * Green: `Q-Anchored (TriviaQA)` * Purple: `Q-Anchored (HotpotQA)` * Pink: `Q-Anchored (NQ)` * **Dashed Lines (A-Anchored):** * Orange: `A-Anchored (PopQA)` * Red: `A-Anchored (TriviaQA)` * Gray: `A-Anchored (HotpotQA)` * Brown: `A-Anchored (NQ)` ### Detailed Analysis **Llama-3-8B Chart (Left):** * **Q-Anchored Series (Solid Lines):** All four solid lines exhibit a strong, consistent downward trend. They start near ΔP = 0 at Layer 0 and decline steeply, reaching values between approximately -60 and -80 by Layer 30. * The Blue (PopQA) and Green (TriviaQA) lines show the most significant drop, ending near -80. * The Purple (HotpotQA) and Pink (NQ) lines follow a similar trajectory but end slightly higher, around -60 to -70. * The lines are jagged, indicating layer-to-layer volatility, but the overall negative slope is unambiguous. * **A-Anchored Series (Dashed Lines):** All four dashed lines remain relatively stable and close to ΔP = 0 throughout all 30 layers. They fluctuate within a narrow band, roughly between -10 and +5, showing no significant downward or upward trend. They are tightly clustered together. **Llama-3-70B Chart (Right):** * **Q-Anchored Series (Solid Lines):** The pattern is similar to the 8B model but extended over 80 layers. The solid lines again show a pronounced downward trend from Layer 0. * They descend rapidly in the first 20-30 layers, reaching a range of -40 to -60. * From Layer 30 to 80, the decline continues but at a slower, more volatile rate, with significant fluctuations. By Layer 80, the lines are spread between approximately -50 and -80. * The relative ordering is less consistent than in the 8B chart, with lines crossing frequently, but the Blue (PopQA) and Green (TriviaQA) lines generally remain among the lowest. * **A-Anchored Series (Dashed Lines):** As in the 8B chart, the dashed lines are stable and hover near the ΔP = 0 baseline across all 80 layers. They show minor fluctuations but no systematic drift, remaining clustered in the -10 to +5 range. ### Key Observations 1. **Anchoring Method Dominance:** The most striking pattern is the stark contrast between Q-Anchored (solid) and A-Anchored (dashed) conditions. Q-Anchoring leads to a large, progressive decrease in ΔP across model layers, while A-Anchoring results in a stable ΔP near zero. 2. **Model Scale Effect:** The trend for Q-Anchored lines is present in both model sizes (8B and 70B parameters). The 70B model chart shows the trend persisting over a greater number of layers (80 vs. 30), with increased volatility in the deeper layers. 3. **Dataset Variation:** Within the Q-Anchored group, the PopQA (blue) and TriviaQA (green) datasets consistently show the largest negative ΔP, especially in the 8B model. The NQ (pink) and HotpotQA (purple) datasets show a slightly attenuated effect. 4. **Spatial Layout:** The legend is positioned at the bottom, centered between the two charts. The charts themselves are aligned horizontally, sharing the same y-axis scale for direct comparison. ### Interpretation This data suggests a fundamental difference in how the Llama-3 model processes information depending on the anchoring prompt. "ΔP" likely represents a change in probability or performance metric. * **Q-Anchored (Question-Anchored) prompting** appears to cause a significant and layer-dependent degradation in the measured metric (ΔP becomes increasingly negative). This could indicate that when the model's processing is "anchored" to the question format, its internal representations or outputs shift dramatically as information propagates through the network layers, potentially moving away from a correct or stable answer distribution. * **A-Anchored (Answer-Anchored) prompting** maintains a stable ΔP near zero across all layers. This suggests that anchoring the model to the answer format results in more consistent internal processing, where the metric does not drift significantly from its initial value regardless of depth. * The consistency of this pattern across two model scales (8B and 70B) and four different QA datasets implies it is a robust phenomenon related to the prompting strategy itself, not a quirk of a specific model size or data domain. The increased volatility in the 70B model's deeper layers might reflect more complex or specialized processing in the larger model. * **Practical Implication:** For tasks where maintaining a stable probability or performance metric across model layers is desirable, A-Anchored prompting appears far more effective than Q-Anchored prompting based on this analysis. The choice of dataset (PopQA/TriviaQA vs. HotpotQA/NQ) modulates the effect's magnitude but does not change its fundamental direction.

## Line Graph: Performance Comparison of Llama-3 Models Across Layers ### Overview The image contains two side-by-side line graphs comparing the performance of different Llama-3 model configurations (8B and 70B) across layers. Each graph tracks the change in ΔP (likely a performance metric) across layers, with multiple data series representing different anchoring strategies and datasets. ### Components/Axes - **X-axis**: Layer (0 to 30 for 8B, 0 to 80 for 70B) - **Y-axis**: ΔP (ranging from -80 to 0) - **Legends**: - **Left Graph (Llama-3-8B)**: - Blue: Q-Anchored (PopQA) - Green: Q-Anchored (TriviaQA) - Red: Q-Anchored (HotpotQA) - Pink: Q-Anchored (NQ) - Orange: A-Anchored (PopQA) - Purple: A-Anchored (TriviaQA) - Gray: A-Anchored (HotpotQA) - Pink Dashed: A-Anchored (NQ) - **Right Graph (Llama-3-70B)**: - Same legend as above, with lines extending to 80 layers. ### Detailed Analysis #### Llama-3-8B (Left Graph) - **Q-Anchored (PopQA)**: Starts at 0, drops sharply to ~-60 by layer 20, then stabilizes with minor fluctuations. - **Q-Anchored (TriviaQA)**: Begins at 0, declines to ~-40 by layer 20, then fluctuates between -30 and -50. - **Q-Anchored (HotpotQA)**: Similar to TriviaQA but with more pronounced oscillations. - **Q-Anchored (NQ)**: Remains near 0 with slight oscillations. - **A-Anchored (PopQA)**: Starts at 0, drops to ~-40 by layer 20, then stabilizes. - **A-Anchored (TriviaQA)**: Declines to ~-30 by layer 20, then fluctuates between -20 and -40. - **A-Anchored (HotpotQA)**: Similar to TriviaQA but with more variability. - **A-Anchored (NQ)**: Stays near 0 with minimal changes. #### Llama-3-70B (Right Graph) - **Q-Anchored (PopQA)**: Starts at 0, drops to ~-40 by layer 40, then stabilizes. - **Q-Anchored (TriviaQA)**: Declines to ~-30 by layer 40, then fluctuates between -20 and -40. - **Q-Anchored (HotpotQA)**: Similar to TriviaQA but with more pronounced oscillations. - **Q-Anchored (NQ)**: Remains near 0 with slight oscillations. - **A-Anchored (PopQA)**: Starts at 0, drops to ~-30 by layer 40, then stabilizes. - **A-Anchored (TriviaQA)**: Declines to ~-20 by layer 40, then fluctuates between -10 and -30. - **A-Anchored (HotpotQA)**: Similar to TriviaQA but with more variability. - **A-Anchored (NQ)**: Stays near 0 with minimal changes. ### Key Observations 1. **Q-Anchored vs. A-Anchored**: Q-Anchored models generally show steeper declines in ΔP compared to A-Anchored models, especially in the 8B version. 2. **Dataset Impact**: PopQA and TriviaQA datasets exhibit more variability than NQ (No Query) models, which remain near 0. 3. **Model Size**: The 70B model shows more stability across layers compared to the 8B model, with less extreme ΔP values. 4. **Layer-Specific Trends**: In the 8B model, the sharpest drops occur in the first 20 layers, while the 70B model shows gradual changes. ### Interpretation The data suggests that anchoring strategies (Q vs. A) significantly influence performance, with Q-Anchored models experiencing more pronounced declines in ΔP. The 70B model's larger size appears to mitigate these declines, resulting in more stable performance across layers. The NQ models (no anchoring) maintain near-zero ΔP, indicating baseline performance without optimization. The dataset-specific trends (e.g., PopQA vs. TriviaQA) highlight how different data types interact with anchoring methods, suggesting that model architecture and data characteristics jointly determine performance outcomes.