## Line Chart: Model Performance Comparison ### Overview The image presents two line charts comparing the performance of different language models (Llama-3.2-3B-Instruct and Llama-3-8B-Instruct) across various layers. The charts depict the change in performance (ΔP) as a function of the layer number for different question-answering tasks. ### Components/Axes * **Titles:** * Left Chart: Llama-3.2-3B-Instruct * Right Chart: Llama-3-8B-Instruct * **X-axis (Layer):** Represents the layer number of the model. * Left Chart: Scale from 0 to 25, incrementing by 5. * Right Chart: Scale from 0 to 30, incrementing by 10. * **Y-axis (ΔP):** Represents the change in performance. * Both Charts: Scale from -80 to 0, incrementing by 20 on the left chart. Scale from -100 to 0, incrementing by 20 on the right chart. * **Legend (Bottom):** * Blue solid line: Q-Anchored (PopQA) * Brown dashed line: A-Anchored (PopQA) * Green dotted line: Q-Anchored (TriviaQA) * Purple dash-dotted line: Q-Anchored (HotpotQA) * Orange dash-dot-dotted line: A-Anchored (TriviaQA) * Gray dotted line: A-Anchored (HotpotQA) * Pink dashed line: Q-Anchored (NQ) * Black dotted line: A-Anchored (NQ) ### Detailed Analysis **Left Chart (Llama-3.2-3B-Instruct):** * **Q-Anchored (PopQA) (Blue solid line):** Starts at approximately 0 at layer 0, decreases to approximately -75 at layer 25. * **A-Anchored (PopQA) (Brown dashed line):** Remains relatively constant around 0 throughout all layers. * **Q-Anchored (TriviaQA) (Green dotted line):** Starts at approximately 0 at layer 0, decreases to approximately -70 at layer 25. * **Q-Anchored (HotpotQA) (Purple dash-dotted line):** Starts at approximately 0 at layer 0, decreases to approximately -65 at layer 25. * **A-Anchored (TriviaQA) (Orange dash-dot-dotted line):** Remains relatively constant around 0 throughout all layers. * **A-Anchored (HotpotQA) (Gray dotted line):** Remains relatively constant around 0 throughout all layers. * **Q-Anchored (NQ) (Pink dashed line):** Remains relatively constant around 0 throughout all layers. * **A-Anchored (NQ) (Black dotted line):** Remains relatively constant around 0 throughout all layers. **Right Chart (Llama-3-8B-Instruct):** * **Q-Anchored (PopQA) (Blue solid line):** Starts at approximately 0 at layer 0, decreases to approximately -90 at layer 30. * **A-Anchored (PopQA) (Brown dashed line):** Remains relatively constant around 0 throughout all layers. * **Q-Anchored (TriviaQA) (Green dotted line):** Starts at approximately 0 at layer 0, decreases to approximately -60 at layer 30. * **Q-Anchored (HotpotQA) (Purple dash-dotted line):** Starts at approximately 0 at layer 0, decreases to approximately -50 at layer 30. * **A-Anchored (TriviaQA) (Orange dash-dot-dotted line):** Remains relatively constant around 0 throughout all layers. * **A-Anchored (HotpotQA) (Gray dotted line):** Remains relatively constant around 0 throughout all layers. * **Q-Anchored (NQ) (Pink dashed line):** Remains relatively constant around 0 throughout all layers. * **A-Anchored (NQ) (Black dotted line):** Remains relatively constant around 0 throughout all layers. ### Key Observations * For both models, the "Q-Anchored" tasks (PopQA, TriviaQA, HotpotQA) show a significant decrease in performance (ΔP) as the layer number increases. * The "A-Anchored" tasks (PopQA, TriviaQA, HotpotQA, NQ) maintain a relatively constant performance (ΔP) around 0 across all layers for both models. * The Llama-3-8B-Instruct model shows a more pronounced decrease in performance for the Q-Anchored (PopQA) task compared to the Llama-3.2-3B-Instruct model. ### Interpretation The data suggests that anchoring the question (Q-Anchored) in the question-answering tasks leads to a degradation in performance as the model processes deeper layers. This could indicate that the model is losing relevant information or becoming more susceptible to noise as it goes through the layers when the question is anchored. Conversely, anchoring the answer (A-Anchored) results in stable performance across all layers, suggesting a more robust processing mechanism when the answer is the focal point. The difference in performance degradation between the two models for the Q-Anchored (PopQA) task may indicate that the larger model (Llama-3-8B-Instruct) is more sensitive to the anchoring of the question in this specific task.

\n ## Line Chart: ΔP vs. Layer for Different Models and Datasets ### Overview The image presents two line charts, side-by-side, displaying the change in probability (ΔP) as a function of layer number. The left chart focuses on the "Llama-3.2-3B-Instruct" model, while the right chart focuses on the "Llama-3-8B-Instruct" model. Each chart shows multiple lines representing different datasets and anchoring methods. The charts appear to be evaluating the impact of model depth (layers) on performance, potentially related to knowledge retention or transfer. ### Components/Axes * **X-axis:** "Layer" - Ranges from 0 to 25 for the left chart and 0 to 30 for the right chart. * **Y-axis:** "ΔP" - Ranges from approximately -100 to 0. * **Legend:** Located at the bottom of the image, identifying each line with its corresponding dataset and anchoring method. * Q-Anchored (PopQA) - Blue line * A-Anchored (PopQA) - Light Brown line * Q-Anchored (TriviaQA) - Purple line * A-Anchored (TriviaQA) - Green line * Q-Anchored (HotpotQA) - Orange dashed line * A-Anchored (HotpotQA) - Pink dashed line * Q-Anchored (NQ) - Cyan line * A-Anchored (NQ) - Magenta line ### Detailed Analysis or Content Details **Left Chart (Llama-3.2-3B-Instruct):** * **Q-Anchored (PopQA):** Starts at approximately 0, rapidly decreases to around -60 by layer 5, and continues to decrease, reaching approximately -80 by layer 25. * **A-Anchored (PopQA):** Starts at approximately 0, decreases to around -20 by layer 5, and then plateaus around -30 to -40 from layer 10 to 25. * **Q-Anchored (TriviaQA):** Starts at approximately 0, decreases to around -40 by layer 5, and continues to decrease, reaching approximately -70 by layer 25. * **A-Anchored (TriviaQA):** Starts at approximately 0, decreases to around -20 by layer 5, and then plateaus around -30 to -40 from layer 10 to 25. * **Q-Anchored (HotpotQA):** Starts at approximately 0, decreases to around -30 by layer 5, and then plateaus around -40 to -50 from layer 10 to 25. * **A-Anchored (HotpotQA):** Starts at approximately 0, decreases to around -20 by layer 5, and then plateaus around -30 to -40 from layer 10 to 25. * **Q-Anchored (NQ):** Starts at approximately 0, decreases to around -30 by layer 5, and then plateaus around -40 to -50 from layer 10 to 25. * **A-Anchored (NQ):** Starts at approximately 0, decreases to around -20 by layer 5, and then plateaus around -30 to -40 from layer 10 to 25. **Right Chart (Llama-3-8B-Instruct):** * **Q-Anchored (PopQA):** Starts at approximately 0, rapidly decreases to around -60 by layer 5, and continues to decrease, reaching approximately -90 by layer 30. * **A-Anchored (PopQA):** Starts at approximately 0, decreases to around -20 by layer 5, and then plateaus around -30 to -40 from layer 10 to 30. * **Q-Anchored (TriviaQA):** Starts at approximately 0, decreases to around -40 by layer 5, and continues to decrease, reaching approximately -70 by layer 30. * **A-Anchored (TriviaQA):** Starts at approximately 0, decreases to around -20 by layer 5, and then plateaus around -30 to -40 from layer 10 to 30. * **Q-Anchored (HotpotQA):** Starts at approximately 0, decreases to around -30 by layer 5, and then plateaus around -40 to -50 from layer 10 to 30. * **A-Anchored (HotpotQA):** Starts at approximately 0, decreases to around -20 by layer 5, and then plateaus around -30 to -40 from layer 10 to 30. * **Q-Anchored (NQ):** Starts at approximately 0, decreases to around -30 by layer 5, and then plateaus around -40 to -50 from layer 10 to 30. * **A-Anchored (NQ):** Starts at approximately 0, decreases to around -20 by layer 5, and then plateaus around -30 to -40 from layer 10 to 30. ### Key Observations * In both charts, the "Q-Anchored" lines consistently show a steeper decline in ΔP compared to the "A-Anchored" lines. This suggests that question-anchored methods lead to a more significant loss of probability as the model depth increases. * The "A-Anchored" lines tend to plateau after a certain number of layers, indicating that the change in probability stabilizes with depth. * The 8B model (right chart) exhibits a more pronounced decline in ΔP for the Q-Anchored lines, reaching lower values than the 3B model (left chart). * The datasets (PopQA, TriviaQA, HotpotQA, NQ) show relatively similar trends within each anchoring method. ### Interpretation The data suggests that increasing model depth (layers) can lead to a loss of information or a decrease in the model's ability to accurately represent the initial probability distribution, as measured by ΔP. This effect is more pronounced when using question-anchored methods. The plateauing of the "A-Anchored" lines suggests that answer-anchored methods may be more robust to the effects of depth, potentially by preserving information related to the answer itself. The larger decline observed in the 8B model could indicate that larger models are more susceptible to this loss of information, or that the effect is simply more noticeable due to the model's increased capacity. The consistent trends across different datasets suggest that this phenomenon is not specific to any particular type of knowledge or question-answering task. This data could be used to inform decisions about model architecture and training strategies, such as exploring methods to mitigate the loss of information with depth or focusing on answer-anchored approaches for deeper models. The negative ΔP values suggest a divergence from the initial probability distribution, which could be interpreted as a form of catastrophic forgetting or a loss of calibration.

## Line Charts: Llama-3.2 Model Layer-wise ΔP Analysis ### Overview The image displays two side-by-side line charts comparing the layer-wise change in probability (ΔP) for two different-sized language models from the Llama-3.2 series: a 3-billion parameter model (3B-Instruct) on the left and an 8-billion parameter model (8B-Instruct) on the right. Each chart plots the ΔP metric across the model's layers for four different question-answering datasets, using two distinct anchoring methods (Q-Anchored and A-Anchored). ### Components/Axes * **Titles:** * Left Chart: `Llama-3.2-3B-Instruct` * Right Chart: `Llama-3.2-8B-Instruct` * **Axes:** * **X-axis (Both Charts):** Labeled `Layer`. The scale runs from 0 to approximately 30, with major tick marks at 0, 5, 10, 15, 20, 25, and 30. * **Y-axis (Both Charts):** Labeled `ΔP`. The scale runs from -100 to 0, with major tick marks at -100, -80, -60, -40, -20, and 0. * **Legend (Bottom, spanning both charts):** Contains 8 entries, differentiating lines by color, line style (solid vs. dashed), and dataset. * **Solid Lines (Q-Anchored):** * Blue: `Q-Anchored (PopQA)` * Green: `Q-Anchored (TriviaQA)` * Purple: `Q-Anchored (HotpotQA)` * Pink: `Q-Anchored (NQ)` * **Dashed Lines (A-Anchored):** * Orange: `A-Anchored (PopQA)` * Red: `A-Anchored (TriviaQA)` * Brown: `A-Anchored (HotpotQA)` * Gray: `A-Anchored (NQ)` * **Visual Elements:** Each data series is represented by a colored line with a semi-transparent shaded region around it, likely indicating confidence intervals or standard deviation. ### Detailed Analysis **Left Chart: Llama-3.2-3B-Instruct** * **Q-Anchored Series (Solid Lines):** All four datasets show a strong, consistent downward trend. ΔP starts near 0 at Layer 0 and decreases sharply, reaching values between approximately -60 and -80 by Layer 27. The lines are tightly clustered, with the blue (PopQA) and purple (HotpotQA) lines often at the lower end of the range. * **A-Anchored Series (Dashed Lines):** All four datasets show a flat, stable trend. ΔP remains very close to 0 across all layers, with minor fluctuations. The lines are tightly clustered near the top of the chart. **Right Chart: Llama-3.2-8B-Instruct** * **Q-Anchored Series (Solid Lines):** The downward trend is present but more varied compared to the 3B model. The blue line (PopQA) shows the steepest and most volatile decline, dropping to near -100 around Layer 20 before a slight recovery. The green (TriviaQA), purple (HotpotQA), and pink (NQ) lines follow a smoother downward path, ending between -60 and -80 by Layer 32. * **A-Anchored Series (Dashed Lines):** Similar to the 3B model, these series remain stable and close to 0 across all layers, with minimal fluctuation. ### Key Observations 1. **Anchoring Method Dominance:** The most striking pattern is the drastic difference between Q-Anchored and A-Anchored methods. Q-Anchoring leads to a significant negative ΔP that grows with layer depth, while A-Anchoring maintains a ΔP near zero. 2. **Model Size Effect:** The 8B model exhibits more pronounced volatility in the Q-Anchored PopQA series (blue line) compared to the 3B model. The other Q-Anchored series in the 8B model also show slightly more separation from each other. 3. **Dataset Similarity:** Within each anchoring method, the trends across the four datasets (PopQA, TriviaQA, HotpotQA, NQ) are broadly similar, suggesting the anchoring technique is a stronger factor than the specific dataset in determining the ΔP trajectory. 4. **Layer Dependence:** For Q-Anchored methods, the effect (negative ΔP) is not uniform; it intensifies progressively through the network layers. ### Interpretation The data demonstrates a fundamental difference in how information is processed or retained within the model layers depending on the anchoring technique. "ΔP" likely represents a change in probability or confidence. The results suggest: * **Q-Anchored (Question-Anchored) processing** causes a progressive and significant decrease in the measured probability metric as information flows deeper into the network. This could indicate a process of evidence accumulation, refinement, or a shift in focus away from the initial question's framing as the model generates an answer. * **A-Anchored (Answer-Anchored) processing** maintains a stable probability metric throughout the layers. This implies that when anchored to the answer, the model's internal state regarding this metric does not change significantly from input to output, suggesting a more consistent or fixed processing pathway. * The increased volatility in the larger 8B model's Q-Anchored PopQA series might reflect greater model capacity leading to more complex or non-linear internal transformations for that specific dataset. In essence, the charts reveal that the choice of anchoring (question vs. answer) fundamentally alters the layer-wise dynamics of the model's internal probability landscape, with the question-anchored approach inducing a strong, depth-dependent decay effect.

## Line Graphs: LLaMA-3-2B-Instruct and LLaMA-3-8B-Instruct Performance Comparison ### Overview The image contains two side-by-side line graphs comparing the performance of different anchoring methods (Q-Anchored and A-Anchored) across datasets (PopQA, TriviaQA, HotpotQA, NQ) in LLaMA-3-2B-Instruct and LLaMA-3-8B-Instruct models. The y-axis represents ΔP (perplexity change), and the x-axis represents model layers. Each graph includes shaded regions indicating confidence intervals. --- ### Components/Axes - **Left Graph (LLaMA-3-2B-Instruct)**: - **X-axis**: Layer (0 to 25) - **Y-axis**: ΔP (range: -100 to 0) - **Legend**: - Blue: Q-Anchored (PopQA) - Green: Q-Anchored (TriviaQA) - Red: Q-Anchored (HotpotQA) - Purple: Q-Anchored (NQ) - Dashed Orange: A-Anchored (PopQA) - Dashed Green: A-Anchored (TriviaQA) - Dashed Red: A-Anchored (HotpotQA) - Dashed Purple: A-Anchored (NQ) - **Right Graph (LLaMA-3-8B-Instruct)**: - **X-axis**: Layer (0 to 30) - **Y-axis**: ΔP (range: -100 to 0) - **Legend**: Same as the left graph. --- ### Detailed Analysis #### LLaMA-3-2B-Instruct (Left Graph) 1. **Q-Anchored (PopQA)** (Blue): - Starts at 0 (layer 0), drops sharply to -60 by layer 25. - Fluctuates between -40 and -60 in mid-layers (layers 5–15). 2. **Q-Anchored (TriviaQA)** (Green): - Starts at 0, declines to -50 by layer 25. - Shows moderate fluctuations (-30 to -50) in mid-layers. 3. **Q-Anchored (HotpotQA)** (Red): - Starts at 0, drops to -40 by layer 25. - Fluctuates between -20 and -40 in mid-layers. 4. **Q-Anchored (NQ)** (Purple): - Starts at 0, declines to -70 by layer 25. - Sharp drop to -70 in early layers (layers 5–10), then stabilizes. 5. **A-Anchored (PopQA)** (Dashed Orange): - Starts at 0, ends at -20 by layer 25. - Minimal fluctuations (-10 to -20). 6. **A-Anchored (TriviaQA)** (Dashed Green): - Starts at 0, ends at -30 by layer 25. - Slight dip to -25 in mid-layers. 7. **A-Anchored (HotpotQA)** (Dashed Red): - Starts at 0, ends at -25 by layer 25. - Stable with minor fluctuations (-15 to -25). 8. **A-Anchored (NQ)** (Dashed Purple): - Starts at 0, ends at -40 by layer 25. - Gradual decline with minor fluctuations (-20 to -40). #### LLaMA-3-8B-Instruct (Right Graph) 1. **Q-Anchored (PopQA)** (Blue): - Starts at 0, drops sharply to -100 by layer 30. - Steep decline in early layers (layers 5–15), then stabilizes. 2. **Q-Anchored (TriviaQA)** (Green): - Starts at 0, declines to -80 by layer 30. - Sharp drop to -60 in early layers, then stabilizes. 3. **Q-Anchored (HotpotQA)** (Red): - Starts at 0, drops to -60 by layer 30. - Moderate decline (-40 to -60) in mid-layers. 4. **Q-Anchored (NQ)** (Purple): - Starts at 0, drops to -90 by layer 30. - Steep decline to -70 in early layers, then stabilizes. 5. **A-Anchored (PopQA)** (Dashed Orange): - Starts at 0, ends at -40 by layer 30. - Gradual decline (-20 to -40). 6. **A-Anchored (TriviaQA)** (Dashed Green): - Starts at 0, ends at -50 by layer 30. - Slight dip to -35 in mid-layers. 7. **A-Anchored (HotpotQA)** (Dashed Red): - Starts at 0, ends at -35 by layer 30. - Stable with minor fluctuations (-25 to -35). 8. **A-Anchored (NQ)** (Dashed Purple): - Starts at 0, ends at -60 by layer 30. - Gradual decline (-30 to -60). --- ### Key Observations 1. **Model Size Impact**: - The 8B model shows steeper ΔP declines compared to the 2B model, especially for Q-Anchored methods. - Example: Q-Anchored (NQ) in 8B drops to -90 vs. -70 in 2B. 2. **Anchoring Method Differences**: - **Q-Anchored** methods exhibit larger ΔP drops, particularly for NQ and HotpotQA datasets. - **A-Anchored** methods show smaller, more stable ΔP values across layers. 3. **Dataset Variability**: - NQ consistently shows the largest ΔP drops, suggesting it is the most challenging dataset. - PopQA and TriviaQA have moderate ΔP declines, while HotpotQA has the smallest drops. 4. **Confidence Intervals**: - Shaded regions indicate variability in ΔP measurements. Larger models (8B) show wider confidence intervals, especially in Q-Anchored methods. --- ### Interpretation - **Model Size and Performance**: The 8B model’s larger ΔP drops suggest that increased model size amplifies the impact of anchoring methods, particularly for complex datasets like NQ. - **Anchoring Robustness**: A-Anchored methods demonstrate greater stability, implying they may be more effective in maintaining performance across layers. - **Dataset Sensitivity**: NQ’s poor performance across both models highlights its inherent difficulty, possibly due to its reliance on reasoning or knowledge-intensive tasks. - **Layer-Specific Trends**: Early layers (0–10) show the most significant ΔP changes, indicating that anchoring methods have a stronger effect in initial processing stages. This analysis underscores the importance of anchoring strategies in model performance, with A-Anchored methods offering potential advantages in stability and robustness.