## Chart Type: Line Graphs ### Overview The image contains two line graphs comparing the performance of Llama-3.2-1B and Llama-3.2-3B models across different layers. The y-axis represents ΔP (Delta P), and the x-axis represents the Layer number. Each graph displays six data series, representing "Q-Anchored" and "A-Anchored" performance on four different question-answering datasets: PopQA, TriviaQA, HotpotQA, and NQ. ### Components/Axes * **Titles:** * Left Graph: "Llama-3.2-1B" * Right Graph: "Llama-3.2-3B" * **Y-Axis:** * Label: "ΔP" * Scale: -80 to 0, with increments of 20 (-60, -40, -20, 0) * **X-Axis:** * Label: "Layer" * Left Graph Scale: 0 to 15, with increments of 5 (5, 10, 15) * Right Graph Scale: 0 to 25, with increments of 5 (5, 10, 15, 20, 25) * **Legend:** Located at the bottom of the image. * Q-Anchored (PopQA): Solid Blue Line * A-Anchored (PopQA): Dashed Orange Line * Q-Anchored (TriviaQA): Dotted Green Line * A-Anchored (TriviaQA): Dash-Dotted Gray Line * Q-Anchored (HotpotQA): Dash-Dot-Dotted Pink Line * A-Anchored (HotpotQA): Dotted Brown Line * Q-Anchored (NQ): Dash-Dotted Pink Line * A-Anchored (NQ): Dotted Gray Line ### Detailed Analysis **Left Graph (Llama-3.2-1B):** * **Q-Anchored (PopQA):** (Solid Blue Line) Starts at approximately 0 and decreases sharply to around -60 by layer 7, then fluctuates between -50 and -60 until layer 15. * **A-Anchored (PopQA):** (Dashed Orange Line) Remains relatively stable around 0, fluctuating slightly between -5 and 5. * **Q-Anchored (TriviaQA):** (Dotted Green Line) Starts at approximately 0 and decreases to around -50 by layer 7, then fluctuates between -40 and -50 until layer 15. * **A-Anchored (TriviaQA):** (Dash-Dotted Gray Line) Remains relatively stable around 0, fluctuating slightly between -5 and 5. * **Q-Anchored (HotpotQA):** (Dash-Dot-Dotted Pink Line) Starts at approximately 0 and decreases to around -50 by layer 7, then fluctuates between -40 and -50 until layer 15. * **A-Anchored (HotpotQA):** (Dotted Brown Line) Remains relatively stable around 0, fluctuating slightly between -5 and 5. * **Q-Anchored (NQ):** (Dash-Dotted Pink Line) Starts at approximately 0 and decreases to around -50 by layer 7, then fluctuates between -40 and -50 until layer 15. * **A-Anchored (NQ):** (Dotted Gray Line) Remains relatively stable around 0, fluctuating slightly between -5 and 5. **Right Graph (Llama-3.2-3B):** * **Q-Anchored (PopQA):** (Solid Blue Line) Starts at approximately 0 and decreases sharply to around -70 by layer 10, then fluctuates between -50 and -70 until layer 25. * **A-Anchored (PopQA):** (Dashed Orange Line) Remains relatively stable around 0, fluctuating slightly between -5 and 5. * **Q-Anchored (TriviaQA):** (Dotted Green Line) Starts at approximately 0 and decreases to around -60 by layer 10, then fluctuates between -40 and -60 until layer 25. * **A-Anchored (TriviaQA):** (Dash-Dotted Gray Line) Remains relatively stable around 0, fluctuating slightly between -5 and 5. * **Q-Anchored (HotpotQA):** (Dash-Dot-Dotted Pink Line) Starts at approximately 0 and decreases to around -60 by layer 10, then fluctuates between -40 and -60 until layer 25. * **A-Anchored (HotpotQA):** (Dotted Brown Line) Remains relatively stable around 0, fluctuating slightly between -5 and 5. * **Q-Anchored (NQ):** (Dash-Dotted Pink Line) Starts at approximately 0 and decreases to around -60 by layer 10, then fluctuates between -40 and -60 until layer 25. * **A-Anchored (NQ):** (Dotted Gray Line) Remains relatively stable around 0, fluctuating slightly between -5 and 5. ### Key Observations * The "Q-Anchored" data series (PopQA, TriviaQA, HotpotQA, and NQ) show a significant decrease in ΔP as the layer number increases, indicating a performance change. * The "A-Anchored" data series (PopQA, TriviaQA, HotpotQA, and NQ) remain relatively stable around 0, suggesting minimal performance change across layers. * The Llama-3.2-3B model (right graph) has a longer x-axis (more layers) compared to the Llama-3.2-1B model (left graph). * The Q-Anchored lines for Llama-3.2-3B appear to reach lower values than those for Llama-3.2-1B. ### Interpretation The graphs suggest that anchoring the question ("Q-Anchored") has a more significant impact on performance across different layers compared to anchoring the answer ("A-Anchored"). The decrease in ΔP for "Q-Anchored" indicates a change in performance as the model processes information through its layers. The stable ΔP for "A-Anchored" suggests that anchoring the answer has a less pronounced effect on performance across layers. The Llama-3.2-3B model, with its increased number of layers, shows a similar trend to the Llama-3.2-1B model, but the Q-Anchored performance appears to decrease further, potentially indicating that the impact of question anchoring becomes more pronounced with increased model depth. The data suggests that the way the question is anchored significantly affects how the model processes information across its layers, while the answer anchoring has a comparatively negligible effect.

\n ## Line Chart: ΔP vs. Layer for Llama-3.2 Models ### Overview The image presents two line charts, side-by-side, displaying the change in performance (ΔP) as a function of layer number for two different Llama-3.2 models: 1B and 3B. Each chart shows multiple lines representing different question-answering datasets and anchoring methods. The charts are visually similar, both showing a steep initial decline in ΔP followed by a leveling off. ### Components/Axes * **X-axis:** Layer (ranging from approximately 0 to 15 for the 1B model and 0 to 25 for the 3B model). * **Y-axis:** ΔP (ranging from approximately -80 to 0). * **Models:** Llama-3.2-1B (left chart), Llama-3.2-3B (right chart). * **Datasets/Anchoring Methods (Legend):** * Q-Anchored (PopQA) - Blue line * A-Anchored (PopQA) - Light Orange dashed line * Q-Anchored (TriviaQA) - Green line * A-Anchored (TriviaQA) - Light Purple line * Q-Anchored (HotpotQA) - Red dashed line * A-Anchored (HotpotQA) - Light Blue line * Q-Anchored (NQ) - Dark Orange line * A-Anchored (NQ) - Light Gray line The legend is positioned at the bottom of the image, spanning both charts. ### Detailed Analysis or Content Details **Llama-3.2-1B Chart (Left):** * **Q-Anchored (PopQA):** Starts at approximately 0 ΔP, rapidly declines to around -60 ΔP by layer 10, then plateaus around -60 ΔP. * **A-Anchored (PopQA):** Starts at approximately 0 ΔP, declines to around -40 ΔP by layer 10, then plateaus around -40 ΔP. * **Q-Anchored (TriviaQA):** Starts at approximately 0 ΔP, declines to around -50 ΔP by layer 8, then plateaus around -50 ΔP. * **A-Anchored (TriviaQA):** Starts at approximately 0 ΔP, declines to around -45 ΔP by layer 8, then plateaus around -45 ΔP. * **Q-Anchored (HotpotQA):** Starts at approximately 0 ΔP, declines to around -30 ΔP by layer 5, then plateaus around -30 ΔP. * **A-Anchored (HotpotQA):** Starts at approximately 0 ΔP, declines to around -25 ΔP by layer 5, then plateaus around -25 ΔP. * **Q-Anchored (NQ):** Starts at approximately 0 ΔP, declines to around -20 ΔP by layer 5, then plateaus around -20 ΔP. * **A-Anchored (NQ):** Starts at approximately 0 ΔP, declines to around -15 ΔP by layer 5, then plateaus around -15 ΔP. **Llama-3.2-3B Chart (Right):** * **Q-Anchored (PopQA):** Starts at approximately 0 ΔP, rapidly declines to around -70 ΔP by layer 15, then plateaus around -70 ΔP. * **A-Anchored (PopQA):** Starts at approximately 0 ΔP, declines to around -50 ΔP by layer 15, then plateaus around -50 ΔP. * **Q-Anchored (TriviaQA):** Starts at approximately 0 ΔP, declines to around -60 ΔP by layer 12, then plateaus around -60 ΔP. * **A-Anchored (TriviaQA):** Starts at approximately 0 ΔP, declines to around -50 ΔP by layer 12, then plateaus around -50 ΔP. * **Q-Anchored (HotpotQA):** Starts at approximately 0 ΔP, declines to around -35 ΔP by layer 10, then plateaus around -35 ΔP. * **A-Anchored (HotpotQA):** Starts at approximately 0 ΔP, declines to around -30 ΔP by layer 10, then plateaus around -30 ΔP. * **Q-Anchored (NQ):** Starts at approximately 0 ΔP, declines to around -25 ΔP by layer 10, then plateaus around -25 ΔP. * **A-Anchored (NQ):** Starts at approximately 0 ΔP, declines to around -20 ΔP by layer 10, then plateaus around -20 ΔP. ### Key Observations * The ΔP consistently decreases with increasing layer number for all datasets and anchoring methods. * Q-Anchored methods generally exhibit a larger decrease in ΔP compared to A-Anchored methods. * PopQA and TriviaQA datasets show the most significant performance decline, while HotpotQA and NQ datasets show a more moderate decline. * The 3B model exhibits a more pronounced decline in ΔP than the 1B model, suggesting that increasing model size exacerbates the performance degradation with layer depth. * The rate of decline appears to slow down after a certain layer number (around 10-15 for the 1B model and 15-20 for the 3B model). ### Interpretation These charts likely represent the impact of increasing model depth on performance, measured by ΔP (presumably a performance metric). The consistent decline in ΔP suggests that adding more layers to the Llama-3.2 models leads to a degradation in performance, potentially due to issues like vanishing gradients or overfitting. The difference between Q-Anchored and A-Anchored methods suggests that the method used to anchor the questions or answers influences the extent of performance degradation. Q-Anchoring appears to be more susceptible to performance loss as the model deepens. The varying degrees of decline across different datasets (PopQA, TriviaQA, HotpotQA, NQ) indicate that the type of question-answering task also plays a role. More complex or challenging datasets (PopQA and TriviaQA) seem to suffer more from increased model depth. The larger decline observed in the 3B model compared to the 1B model suggests that the performance degradation becomes more pronounced as the model size increases. This could be due to the increased complexity of larger models making them more prone to overfitting or other issues associated with deep learning. The leveling off of the curves after a certain layer number suggests that there might be a point of diminishing returns when adding more layers to these models. Beyond that point, the performance gains are minimal, and the risk of degradation outweighs the potential benefits.

## Line Charts: Llama-3.2 Model Layer-wise ΔP Analysis ### Overview The image displays two side-by-side line charts comparing the performance metric "ΔP" across the layers of two different language models: Llama-3.2-1B (left) and Llama-3.2-3B (right). Each chart plots multiple data series representing different experimental conditions (Q-Anchored vs. A-Anchored) evaluated on four distinct question-answering datasets. ### Components/Axes * **Titles:** * Left Chart: `Llama-3.2-1B` * Right Chart: `Llama-3.2-3B` * **Y-Axis (Both Charts):** * Label: `ΔP` (Delta P) * Scale: Linear, ranging from 0 at the top to negative values at the bottom. * Left Chart Range: 0 to -80 (major ticks at 0, -20, -40, -60). * Right Chart Range: 0 to -80 (major ticks at 0, -20, -40, -60, -80). * **X-Axis (Both Charts):** * Label: `Layer` * Left Chart Scale: 0 to 15 (major ticks at 0, 5, 10, 15). * Right Chart Scale: 0 to 25 (major ticks at 0, 5, 10, 15, 20, 25). * **Legend (Bottom Center, spanning both charts):** * Contains 8 entries, differentiating lines by color and style (solid vs. dashed). * **Q-Anchored Series (Solid Lines):** * Blue: `Q-Anchored (PopQA)` * Green: `Q-Anchored (TriviaQA)` * Purple: `Q-Anchored (HotpotQA)` * Pink: `Q-Anchored (NQ)` * **A-Anchored Series (Dashed Lines):** * Orange: `A-Anchored (PopQA)` * Red: `A-Anchored (TriviaQA)` * Gray: `A-Anchored (HotpotQA)` * Brown: `A-Anchored (NQ)` * **Data Representation:** Each data series is shown as a line with a semi-transparent shaded band around it, likely representing a confidence interval or standard deviation. ### Detailed Analysis **Llama-3.2-1B (Left Chart):** * **Q-Anchored (Solid Lines) Trend:** All four solid lines show a strong, consistent downward trend. Starting near ΔP = 0 at Layer 0, they decline steeply, reaching their lowest points between Layers 10-15. The values at Layer 15 are approximately: * Q-Anchored (PopQA) [Blue]: ~ -60 * Q-Anchored (TriviaQA) [Green]: ~ -55 * Q-Anchored (HotpotQA) [Purple]: ~ -50 * Q-Anchored (NQ) [Pink]: ~ -45 * **A-Anchored (Dashed Lines) Trend:** All four dashed lines remain relatively stable and close to zero throughout all layers. They fluctuate slightly but generally stay within the range of ΔP = 0 to -10. There is no significant downward trend. **Llama-3.2-3B (Right Chart):** * **Q-Anchored (Solid Lines) Trend:** Similar to the 1B model, the solid lines exhibit a pronounced downward trajectory. The decline appears more volatile, with deeper troughs. The lowest points occur around Layers 20-25. Approximate values at Layer 25: * Q-Anchored (PopQA) [Blue]: ~ -70 * Q-Anchored (TriviaQA) [Green]: ~ -65 * Q-Anchored (HotpotQA) [Purple]: ~ -60 * Q-Anchored (NQ) [Pink]: ~ -55 * **A-Anchored (Dashed Lines) Trend:** Consistent with the 1B model, the dashed lines for A-Anchored conditions hover near ΔP = 0 across all layers, showing minor fluctuations but no major decline. ### Key Observations 1. **Clear Dichotomy:** There is a stark and consistent separation between the behavior of Q-Anchored (solid lines) and A-Anchored (dashed lines) conditions across both models. This is the most prominent feature of the data. 2. **Layer-Dependent Degradation:** For Q-Anchored conditions, the metric ΔP degrades significantly (becomes more negative) as information propagates through the network layers. This degradation is progressive. 3. **Model Scale Effect:** The larger model (3B) shows a similar pattern but extends over more layers (25 vs. 15) and reaches slightly more negative ΔP values for the Q-Anchored conditions, suggesting the effect may be amplified or more measurable in a deeper network. 4. **Dataset Consistency:** The relative ordering of the four datasets within the Q-Anchored group is roughly consistent between models: PopQA (blue) tends to show the most negative ΔP, followed by TriviaQA (green), HotpotQA (purple), and NQ (pink). The A-Anchored lines are tightly clustered near zero with no clear dataset ordering. ### Interpretation The data suggests a fundamental difference in how "question-anchored" (Q-Anchored) versus "answer-anchored" (A-Anchored) information is processed or retained across the layers of these Llama models. * **Q-Anchored Processing:** The steep negative trend in ΔP for Q-Anchored conditions indicates that the model's internal representation or processing related to the question itself changes dramatically and progressively as it moves through the network layers. The metric ΔP, which likely measures some form of probability shift or performance delta, deteriorates. This could imply that the question context becomes less "stable" or is transformed in a way that negatively impacts this specific metric as depth increases. * **A-Anchored Processing:** In contrast, the stability of ΔP near zero for A-Anchored conditions suggests that anchoring to the answer provides a robust signal that maintains its integrity or influence throughout the network's depth. The model's processing related to the answer appears less susceptible to layer-wise degradation on this metric. * **Implication:** This contrast may highlight a potential vulnerability or characteristic of how these models handle query-based (question) versus target-based (answer) information flow. The findings could be relevant for understanding model interpretability, the mechanics of information propagation in transformers, or for designing more robust prompting and fine-tuning strategies. The consistency across four different QA datasets strengthens the generalizability of this observed pattern.

## Line Graphs: ΔP vs Layer for QA Models in LLaMA-3.2-1B and 3B ### Overview The image contains two line graphs comparing the change in perplexity (ΔP) across transformer layers for different question-answering (QA) models in LLaMA-3.2-1B (left) and LLaMA-3.2-3B (right). The graphs show six data series with distinct line styles and colors, representing combinations of anchoring strategies (Q-Anchored vs A-Anchored) and datasets (PopQA, TriviaQA, HotpotQA, NQ). Shaded regions indicate confidence intervals. ### Components/Axes - **X-axis (Layer)**: Integer values from 0 to 15 (1B) and 0 to 25 (3B), representing transformer layers. - **Y-axis (ΔP)**: Change in perplexity, ranging from -80 to 0. - **Legends**: - **Q-Anchored (PopQA)**: Solid blue - **A-Anchored (PopQA)**: Dashed orange - **Q-Anchored (TriviaQA)**: Dotted green - **A-Anchored (TriviaQA)**: Dash-dot pink - **Q-Anchored (HotpotQA)**: Solid purple - **A-Anchored (NQ)**: Dotted gray ### Detailed Analysis #### LLaMA-3.2-1B (Left Graph) 1. **Q-Anchored (PopQA)**: Starts at 0, sharply drops to ~-60 at layer 10, then recovers to ~-20 by layer 15. 2. **A-Anchored (PopQA)**: Remains relatively stable, fluctuating between 0 and -10. 3. **Q-Anchored (TriviaQA)**: Drops to ~-50 at layer 5, recovers to ~-10 by layer 15. 4. **A-Anchored (TriviaQA)**: Stable between 0 and -5. 5. **Q-Anchored (HotpotQA)**: Dips to ~-40 at layer 10, recovers to ~-15 by layer 15. 6. **A-Anchored (NQ)**: Stable between 0 and -5. #### LLaMA-3.2-3B (Right Graph) 1. **Q-Anchored (PopQA)**: Starts at 0, drops to ~-70 at layer 10, recovers to ~-30 by layer 25. 2. **A-Anchored (PopQA)**: Stable between 0 and -5. 3. **Q-Anchored (TriviaQA)**: Drops to ~-60 at layer 5, recovers to ~-20 by layer 25. 4. **A-Anchored (TriviaQA)**: Stable between 0 and -5. 5. **Q-Anchored (HotpotQA)**: Sharp dip to ~-80 at layer 20, recovers to ~-40 by layer 25. 6. **A-Anchored (NQ)**: Stable between 0 and -5. ### Key Observations 1. **Q-Anchored models** consistently show larger ΔP dips than A-Anchored counterparts, especially in deeper layers. 2. **HotpotQA** datasets exhibit the most extreme ΔP fluctuations, particularly in the 3B model. 3. **A-Anchored (NQ)** remains the most stable across all layers and model sizes. 4. **Confidence intervals** (shaded regions) are widest for Q-Anchored models, indicating higher variability. 5. **Layer-specific trends**: ΔP dips correlate with mid-to-late layers (5–20), suggesting architectural or training dynamics in these regions. ### Interpretation The data suggests that **Q-Anchored models** are more sensitive to layer-specific changes in perplexity, particularly when trained on complex datasets like HotpotQA. The pronounced dips in ΔP for Q-Anchored models may reflect: - **Architectural bottlenecks**: Certain layers struggle more with QA tasks when anchored to specific datasets. - **Training dynamics**: Q-Anchored models might overfit to dataset-specific patterns in later layers. - **Scalability differences**: The 3B model’s extended layers show similar trends, implying that larger models amplify these effects. The stability of A-Anchored models (especially NQ) suggests they are less prone to layer-specific degradation, making them more robust for general QA tasks. The extreme fluctuations in HotpotQA datasets highlight challenges in handling multi-hop reasoning or domain-specific knowledge in later transformer layers.