## Chart: Layer vs. Delta P for Llama-3-8B and Llama-3-70B ### Overview The image presents two line charts comparing the performance of Llama-3-8B and Llama-3-70B models across different layers. The y-axis represents the change in probability (ΔP), while the x-axis represents the layer number. Each chart displays six data series, representing different question-answering datasets (PopQA, TriviaQA, HotpotQA, and NQ) anchored by either the question (Q-Anchored) or the answer (A-Anchored). ### Components/Axes * **Titles:** * Left Chart: "Llama-3-8B" * Right Chart: "Llama-3-70B" * **Y-Axis:** * Label: "ΔP" * Scale: -80 to 0, with tick marks at -80, -60, -40, -20, and 0. * **X-Axis:** * Label: "Layer" * Left Chart Scale: 0 to 30, with tick marks at 0, 10, 20, and 30. * Right Chart Scale: 0 to 80, with tick marks at 0, 20, 40, 60, and 80. * **Legend:** Located at the bottom of the image. * **Q-Anchored (PopQA):** Solid Blue Line * **A-Anchored (PopQA):** Dashed Brown Line * **Q-Anchored (TriviaQA):** Dotted Green Line * **A-Anchored (TriviaQA):** Dashed-Dotted Light Brown Line * **Q-Anchored (HotpotQA):** Dashed-Dotted Purple Line * **A-Anchored (HotpotQA):** Dotted Light Brown Line * **Q-Anchored (NQ):** Dashed Purple Line * **A-Anchored (NQ):** Dotted Light Gray Line ### Detailed Analysis **Left Chart: Llama-3-8B** * **Q-Anchored (PopQA):** (Solid Blue) Starts at approximately 0 and decreases sharply to approximately -80 by layer 30. * **A-Anchored (PopQA):** (Dashed Brown) Remains relatively stable around 0 throughout all layers. * **Q-Anchored (TriviaQA):** (Dotted Green) Starts at approximately 0 and decreases to approximately -65 by layer 30. * **A-Anchored (TriviaQA):** (Dashed-Dotted Light Brown) Remains relatively stable around 0 throughout all layers. * **Q-Anchored (HotpotQA):** (Dashed-Dotted Purple) Starts at approximately 0 and decreases to approximately -70 by layer 30. * **A-Anchored (HotpotQA):** (Dotted Light Brown) Remains relatively stable around 0 throughout all layers. * **Q-Anchored (NQ):** (Dashed Purple) Starts at approximately 0 and decreases to approximately -60 by layer 30. * **A-Anchored (NQ):** (Dotted Light Gray) Remains relatively stable around 0 throughout all layers. **Right Chart: Llama-3-70B** * **Q-Anchored (PopQA):** (Solid Blue) Starts at approximately 0 and decreases sharply to approximately -80 by layer 30, then fluctuates between -60 and -80 until layer 80. * **A-Anchored (PopQA):** (Dashed Brown) Remains relatively stable around 0 throughout all layers. * **Q-Anchored (TriviaQA):** (Dotted Green) Starts at approximately 0 and decreases to approximately -65 by layer 30, then fluctuates between -50 and -70 until layer 80. * **A-Anchored (TriviaQA):** (Dashed-Dotted Light Brown) Remains relatively stable around 0 throughout all layers. * **Q-Anchored (HotpotQA):** (Dashed-Dotted Purple) Starts at approximately 0 and decreases to approximately -70 by layer 30, then fluctuates between -50 and -70 until layer 80. * **A-Anchored (HotpotQA):** (Dotted Light Brown) Remains relatively stable around 0 throughout all layers. * **Q-Anchored (NQ):** (Dashed Purple) Starts at approximately 0 and decreases to approximately -60 by layer 30, then fluctuates between -50 and -70 until layer 80. * **A-Anchored (NQ):** (Dotted Light Gray) Remains relatively stable around 0 throughout all layers. ### Key Observations * For both models, the "Q-Anchored" series (PopQA, TriviaQA, HotpotQA, and NQ) show a significant decrease in ΔP as the layer number increases, indicating a change in probability when anchoring with the question. * The "A-Anchored" series (PopQA, TriviaQA, HotpotQA, and NQ) remain relatively stable around 0 for both models, suggesting that anchoring with the answer does not significantly affect the probability. * The Llama-3-70B model shows more fluctuation in the "Q-Anchored" series after layer 30 compared to the Llama-3-8B model. ### Interpretation The data suggests that anchoring the question-answering process with the question itself ("Q-Anchored") leads to a substantial change in probability as the model processes deeper layers. This could indicate that the model is refining its understanding or focus as it progresses through the layers. In contrast, anchoring with the answer ("A-Anchored") does not significantly alter the probability, possibly because the answer provides a fixed reference point. The fluctuations observed in the Llama-3-70B model after layer 30 for the "Q-Anchored" series might indicate that the larger model continues to adjust its understanding or confidence even in later layers, whereas the smaller Llama-3-8B model stabilizes earlier. This could be due to the larger model's greater capacity to process and refine information. The consistent behavior of the "A-Anchored" series suggests that providing the answer upfront stabilizes the model's probability assessment, regardless of the layer. This could be useful in applications where a consistent and reliable probability score is desired.

## Line Chart: ΔP vs. Layer for Llama Models ### Overview The image presents two line charts, side-by-side, comparing the change in probability (ΔP) across layers for two Llama models: Llama-3-8B and Llama-3-70B. Each chart displays multiple lines representing different question-answering datasets and anchoring methods. The x-axis represents the layer number, and the y-axis represents ΔP. ### Components/Axes * **X-axis:** Layer (ranging from 0 to 30 for Llama-3-8B and 0 to 80 for Llama-3-70B). * **Y-axis:** ΔP (ranging from approximately -90 to 0). * **Models:** Llama-3-8B (left chart), Llama-3-70B (right chart). * **Datasets/Anchoring Methods (Legend):** * Q-Anchored (PopQA) - Blue solid line * A-Anchored (PopQA) - Orange dashed line * Q-Anchored (TriviaQA) - Purple solid line * A-Anchored (TriviaQA) - Green dashed line * Q-Anchored (HotpotQA) - Brown dashed-dotted line * A-Anchored (HotpotQA) - Red dashed-dotted line * Q-Anchored (NQ) - Teal solid line * A-Anchored (NQ) - Gray solid line ### Detailed Analysis or Content Details **Llama-3-8B (Left Chart):** * **Q-Anchored (PopQA):** The line starts at approximately ΔP = -2, decreases steadily to approximately ΔP = -70 at layer 25, and then plateaus. * **A-Anchored (PopQA):** The line starts at approximately ΔP = -1, decreases gradually to approximately ΔP = -50 at layer 25, and then plateaus. * **Q-Anchored (TriviaQA):** The line starts at approximately ΔP = -3, decreases rapidly to approximately ΔP = -60 at layer 15, and then continues to decrease to approximately ΔP = -80 at layer 30. * **A-Anchored (TriviaQA):** The line starts at approximately ΔP = -2, decreases gradually to approximately ΔP = -50 at layer 20, and then continues to decrease to approximately ΔP = -70 at layer 30. * **Q-Anchored (HotpotQA):** The line starts at approximately ΔP = -1, decreases rapidly to approximately ΔP = -60 at layer 10, and then continues to decrease to approximately ΔP = -75 at layer 30. * **A-Anchored (HotpotQA):** The line starts at approximately ΔP = -1, decreases gradually to approximately ΔP = -40 at layer 20, and then continues to decrease to approximately ΔP = -60 at layer 30. * **Q-Anchored (NQ):** The line starts at approximately ΔP = -2, decreases steadily to approximately ΔP = -60 at layer 20, and then continues to decrease to approximately ΔP = -75 at layer 30. * **A-Anchored (NQ):** The line starts at approximately ΔP = -1, decreases gradually to approximately ΔP = -50 at layer 20, and then continues to decrease to approximately ΔP = -65 at layer 30. **Llama-3-70B (Right Chart):** * **Q-Anchored (PopQA):** The line starts at approximately ΔP = -2, decreases steadily to approximately ΔP = -60 at layer 40, and then plateaus. * **A-Anchored (PopQA):** The line starts at approximately ΔP = -1, decreases gradually to approximately ΔP = -50 at layer 40, and then plateaus. * **Q-Anchored (TriviaQA):** The line starts at approximately ΔP = -3, decreases rapidly to approximately ΔP = -60 at layer 20, and then continues to decrease to approximately ΔP = -80 at layer 70. * **A-Anchored (TriviaQA):** The line starts at approximately ΔP = -2, decreases gradually to approximately ΔP = -50 at layer 30, and then continues to decrease to approximately ΔP = -70 at layer 70. * **Q-Anchored (HotpotQA):** The line starts at approximately ΔP = -1, decreases rapidly to approximately ΔP = -60 at layer 10, and then continues to decrease to approximately ΔP = -75 at layer 70. * **A-Anchored (HotpotQA):** The line starts at approximately ΔP = -1, decreases gradually to approximately ΔP = -40 at layer 20, and then continues to decrease to approximately ΔP = -60 at layer 70. * **Q-Anchored (NQ):** The line starts at approximately ΔP = -2, decreases steadily to approximately ΔP = -60 at layer 30, and then continues to decrease to approximately ΔP = -80 at layer 70. * **A-Anchored (NQ):** The line starts at approximately ΔP = -1, decreases gradually to approximately ΔP = -50 at layer 30, and then continues to decrease to approximately ΔP = -70 at layer 70. ### Key Observations * All lines exhibit a downward trend, indicating a decrease in ΔP as the layer number increases. * The rate of decrease varies depending on the dataset and anchoring method. * Q-Anchored lines generally decrease more rapidly than A-Anchored lines. * The Llama-3-70B model shows a more extended decrease in ΔP across more layers compared to the Llama-3-8B model. * The HotpotQA dataset consistently shows a steeper decline in ΔP compared to other datasets. ### Interpretation The charts demonstrate how the change in probability (ΔP) evolves across different layers of the Llama models for various question-answering tasks. The negative ΔP values suggest a decreasing confidence or probability associated with the model's predictions as information propagates through deeper layers. The difference between Q-Anchored and A-Anchored lines suggests that anchoring based on the question (Q-Anchored) leads to a more pronounced decrease in ΔP compared to anchoring based on the answer (A-Anchored). This could indicate that the question provides more informative cues for the model's reasoning process. The steeper decline observed for the HotpotQA dataset might be attributed to the complexity of the questions in this dataset, requiring more extensive reasoning and potentially leading to greater uncertainty in deeper layers. The extended decrease in ΔP for the Llama-3-70B model, compared to the Llama-3-8B model, could be a result of the larger model size and increased capacity for learning complex relationships, which also leads to a more nuanced and potentially less confident representation of information in deeper layers. The plateauing of the lines suggests a point where further processing through additional layers does not significantly alter the model's probability distribution.

## Line Charts: Layer-wise ΔP for Llama-3 Models ### Overview The image displays two side-by-side line charts comparing the change in probability (ΔP) across neural network layers for two different model sizes: Llama-3-8B (left) and Llama-3-70B (right). The charts analyze the performance of two anchoring methods (Q-Anchored and A-Anchored) across four different question-answering datasets. ### Components/Axes * **Titles:** * Left Chart: "Llama-3-8B" * Right Chart: "Llama-3-70B" * **Y-Axis (Both Charts):** Label is "ΔP". Scale ranges from -80 to 0, with major tick marks at intervals of 20 (-80, -60, -40, -20, 0). * **X-Axis (Left Chart - Llama-3-8B):** Label is "Layer". Scale ranges from 0 to 30, with major tick marks at 0, 10, 20, 30. * **X-Axis (Right Chart - Llama-3-70B):** Label is "Layer". Scale ranges from 0 to 80, with major tick marks at 0, 20, 40, 60, 80. * **Legend (Bottom Center):** Contains 8 entries, differentiating lines by color and style (solid vs. dashed). * **Q-Anchored (Solid Lines):** * Blue: Q-Anchored (PopQA) * Green: Q-Anchored (TriviaQA) * Purple: Q-Anchored (HotpotQA) * Pink: Q-Anchored (NQ) * **A-Anchored (Dashed Lines):** * Orange: A-Anchored (PopQA) * Red: A-Anchored (TriviaQA) * Gray: A-Anchored (HotpotQA) * Brown: A-Anchored (NQ) ### Detailed Analysis **Llama-3-8B Chart (Left):** * **Q-Anchored Lines (Solid):** All four solid lines (Blue, Green, Purple, Pink) exhibit a strong, consistent downward trend. They start near ΔP = 0 at Layer 0 and decline steeply, converging in a cluster between approximately -60 and -80 by Layer 30. The lines are tightly grouped, with the Pink line (NQ) appearing slightly higher (less negative) than the others in the mid-layers (10-20). * **A-Anchored Lines (Dashed):** All four dashed lines (Orange, Red, Gray, Brown) remain relatively flat and close to ΔP = 0 across all layers (0-30). They show minor fluctuations but no significant downward or upward trend, staying within a narrow band roughly between -10 and +5. **Llama-3-70B Chart (Right):** * **Q-Anchored Lines (Solid):** The same four solid lines show a similar downward trend but over a longer layer span. They start near 0 and decline to a cluster between -60 and -80 by Layer 80. The decline is less steep per layer compared to the 8B model. The Purple line (HotpotQA) appears to have the most pronounced dip around Layer 40 before recovering slightly. * **A-Anchored Lines (Dashed):** Consistent with the 8B model, the dashed lines remain stable near ΔP = 0 across all 80 layers, with minor noise. ### Key Observations 1. **Anchoring Method Dichotomy:** There is a stark and consistent contrast between the two anchoring methods across both model sizes and all four datasets. Q-Anchored performance (ΔP) degrades significantly with depth, while A-Anchored performance remains stable. 2. **Model Size Effect:** The trend for Q-Anchored lines is similar in shape but stretched across more layers in the larger (70B) model. The final ΔP values at the deepest layers are comparable (~ -80), but the rate of decline is slower in the 70B model. 3. **Dataset Similarity:** Within each anchoring group (Q or A), the lines for different datasets (PopQA, TriviaQA, HotpotQA, NQ) follow very similar trajectories, suggesting the observed phenomenon is robust across these QA benchmarks. 4. **Spatial Layout:** The legend is positioned centrally below both charts, allowing for direct cross-referencing. The charts share the same y-axis scale and label, facilitating direct comparison of the ΔP magnitude. ### Interpretation The data suggests a fundamental difference in how information is processed or retained across layers depending on the anchoring method. "ΔP" likely represents a change in probability or performance metric relative to a baseline. * **Q-Anchored (Question-Anchored):** The consistent negative trend indicates that as information propagates through deeper layers of the model, the probability or confidence associated with the question-anchored representation diminishes significantly. This could imply that deeper layers are less effective at maintaining or utilizing the initial question context, or that the representation becomes "diluted." * **A-Anchored (Answer-Anchored):** The stability near zero suggests that the answer-anchored representation is robustly maintained throughout the network's depth. The model's processing does not degrade the answer-related signal as it moves through the layers. The contrast between the 8B and 70B models for the Q-Anchored lines is particularly insightful. The more gradual decline in the larger model might indicate that increased model capacity allows for a better preservation of the question-anchored signal across a deeper architecture, even if the ultimate degradation at the final layer is similar. This visualization provides strong evidence that the choice of anchoring (question vs. answer) has a profound and systematic impact on internal model dynamics across layers.

## Line Graphs: Performance Comparison of Q-Anchored and A-Anchored Methods in LLaMA-3 Models ### Overview The image contains two side-by-side line graphs comparing the performance degradation (ΔP) of Q-Anchored and A-Anchored methods across different layers in LLaMA-3-8B and LLaMA-3-70B models. The graphs visualize how performance changes (ΔP) vary with model depth for four question-answering datasets: PopQA, TriviaQA, HotpotQA, and NQ. Each line represents a specific anchoring method and dataset combination. ### Components/Axes - **X-axis (Layer)**: Model depth, ranging from 0 to 30 for LLaMA-3-8B and 0 to 80 for LLaMA-3-70B. - **Y-axis (ΔP)**: Performance change, measured in arbitrary units (range: -80 to 0). - **Legends**: - **LLaMA-3-8B** (left graph): - Solid blue: Q-Anchored (PopQA) - Dashed orange: A-Anchored (PopQA) - Solid green: Q-Anchored (TriviaQA) - Dashed red: A-Anchored (TriviaQA) - Solid purple: Q-Anchored (HotpotQA) - Dashed pink: A-Anchored (HotpotQA) - Solid gray: Q-Anchored (NQ) - Dashed brown: A-Anchored (NQ) - **LLaMA-3-70B** (right graph): - Same color/style coding as above, with lines extending to layer 80. ### Detailed Analysis #### LLaMA-3-8B (Left Graph) - **Q-Anchored (PopQA)**: Starts at 0, slopes downward sharply to ~-80 by layer 30 (peak ΔP: -80). - **A-Anchored (PopQA)**: Remains near 0 with minor fluctuations (ΔP: -2 to 0). - **Q-Anchored (TriviaQA)**: Drops to ~-60 by layer 30 (ΔP: -60). - **A-Anchored (TriviaQA)**: Stable near 0 (ΔP: -1 to 0). - **Q-Anchored (HotpotQA)**: Declines to ~-70 (ΔP: -70). - **A-Anchored (HotpotQA)**: Stable near 0 (ΔP: -2 to 0). - **Q-Anchored (NQ)**: Plummets to ~-85 (ΔP: -85). - **A-Anchored (NQ)**: Stable near 0 (ΔP: -1 to 0). #### LLaMA-3-70B (Right Graph) - **Q-Anchored (PopQA)**: Starts at 0, dips to ~-60 by layer 40, then stabilizes (ΔP: -60). - **A-Anchored (PopQA)**: Fluctuates slightly above 0 (ΔP: -1 to 2). - **Q-Anchored (TriviaQA)**: Drops to ~-50 by layer 60 (ΔP: -50). - **A-Anchored (TriviaQA)**: Stable near 0 (ΔP: -1 to 1). - **Q-Anchored (HotpotQA)**: Declines to ~-75 by layer 80 (ΔP: -75). - **A-Anchored (HotpotQA)**: Stable near 0 (ΔP: -1 to 1). - **Q-Anchored (NQ)**: Plummets to ~-90 by layer 80 (ΔP: -90). - **A-Anchored (NQ)**: Stable near 0 (ΔP: -1 to 1). ### Key Observations 1. **Performance Degradation**: Q-Anchored methods show significant performance drops (ΔP) across all datasets, while A-Anchored methods remain stable (ΔP ≈ 0). 2. **Model Size Impact**: LLaMA-3-70B exhibits more gradual degradation than LLaMA-3-8B, suggesting larger models handle anchoring better. 3. **Dataset Variability**: NQ (Natural Questions) shows the steepest decline for Q-Anchored methods, indicating higher sensitivity to anchoring choices. 4. **Layer Stability**: A-Anchored methods maintain near-zero ΔP across all layers, while Q-Anchored methods degrade sharply in early layers. ### Interpretation The data demonstrates that **A-Anchored methods preserve performance stability** across model layers, whereas **Q-Anchored methods degrade significantly**, especially in smaller models (LLaMA-3-8B). The larger LLaMA-3-70B model mitigates this degradation but does not eliminate it. The NQ dataset’s extreme sensitivity to anchoring suggests it poses unique challenges for Q-Anchored approaches. These trends highlight the importance of anchoring strategy in maintaining model performance during scaling.