## Density Plot: Saliency Score Distribution for Llama-3-8B and Llama-3-70B ### Overview The image presents two density plots comparing the saliency score distributions for Llama-3-8B and Llama-3-70B models. Each plot shows the distribution of saliency scores for two different question-answering datasets (TriviaQA and NQ) and two different types of saliency scores (S_Eq→E_A and S_Eq→*). The x-axis represents the saliency score, and the y-axis represents the density. ### Components/Axes * **Titles:** * Left Plot: Llama-3-8B * Right Plot: Llama-3-70B * **X-axis (Saliency Score):** * Left Plot: Range from 0.0 to 1.5, with markers at 0.0, 0.5, 1.0, and 1.5. * Right Plot: Range from 0.0 to 0.2, with markers at 0.0, 0.1, and 0.2. * **Y-axis (Density):** * Left Plot: Range from 0.00 to 0.75, with markers at 0.00, 0.25, 0.50, and 0.75. * Right Plot: Range from 0 to 4, with markers at 0, 2, and 4. * **Legend (located at the bottom):** * Blue: S_Eq→E_A (TriviaQA) * Green: S_Eq→E_A (NQ) * Orange: S_Eq→* (TriviaQA) * Red: S_Eq→* (NQ) ### Detailed Analysis **Llama-3-8B (Left Plot):** * **S_Eq→E_A (TriviaQA) - Blue:** The distribution peaks around 0.1, with a smaller peak around 0.5. The density gradually decreases after 0.5, extending to approximately 1.2. * **S_Eq→E_A (NQ) - Green:** The distribution has a sharp peak near 0.0, followed by a smaller peak around 0.2. The density decreases significantly after 0.2, extending to approximately 0.8. * **S_Eq→* (TriviaQA) - Orange:** The distribution peaks around 0.6, with a long tail extending to approximately 1.5. * **S_Eq→* (NQ) - Red:** The distribution peaks around 0.4, with a tail extending to approximately 1.0. **Llama-3-70B (Right Plot):** * **S_Eq→E_A (TriviaQA) - Blue:** The distribution peaks around 0.02, with a smaller peak around 0.07. The density decreases after 0.07, extending to approximately 0.18. * **S_Eq→E_A (NQ) - Green:** The distribution has a sharp peak near 0.0, followed by a smaller peak around 0.03. The density decreases significantly after 0.03, extending to approximately 0.1. * **S_Eq→* (TriviaQA) - Orange:** The distribution peaks around 0.1, with a tail extending to approximately 0.2. * **S_Eq→* (NQ) - Red:** The distribution peaks around 0.06, with a tail extending to approximately 0.15. ### Key Observations * For both models, the S_Eq→E_A scores tend to be concentrated at lower saliency scores compared to S_Eq→* scores. * The Llama-3-70B model shows a more concentrated distribution of saliency scores compared to the Llama-3-8B model, with the x-axis scale being much smaller. * The NQ dataset generally has lower saliency scores compared to the TriviaQA dataset for both models. ### Interpretation The density plots illustrate the distribution of saliency scores for different configurations of Llama models and datasets. The shift in distributions between the 8B and 70B models suggests that model size influences the saliency scores. The difference between TriviaQA and NQ datasets indicates that the nature of the questions and answers affects the saliency scores. The distinction between S_Eq→E_A and S_Eq→* suggests that the method of calculating saliency scores also plays a significant role. The data suggests that the Llama-3-70B model assigns lower saliency scores overall compared to the Llama-3-8B model, potentially indicating a more focused or refined attention mechanism.

\n ## Density Plots: Salience Score Distributions for Llama Models ### Overview The image presents two density plots, side-by-side, visualizing the distribution of "Saliency Scores" for two different Llama models: Llama-3-8B (left) and Llama-3-70B (right). Each plot displays the density of scores for two different datasets: TriviaQA and NQ (Natural Questions). The y-axis represents "Density", and the x-axis represents "Saliency Score". ### Components/Axes * **X-axis:** "Saliency Score". The scale ranges from approximately -0.5 to 1.5 for Llama-3-8B and from approximately -0.1 to 0.2 for Llama-3-70B. * **Y-axis:** "Density". The scale ranges from 0.0 to 0.75 for Llama-3-8B and from 0.0 to 4.0 for Llama-3-70B. * **Legend:** Located at the bottom of the image, the legend identifies the data series using color-coding: * Light Blue: `S_E_Q -> E_A (TriviaQA)` * Orange: `S_E_Q -> * (TriviaQA)` * Light Green: `S_E_Q -> E_A (NQ)` * Pink: `S_E_Q -> * (NQ)` * **Titles:** Each plot has a title indicating the Llama model being analyzed: "Llama-3-8B" (left) and "Llama-3-70B" (right). ### Detailed Analysis or Content Details **Llama-3-8B (Left Plot):** * **S_E_Q -> E_A (TriviaQA) - Light Blue:** This line exhibits a primary peak around a Saliency Score of approximately 0.1, with a secondary, smaller peak around 0.7. The density decreases towards both ends of the scale. * **S_E_Q -> * (TriviaQA) - Orange:** This line shows a broad distribution with a peak around a Saliency Score of 0.4. The density is relatively low across most of the range, with a slight increase towards the right. * **S_E_Q -> E_A (NQ) - Light Green:** This line has a prominent peak around a Saliency Score of 0.0, with a rapid decline in density as the score increases. * **S_E_Q -> * (NQ) - Pink:** This line displays a similar pattern to the green line, with a peak near 0.0, but with a slightly broader distribution and lower overall density. **Llama-3-70B (Right Plot):** * **S_E_Q -> E_A (TriviaQA) - Light Blue:** This line has a sharp peak around a Saliency Score of approximately 0.0, with a very rapid decline in density. * **S_E_Q -> * (TriviaQA) - Orange:** This line shows a peak around a Saliency Score of approximately 0.05, with a similar rapid decline in density as the blue line. * **S_E_Q -> E_A (NQ) - Light Green:** This line exhibits a peak around a Saliency Score of approximately 0.0, with a rapid decline in density. * **S_E_Q -> * (NQ) - Pink:** This line shows a peak around a Saliency Score of approximately 0.1, with a rapid decline in density. ### Key Observations * The Llama-3-70B model exhibits much higher density values overall compared to the Llama-3-8B model. * For both models, the distributions for the NQ dataset (green and pink lines) are generally concentrated closer to a Saliency Score of 0.0 than the distributions for the TriviaQA dataset (blue and orange lines). * The Llama-3-8B model shows a wider range of Saliency Scores, particularly for the TriviaQA dataset, compared to the Llama-3-70B model. * The `S_E_Q -> *` distributions (orange and pink) are generally lower in density than the `S_E_Q -> E_A` distributions (blue and green) for both models and datasets. ### Interpretation The plots demonstrate the distribution of saliency scores for two Llama models across two different question-answering datasets. Saliency scores likely represent the importance or relevance of certain parts of the input text when answering a question. The higher density values for the Llama-3-70B model suggest that it may be more confident or consistent in its saliency assessments compared to the Llama-3-8B model. The concentration of scores near 0.0 for the NQ dataset indicates that the model finds less need to highlight specific parts of the input text when answering questions from this dataset, potentially because the questions are more straightforward or the relevant information is more readily available. The difference between the `S_E_Q -> E_A` and `S_E_Q -> *` distributions suggests that the method used to calculate saliency (represented by `E_A` and `*`) impacts the resulting scores. The lower density of the `S_E_Q -> *` distributions could indicate that this method identifies fewer important parts of the input text. The wider distribution of scores for the Llama-3-8B model on the TriviaQA dataset suggests that this dataset presents more complex or ambiguous questions, requiring the model to consider a broader range of input text segments.

\n ## Density Plots: Saliency Score Distributions for Llama-3 Models ### Overview The image displays two side-by-side kernel density estimation (KDE) plots comparing the distribution of "Saliency Scores" for two different-sized language models (Llama-3-8B and Llama-3-70B) across two question-answering datasets (TriviaQA and NQ). The plots visualize how model attention or attribution (saliency) is distributed for different input-output mappings. ### Components/Axes * **Titles:** * Left Plot: `Llama-3-8B` * Right Plot: `Llama-3-70B` * **Axes:** * **X-axis (both plots):** `Saliency Score` * Llama-3-8B scale: 0.0 to 1.5, with major ticks at 0.0, 0.5, 1.0, 1.5. * Llama-3-70B scale: 0.0 to 0.2, with major ticks at 0.0, 0.1, 0.2. **Note the significant difference in scale between the two models.** * **Y-axis (both plots):** `Density` * Llama-3-8B scale: 0.00 to 0.75, with major ticks at 0.00, 0.25, 0.50, 0.75. * Llama-3-70B scale: 0 to 4, with major ticks at 0, 2, 4. * **Legend:** Located at the bottom center, spanning both plots. It defines four data series by color and label: 1. **Light Blue:** `S_{E_Q -> E_A} (TriviaQA)` - Saliency from Question Embedding to Answer Embedding for TriviaQA. 2. **Light Orange:** `S_{E_Q -> *} (TriviaQA)` - Saliency from Question Embedding to all tokens (`*`) for TriviaQA. 3. **Light Green:** `S_{E_Q -> E_A} (NQ)` - Saliency from Question Embedding to Answer Embedding for Natural Questions (NQ). 4. **Light Red/Pink:** `S_{E_Q -> *} (NQ)` - Saliency from Question Embedding to all tokens (`*`) for NQ. ### Detailed Analysis **Llama-3-8B (Left Plot):** * **Trend Verification:** All four distributions are right-skewed, with the bulk of density concentrated at lower saliency scores (0.0 to 0.75) and long tails extending to higher values. * **Data Series Analysis:** * `S_{E_Q -> E_A} (TriviaQA)` (Blue): Has the highest peak density (~0.8) at a very low score (~0.05). Shows a secondary, smaller peak/hump around 0.3. * `S_{E_Q -> E_A} (NQ)` (Green): Has the second-highest peak (~0.75) also near 0.05. Its distribution is slightly broader than the blue series, with a notable shoulder around 0.2. * `S_{E_Q -> *}` (TriviaQA & NQ) (Orange & Red): These distributions are much flatter and broader. Their peaks are lower (~0.4 for Orange, ~0.35 for Red) and occur at higher saliency scores (around 0.5-0.6). They have significantly longer and heavier tails extending past 1.0. **Llama-3-70B (Right Plot):** * **Trend Verification:** Distributions are more peaked and concentrated within a much narrower range (0.0 to 0.2) compared to the 8B model. They are less skewed. * **Data Series Analysis:** * `S_{E_Q -> E_A} (TriviaQA)` (Blue): Exhibits the highest and sharpest peak (density >4) at a very low score (~0.02). * `S_{E_Q -> E_A} (NQ)` (Green): Has the second-highest peak (~3.8) at a similarly low score (~0.03). It shows a distinct secondary peak/hump around 0.08. * `S_{E_Q -> *}` (TriviaQA & NQ) (Orange & Red): These are again broader and flatter than the `E_A` series. Their peaks are lower (density ~2.5-3) and occur at slightly higher scores (around 0.05-0.07). Their tails are shorter, mostly contained below 0.15. ### Key Observations 1. **Model Size Effect:** The 70B model's saliency scores are an order of magnitude smaller (x-axis max 0.2 vs. 1.5) and more densely concentrated (y-axis max 4 vs. 0.75) than the 8B model's. This suggests the larger model's attributions are more focused and consistent. 2. **Attribution Target Effect:** For both models and both datasets, the saliency from Question to Answer (`S_{E_Q -> E_A}`) distributions are sharper and peak at lower scores than the saliency from Question to all tokens (`S_{E_Q -> *}`). This indicates that attribution to the specific answer is more concentrated than diffuse attribution to the entire output. 3. **Dataset Effect:** The difference between datasets (TriviaQA vs. NQ) is less pronounced than the difference between models or attribution targets. However, for the `S_{E_Q -> E_A}` metric, the NQ distribution (Green) consistently shows a more prominent secondary hump compared to TriviaQA (Blue). ### Interpretation This visualization provides a technical comparison of model interpretability metrics. The data suggests that **larger models (70B) develop more precise and consistent internal attribution pathways** (lower, narrower saliency score distributions) compared to smaller models (8B), which have more variable and diffuse attributions. Furthermore, the analysis distinguishes between **targeted attribution** (to the answer) and **diffuse attribution** (to all tokens). The consistently sharper peaks for `S_{E_Q -> E_A}` imply that when the model's attention is measured specifically on the answer-generating process, the signal is cleaner and more localized. The broader `S_{E_Q -> *}` distributions reflect the noisier, more distributed nature of attention across an entire generated sequence. The secondary humps, particularly visible in the NQ `S_{E_Q -> E_A}` distributions for both models, may indicate a sub-population of questions or answer types where the model's saliency pattern differs systematically. This could be an avenue for further investigation into dataset properties or model behavior. The stark contrast in scale between the 8B and 70B models is the most striking finding, highlighting a fundamental shift in how larger models process and attribute information.

## Line Charts: Llama-3-8B and Llama-3-70B Saliency Score Distributions ### Overview Two side-by-side density plots compare saliency score distributions for two Llama-3 model variants (8B and 70B parameters). Each chart shows four overlapping density curves representing different evaluation scenarios, with distinct color coding for TriviaQA and NQ datasets. ### Components/Axes - **X-axis (Saliency Score)**: - Llama-3-8B: 0.0–1.5 - Llama-3-70B: 0.0–0.2 - **Y-axis (Density)**: - Llama-3-8B: 0.00–0.75 - Llama-3-70B: 0–4 - **Legend** (bottom center): - Blue: S_Eq → E_A (TriviaQA) - Green: S_Eq → E_A (NQ) - Orange: S_Eq → * (TriviaQA) - Red: S_Eq → * (NQ) ### Detailed Analysis **Llama-3-8B Chart**: - **Blue (TriviaQA)**: Peaks at ~0.5 with density ~0.6, tapering to ~0.1 at 1.5 - **Green (NQ)**: Peaks at ~0.4 with density ~0.7, broader spread than blue - **Orange (TriviaQA*)**: Peaks at ~0.6 with density ~0.5, flatter curve - **Red (NQ*)**: Peaks at ~0.7 with density ~0.4, most right-shifted distribution **Llama-3-70B Chart**: - **Blue (TriviaQA)**: Peaks at ~0.05 with density ~3.5, sharp drop-off - **Green (NQ)**: Peaks at ~0.03 with density ~4, narrowest distribution - **Orange (TriviaQA*)**: Peaks at ~0.07 with density ~2.5, wider than blue - **Red (NQ*)**: Peaks at ~0.09 with density ~2, most right-shifted in 70B model ### Key Observations 1. **Scale Differences**: 70B model's saliency scores are compressed (0–0.2 vs 0–1.5), suggesting different normalization or measurement scales. 2. **Peak Density**: 8B model shows higher maximum densities (up to 0.7 vs 4 in 70B), but 70B's values are scaled differently. 3. **Distribution Shapes**: - 8B model shows broader, more varied distributions - 70B model exhibits sharper, more concentrated peaks 4. **Dataset Performance**: - TriviaQA (blue/orange) consistently shows higher saliency scores than NQ (green/red) - NQ* (red) in 8B model has the highest saliency scores (~0.7) ### Interpretation The charts demonstrate that: 1. **Model Size Impact**: The 70B model shows more focused attention (narrower distributions) compared to 8B, but with lower absolute saliency values. 2. **Dataset Characteristics**: TriviaQA consistently elicits higher saliency scores than NQ across both models, suggesting different cognitive demands. 3. **Performance Tradeoffs**: While 70B has more precise attention (sharper peaks), 8B maintains broader coverage of saliency scores, potentially indicating better generalization. 4. **Normalization Concerns**: The stark difference in x-axis ranges between models suggests potential methodological differences in saliency score calculation or scaling between model sizes. The data implies that larger models develop more specialized attention patterns, but smaller models may retain broader contextual awareness. The consistent TriviaQA > NQ pattern across models suggests dataset-specific cognitive processing differences.