## Line Charts: Performance Comparison with Masked Heads ### Overview The image contains four line charts comparing the performance of "TopK" and "RandomK" methods across different tasks: Retrieval, Knowledge Recall, Math Calculation, and Inference. The x-axis represents the number of masked heads (16, 32, 64, 128), and the y-axis represents the score (from 0.0 to 1.0). Each chart displays two solid lines ("TopK Accuracy" and "TopK Comet") and two dashed lines ("RandomK Accuracy" and "RandomK Comet"). ### Components/Axes * **X-axis:** "# Masked Heads" with values 16, 32, 64, and 128. * **Y-axis:** "Score" ranging from 0.0 to 1.0 in increments of 0.2. * **Chart Titles:** Retrieval, Knowledge Recall, Math Calculation, Inference. * **Legend (Top):** * Blue solid line: "TopK Accuracy" * Blue dashed line: "RandomK Accuracy" * Red solid line: "TopK Comet" * Red dashed line: "RandomK Comet" ### Detailed Analysis #### Retrieval Chart * **TopK Accuracy (Blue Solid):** Starts at approximately 0.95 at 16 masked heads, decreases to approximately 0.55 at 32 masked heads, remains relatively stable around 0.60 until 64 masked heads, then drops sharply to approximately 0.0 at 128 masked heads. * **RandomK Accuracy (Blue Dashed):** Starts at approximately 0.95 and remains relatively stable between 0.95 and 0.80 across all values of masked heads. * **TopK Comet (Red Solid):** Starts at approximately 0.95 at 16 masked heads, decreases to approximately 0.75 at 32 masked heads, remains relatively stable around 0.75 until 64 masked heads, then decreases to approximately 0.40 at 128 masked heads. * **RandomK Comet (Red Dashed):** Starts at approximately 0.95 and remains relatively stable between 0.95 and 0.80 across all values of masked heads. #### Knowledge Recall Chart * **TopK Accuracy (Blue Solid):** Starts at approximately 0.90 at 16 masked heads, decreases to approximately 0.10 at 128 masked heads, with a slight increase at 64 masked heads. * **RandomK Accuracy (Blue Dashed):** Starts at approximately 0.95 and decreases to approximately 0.80 at 128 masked heads. * **TopK Comet (Red Solid):** Starts at approximately 0.95 at 16 masked heads, decreases to approximately 0.20 at 128 masked heads. * **RandomK Comet (Red Dashed):** Starts at approximately 0.95 and decreases to approximately 0.85 at 128 masked heads. #### Math Calculation Chart * **TopK Accuracy (Blue Solid):** Starts at approximately 0.95 at 16 masked heads, decreases to approximately 0.20 at 128 masked heads. * **RandomK Accuracy (Blue Dashed):** Starts at approximately 0.95 and decreases to approximately 0.90 at 128 masked heads. * **TopK Comet (Red Solid):** Starts at approximately 0.95 at 16 masked heads, decreases to approximately 0.60 at 128 masked heads. * **RandomK Comet (Red Dashed):** Starts at approximately 0.95 and decreases to approximately 0.90 at 128 masked heads. #### Inference Chart * **TopK Accuracy (Blue Solid):** Starts at approximately 0.95 at 16 masked heads, decreases to approximately 0.65 at 128 masked heads. * **RandomK Accuracy (Blue Dashed):** Starts at approximately 0.95 and decreases to approximately 0.80 at 128 masked heads. * **TopK Comet (Red Solid):** Starts at approximately 0.95 at 16 masked heads, decreases to approximately 0.75 at 128 masked heads. * **RandomK Comet (Red Dashed):** Starts at approximately 0.95 and decreases to approximately 0.85 at 128 masked heads. ### Key Observations * In all four tasks, the "RandomK Accuracy" and "RandomK Comet" lines (dashed) show more stable performance as the number of masked heads increases, compared to the "TopK Accuracy" and "TopK Comet" lines (solid). * The "TopK Accuracy" line experiences the most significant drop in performance, especially in the Retrieval and Knowledge Recall tasks. * The "TopK Comet" line also shows a decrease in performance, but not as drastic as the "TopK Accuracy" line. ### Interpretation The charts suggest that the "RandomK" methods are more robust to the masking of heads compared to the "TopK" methods. As the number of masked heads increases, the performance of "TopK" methods decreases significantly, indicating that these methods are more sensitive to the loss of information from specific heads. The "RandomK" methods, on the other hand, maintain a more stable performance, suggesting that they are better at utilizing the remaining information when heads are masked. This could be because "RandomK" methods distribute the attention more evenly across the heads, while "TopK" methods rely more heavily on a specific subset of heads. The Retrieval and Knowledge Recall tasks seem to be more affected by the masking of heads than the Math Calculation and Inference tasks, suggesting that these tasks may rely more on specific attention patterns.

## Line Chart: Performance of Different Models with Varying Masked Heads ### Overview The image presents four separate line charts, each representing the performance of different models (TopK Accuracy, RandomK Accuracy, TopK Comet, and RandomK Comet) across four different tasks: Retrieval, Knowledge Recall, Math Calculation, and Inference. The x-axis represents the number of masked heads, ranging from 16 to 128, while the y-axis represents the score, ranging from 0.0 to 1.0. ### Components/Axes * **X-axis Label:** "# Masked Heads" * **Y-axis Label:** "Score" * **Chart Titles (from left to right):** "Retrieval", "Knowledge Recall", "Math Calculation", "Inference" * **Legend:** * TopK Accuracy (Blue Solid Line) * RandomK Accuracy (Blue Dashed Line) * TopK Comet (Red Solid Line) * RandomK Comet (Red Dashed Line) ### Detailed Analysis or Content Details **1. Retrieval Chart:** * **TopK Accuracy (Blue Solid):** Starts at approximately 0.92 at 16 masked heads, sharply declines to approximately 0.05 at 64 masked heads, and then slightly increases to approximately 0.1 at 128 masked heads. * **RandomK Accuracy (Blue Dashed):** Starts at approximately 0.91 at 16 masked heads, remains relatively stable around 0.85-0.90 until 64 masked heads, then declines to approximately 0.75 at 128 masked heads. * **TopK Comet (Red Solid):** Starts at approximately 0.88 at 16 masked heads, declines to approximately 0.75 at 64 masked heads, and then increases to approximately 0.8 at 128 masked heads. * **RandomK Comet (Red Dashed):** Starts at approximately 0.89 at 16 masked heads, remains relatively stable around 0.85-0.90 until 64 masked heads, then declines to approximately 0.8 at 128 masked heads. **2. Knowledge Recall Chart:** * **TopK Accuracy (Blue Solid):** Starts at approximately 0.88 at 16 masked heads, declines to approximately 0.3 at 64 masked heads, and then increases to approximately 0.4 at 128 masked heads. * **RandomK Accuracy (Blue Dashed):** Starts at approximately 0.87 at 16 masked heads, remains relatively stable around 0.85-0.90 until 64 masked heads, then declines to approximately 0.75 at 128 masked heads. * **TopK Comet (Red Solid):** Starts at approximately 0.90 at 16 masked heads, declines to approximately 0.80 at 64 masked heads, and then remains relatively stable around 0.8 at 128 masked heads. * **RandomK Comet (Red Dashed):** Starts at approximately 0.89 at 16 masked heads, remains relatively stable around 0.85-0.90 until 64 masked heads, then declines to approximately 0.8 at 128 masked heads. **3. Math Calculation Chart:** * **TopK Accuracy (Blue Solid):** Starts at approximately 0.75 at 16 masked heads, declines to approximately 0.1 at 64 masked heads, and then increases to approximately 0.2 at 128 masked heads. * **RandomK Accuracy (Blue Dashed):** Starts at approximately 0.78 at 16 masked heads, declines to approximately 0.65 at 64 masked heads, and then remains relatively stable around 0.6 at 128 masked heads. * **TopK Comet (Red Solid):** Starts at approximately 0.85 at 16 masked heads, declines to approximately 0.7 at 64 masked heads, and then remains relatively stable around 0.7 at 128 masked heads. * **RandomK Comet (Red Dashed):** Starts at approximately 0.82 at 16 masked heads, declines to approximately 0.75 at 64 masked heads, and then remains relatively stable around 0.7 at 128 masked heads. **4. Inference Chart:** * **TopK Accuracy (Blue Solid):** Starts at approximately 0.85 at 16 masked heads, declines to approximately 0.7 at 64 masked heads, and then remains relatively stable around 0.7 at 128 masked heads. * **RandomK Accuracy (Blue Dashed):** Starts at approximately 0.86 at 16 masked heads, declines to approximately 0.75 at 64 masked heads, and then remains relatively stable around 0.7 at 128 masked heads. * **TopK Comet (Red Solid):** Starts at approximately 0.88 at 16 masked heads, declines to approximately 0.8 at 64 masked heads, and then remains relatively stable around 0.8 at 128 masked heads. * **RandomK Comet (Red Dashed):** Starts at approximately 0.87 at 16 masked heads, declines to approximately 0.8 at 64 masked heads, and then remains relatively stable around 0.8 at 128 masked heads. ### Key Observations * Generally, increasing the number of masked heads leads to a decrease in performance for TopK Accuracy across all tasks. * RandomK Accuracy tends to be more stable than TopK Accuracy as the number of masked heads increases. * TopK Comet and RandomK Comet generally outperform their respective non-Comet counterparts, especially at higher numbers of masked heads. * The most significant performance drop for TopK Accuracy is observed in the Retrieval and Math Calculation tasks. ### Interpretation The data suggests that increasing the number of masked heads negatively impacts the performance of the TopK Accuracy model, particularly in tasks requiring precise information retrieval (Retrieval) and mathematical reasoning (Math Calculation). The RandomK Accuracy model demonstrates more robustness to increasing masked heads, indicating that it may be less reliant on specific input features. The Comet models consistently outperform their non-Comet counterparts, suggesting that the Comet architecture provides a benefit in handling masked inputs. The varying degrees of performance decline across different tasks indicate that the sensitivity to masked heads is task-dependent. This could be due to the inherent complexity of each task and the model's ability to generalize from incomplete information. The consistent performance of the Comet models suggests that they are better equipped to handle the challenges posed by masked inputs, potentially through more effective attention mechanisms or representation learning.

## Multi-Panel Line Chart: Impact of Masked Attention Heads on Model Performance ### Overview The image displays a set of four line charts arranged horizontally, comparing the performance of two methods ("TopK" and "RandomK") across four different tasks as the number of masked attention heads increases. Each chart plots two metrics: "Accuracy" (solid lines, left y-axis) and "Comet" (dashed lines, right y-axis). The overall trend shows that performance, particularly Accuracy, degrades as more heads are masked, with the "TopK" method generally being more sensitive than "RandomK". ### Components/Axes * **Legend:** Positioned at the top center of the entire figure. It defines four data series: * `TopK Accuracy`: Solid blue line with circular markers. * `RandomK Accuracy`: Dashed blue line with circular markers. * `TopK Comet`: Solid red line with circular markers. * `RandomK Comet`: Dashed red line with circular markers. * **X-Axis (Common to all subplots):** Labeled "# Masked Heads". It has major tick marks at 16, 32, 64, and 128. The scale appears to be logarithmic (base 2). * **Y-Axis (Left, for Accuracy):** Labeled "Score" on the leftmost chart. The scale ranges from 0.0 to 1.0 with increments of 0.2. * **Y-Axis (Right, for Comet):** Not explicitly labeled with text, but implied by the dashed lines and the legend. The scale is also 0.0 to 1.0. * **Subplot Titles:** Each of the four panels has a title at its top center: 1. **Retrieval** (Leftmost panel) 2. **Knowledge Recall** (Second from left) 3. **Math Calculation** (Third from left) 4. **Inference** (Rightmost panel) ### Detailed Analysis **1. Retrieval Task (Left Panel)** * **TopK Accuracy (Solid Blue):** Starts high (~0.95 at 0-16 heads). Drops sharply after 32 heads, falling to ~0.55 at 64 heads, and plummets to near 0.0 by 128 heads. * **RandomK Accuracy (Dashed Blue):** Starts high (~0.95). Shows a very gradual, slight decline, remaining above ~0.85 even at 128 heads. * **TopK Comet (Solid Red):** Starts high (~0.95). Declines steadily after 16 heads, reaching ~0.4 by 128 heads. * **RandomK Comet (Dashed Red):** Starts high (~0.95). Remains very stable and high, showing only a minimal decrease to ~0.9 by 128 heads. **2. Knowledge Recall Task (Second Panel)** * **TopK Accuracy (Solid Blue):** Starts around ~0.8. Drops dramatically after 16 heads, hitting a low of ~0.3 at 32 heads, recovers slightly to ~0.45 at 64 heads, then falls to near 0.0 by 128 heads. * **RandomK Accuracy (Dashed Blue):** Starts around ~0.8. Declines gradually and linearly to ~0.75 by 128 heads. * **TopK Comet (Solid Red):** Starts around ~0.95. Declines steadily to ~0.4 by 128 heads. * **RandomK Comet (Dashed Red):** Starts around ~0.95. Remains very stable, ending near ~0.85. **3. Math Calculation Task (Third Panel)** * **TopK Accuracy (Solid Blue):** Starts high (~0.95). Declines gradually until 64 heads (~0.7), then drops sharply to ~0.35 at 128 heads. * **RandomK Accuracy (Dashed Blue):** Starts high (~0.95). Shows a very slow, linear decline to ~0.8 by 128 heads. * **TopK Comet (Solid Red):** Starts high (~0.95). Remains stable until 64 heads, then drops sharply to ~0.6 at 128 heads. * **RandomK Comet (Dashed Red):** Starts high (~0.95). Remains very stable, ending near ~0.85. **4. Inference Task (Right Panel)** * **TopK Accuracy (Solid Blue):** Starts around ~0.85. Shows a fluctuating but generally downward trend, with a notable dip at 32 heads (~0.75), a recovery at 64 heads (~0.85), and a final drop to ~0.5 by 128 heads. * **RandomK Accuracy (Dashed Blue):** Starts around ~0.85. Declines very gradually to ~0.7 by 128 heads. * **TopK Comet (Solid Red):** Starts around ~0.95. Fluctuates but maintains a high level, ending near ~0.85. * **RandomK Comet (Dashed Red):** Starts around ~0.95. Remains very stable and high, ending near ~0.9. ### Key Observations 1. **Method Sensitivity:** The "TopK" method (solid lines) is consistently and significantly more sensitive to the number of masked heads than the "RandomK" method (dashed lines). This is true for both Accuracy and Comet metrics across all tasks. 2. **Metric Divergence:** For the "TopK" method, the Accuracy metric (blue solid) degrades much more severely and rapidly than the Comet metric (red solid). For "RandomK", both metrics remain relatively stable. 3. **Task Variability:** The "Knowledge Recall" task shows the most severe and early drop in TopK Accuracy. The "Inference" task shows the most fluctuation in its TopK Accuracy trend. 4. **Threshold Effect:** For TopK Accuracy, there appears to be a critical threshold between 32 and 64 masked heads where performance begins to collapse in most tasks (Retrieval, Math Calculation). ### Interpretation This data demonstrates a fundamental difference between two strategies for selecting which attention heads to mask. The "RandomK" approach is highly robust; masking heads randomly has a minimal negative impact on both task accuracy and the "Comet" metric (likely a measure of output quality or coherence). This suggests that many attention heads are redundant or can be compensated for by others. In stark contrast, the "TopK" approach—presumably masking the heads deemed most important by some criterion—is highly destructive. The catastrophic drop in Accuracy indicates that these "TopK" heads are indeed critical for the model's task performance. The fact that the Comet score degrades more slowly suggests that while the model's ability to produce a correct answer (Accuracy) is crippled, its general output quality or fluency (Comet) is somewhat more resilient, though still negatively affected. The findings imply that the model's knowledge and reasoning capabilities are concentrated in a subset of attention heads. Identifying and preserving these heads is crucial for maintaining performance under parameter reduction or efficiency constraints. Conversely, random pruning is a surprisingly effective strategy for reducing model size with minimal performance loss. The variability across tasks (e.g., Knowledge Recall being most sensitive) also indicates that different capabilities rely on different internal structures within the model.

## Line Graphs: Accuracy Metrics vs. Masked Heads ### Overview The image contains four line graphs comparing accuracy metrics across four tasks (Retrieval, Knowledge Recall, Math Calculation, Inference) as the number of masked heads increases (16, 32, 64, 128). Each graph tracks four metrics: TopK Accuracy (solid blue), RandomK Accuracy (dashed blue), TopK Comet (solid red), and RandomK Comet (dashed red). Scores range from 0 to 1.0. ### Components/Axes - **X-axis**: "# Masked Heads" (16, 32, 64, 128) - **Y-axis**: "Score" (0.0 to 1.0) - **Legends**: - Top-left: TopK Accuracy (solid blue), RandomK Accuracy (dashed blue), TopK Comet (solid red), RandomK Comet (dashed red) - **Subplots**: - Top-left: Retrieval - Top-right: Knowledge Recall - Bottom-left: Math Calculation - Bottom-right: Inference ### Detailed Analysis #### Retrieval - **TopK Accuracy**: Starts at ~0.95 (16 masked heads), drops sharply to ~0.8 (32), ~0.6 (64), and ~0.2 (128). - **RandomK Accuracy**: Remains stable (~0.8) across all masked heads. - **TopK Comet**: Declines gradually from ~0.95 to ~0.75. - **RandomK Comet**: Stable (~0.85) with minor fluctuations. #### Knowledge Recall - **TopK Accuracy**: Starts at ~0.9, drops to ~0.7 (32), ~0.5 (64), and ~0.3 (128). - **RandomK Accuracy**: Stable (~0.8) with a slight dip at 64 (~0.75). - **TopK Comet**: Declines from ~0.9 to ~0.6. - **RandomK Comet**: Stable (~0.85). #### Math Calculation - **TopK Accuracy**: Starts at ~0.95, drops to ~0.8 (32), ~0.6 (64), and ~0.4 (128). - **RandomK Accuracy**: Stable (~0.85) with a minor dip at 64 (~0.8). - **TopK Comet**: Declines from ~0.95 to ~0.75. - **RandomK Comet**: Stable (~0.85). #### Inference - **TopK Accuracy**: Starts at ~0.9, drops to ~0.7 (32), ~0.6 (64), and ~0.4 (128). - **RandomK Accuracy**: Stable (~0.85) with a slight dip at 64 (~0.8). - **TopK Comet**: Declines from ~0.9 to ~0.75. - **RandomK Comet**: Stable (~0.85). ### Key Observations 1. **TopK metrics degrade sharply** as masked heads increase, especially in Retrieval and Math Calculation. 2. **RandomK metrics remain stable** across all tasks and masked heads, suggesting robustness. 3. **TopK Comet** consistently outperforms RandomK Comet in Retrieval and Math Calculation but underperforms in Knowledge Recall and Inference. 4. **RandomK Comet** maintains near-constant performance (~0.85) across all tasks. ### Interpretation The data suggests that **TopK methods are sensitive to masked heads**, with performance collapsing as masking increases. In contrast, **RandomK methods show resilience**, maintaining stable scores regardless of masking. The Comet metrics (TopK/RandomK) appear more robust than Accuracy metrics, particularly in Knowledge Recall and Inference. This implies that Comet-based evaluations might better capture task-specific nuances under varying masking conditions. The sharp decline in TopK Accuracy for Retrieval and Math Calculation at 128 masked heads highlights a critical vulnerability in these methods when extensive masking is applied.