## Bar Chart: Attention Sink Compression Ablation
### Overview
This image displays a grouped bar chart titled "(a) Attention sink compression ablation." It compares the performance scores of two distinct methods—"kvtc" and "kvtc + sink compression"—across five different benchmarks: GSM8K, MMLU, QASPER, LITM, and RULER-VT. The chart illustrates the impact of applying "sink compression" to the "kvtc" method.
### Components/Axes
* **Y-Axis:** Labeled "Score," representing performance metrics on a scale from 0 to 100, with horizontal grid lines at 10-unit intervals.
* **X-Axis:** Categorical labels representing five benchmarks: GSM8K, MMLU, QASPER, LITM, and RULER-VT.
* **Legend (Top-Left):**
* **Dark Grey Bar:** Represents "kvtc + sink compression."
* **Light Green Bar:** Represents "kvtc."
* **Caption (Bottom):** "(a) Attention sink compression ablation."
### Detailed Analysis
The following values are approximate visual estimates based on the provided grid lines:
| Benchmark | kvtc (Light Green) | kvtc + sink compression (Dark Grey) |
| :--- | :--- | :--- |
| **GSM8K** | ~57 | ~1.5 |
| **MMLU** | ~60 | ~10 |
| **QASPER** | ~38 | ~28 |
| **LITM** | ~90 | 0 (No bar visible) |
| **RULER-VT** | ~96 | ~62 |
**Trend Verification:**
* **kvtc (Light Green):** Shows a generally high performance across all categories, with the lowest score at QASPER (~38) and the highest at RULER-VT (~96).
* **kvtc + sink compression (Dark Grey):** Shows a significant performance deficit compared to the baseline. It starts near zero at GSM8K, shows a slight increase at MMLU and QASPER, is absent at LITM, and reaches its peak at RULER-VT (~62).
### Key Observations
* **Consistent Underperformance:** The "kvtc + sink compression" method consistently scores lower than the "kvtc" method across every benchmark tested.
* **Catastrophic Failure at LITM:** The "kvtc + sink compression" method appears to fail entirely on the LITM benchmark, as there is no visible bar, indicating a score of 0 or negligible performance.
* **Performance Gap:** The gap between the two methods is most pronounced in the LITM and GSM8K benchmarks. The gap is narrowest in the QASPER benchmark.
* **Relative Improvement:** While "kvtc + sink compression" performs poorly, it does show a trend of increasing scores as the benchmarks progress from left to right (excluding the LITM anomaly), suggesting it may be more effective on specific types of tasks (like RULER-VT) than others.
### Interpretation
This ablation study demonstrates that the "sink compression" technique is highly detrimental to the performance of the "kvtc" model.
The data suggests that "sink compression" likely causes significant information loss or "catastrophic forgetting" of the context required to solve these reasoning and retrieval tasks. The fact that the performance drops to near-zero in benchmarks like GSM8K and LITM indicates that the compression mechanism is likely discarding critical tokens or attention heads necessary for the model to function correctly.
From a technical standpoint, this chart serves as a negative result for the "sink compression" method, proving that while it may be intended to optimize memory or compute, the trade-off in accuracy is severe and unacceptable for these specific benchmarks. The "kvtc" baseline is clearly superior, and the "sink compression" modification requires significant refinement or a different approach to be viable.
