## Line Graphs: PCA Eigenvalue Spectra in Bilinear Form Space (2.5B)
### Overview
The image contains three line graphs comparing PCA eigenvalue spectra across three transformer layers (Early: L=0, Middle: L=18, Late: L=35). Each graph plots PCA index (x-axis) against variance (y-axis) for five methods: Standard MHA, LRKV (r=64), GQA, MQA, and MLA. The graphs reveal how variance distribution across principal components (PCs) differs by layer and method.
### Components/Axes
- **X-axis**: "PC Index" (2.5 to 17.5, increments of 2.5)
- **Y-axis**: "Variance (PCA Eigenvalue)" (0 to 7.5 for Early Layer, 0 to 2.5 for Middle Layer, 0 to 1.4 for Late Layer)
- **Legends**:
- Standard MHA (blue)
- LRKV (r=64) (orange)
- GQA (green)
- MQA (red)
- MLA (purple)
### Detailed Analysis
#### Early Layer (L=0)
- **Standard MHA**: Starts at ~7.5 variance at PC Index 2.5, drops sharply to ~0.5 by PC Index 17.5.
- **LRKV (r=64)**: Begins at ~3.5, declines to ~0.2.
- **GQA**: Starts at ~1.5, decreases to ~0.3.
- **MQA**: Begins at ~0.8, drops to ~0.1.
- **MLA**: Starts at ~0.5, declines to ~0.05.
- **Trend**: All methods show rapid variance decay after PC Index 5. Standard MHA retains the highest early variance.
#### Middle Layer (L=18)
- **Standard MHA**: Starts at ~2.5, declines to ~1.0.
- **LRKV (r=64)**: Begins at ~1.2, decreases to ~0.8.
- **GQA**: Starts at ~1.0, drops to ~0.6.
- **MQA**: Begins at ~0.9, declines to ~0.4.
- **MLA**: Starts at ~0.8, decreases to ~0.3.
- **Trend**: Variance decay slows after PC Index 10. Standard MHA maintains the highest values.
#### Late Layer (L=35)
- **Standard MHA**: Starts at ~1.4, drops to ~0.8.
- **LRKV (r=64)**: Begins at ~1.0, decreases to ~0.6.
- **GQA**: Starts at ~1.2, drops to ~0.5.
- **MQA**: Begins at ~1.3, declines to ~0.4.
- **MLA**: Starts at ~1.1, decreases to ~0.3.
- **Trend**: Sharp variance drop after PC Index 15. GQA and MQA show steeper late-stage declines.
### Key Observations
1. **Layer-Specific Variance**: Early Layer retains the highest variance overall, while Late Layer shows the most pronounced decay.
2. **Method Differences**:
- Standard MHA consistently retains the highest early variance.
- MLA exhibits the lowest variance across all layers.
- LRKV (r=64) and GQA show intermediate decay rates.
3. **Anomalies**:
- Late Layer graphs exhibit abrupt drops near PC Index 17.5, suggesting potential numerical instability or sparsity in later PCs.
- MQA and GQA show similar decay patterns but with higher initial variance in Late Layer.
### Interpretation
The data suggests that PCA eigenvalue spectra vary significantly by transformer layer and attention mechanism. Standard MHA retains more early-stage variance, indicating stronger feature representation in initial PCs. The sharp late-layer drops (especially in GQA and MQA) may reflect overfitting or reduced information content in deeper layers. MLA's consistent low variance suggests a more regularized or efficient design. These patterns align with expectations for attention mechanisms balancing capacity and generalization across transformer depths.
