## Line Graphs: Frobenius Norm Comparison Across Layers
### Overview
The image contains two line graphs comparing the Frobenius norm (in keys) of different attention mechanisms across neural network layers. Graph (a) represents a 128M parameter model, while graph (b) represents a 2.5B parameter model. Both graphs compare **Standard MHA** (Multi-Head Attention) with **LRKV** (Layer-wise Residual Key-Value) variants: total, shared, and residual components.
---
### Components/Axes
#### Graph (a) 128M
- **X-axis (Layer)**: 0 to 10 layers (discrete intervals).
- **Y-axis (Frobenius norm)**: 0 to 3000 keys.
- **Legend**: Top-right corner, color-coded:
- Gray: Standard MHA
- Red: LRKV (total)
- Green: LRKV (shared)
- Orange: LRKV (residual)
- **LRKV/Standard Ratio**: 4.2× (annotated near bottom-left).
#### Graph (b) 2.5B
- **X-axis (Layer)**: 0 to 35 layers (discrete intervals).
- **Y-axis (Frobenius norm)**: 0 to 1400 keys.
- **Legend**: Top-right corner, identical color coding as graph (a).
- **LRKV/Standard Ratio**: 5.1× (annotated near bottom-left).
---
### Detailed Analysis
#### Graph (a) 128M
1. **Standard MHA**: Flat line at ~300 keys across all layers.
2. **LRKV (total)**:
- Starts at ~3000 keys (layer 0), drops sharply to ~800 keys at layer 1.
- Fluctuates between ~600–1000 keys for layers 2–10.
3. **LRKV (shared)**:
- Starts at ~500 keys (layer 0), stabilizes at ~600–700 keys for layers 2–10.
4. **LRKV (residual)**:
- Starts at ~400 keys (layer 0), stabilizes at ~500–600 keys for layers 2–10.
5. **Trend**: LRKV (total) dominates, with shared/residual components contributing smaller residuals.
#### Graph (b) 2.5B
1. **Standard MHA**: Flat line at ~200 keys across all layers.
2. **LRKV (total)**:
- Starts at ~1200 keys (layer 0), drops to ~800 keys by layer 5.
- Stabilizes at ~1000–1200 keys for layers 10–35.
3. **LRKV (shared)**:
- Starts at ~400 keys (layer 0), stabilizes at ~500–600 keys for layers 5–35.
4. **LRKV (residual)**:
- Starts at ~300 keys (layer 0), stabilizes at ~400–500 keys for layers 5–35.
5. **Trend**: LRKV (total) remains significantly higher than Standard MHA, with shared/residual components contributing smaller residuals.
---
### Key Observations
1. **LRKV Efficiency**:
- LRKV (total) consistently exceeds Standard MHA by 4.2× (128M) and 5.1× (2.5B), indicating higher parameter efficiency.
- Shared/residual components account for ~20–30% of LRKV (total) in both models.
2. **Layer-Specific Behavior**:
- Sharp initial drop in LRKV (total) at layer 1 (128M) and layer 5 (2.5B), suggesting architectural optimizations in early layers.
- Stabilization of residuals (shared/residual) after early layers implies layer-wise parameter sharing.
3. **Scalability**:
- Higher LRKV/Standard ratio in the 2.5B model (5.1× vs. 4.2×) suggests improved efficiency at scale.
---
### Interpretation
The data demonstrates that **LRKV outperforms Standard MHA in parameter efficiency**, with the gap widening in larger models (2.5B). The **shared and residual components** of LRKV reduce redundancy, enabling better scalability. The sharp initial drop in LRKV (total) may reflect architectural choices to minimize early-layer complexity, while stabilization in later layers indicates optimized parameter reuse. These trends align with efforts to reduce computational overhead in large language models while maintaining performance.
