## Line Chart: KV-Cache Memory Usage vs. Latent Dimension Ratio
### Overview
The chart compares KV-Cache memory usage (MB/token) across three methods (MLA KV-Cache, MLA+RoPE KV-Cache, and MHA Baseline) as the latent dimension ratio (r) increases from d/8 to d. The MHA Baseline remains constant, while the other two methods show increasing memory usage with higher r values.
### Components/Axes
- **X-axis**: Latent Dimension Ratio (r) with markers at d/8, d/4, d/2, 3d/4, d.
- **Y-axis**: KV-Cache Memory Usage (MB/token), ranging from 0 to 0.035 in increments of 0.005.
- **Legend**:
- Orange: MLA KV-Cache Memory
- Purple: MLA+RoPE KV-Cache Memory
- Blue dashed: MHA Baseline
### Detailed Analysis
1. **MLA KV-Cache Memory (Orange Line)**:
- Starts at ~0.004 MB/token at r = d/8.
- Increases to ~0.008 MB/token at r = d/4.
- Rises to ~0.016 MB/token at r = d/2.
- Further increases to ~0.022 MB/token at r = 3d/4.
- Reaches ~0.029 MB/token at r = d.
2. **MLA+RoPE KV-Cache Memory (Purple Line)**:
- Starts slightly below MLA at ~0.0035 MB/token at r = d/8.
- Follows a similar upward trend, reaching ~0.028 MB/token at r = d.
- Remains consistently ~0.001–0.002 MB/token lower than MLA across all r values.
3. **MHA Baseline (Blue Dashed Line)**:
- Constant at ~0.029 MB/token for all r values.
### Key Observations
- Both MLA and MLA+Rope KV-Cache memory usage increases monotonically with r, approaching the MHA Baseline as r approaches d.
- The MHA Baseline remains flat, suggesting fixed memory requirements independent of r.
- The gap between MLA+RoPE and MLA narrows as r increases, with MLA+RoPE maintaining a ~10–15% lower memory footprint.
### Interpretation
The data suggests that:
1. **Latent Dimension Ratio Impact**: Higher r values (closer to d) reduce the memory efficiency gap between MLA/MLA+RoPE and the MHA Baseline. This implies that larger latent dimensions may mitigate memory overhead differences between these methods.
2. **MLA+RoPE Optimization**: The persistent lower memory usage of MLA+RoPE indicates potential architectural optimizations (e.g., RoPE integration) that reduce KV-Cache demands compared to vanilla MLA.
3. **Baseline Consistency**: The MHA Baseline's flat line suggests it represents a theoretical upper bound or fixed reference point, possibly reflecting hardware constraints or fixed memory allocation strategies.
The trends highlight trade-offs between model complexity (via r) and memory efficiency, with MLA+RoPE offering a middle ground between performance and resource usage.
