## Line Graph: Log of Perplexity vs. RoPE Removal Ratio
### Overview
The graph compares the performance of various methods in maintaining low perplexity as the RoPE (Rotary Positional Encoding) removal ratio increases. The y-axis uses a logarithmic scale to emphasize differences in perplexity values across methods.
### Components/Axes
- **X-axis**: RoPE Removal Ratio (%) ranging from 0% to 80% in 20% increments.
- **Y-axis**: Log of Perplexity (scale from ~2 to ~8).
- **Legend**: Located on the right, with color-coded lines and markers for each method:
- **Vanilla PCA**: Green circles (flat line at ~8).
- **MHA2MLA**: Blue squares (flat until 80%, then spikes to ~5.5).
- **RoRoPE**: Orange diamonds (flat at ~2).
- **RoRoPE & 2D-FreqFold**: Yellow stars (flat at ~2).
- **RoRoPE & 4D-FreqFold**: Red diamonds (flat at ~2).
- **RoRoPE & 8D-FreqFold**: Orange stars (flat at ~2, slight upward trend).
### Detailed Analysis
1. **Vanilla PCA** (green circles):
- Remains nearly flat at ~8 across all removal ratios.
- Slight dip to ~7.8 at 80% removal.
2. **MHA2MLA** (blue squares):
- Stays near ~2 for 0–60% removal.
- Sharp increase to ~5.5 at 80% removal.
3. **RoRoPE** (orange diamonds):
- Flat at ~2 across all ratios.
4. **RoRoPE & 2D-FreqFold** (yellow stars):
- Flat at ~2 across all ratios.
5. **RoRoPE & 4D-FreqFold** (red diamonds):
- Flat at ~2 across all ratios.
6. **RoRoPE & 8D-FreqFold** (orange stars):
- Flat at ~2 for 0–60% removal.
- Slight upward trend to ~2.3 at 80% removal.
### Key Observations
- **Vanilla PCA** exhibits the highest perplexity but remains stable.
- **MHA2MLA** shows a dramatic increase in perplexity only at 80% removal, suggesting sensitivity to high removal ratios.
- **RoRoPE combinations** (2D, 4D, 8D-FreqFold) maintain the lowest perplexity (~2), with 8D-FreqFold showing marginal improvement at high removal ratios.
- The logarithmic y-axis amplifies the visual impact of MHA2MLA's spike at 80%.
### Interpretation
The data suggests that **RoRoPE-based methods** (especially with higher FreqFold dimensions like 8D) are more robust to RoPE removal, maintaining low perplexity even as removal ratios increase. **Vanilla PCA** performs poorly in terms of perplexity but is stable, while **MHA2MLA** struggles at high removal ratios. The slight upward trend in the 8D-FreqFold combination indicates potential scalability benefits, though further validation is needed. This highlights the trade-offs between stability and performance across different architectural choices.
