## Line Graphs: MHA2MLA and GQA2MLA Training Loss Curves
### Overview
Two line graphs compare training loss curves for different model configurations across 12,000 training steps. The graphs are labeled "MHA2MLA: SmolLM-1B7" (top) and "GQA2MLA: SmolLM-135M" (bottom). Each graph tracks four data series representing variations of a model architecture, with loss values decreasing over time.
### Components/Axes
- **X-axis**: "#Step" (0 to 12,000 in increments of 2,000).
- **Y-axis**: "Loss" (ranges from ~1.75 to 3.25).
- **Legend**: Positioned at the top-right corner of each graph.
- **Colors/Labels**:
- Black: `full-rope`
- Dark purple: `S2-norm`
- Light purple: `S2-norm + SVD_joint`
- Pink: `S2-norm + SVD_split`
### Detailed Analysis
#### MHA2MLA: SmolLM-1B7
- **full-rope** (black): Starts at ~2.75 loss, decreases steadily to ~1.9 by step 12,000.
- **S2-norm** (dark purple): Begins at ~2.5, drops to ~1.9, with minor fluctuations.
- **S2-norm + SVD_joint** (light purple): Starts at ~2.25, declines to ~2.0, showing slight volatility.
- **S2-norm + SVD_split** (pink): Initiates at ~2.5, reduces to ~2.0, with a smoother trajectory than `SVD_joint`.
#### GQA2MLA: SmolLM-135M
- **full-rope** (black): Starts at ~3.25, decreases to ~2.4, with sharper early declines.
- **S2-norm** (dark purple): Begins at ~2.75, drops to ~2.4, with moderate fluctuations.
- **S2-norm + SVD_joint** (light purple): Starts at ~2.5, reduces to ~2.3, with minor oscillations.
- **S2-norm + SVD_split** (pink): Initiates at ~2.75, declines to ~2.4, with a steadier curve than `SVD_joint`.
### Key Observations
1. **Consistent Trends**: All methods show a downward trend in loss, indicating improved performance over time.
2. **Performance Hierarchy**:
- `full-rope` and `S2-norm` outperform `SVD`-augmented variants in both graphs.
- `SVD_split` consistently lags behind `SVD_joint` in loss reduction.
3. **Initial Divergence**: The `full-rope` method exhibits the steepest initial drop in both graphs, suggesting faster early learning.
4. **Volatility**: `SVD_joint` and `SVD_split` lines show more fluctuations compared to `full-rope` and `S2-norm`.
### Interpretation
- **Model Efficiency**: The `full-rope` method achieves the lowest loss in both configurations, implying it is the most effective architecture for these tasks.
- **SVD Augmentation**: Adding SVD components (`SVD_joint` or `SVD_split`) to `S2-norm` does not significantly improve performance over the base `S2-norm` method, and may even introduce instability (evidenced by higher volatility).
- **Task-Specific Behavior**:
- MHA2MLA (1B7) starts with higher loss but converges faster than GQA2MLA (135M), which begins with a larger loss margin.
- The `SVD_split` method’s higher loss in both graphs suggests it may be less stable or less effective than other variants.
- **Practical Implications**: For resource-constrained scenarios, `full-rope` or `S2-norm` would be preferable over SVD-augmented methods, which offer diminishing returns.
### Spatial Grounding & Verification
- **Legend Placement**: Top-right corner in both graphs, ensuring clear visibility.
- **Color Consistency**:
- `full-rope` (black) matches all black lines.
- `S2-norm` (dark purple) aligns with dark purple lines.
- `SVD_joint` (light purple) and `SVD_split` (pink) are correctly mapped.
- **Axis Labels**: Positioned at the bottom (x-axis) and left (y-axis), with gridlines for reference.
### Content Details
- **Loss Values**:
- MHA2MLA: Initial loss ranges from 1.75–2.75; final loss ~1.9–2.0.
- GQA2MLA: Initial loss ranges from 2.25–3.25; final loss ~2.3–2.4.
- **Step Progression**: All lines show gradual convergence, with no plateaus or spikes beyond expected training noise.
### Notable Anomalies
- **SVD_split Underperformance**: Consistently higher loss than other methods, even when starting from similar initial values.
- **GQA2MLA Volatility**: `S2-norm + SVD_joint` exhibits sharper fluctuations compared to its counterpart in MHA2MLA.
### Conclusion
The graphs demonstrate that simpler architectures (`full-rope` and `S2-norm`) outperform SVD-augmented variants in reducing training loss. This suggests that SVD components may introduce unnecessary complexity without meaningful benefits for these tasks. Further investigation into the role of SVD in model training is warranted to clarify its utility.
