## Line Graph: Compression Gradient Evolution ### Overview The graph illustrates the evolution of the compression gradient (SR Gradient) over training steps for three model sizes: D8 (30.4M), D12 (92.8M), and D16 (285.2M). The y-axis measures the gradient difference between early and late training phases, while the x-axis represents training steps from 0 to 10,000. All three models exhibit similar trends with initial sharp declines followed by stabilization. ### Components/Axes - **X-axis (Training Step)**: Ranges from 0 to 10,000 in increments of 2,000. - **Y-axis (SR Gradient)**: Ranges from -200 to 0 in increments of 50. - **Legend**: Located in the bottom-right corner, mapping: - Blue circles: D8 (30.4M) - Orange squares: D12 (92.8M) - Green triangles: D16 (285.2M) ### Detailed Analysis 1. **Initial Drop (0–2,000 steps)**: - All models experience a sharp decline in SR Gradient. - D8 (blue) drops steeply to ~-200, then rebounds to ~-50 by step 1,000. - D12 (orange) declines to ~-150, then rises to ~-75 by step 1,000. - D16 (green) plunges to ~-200, then climbs to ~-100 by step 1,000. 2. **Stabilization Phase (2,000–10,000 steps)**: - All lines converge and flatten near 0. - D8 and D16 show minor fluctuations but remain tightly grouped. - D12 exhibits smoother convergence with less variability. ### Key Observations - **Model Size Correlation**: Larger models (D16) show deeper initial drops but similar stabilization patterns. - **Convergence**: By step 4,000, all models stabilize within ±10 of 0. - **Anomalies**: D8 and D16 exhibit sharper early-phase volatility compared to D12. ### Interpretation The data suggests that compression gradients initially diverge significantly during early training but converge as training progresses. Larger models (D16) demonstrate greater sensitivity to early training dynamics, while smaller models (D8) show more erratic early behavior. The stabilization near 0 implies that gradient magnitudes become negligible after ~4,000 steps, indicating potential convergence of the compression mechanism. The consistent flattening across model sizes suggests that gradient evolution is less dependent on model capacity beyond a certain training threshold.