## Line Chart: Acoustic MLM Loss on Codebook-0 vs Training Steps ### Overview The chart visualizes the convergence behavior of four different training configurations for an Acoustic MLM model. All lines show rapid initial loss reduction followed by stabilization, with notable differences in convergence speed and stability. ### Components/Axes - **X-axis**: Training Step (0K to 100K, logarithmic scale) - **Y-axis**: Acoustic MLM Loss on Codebook-0 (5 to 10, linear scale) - **Legend**: Located in top-right corner with four entries: 1. Blue circles: Pre-Norm | Gradient Clip=10 | Run 1 2. Orange triangles: Pre-Norm | Gradient Clip=1 | Run 2 3. Green triangles: Post-Norm | DeepNorm | Gradient Clip=1 4. Pink triangles: Pre-Norm | Attn. Relax | Gradient Clip=1 ### Detailed Analysis 1. **Blue Line (Gradient Clip=10)**: - Starts at ~9.5 loss at 0K - Drops sharply to ~5.8 by 10K - Stabilizes with minor fluctuations (~5.7-5.8) after 20K - Fastest convergence among all configurations 2. **Green Line (Post-Norm | DeepNorm)**: - Starts at ~9.5 loss at 0K - Sharp decline to ~5.9 by 15K - Abrupt spike to ~9.2 at 20K (anomaly) - Resumes decline to ~5.7 by 30K - Stabilizes with minor fluctuations (~5.6-5.7) after 40K 3. **Pink Line (Attn. Relax)**: - Starts at ~9.4 loss at 0K - Gradual decline to ~5.8 by 30K - Stabilizes with minor fluctuations (~5.7-5.8) after 40K 4. **Orange Line (Gradient Clip=1)**: - Starts at ~9.3 loss at 0K - Slowest decline, reaching ~5.9 by 50K - Stabilizes with minor fluctuations (~5.8-5.9) after 60K ### Key Observations - All configurations show similar initial loss values (~9.3-9.5) - Gradient Clip=10 (blue) achieves lowest final loss (~5.7) - Post-Norm (green) has fastest initial convergence but includes an anomalous spike - Gradient Clip=1 (orange) shows slowest convergence - Attn. Relax (pink) performs between Gradient Clip=10 and Gradient Clip=1 ### Interpretation The data suggests that higher gradient clipping (10) enables faster and more stable convergence compared to lower clipping (1). The Post-Norm configuration with DeepNorm demonstrates rapid initial improvement but includes an unexplained anomaly at 20K training steps, potentially indicating instability in that specific configuration. The Attn. Relax method shows intermediate performance, suggesting attention relaxation mechanisms provide moderate benefits without the instability seen in Post-Norm. The consistent stabilization patterns across all lines after ~30K-40K steps indicate that most configurations reach their asymptotic loss values within this range.