## Line Graph: Average Loss Across Token Positions
### Overview
The image displays a multi-line graph comparing the average loss of five different RWKV models across token positions (1k-32k). The y-axis represents average loss values (2.1-2.6), while the x-axis shows token positions in 2k increments. Five distinct data series are plotted with unique colors and markers.
### Components/Axes
- **X-axis (Token Positions)**: Labeled "Token Positions" with markers at 1k, 2k, ..., 32k intervals.
- **Y-axis (Average Loss)**: Labeled "Average Loss" with increments of 0.1 from 2.1 to 2.6.
- **Legend**: Located in the top-right corner, mapping colors to models:
- Blue: RWKV7-2.9B-World
- Green: RWKV6-3B-World
- Orange: RWKV5-3B-World
- Red: RWKV4-3B-World
- Purple: RWKV7-2.9B-World3-128k-tuned
### Detailed Analysis
1. **RWKV4-3B-World (Red Line)**:
- Starts at ~2.48 (1k) and exhibits a sharp upward spike to ~2.6 at token 12k.
- Declines to ~2.42 by token 32k.
- **Key Outlier**: The 12k token position shows a 0.18 loss increase compared to neighboring positions.
2. **RWKV7-2.9B-World3-128k-tuned (Purple Line)**:
- Maintains the lowest average loss (~2.12-2.18) across all positions.
- Shows minimal fluctuation, with a slight dip to ~2.11 at token 10k.
3. **RWKV7-2.9B-World (Blue Line)**:
- Starts at ~2.3 (1k) and fluctuates between ~2.15-2.25.
- Peaks at ~2.28 at token 31k before stabilizing at ~2.25 at 32k.
4. **RWKV6-3B-World (Green Line)**:
- Begins at ~2.37 (1k) and trends downward to ~2.26 at 32k.
- Shows periodic fluctuations, peaking at ~2.29 at token 29k.
5. **RWKV5-3B-World (Orange Line)**:
- Starts at ~2.39 (1k) and trends downward to ~2.31 at 32k.
- Exhibits a sawtooth pattern with peaks at tokens 18k (~2.34) and 30k (~2.33).
### Key Observations
- **Performance Hierarchy**: The tuned model (purple) consistently outperforms others, maintaining ~15% lower loss than the worst-performing model (red).
- **Anomaly Detection**: The red line's 12k spike suggests a potential bug or training instability in RWKV4-3B-World.
- **Model Scaling**: Newer models (RWKV7 variants) generally show better loss compression than older versions (RWKV4-6).
- **Positional Sensitivity**: All models exhibit increased loss at higher token positions, with RWKV4-3B-World showing the most dramatic degradation.
### Interpretation
The data demonstrates significant performance disparities between RWKV model versions, with the 128k-tuned variant achieving state-of-the-art results. The red line's anomalous spike at token 12k warrants investigation into potential training data corruption or architectural limitations in the RWKV4-3B-World model. The consistent performance of the tuned model suggests that extended training on larger datasets (128k tokens) significantly improves generalization. These findings highlight the importance of model versioning and hyperparameter tuning in transformer-based architectures.
