## Line Graphs: RMS of RWKV State Entries and Average Stable Rank of WKV Matrices
### Overview
The image contains two line graphs comparing performance metrics of RWKV models (RWKV5-1.5B, RWKV6-1.6B, RWKV7-1.5B) across varying context lengths (1 to 8192 tokens). The left graph measures RMS of state entries, while the right graph tracks average stable rank of WKV matrices. Both graphs use logarithmic scales for context length and linear scales for their respective y-axes.
---
### Components/Axes
#### Left Graph: RMS of RWKV State Entries
- **X-axis**: Context Length (logarithmic scale: 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192)
- **Y-axis**: RMS (logarithmic scale: 10⁻¹ to 10¹)
- **Legend**:
- Blue: RWKV5-1.5B
- Orange: RWKV6-1.6B
- Green: RWKV7-1.5B
- **Positioning**: Legend in top-right corner; lines anchored to legend labels.
#### Right Graph: Average Stable Rank of WKV Matrices
- **X-axis**: Context Length (same logarithmic scale as left graph)
- **Y-axis**: Average Stable Rank (linear scale: 1.0 to 1.6)
- **Legend**: Same color coding as left graph (blue, orange, green).
- **Positioning**: Legend in top-right corner; lines anchored to legend labels.
---
### Detailed Analysis
#### Left Graph: RMS of RWKV State Entries
- **RWKV5-1.5B (Blue)**:
- Starts at ~10¹.² (≈15.85) at context length 1.
- Dips slightly at context length 2 (~10¹.1, ≈12.6).
- Rises steadily to ~10¹.5 (≈31.6) at 8192 tokens.
- **RWKV6-1.6B (Orange)**:
- Starts at ~10¹.1 (≈12.6) at context length 1.
- Increases gradually to ~10¹.4 (≈25.1) at 8192 tokens.
- **RWKV7-1.5B (Green)**:
- Remains flat at ~10⁻¹ (≈0.1) across all context lengths.
#### Right Graph: Average Stable Rank of WKV Matrices
- **RWKV5-1.5B (Blue)**:
- Starts at 1.0 at context length 1.
- Rises sharply to 1.6 at 8192 tokens.
- **RWKV6-1.6B (Orange)**:
- Starts at 1.0 at context length 1.
- Increases gradually to 1.5 at 8192 tokens.
- **RWKV7-1.5B (Green)**:
- Starts at 1.0 at context length 1.
- Rises steadily to 1.4 at 8192 tokens, with a minor dip at 4096 tokens (~1.35).
---
### Key Observations
1. **RMS Trends**:
- RWKV5-1.5B exhibits the highest RMS values, increasing exponentially with context length.
- RWKV7-1.5B maintains near-zero RMS, suggesting stable state management.
- RWKV6-1.6B shows intermediate RMS growth, slower than RWKV5-1.5B.
2. **Stable Rank Trends**:
- RWKV5-1.5B has the steepest increase in stable rank, indicating reduced stability at longer contexts.
- RWKV7-1.5B demonstrates the most stable behavior, with the lowest final stable rank (1.4 vs. 1.6 for RWKV5-1.5B).
- RWKV6-1.6B balances growth and stability, with a moderate increase to 1.5.
---
### Interpretation
1. **Model Efficiency**:
- RWKV7-1.5B’s near-zero RMS and low stable rank suggest superior state management efficiency, critical for long-context tasks.
- RWKV5-1.5B’s high RMS and stable rank imply higher computational variability, potentially limiting scalability.
2. **Context Length Impact**:
- All models show increased RMS and stable rank with longer contexts, but RWKV7-1.5B mitigates this effect most effectively.
- The divergence in trends highlights architectural differences: RWKV7’s design likely prioritizes stability over raw capacity.
3. **Anomalies**:
- RWKV7-1.5B’s slight dip in stable rank at 4096 tokens may indicate transient optimization or noise in the dataset.
- RWKV5-1.5B’s sharp rise in stable rank suggests sensitivity to context length, possibly due to memory constraints.
---
### Conclusion
The data underscores trade-offs between model size, stability, and context handling. RWKV7-1.5B’s performance aligns with use cases requiring long-context stability, while RWKV5-1.5B may suit scenarios prioritizing raw capacity despite higher variability. Further investigation into RWKV7’s architectural innovations (e.g., attention mechanisms, state initialization) could clarify its efficiency advantages.
