## Heatmaps: D16 Spectral Properties by Matrix Type and Layer
### Overview
The image contains two side-by-side heatmaps visualizing spectral properties of a D16 model. The left heatmap shows "Final Compression %" (50-100%), while the right shows "Final Weighted α" (0.1-0.6). Both matrices are organized by layer (0-15) on the x-axis and matrix type (Q, K, V, O, MLP↑, MLP↓) on the y-axis.
### Components/Axes
- **X-axis (Layer)**: 0 to 15 (integer increments)
- **Y-axis (Matrix Type)**:
- Q (Query)
- K (Key)
- V (Value)
- O (Other)
- MLP↑ (MLP Up)
- MLP↓ (MLP Down)
- **Color Scales**:
- Left: Dark red (100%) to light yellow (50%)
- Right: Yellow (0.6) to dark purple (0.1)
### Detailed Analysis
#### Final Compression %
- **Q, K, V, O**: Consistently high compression (85-100%) across all layers, with darkest red in layers 0-3.
- **MLP↑/MLP↓**: Lower compression (60-70%), with MLP↓ slightly higher than MLP↑.
- **Layer 0**: All matrix types show peak compression (90-100% for Q/K/V/O).
#### Final Weighted α
- **K**: Highest α values (0.5-0.6) in layers 0-3, dropping to 0.2-0.3 in later layers.
- **Q**: Moderate α (0.3-0.5), peaking in layers 0-2.
- **V**: Lower α (0.2-0.4), with a gradual decline after layer 3.
- **O**: Lowest α (0.1-0.3), minimal variation across layers.
- **MLP↑/MLP↓**: Very low α (0.1-0.2), with MLP↓ slightly higher than MLP↑.
### Key Observations
1. **Early Layer Dominance**: Layers 0-3 exhibit the highest compression and α values for Q/K/V/O.
2. **MLP Layer Consistency**: MLP↑/MLP↓ show minimal variation in both metrics, suggesting stable but low importance.
3. **α Decay**: α values for Q/K/V/O decrease significantly after layer 3, while compression remains high.
### Interpretation
The data suggests that early layers (0-3) of the D16 model are critical for both compression efficiency and spectral importance (α). Q/K/V/O matrices dominate these layers, indicating their role in capturing essential information. MLP layers (↑/↓) are less compressed and have lower α, implying they contribute less to the model's core functionality. The sharp decline in α after layer 3 may reflect diminishing returns in later layers, while sustained high compression suggests efficient parameter sharing or redundancy in later stages. This pattern aligns with typical transformer architectures, where early layers handle semantic features and later layers refine representations.
