## Heatmap: LLaMA-7b Channel-Key and Channel-Value Correlation Patterns Across Layers ### Overview The image presents two sets of heatmaps comparing correlation patterns between channels and keys/values across 8 layers of the LLaMA-7b neural network architecture. The top row shows "Key" correlations, while the bottom row shows "Value" correlations. Each heatmap visualizes 321 channels against 16 key/value dimensions across layers 1, 2, 13, 14, 28, 29, 31, and 32. ### Components/Axes - **X-axis**: Channels (1 to 321) - **Y-axis**: Correlations (1 to 32) - **Legend**: Color scale from -1.0 (blue) to 1.0 (red), with 0.0 as white - **Layer labels**: Positioned above each heatmap column (Layer 1, 2, 13, 14, 28, 29, 31, 32) - **Diagonal reference**: Red dashed line indicating perfect correlation (1.0) ### Detailed Analysis **Key Correlations (Top Row):** - **Layer 1**: Strong block structure with alternating red/blue squares (ranging from -0.8 to 0.8) - **Layer 2**: Similar block pattern but with reduced intensity (ranging from -0.6 to 0.6) - **Layer 13**: Emerging diagonal patterns with moderate correlation strength (ranging from -0.4 to 0.4) - **Layer 14**: More diffuse pattern with scattered red/blue patches (ranging from -0.3 to 0.3) - **Layer 28**: Weak correlation structure with mostly white areas (ranging from -0.2 to 0.2) - **Layer 29**: Further diffusion with minimal correlation patterns (ranging from -0.1 to 0.1) - **Layer 31**: Near-uniform white with faint diagonal traces (ranging from -0.05 to 0.05) - **Layer 32**: Almost entirely white with only residual red/blue spots (ranging from -0.01 to 0.01) **Value Correlations (Bottom Row):** - **Layer 1**: Similar block structure to Key correlations but with slightly reduced intensity (ranging from -0.7 to 0.7) - **Layer 2**: Maintains block pattern with comparable strength to Layer 1 (ranging from -0.6 to 0.6) - **Layer 13**: Shows emerging diagonal patterns with moderate strength (ranging from -0.4 to 0.4) - **Layer 14**: More diffuse pattern with scattered patches (ranging from -0.3 to 0.3) - **Layer 28**: Weak correlation structure with mostly white areas (ranging from -0.2 to 0.2) - **Layer 29**: Further diffusion with minimal patterns (ranging from -0.1 to 0.1) - **Layer 31**: Near-uniform white with faint traces (ranging from -0.05 to 0.05) - **Layer 32**: Almost entirely white with residual spots (ranging from -0.01 to 0.01) ### Key Observations 1. **Diagonal Dominance**: All heatmaps show perfect correlation (1.0) along the diagonal, confirmed by consistent red dashed lines 2. **Layer Evolution**: Early layers (1-2) show strong structured patterns that progressively diffuse through later layers 3. **Key vs Value**: Key correlations maintain stronger structural patterns than Value correlations across all layers 4. **Layer 32 Anomaly**: Both Key and Value correlations approach near-zero values, suggesting minimal channel interactions in final layers 5. **Color Consistency**: Red (positive) and blue (negative) correlations maintain consistent positioning relative to the diagonal across all layers ### Interpretation The data demonstrates a clear progression from structured, task-specific representations in early layers to more generalized, abstract representations in later layers. The strong block patterns in Layer 1 suggest initial attention mechanisms focus on specific channel relationships, while the diffusion in deeper layers indicates increasing model capacity to handle complex, multi-dimensional interactions. The near-zero correlations in Layer 32 imply that final representations have largely disentangled channel dependencies, potentially optimizing for generalization. The consistent diagonal dominance across all layers confirms the expected perfect self-correlation property of attention mechanisms.