## Heatmaps: Attention Distribution Across MHA Variants ### Overview The image displays five heatmaps comparing attention distribution patterns across different Multi-Head Attention (MHA) variants. Each heatmap visualizes the correlation between attention heads (x-axis) and their corresponding output indices (y-axis), with color intensity representing correlation strength (-1.00 to 1.00). All heatmaps share identical axis ranges (0-16) and color scales. ### Components/Axes - **X-axis**: "Head Index" (0-16) - **Y-axis**: "Head Index" (0-16) - **Color Scale**: - Red: Positive correlation (1.00) - Blue: Negative correlation (-1.00) - Yellow: Neutral (0.00) - **Legend**: Positioned right of each heatmap, consistent across all variants - **Heatmap Titles** (top-center): 1. Standard MHA 2. LRKV (r=64) 3. GQA 4. MQA 5. MLA ### Detailed Analysis 1. **Standard MHA** - Diagonal dominance with dark red squares (0.8-1.0 correlation) - Sparse off-diagonal activity (≤0.2 correlation) - No significant outliers 2. **LRKV (r=64)** - Strong diagonal (0.9-1.0) - Distinct off-diagonal pattern: - Secondary diagonal (i+1,j) shows moderate correlation (0.3-0.6) - Checkerboard-like pattern in lower-left quadrant 3. **GQA** - Diagonal dominance (0.7-0.9) - Increased off-diagonal noise: - Random clusters of 0.2-0.5 correlation - Blue patches (-0.1 to -0.3) in upper-right quadrant 4. **MQA** - Weakest diagonal (0.5-0.7) - Mixed off-diagonal signals: - Strong negative correlation (-0.4 to -0.6) in lower-left - Yellow/white dominance (0.0-0.2) elsewhere 5. **MLA** - Strongest diagonal (0.9-1.0) - Notable off-diagonal: - Dark red secondary diagonal (i+2,j) - Blue clusters (-0.3 to -0.5) in upper-right ### Key Observations - All variants show diagonal dominance (self-attention) - LRKV and MLA exhibit structured off-diagonal patterns - GQA shows highest noise variance - MQA demonstrates weakest diagonal correlation - Color consistency across heatmaps confirms uniform correlation scale ### Interpretation The heatmaps reveal fundamental differences in attention mechanism behavior: 1. **Standard MHA** represents baseline self-attention with minimal cross-head interaction 2. **LRKV's** structured off-diagonal pattern suggests engineered attention routing for specific head interactions 3. **GQA's** noise indicates potential instability in attention distribution 4. **MQA's** weak diagonal correlation aligns with its multi-query design reducing head specialization 5. **MLA's** strong secondary diagonal implies enhanced cross-head coordination The variations highlight how architectural choices (e.g., key-value sharing in MQA, relative position encoding in LRKV) fundamentally alter attention distribution patterns. The consistent diagonal dominance across all variants confirms the preservation of self-attention as a core mechanism, while off-diagonal patterns reveal unique variant characteristics.