## Heatmap: High-Low difference (trust-span attention mass) | q=last_prompt_token
### Overview
This heatmap visualizes the "High-Low difference" metric across neural network layers and attention heads, with values ranging from -0.15 (purple) to +0.10 (yellow). The metric likely quantifies variability in attention mechanisms, with "trust-span" suggesting a focus on reliable or consistent attention patterns. The majority of values cluster near zero (teal), with scattered extremes in yellow and purple.
### Components/Axes
- **X-axis (Head)**: 0 to 30 (31 categories, integer steps)
- **Y-axis (Layer)**: 0 to 60 (61 categories, integer steps)
- **Color Scale**:
- Purple (-0.15) to Yellow (+0.10)
- Neutral teal (~0.00) dominates
- **Legend**: Positioned right-aligned, with gradient from purple (low) to yellow (high)
### Detailed Analysis
- **High Values (+0.10, Yellow)**:
- Layer 25, Head 10
- Layer 30, Head 25
- Layer 40, Head 15
- Layer 50, Head 20
- **Low Values (-0.15, Purple)**:
- Layer 10, Head 5
- **Neutral Values (~0.00, Teal)**:
- Most regions (e.g., Layer 0-10, Heads 0-30; Layer 30-50, Heads 0-30)
### Key Observations
1. **Sparse Extremes**: Only 5 distinct extreme values (4 high, 1 low) across 61x31 grid.
2. **Vertical Clustering**: High values cluster in mid-to-lower layers (25-50).
3. **Horizontal Distribution**: High values appear in heads 10, 15, 20, 25—no clear horizontal pattern.
4. **Low Value Anomaly**: Single purple cell at Layer 10, Head 5 stands out as the most negative value.
### Interpretation
The heatmap reveals that trust-span attention mass differences are **highly sparse**, with most attention heads showing minimal variability (teal). The isolated yellow cells in mid-lower layers (25-50) suggest specific heads in these layers exhibit significantly higher attention mass variability, potentially indicating:
- **Layer-Specific Sensitivity**: Later layers (25-50) may process more complex features requiring adaptive attention.
- **Head Specialization**: Heads 10, 15, 20, 25 might specialize in tasks requiring greater attention flexibility.
- **Outlier Behavior**: The purple cell at Layer 10, Head 5 could represent a head with unusually consistent (or rigid) attention patterns.
The data implies that attention mechanisms in neural networks exhibit **layer-dependent variability**, with later layers showing more pronounced differences in attention mass. This could reflect architectural design choices or emergent properties of attention dynamics in deep networks.
