## Histograms: Activation Value Distributions Across Attention Heads ### Overview The image displays four histograms comparing activation value distributions for attention heads 2, 4, 6, and 8 in a neural network model. Each histogram shows a teal-colored bar distribution overlaid with a black dashed normal distribution curve (μ=0, σ=1). The x-axis represents activation values (-5 to +5), and the y-axis shows frequency counts. ### Components/Axes - **X-axis**: "Activation Value" (range: -5 to +5, increments of 1) - **Y-axis**: "Frequency" (no explicit scale, but bars vary in height) - **Legend**: Implicit (black dashed line = standard normal distribution) - **Panel Titles**: - "Queries - Head 2" (leftmost) - "Queries - Head 4" (second from left) - "Queries - Head 6" (third from left) - "Queries - Head 8" (rightmost) ### Detailed Analysis 1. **Head 2**: - Peak activation value: ~-0.2 (left of μ=0) - Spread: Narrowest (σ ≈ 0.5) - Normal distribution overlay: Bars align closely with curve in central region but underrepresent tails 2. **Head 4**: - Peak activation value: ~0.0 (centered at μ=0) - Spread: Moderate (σ ≈ 0.6) - Normal distribution overlay: Bars match curve shape but show slight deviation in extreme tails 3. **Head 6**: - Peak activation value: ~0.3 (right of μ=0) - Spread: Broader (σ ≈ 0.7) - Normal distribution overlay: Bars show increased deviation from curve in right tail 4. **Head 8**: - Peak activation value: ~0.5 (further right of μ=0) - Spread: Widest (σ ≈ 0.8) - Normal distribution overlay: Significant deviation from curve in both tails, with underrepresentation in left tail ### Key Observations - **Trend**: Activation value distributions shift rightward and broaden as head number increases (Head 2 → Head 8) - **Normal Distribution Deviations**: - Heads 2-4 show mild deviations from Gaussian distribution - Heads 6-8 exhibit pronounced non-Gaussian behavior, particularly in extreme activation values - **Spatial Pattern**: Rightward shift in peak activation values correlates with increasing head number ### Interpretation The data suggests a systematic progression in attention head behavior: 1. **Early Heads (2-4)**: Process more focused, Gaussian-like activation patterns, potentially handling basic feature extraction 2. **Later Heads (6-8)**: Exhibit broader, right-skewed distributions, indicating: - Increased sensitivity to higher activation values - Potential handling of more complex/long-range dependencies - Greater variability in processing strategies The deviation from normal distribution in later heads implies non-linear processing characteristics that may be critical for capturing nuanced patterns in the input data. The rightward shift could reflect a bias toward positive activation values in later processing stages, possibly related to value assignment or decision-making in the model's architecture.