## Heatmap: Information propagation path
### Overview
The image is a heatmap visualizing the log-probability of the word "Paris" across different layers and linguistic components. Darker purple shades represent lower log-probability values (more negative), while lighter shades indicate higher values (closer to zero).
### Components/Axes
- **X-axis (Layer)**: Discrete numerical values at positions 1, 6, 11, 16, and 21. Represents sequential processing layers.
- **Y-axis (Linguistic components)**: Phrases stacked vertically:
1. "The"
2. "E"
3. "iff"
4. "el"
5. "Tower"
6. "is"
7. "located"
8. "in"
9. "the"
10. "city"
11. "of"
- **Color scale (Log-probability of "Paris")**: Ranges from -1 (lightest purple) to -7 (darkest purple), with intermediate values at -2, -3, -4, -5, -6.
### Detailed Analysis
- **Layer 1**: All cells are light purple, indicating log-probabilities near -1.
- **Layer 6**: Gradual darkening from "The" to "el," with "el" reaching -3.
- **Layer 11**: Darkest cells in "el" (-4) and "Tower" (-3).
- **Layer 16**: "Tower" (-4), "is" (-3), and "located" (-2).
- **Layer 21**: Darkest cells in "city of" (-7), "the" (-6), and "located" (-5).
### Key Observations
1. **Trend**: Log-probability decreases (becomes more negative) as layers increase, particularly in later layers (16–21).
2. **Critical regions**:
- "city of" consistently shows the lowest values (-7 at layer 21).
- "el" and "Tower" exhibit significant drops in layers 6–11.
3. **Anomalies**:
- "iff" and "E" remain relatively light across all layers.
- "is" and "in" show moderate darkening only in later layers.
### Interpretation
The heatmap suggests that the word "Paris" becomes increasingly improbable as linguistic components are processed in deeper layers. The darkest cells in "city of" (layer 21) imply that phrases describing spatial relationships (e.g., "located in the city of") are least likely to include "Paris" in later processing stages. This could reflect a model's prioritization of local context over global semantic relationships in later layers. The persistence of light shades in "iff" and "E" indicates these components retain higher uncertainty about "Paris" throughout processing. The structured darkening pattern aligns with typical attention mechanisms in transformer models, where early layers capture syntactic features while later layers focus on semantic disambiguation.
