## Heatmaps: GPT-2 Medium - Name Copying and Country to Capital Heads
### Overview
The image contains two side-by-side heatmaps comparing attention head activity in GPT-2 medium for two tasks: "Name Copying" (left) and "Country to Capital" (right). Both heatmaps use a color gradient (purple to yellow) to represent scores from 0 to 1, with black crosses marking specific data points. A text box labeled "Circuits Components Reused" appears in both charts, with "Mover Heads" and "Capital Heads" annotations.
---
### Components/Axes
- **X-axis (Layer)**: Labeled "Layer" with values 0–22 (integers).
- **Y-axis (Head)**: Labeled "Head" with values 15–22 (integers).
- **Color Scale**:
- Left heatmap: "Name Copying score" (0–1, purple to yellow).
- Right heatmap: "Country to capital score" (0–1, purple to yellow).
- **Annotations**:
- Text box: "Circuits Components Reused" (positioned near bottom-left of both heatmaps).
- Black crosses:
- Left: Labeled "Mover Heads" (e.g., Layer 14, Head 16; Layer 18, Head 19).
- Right: Labeled "Capital Heads" (e.g., Layer 16, Head 17; Layer 20, Head 21).
---
### Detailed Analysis
#### Left Heatmap (Name Copying)
- **Color Distribution**:
- High scores (yellow/green) cluster in:
- Upper-right quadrant (Layers 16–22, Heads 17–22).
- Lower-left quadrant (Layers 0–8, Heads 15–18).
- Low scores (purple) dominate the central region (Layers 8–16, Heads 15–18).
- **Black Crosses**:
- Located at:
- Layer 14, Head 16 (score ~0.8).
- Layer 18, Head 19 (score ~0.7).
- Layer 20, Head 21 (score ~0.6).
#### Right Heatmap (Country to Capital)
- **Color Distribution**:
- High scores (yellow/green) cluster in:
- Upper-right quadrant (Layers 16–22, Heads 17–22).
- Lower-left quadrant (Layers 0–8, Heads 15–18).
- Low scores (purple) dominate the central region (Layers 8–16, Heads 15–18).
- **Black Crosses**:
- Located at:
- Layer 16, Head 17 (score ~0.8).
- Layer 20, Head 21 (score ~0.7).
- Layer 22, Head 22 (score ~0.6).
---
### Key Observations
1. **Similar Patterns**: Both heatmaps show high scores in the upper-right and lower-left quadrants, suggesting shared mechanisms for these tasks.
2. **Black Crosses**: Marked heads in both charts align with high-scoring regions, indicating these heads are critical for their respective tasks.
3. **Text Box**: The "Circuits Components Reused" annotation implies overlapping functional components across tasks.
---
### Interpretation
- **Task-Specific Heads**: The black crosses ("Mover Heads" and "Capital Heads") likely represent specialized attention mechanisms for name copying and country-capital mapping.
- **Reused Circuits**: The overlapping high-score regions suggest GPT-2 medium repurposes similar attention patterns for structurally analogous tasks (e.g., mapping entities to their attributes).
- **Layer Dependency**: High scores in upper layers (16–22) may reflect hierarchical processing, where later layers refine task-specific representations.
The data highlights how transformer models leverage modular attention mechanisms, with certain heads specializing in specific tasks while sharing broader functional components.
