## Heatmap: Category Distribution Across Layers and Heads ### Overview The image displays four heatmaps visualizing the distribution of linguistic, knowledge, and algorithmic categories across neural network layers (x-axis) and attention heads (y-axis). Each heatmap uses distinct colors to represent specific categories, with the "All Categories" view showing a composite distribution. ### Components/Axes - **X-axis (layer)**: Ranges from 0 to 30 in increments of 6. - **Y-axis (head)**: Ranges from 0 to 30 in increments of 6. - **Legend**: Located on the left of the "All Categories" heatmap, mapping colors to categories: - Brown: 3 categories - Purple: 2 categories - Green: Linguistic - Orange: Knowledge - Blue: Algorithmic - Gray: Unclassified ### Detailed Analysis 1. **All Categories**: - Green (Linguistic) and orange (Knowledge) squares dominate, with green concentrated in layers 12–24 and heads 6–24. - Orange (Knowledge) appears most frequently in layers 18–24 and heads 12–18. - Blue (Algorithmic) is sparse, with clusters in layers 0–12 and heads 18–24. - Brown (3 categories) and purple (2 categories) are rare, appearing sporadically. 2. **Algorithmic**: - Blue squares are concentrated in layers 0–12 and heads 18–24, with a dense cluster at layer 18, head 24. - Minimal presence in layers >18 or heads <12. 3. **Knowledge**: - Orange squares are spread across layers 0–30 but peak in layers 6–24 and heads 6–18. - A notable cluster appears at layer 12, head 6. 4. **Linguistic**: - Green squares are distributed across all layers but cluster in layers 6–24 and heads 0–18. - A dense region is observed at layer 18, head 12. ### Key Observations - **Concentration vs. Distribution**: Algorithmic categories are tightly clustered in early layers and high heads, while Linguistic and Knowledge categories are more evenly distributed. - **Overlap**: The "All Categories" heatmap shows significant overlap between Linguistic (green) and Knowledge (orange), particularly in layers 12–24 and heads 6–18. - **Unclassified**: Gray squares (unclassified) are absent in the individual category heatmaps but appear in the composite view, suggesting some heads/layers lack clear categorization. ### Interpretation The data suggests a hierarchical organization of neural processing: 1. **Algorithmic** functions (blue) may dominate early layers (0–12), potentially handling low-level feature extraction, with high-head activity (18–24) indicating complex pattern recognition. 2. **Linguistic** (green) and **Knowledge** (orange) categories show broader engagement across middle layers (6–24), implying integration of semantic and contextual information. 3. The absence of unclassified regions in individual category heatmaps suggests robust categorization, though the composite view reveals residual ambiguity in certain areas. Notably, the clustering of Algorithmic activity in layer 18, head 24, and Linguistic activity in layer 18, head 12, may indicate specialized sub-networks for specific tasks. The even distribution of Knowledge across middle layers aligns with its role in cross-modal integration.