## Heatmap: Category Distribution Across Layers and Heads ### Overview The image presents four heatmaps visualizing the distribution of linguistic, knowledge, and algorithmic categories across neural network layers (x-axis: 0-35) and heads (y-axis: 0-40). The "All Categories" heatmap shows overlapping distributions, while the subsequent panels isolate specific categories. Colors correspond to predefined categories (see legend). ### Components/Axes - **X-axis (layer)**: 0 to 35, representing neural network layers. - **Y-axis (head)**: 0 to 40, representing attention heads. - **Legend**: - Brown: 3 categories - Purple: 2 categories - Green: Linguistic - Orange: Knowledge - Blue: Algorithmic - Gray: Unclassified ### Detailed Analysis #### All Categories - **Distribution**: - Green (Linguistic) and orange (Knowledge) dominate, with green concentrated in layers 14-28 and orange peaking at layer 21. - Blue (Algorithmic) is sparse but present across all layers. - Brown (3 categories) and purple (2 categories) are rare, appearing sporadically in layers 14-35. - Gray (Unclassified) fills gaps between colored points. #### Algorithmic - **Distribution**: - Blue squares are uniformly distributed but denser in layers 14-21. - No clear trend; density remains low compared to other categories. #### Knowledge - **Distribution**: - Orange squares cluster strongly in layers 14-28, with a peak at layer 21. - Density decreases sharply in layers 0-7 and 28-35. #### Linguistic - **Distribution**: - Green squares are evenly spread across all layers but densest in layers 14-28. - Layer 21 shows the highest concentration. ### Key Observations 1. **Layer 21 Dominance**: All three primary categories (Linguistic, Knowledge, Algorithmic) show elevated activity in layer 21. 2. **Knowledge Concentration**: Knowledge (orange) is most tightly clustered around layer 21, suggesting a focal point for this category. 3. **Algorithmic Sparsity**: Algorithmic (blue) points are dispersed but lack the density of other categories. 4. **Unclassified Prevalence**: Gray areas (Unclassified) are most prominent in layers 0-7 and 32-40. ### Interpretation The data suggests that **layer 21** acts as a critical hub for integrating linguistic, knowledge, and algorithmic processing. The tight clustering of Knowledge in this layer implies specialized functionality, while the broader spread of Linguistic points indicates distributed processing. Algorithmic elements appear less localized, possibly reflecting general-purpose operations. The scarcity of 2- and 3-category points (brown/purple) suggests these represent edge cases or transitional states. The dominance of Unclassified regions in peripheral layers (0-7, 32-40) may indicate incomplete categorization or noise in those areas. This pattern aligns with hierarchical processing models, where middle layers specialize in complex feature integration.