## Heatmaps: Category Distribution Across Layers and Heads ### Overview The image displays four heatmaps visualizing the distribution of linguistic, knowledge-based, and algorithmic categories across neural network layers and attention heads. The "All Categories" heatmap combines all classifications, while the subsequent panels isolate specific categories. Spatial patterns reveal how different cognitive functions are localized within the model architecture. ### Components/Axes - **X-axis**: Layer index (0-30), representing neural network depth - **Y-axis**: Head index (0-30), representing attention mechanism components - **Legend**: - Brown: 3 categories - Purple: 2 categories - Green: Linguistic - Orange: Knowledge - Blue: Algorithmic - Gray: Unclassified - **Heatmap Titles**: - All Categories (combined) - Algorithmic (blue) - Knowledge (orange) - Linguistic (green) ### Detailed Analysis 1. **All Categories Heatmap**: - Mixed distribution of brown (3 categories), purple (2 categories), green, orange, and blue squares - Gray squares (unclassified) appear sparsely in upper layers (24-30) - Highest density of colored squares in layers 12-24 2. **Algorithmic Heatmap**: - Exclusively blue squares (algorithmic category) - Concentrated in layers 12-24, heads 6-18 - Notable cluster at layer 18, head 12 3. **Knowledge Heatmap**: - Orange squares dominate layers 6-24 - Strong presence in heads 12-24 - Notable cluster at layer 24, head 24 4. **Linguistic Heatmap**: - Green squares prevalent in layers 0-24 - Dense distribution in heads 0-18 - Notable cluster at layer 6, head 6 ### Key Observations - **Spatial Segregation**: Categories show distinct spatial patterns, with minimal overlap between heatmaps - **Layer Depth Correlation**: Algorithmic and Knowledge categories concentrate in deeper layers (12-24) - **Head Specialization**: Linguistic category dominates early heads (0-18), while Knowledge/Algorithmic occupy mid-to-late heads - **Unclassified Presence**: Gray squares in "All Categories" suggest 8-12% of layer-head combinations remain unclassified ### Interpretation The data demonstrates clear functional specialization within the neural architecture: 1. **Linguistic Processing**: Early layers (0-12) and heads (0-18) specialize in language-related tasks, suggesting foundational language understanding occurs in shallower network regions 2. **Knowledge Integration**: Mid-layers (12-24) show strong Knowledge category presence, indicating hierarchical knowledge representation building upon linguistic foundations 3. **Algorithmic Operations**: Deeper layers (18-30) contain algorithmic processing, possibly handling complex pattern recognition and decision-making 4. **Unclassified Regions**: The presence of gray squares in upper layers suggests either model uncertainty or emergent properties not captured by current categorization This spatial distribution pattern aligns with theories of neural network modularity, where different cognitive functions are localized in specific architectural regions. The clear separation between categories implies effective feature disentanglement, while the unclassified regions warrant further investigation into potential model ambiguities or novel processing mechanisms.