## Heatmap: Avg JS Divergence by Layer and Category ### Overview The image is a heatmap visualizing the average Jensen-Shannon (JS) divergence across different layers (0-30) for three categories: Subj. (Subject), Attn. (Attention), and Last. The color intensity represents the magnitude of the JS divergence, ranging from light blue (low divergence) to dark blue (high divergence). ### Components/Axes * **X-axis:** Layer, with values ranging from 0 to 30 in increments of 2. * **Y-axis:** Categories: Subj., Attn., Last. * **Color Scale (Legend):** Avg JS Divergence, ranging from 0.2 (light blue) to 0.6 (dark blue). The scale has markers at 0.2, 0.3, 0.4, 0.5, and 0.6. ### Detailed Analysis * **Subj. (Subject):** The JS divergence for the "Subj." category is relatively high (blue color) for layers 0 to approximately 16. The color intensity gradually decreases from layer 0 to 16, indicating a decreasing JS divergence. After layer 16, the JS divergence drops significantly and remains low (light blue). * Layer 0: Approximately 0.55 JS Divergence * Layer 8: Approximately 0.45 JS Divergence * Layer 16: Approximately 0.35 JS Divergence * **Attn. (Attention):** The JS divergence for the "Attn." category is consistently low (light blue) across all layers (0-30). The values appear to be close to 0.2. * **Last.:** The JS divergence for the "Last." category is consistently low (light blue) across all layers (0-30). The values appear to be close to 0.2. ### Key Observations * The "Subj." category exhibits a significantly higher JS divergence in the initial layers (0-16) compared to the "Attn." and "Last." categories. * The JS divergence for the "Attn." and "Last." categories remains consistently low across all layers. * There is a clear transition in the "Subj." category around layer 16, where the JS divergence drops sharply. ### Interpretation The heatmap suggests that the "Subject" category has a higher degree of divergence in the earlier layers of the model, indicating that the representations or information related to the subject are more variable or less stable in these initial layers. The "Attention" and "Last" categories, on the other hand, show consistently low divergence across all layers, suggesting more stable representations. The sharp drop in JS divergence for the "Subject" category after layer 16 could indicate a point where the model has effectively learned or stabilized the subject-related information. This could be due to the model learning to extract relevant features or representations related to the subject in the earlier layers, and then refining or stabilizing these representations in the later layers.

# Technical Data Extraction: Heatmap Analysis of JS Divergence ## 1. Image Overview This image is a heatmap visualization representing the **Avg JS Divergence** (Jensen-Shannon Divergence) across different layers of a neural network model, categorized by three specific components or methods. ## 2. Component Isolation ### A. Header / Title * No explicit title is present within the image frame. ### B. Main Chart (Heatmap) * **Type:** Heatmap with a grid structure. * **X-Axis (Horizontal):** Labeled "**Layer**". It contains 32 discrete units, indexed from **0 to 31**. Major numerical markers are provided every 2 units (0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30). * **Y-Axis (Vertical):** Contains three categories: 1. **Subj.** (Top row) 2. **Attn.** (Middle row) 3. **Last.** (Bottom row) ### C. Legend / Color Bar * **Location:** Right side of the plot. * **Label:** "**Avg JS Divergence**" (oriented vertically). * **Scale:** Linear gradient from light blue to dark blue. * **Numerical Markers:** 0.2, 0.3, 0.4, 0.5, 0.6. * **Color Mapping:** * **0.2 (Minimum):** Near-white/very light blue. * **0.6 (Maximum):** Deep navy blue. --- ## 3. Data Extraction and Trend Verification ### Row 1: "Subj." (Subject) * **Visual Trend:** This row shows the highest divergence values in the dataset. It starts with a moderate-to-high blue intensity in the early layers, maintains this through the mid-layers, and then sharply drops to the minimum value (near-white) in the later layers. * **Data Points (Approximate JS Divergence):** * **Layers 0 - 15:** Values range between approximately **0.35 and 0.45**. The color is a consistent medium blue. * **Layers 16 - 31:** Values drop significantly to approximately **0.20 - 0.22**. The color is nearly white. ### Row 2: "Attn." (Attention) * **Visual Trend:** This row is consistently flat and low. There is almost no variation across the layers. * **Data Points (Approximate JS Divergence):** * **Layers 0 - 31:** Values are consistently at the minimum baseline of approximately **0.20**. The entire row appears as a very light, near-white band. ### Row 3: "Last." (Last) * **Visual Trend:** Similar to the "Attn." row, this row remains at the baseline for almost the entire duration, with a very slight, negligible increase at the final layer. * **Data Points (Approximate JS Divergence):** * **Layers 0 - 30:** Values are at the minimum baseline of approximately **0.20**. * **Layer 31:** There is a very slight darkening, suggesting a value of approximately **0.22 - 0.25**, though it remains much lighter than the "Subj." early layers. --- ## 4. Summary Table of Extracted Information | Category (Y-Axis) | Layer Range (X-Axis) | Visual Intensity | Estimated Avg JS Divergence | | :--- | :--- | :--- | :--- | | **Subj.** | 0 - 15 | Medium Blue | 0.35 - 0.45 | | **Subj.** | 16 - 31 | Near White | ~0.20 | | **Attn.** | 0 - 31 | Near White | ~0.20 (Constant) | | **Last.** | 0 - 30 | Near White | ~0.20 | | **Last.** | 31 | Very Light Blue | ~0.23 | ## 5. Technical Observations The data indicates that the "Subj." component experiences significantly higher Jensen-Shannon Divergence in the first half of the model's layers (0-15) compared to the "Attn." and "Last." components. After Layer 15, the divergence for "Subj." converges to the same low baseline seen in the other two categories. This suggests that the specific behavior or information captured by "Subj." is most distinct or volatile in the earlier stages of processing.

\n ## Heatmap: Average Jensen-Shannon Divergence by Layer and Representation ### Overview This image presents a heatmap visualizing the average Jensen-Shannon (JS) Divergence across different layers of a model, for three different representations: Subject ('Subj.'), Attention ('Attn.'), and Last Layer ('Last.'). The heatmap uses a blue color gradient to represent divergence values, with darker blues indicating higher divergence. ### Components/Axes * **X-axis:** Layer, ranging from 0 to 30, with increments of 2. * **Y-axis:** Representation, with three categories: 'Subj.', 'Attn.', and 'Last.'. * **Color Scale:** Average JS Divergence, ranging from 0.2 (light blue) to 0.6 (dark blue). The scale is positioned on the right side of the heatmap. ### Detailed Analysis The heatmap displays JS Divergence values for each combination of layer and representation. * **Subject ('Subj.') Representation:** The JS Divergence remains relatively constant across layers 0 to 30, hovering around approximately 0.55 ± 0.02. There is a slight decrease in divergence around layer 28, dropping to approximately 0.52. * **Attention ('Attn.') Representation:** The JS Divergence is consistently lower than for the 'Subj.' representation, generally around 0.40 ± 0.02. There is a slight increase in divergence from layer 0 to layer 6, rising from approximately 0.38 to 0.42. It remains relatively stable until layer 26, where it begins to increase, reaching approximately 0.45 by layer 30. * **Last Layer ('Last.') Representation:** The JS Divergence starts at approximately 0.25 at layer 0 and increases steadily with increasing layer number. By layer 30, the divergence reaches approximately 0.55. The increase appears roughly linear. ### Key Observations * The 'Subj.' representation exhibits the highest and most stable JS Divergence across all layers. * The 'Last.' representation shows the lowest divergence at early layers, but its divergence increases significantly as the layer number increases. * The 'Attn.' representation has intermediate divergence values, with a slight increase towards the later layers. * There is a clear trend of increasing divergence in the 'Last.' representation as the layer number increases. ### Interpretation The heatmap suggests that the 'Subject' representation maintains a consistent level of information throughout the layers, as indicated by the stable JS Divergence. This could imply that the subject information is well-preserved during processing. The 'Last' layer representation, initially having low divergence, becomes more divergent as the layer number increases, suggesting that the information in this representation becomes more distinct or specialized as it passes through the model. The 'Attention' representation shows a moderate level of divergence, with a slight increase in later layers, potentially indicating that the attention mechanism is learning to focus on different aspects of the input as processing progresses. The increasing divergence in the 'Last' layer could be indicative of feature extraction or transformation, where the representation becomes more specific to the task at hand. The differences in divergence values between the representations suggest that each representation captures different aspects of the input data and evolves differently during processing.

## Heatmap: Average Jensen-Shannon Divergence Across Model Layers ### Overview The image is a heatmap visualizing the "Avg JS Divergence" (Average Jensen-Shannon Divergence) across different layers of a model for three distinct categories. The heatmap uses a blue color gradient to represent divergence values, with darker blue indicating higher divergence. The data suggests a comparison of how different components or representations within a model diverge from a reference distribution as information propagates through its layers. ### Components/Axes * **Chart Type:** Heatmap. * **X-Axis:** Labeled **"Layer"**. It represents model layers, with numerical markers at intervals of 2, ranging from **0 to 30**. * **Y-Axis:** Contains three categorical labels, positioned on the left side of the heatmap: 1. **"Subj."** (Top row) 2. **"Attn."** (Middle row) 3. **"Last."** (Bottom row) * **Color Bar/Legend:** Positioned on the right side of the chart. It is labeled **"Avg JS Divergence"** and provides a scale for interpreting the heatmap colors. * **Scale Range:** Approximately **0.2 to 0.6**. * **Color Gradient:** A sequential blue palette where lighter shades (near white/light blue) correspond to lower values (~0.2) and darker, saturated blue corresponds to higher values (~0.6). ### Detailed Analysis The heatmap displays a 3 (categories) x 16 (layer intervals) grid of colored cells. The color intensity represents the average JS Divergence value for that category at that layer. 1. **"Subj." Row (Top):** * **Trend:** This row shows the highest divergence values, which are concentrated in the earlier layers and gradually decrease. * **Data Points:** The cells from **Layer 0 to approximately Layer 16** are colored in varying shades of medium to dark blue. The darkest blue (highest divergence, ~0.5-0.6) appears in the very first layers (0-4). The color progressively lightens as the layer number increases, becoming very light blue (divergence ~0.25-0.3) by Layer 16 and remaining light for layers beyond. 2. **"Attn." Row (Middle):** * **Trend:** This row exhibits consistently low divergence across all layers. * **Data Points:** All cells in this row are a very light blue or off-white color, indicating divergence values at the low end of the scale, approximately **0.2 to 0.25**. There is no significant visual trend or variation across layers. 3. **"Last." Row (Bottom):** * **Trend:** This row also shows low divergence overall, with a very slight increase visible in the final layers. * **Data Points:** Most cells are light blue, similar to the "Attn." row (~0.2-0.25). However, in the final columns corresponding to **Layers 28 and 30**, the blue shade becomes slightly more pronounced, suggesting a minor increase in divergence to approximately **0.25-0.3**. ### Key Observations * **Dominant Pattern:** The most striking feature is the high divergence in the "Subj." category during the early to mid-layers (0-16), which sharply contrasts with the low, stable divergence of the "Attn." and "Last." categories. * **Layer Sensitivity:** The "Subj." representation appears to be highly sensitive to layer depth, undergoing significant change (high divergence) early in the network before stabilizing. * **Stability of Attention:** The "Attn." component shows remarkable stability (low divergence) throughout the entire depth of the model. * **Late Divergence in "Last.":** There is a subtle but observable uptick in divergence for the "Last." category in the very final layers (28-30). ### Interpretation This heatmap likely analyzes the internal dynamics of a deep neural network, possibly a transformer model given the "Attn." (Attention) label. Jensen-Shannon Divergence measures the similarity between two probability distributions. * **What the data suggests:** The high early divergence for **"Subj."** implies that the model's representation of the "subject" (or a subject-related feature) changes dramatically in the initial processing stages. This could indicate the model is actively constructing or refining this concept from raw input. * **Relationship between elements:** The stark contrast between "Subj." and "Attn." suggests these components play fundamentally different roles. The attention mechanism ("Attn.") appears to operate on a stable, consistent distribution across layers, perhaps serving as a reliable routing or weighting function. In contrast, the subject representation is highly dynamic. * **Notable anomaly/trend:** The slight rise in divergence for **"Last."** in the final layers is intriguing. It may indicate that the final output representation ("Last") begins to diverge slightly from an intermediate representation as it is fine-tuned for the specific task output, or it could be an artifact of the final layer normalization. * **Underlying significance:** This visualization helps diagnose where and how a model transforms information. The early, high-divergence zone for "Subj." pinpoints a critical phase of feature formation, while the stability of "Attn." confirms its role as a consistent computational primitive. This kind of analysis is crucial for understanding model interpretability, debugging representation learning, and guiding architectural improvements.

## Heatmap: Average JS Divergence Across Layers and Components ### Overview The image is a heatmap visualizing the average Jensen-Shannon (JS) divergence across 31 layers (0–30) for three components: "Subj." (subject), "Attn." (attention), and "Last." (final layer). The color gradient ranges from light blue (low divergence, ~0.2) to dark blue (high divergence, ~0.6), with the legend positioned on the right. --- ### Components/Axes - **X-axis (Layer)**: Labeled "Layer," with integer increments from 0 to 30. - **Y-axis (Components)**: Three categories: - "Subj." (subject) - "Attn." (attention) - "Last." (final layer) - **Color Scale**: "Avg JS Divergence" (0.2–0.6), with darker blue indicating higher divergence. --- ### Detailed Analysis 1. **Subj. (Subject)**: - Dark blue bars dominate the first 15 layers (0–14), indicating high divergence (~0.5–0.6). - Gradual lightening from layer 15 onward, with values dropping to ~0.3 by layer 30. - Spatial grounding: Darkest bars are clustered in the top-left region. 2. **Attn. (Attention)**: - Uniform light blue across all layers (~0.2–0.3), with no significant variation. - Slightly darker in the first 10 layers (~0.25), but no clear trend. 3. **Last. (Final Layer)**: - Light blue throughout, with minimal divergence (~0.2–0.3). - No discernible pattern across layers. --- ### Key Observations - **Highest Divergence**: "Subj." in early layers (0–14) shows the strongest variability. - **Stability**: "Attn." and "Last." exhibit consistently low divergence, suggesting stability. - **Transition**: "Subj." divergence decreases sharply after layer 15, implying a shift in focus or representation. --- ### Interpretation The data suggests that subject-related features ("Subj.") are highly variable in early layers but stabilize later, potentially reflecting a model's initial exploration of subject-specific patterns followed by refinement. Attention ("Attn.") and final layer representations ("Last.") remain stable, indicating consistent processing or aggregation of information. The sharp drop in "Subj." divergence after layer 15 may signal a transition from exploration to exploitation in the model's architecture. This pattern could inform layer-wise analysis of neural network behavior, particularly in tasks requiring subject-specific attention.