## Heatmap: Avg JS Divergence by Layer and Category ### Overview The image is a heatmap visualizing the average Jensen-Shannon (JS) divergence across different layers (0-30) for three categories: "Subj.", "Attn.", and "Last.". The color intensity represents the magnitude of the JS divergence, ranging from approximately 0.1 (lightest blue) to 0.6 (darkest blue). ### Components/Axes * **Y-axis:** Categories: "Subj.", "Attn.", "Last." * **X-axis:** Layer number, ranging from 0 to 30 in increments of 2. * **Colorbar (Right):** Represents the "Avg JS Divergence", ranging from 0.1 to 0.6. The color gradient goes from light blue (0.1) to dark blue (0.6). ### Detailed Analysis * **Subj. (Subject):** * From layer 0 to approximately layer 18, the JS divergence is high, indicated by the dark blue color, close to 0.6. * From layer 20 to 30, the JS divergence decreases significantly, indicated by the lighter blue color, around 0.3. * **Attn. (Attention):** * The JS divergence is generally lower than "Subj." across all layers. * From layer 0 to approximately layer 14, the JS divergence is low, around 0.2. * From layer 16 to 30, the JS divergence increases slightly, reaching approximately 0.3. * **Last.:** * The JS divergence is relatively consistent across all layers, with values ranging from approximately 0.2 to 0.3. ### Key Observations * "Subj." has the highest JS divergence in the initial layers (0-18), indicating a greater difference in distributions compared to "Attn." and "Last.". * The JS divergence for "Subj." decreases significantly after layer 18. * "Attn." has the lowest JS divergence in the initial layers. * "Last." exhibits a relatively stable JS divergence across all layers. ### Interpretation The heatmap illustrates how the Jensen-Shannon divergence varies across different layers of a model for three categories: "Subj.", "Attn.", and "Last.". The high JS divergence for "Subj." in the initial layers suggests that the distributions being compared are more dissimilar in these layers compared to the later layers. The lower and more consistent JS divergence for "Attn." and "Last." indicates a more stable and similar distribution across all layers. The data suggests that the "Subj." category undergoes a significant change in its distribution around layer 18, while "Attn." and "Last." remain relatively stable. This could indicate that the model processes or transforms the "Subj." information differently in the later layers.

# Technical Document Extraction: Heatmap Analysis of JS Divergence ## 1. Component Isolation The image is a heatmap visualization representing data across different layers of a neural network model. * **Header/Title:** None present. * **Main Chart Area:** A grid of 3 rows and 32 columns (representing layers 0 through 31). * **Y-Axis (Left):** Categorical labels representing different components or states. * **X-Axis (Bottom):** Numerical labels representing model layers. * **Legend (Right):** A vertical color scale bar indicating the magnitude of the measured metric. --- ## 2. Metadata and Labels * **X-Axis Title:** `Layer` * **X-Axis Markers:** 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30 (Total of 32 columns visible). * **Y-Axis Labels:** * `Subj.` (Top row) * `Attn.` (Middle row) * `Last.` (Bottom row) * **Legend Title:** `Avg JS Divergence` (Jensen-Shannon Divergence) * **Legend Scale:** 0.1 (Lightest blue/white) to 0.6 (Darkest blue). --- ## 3. Legend and Color Mapping The legend is located at the far right of the image. * **Dark Blue (~0.6):** High Average JS Divergence. * **Medium Blue (~0.3 - 0.4):** Moderate Average JS Divergence. * **Light Blue/White (~0.1):** Low Average JS Divergence. --- ## 4. Trend Verification and Data Extraction ### Row 1: Subj. (Subject) * **Visual Trend:** Starts with very high intensity (dark blue) from Layer 0, maintains this intensity through the mid-layers, and then sharply fades to near-white in the final third of the layers. * **Data Points:** * **Layers 0 - 17:** High divergence (approx. 0.5 to 0.6). * **Layers 18 - 22:** Transition period; divergence drops from approx. 0.4 to 0.2. * **Layers 23 - 31:** Low divergence (approx. 0.1). ### Row 2: Attn. (Attention) * **Visual Trend:** Predominantly low intensity (near-white) across almost the entire spectrum, with a very slight, subtle increase in the middle layers. * **Data Points:** * **Layers 0 - 11:** Very low divergence (~0.1). * **Layers 12 - 16:** Slight increase to a very light blue (~0.15 - 0.2). * **Layers 17 - 31:** Returns to very low divergence (~0.1). ### Row 3: Last. (Last Token/State) * **Visual Trend:** Starts at very low intensity and gradually increases in saturation as the layer number increases, peaking and stabilizing in the latter half of the model. * **Data Points:** * **Layers 0 - 7:** Very low divergence (~0.1). * **Layers 8 - 15:** Gradual upward slope in divergence (approx. 0.15 to 0.25). * **Layers 16 - 31:** Sustained moderate divergence (approx. 0.3 to 0.35). --- ## 5. Summary of Findings The heatmap illustrates a clear shift in information processing (measured by JS Divergence) across the 32 layers of the model: 1. **Subject information (`Subj.`)** is most prominent and divergent in the early to middle stages (Layers 0-18). 2. **Attention mechanisms (`Attn.`)** show minimal divergence across all layers, with a negligible peak around Layer 14. 3. **The final state/token (`Last.`)** gains divergence progressively, becoming the dominant feature in the latter half of the model (Layers 16-31), coinciding with the drop-off of the Subject information.

\n ## Heatmap: Jensen-Shannon Divergence by Layer and Subject ### Overview This image presents a heatmap visualizing the average Jensen-Shannon (JS) Divergence across different layers of a model, categorized by subject (Subj.), attention (Attn.), and last layer (Last.). The heatmap displays the divergence values using a color gradient, ranging from dark blue (high divergence) to light blue (low divergence). ### Components/Axes * **X-axis:** Layer, ranging from 0 to 30, with increments of 2. * **Y-axis:** Three categories: "Subj." (Subject), "Attn." (Attention), and "Last." (Last Layer). * **Color Scale:** Represents "Avg JS Divergence", ranging from 0.1 to 0.6. The color scale is positioned on the right side of the heatmap. * **Legend:** Located in the top-right corner, indicating the mapping between color and JS Divergence values. ### Detailed Analysis The heatmap consists of three horizontal rows, each representing one of the categories (Subj., Attn., Last.). Each cell in the heatmap represents the average JS Divergence for a specific layer and category. **Subject (Subj.):** * The JS Divergence is initially high (approximately 0.55-0.6) for layers 0-6. * There is a noticeable decrease in JS Divergence between layers 6 and 8 (down to approximately 0.45). * From layers 8-14, the JS Divergence remains relatively stable around 0.45-0.5. * A further decrease is observed from layers 14-20 (down to approximately 0.3). * From layers 20-30, the JS Divergence continues to decrease, reaching approximately 0.2-0.3. **Attention (Attn.):** * The JS Divergence starts at a low value (approximately 0.15-0.2) for layers 0-4. * It gradually increases from layers 4-10 (reaching approximately 0.35-0.4). * From layers 10-20, the JS Divergence remains relatively stable around 0.4. * A slight decrease is observed from layers 20-30 (down to approximately 0.3). **Last Layer (Last.):** * The JS Divergence starts at a very low value (approximately 0.1) for layers 0-6. * It gradually increases from layers 6-14 (reaching approximately 0.25-0.3). * From layers 14-30, the JS Divergence remains relatively stable around 0.25-0.3. ### Key Observations * The "Subj." category exhibits the highest JS Divergence values, particularly in the initial layers. * The "Last." category consistently shows the lowest JS Divergence values across all layers. * The "Attn." category shows a moderate increase in JS Divergence in the middle layers, then plateaus. * There is a clear trend of decreasing JS Divergence with increasing layer number for the "Subj." category. ### Interpretation The heatmap suggests that the "Subject" component of the model initially has the most significant divergence from its expected distribution, but this divergence decreases as the model processes information through deeper layers. This could indicate that the initial layers are more sensitive to variations in the input subject, while later layers learn to generalize and reduce this sensitivity. The "Last" layer consistently exhibits low divergence, suggesting that the final output of the model is relatively stable and predictable. The "Attention" layer shows a moderate divergence that remains relatively constant after the initial layers, indicating a consistent level of attention-related variation. The decreasing trend in JS Divergence for the "Subj." category could be interpreted as the model learning to extract more abstract and invariant features as it progresses through the layers. The differences in divergence patterns between the three categories highlight the distinct roles and behaviors of each component within the model. The heatmap provides insights into the internal dynamics of the model and how different components contribute to its overall performance.

## Heatmap: Average Jensen-Shannon Divergence Across Model Layers ### Overview The image is a heatmap visualizing the average Jensen-Shannon (JS) Divergence across 31 layers (0-30) of a model for three distinct components or metrics. The heatmap uses a blue color gradient to represent the magnitude of divergence, with a corresponding color scale bar on the right. ### Components/Axes * **Y-Axis (Vertical):** Lists three categories, positioned on the left side of the chart. * `Subj.` (Top row) * `Attn.` (Middle row) * `Last.` (Bottom row) * **X-Axis (Horizontal):** Labeled "Layer" at the bottom center. It displays numerical markers from 0 to 30, incrementing by 2 (0, 2, 4, ..., 30). Each integer layer from 0 to 30 is represented by a vertical column in the heatmap. * **Color Scale (Legend):** Positioned on the right side of the chart. It is a vertical bar labeled "Avg JS Divergence". The scale ranges from a light blue/white at the bottom (value `0.1`) to a dark blue at the top (value `0.6`), with intermediate markers at `0.2`, `0.3`, `0.4`, and `0.5`. ### Detailed Analysis The heatmap displays the following patterns for each row (component) across the layers (columns): 1. **Row: `Subj.`** * **Trend:** Starts with very high divergence in the earliest layers, which gradually decreases and fades to very low divergence in the later layers. * **Approximate Values:** Layers 0-12 show the darkest blue, indicating Avg JS Divergence values near or at the maximum of `~0.6`. The color begins to lighten noticeably around layer 14 (`~0.5`), continues to fade through layers 16-20 (`~0.4` to `~0.2`), and becomes very light (near `0.1`) from layer 22 onward. 2. **Row: `Attn.`** * **Trend:** Shows generally low divergence across most layers, with a subtle, localized increase in the middle layers. * **Approximate Values:** Layers 0-10 are very light, indicating values near `0.1`. A slight darkening is visible from approximately layer 12 to layer 20, suggesting a modest increase in divergence to around `~0.2` to `~0.3`. The divergence returns to very low levels (`~0.1`) from layer 22 to layer 30. 3. **Row: `Last.`** * **Trend:** Starts with very low divergence in the early layers and shows a steady, progressive increase in divergence across the subsequent layers. * **Approximate Values:** Layers 0-6 are very light (`~0.1`). A gradual darkening begins around layer 8 (`~0.15`), becoming more pronounced through the middle layers (e.g., layer 16 is `~0.3`, layer 22 is `~0.4`). The divergence continues to increase, with the final layers (28-30) showing a medium blue, corresponding to values of approximately `~0.45` to `0.5`. ### Key Observations * **Inverse Relationship:** The `Subj.` and `Last.` rows exhibit a near-inverse relationship. `Subj.` divergence is highest in early layers and decays, while `Last.` divergence is lowest in early layers and grows. * **Localized Activity:** The `Attn.` row shows a distinct, bounded region of slightly elevated divergence in the model's middle layers (approx. 12-20), unlike the broad trends of the other two rows. * **Maximum Divergence:** The highest divergence values (`~0.6`) are exclusively found in the `Subj.` row for the first dozen layers. * **Layer 22 Transition:** Layer 22 appears to be a transition point where the `Subj.` row's divergence has nearly vanished, the `Attn.` row's minor elevation ends, and the `Last.` row's divergence becomes firmly established. ### Interpretation This heatmap likely visualizes how different types of information or representations evolve within a deep neural network (e.g., a transformer model) across its layers. * **`Subj.` (Subject/Subject Representation):** The high early-layer divergence suggests that subject-related information is processed and is highly variable or "divergent" in the initial stages of the network. Its decay indicates this representation becomes more stable or converges as information flows deeper. * **`Attn.` (Attention):** The localized increase in the middle layers aligns with the hypothesis that attention mechanisms perform significant, focused computations in the intermediate processing stages of the model, before the representations are finalized. * **`Last.` (Last Layer/Final Representation):** The steadily increasing divergence suggests that the final output representation becomes progressively more distinct or specialized layer-by-layer, accumulating information from earlier processing stages. The high divergence in later layers may reflect the model's preparation for a specific, fine-grained prediction task. **Overall Narrative:** The data suggests a processing pipeline where initial layers heavily work on subject-related features (`Subj.`), middle layers engage in focused relational computations (`Attn.`), and later layers build up a complex, divergent final representation (`Last.`) suitable for the model's ultimate objective. The inverse trend between `Subj.` and `Last.` could indicate a transformation from raw, variable input features to a refined, task-specific output.

## Heatmap: Average JS Divergence Across Layers and Categories ### Overview The image is a heatmap visualizing the average JS divergence across three categories ("Subj.", "Attn.", "Last.") and 31 layers (0–30). The color intensity corresponds to divergence values, with darker blue indicating higher divergence (0.6) and lighter blue indicating lower divergence (0.1). The legend on the right maps color intensity to divergence values. --- ### Components/Axes - **Y-Axis (Categories)**: - "Subj." (Subject) - "Attn." (Attention) - "Last." (Last) - **X-Axis (Layers)**: - Labeled "Layer" with integer values from 0 to 30. - **Legend**: - Positioned on the right, labeled "Avg JS Divergence." - Color gradient: Dark blue (0.6) to light blue (0.1). --- ### Detailed Analysis 1. **"Subj." (Subject)**: - Layers 0–14: Dark blue (high divergence, ~0.5–0.6). - Layers 15–30: Gradual lightening (diminishing divergence, ~0.3–0.5). - **Trend**: Sharp decline in divergence after layer 14. 2. **"Attn." (Attention)**: - Layers 0–14: Light blue (low divergence, ~0.1–0.2). - Layers 15–20: Gradual darkening (increasing divergence, ~0.2–0.4). - Layers 21–30: Light blue again (diminishing divergence, ~0.1–0.2). - **Trend**: Peak divergence around layers 15–20. 3. **"Last." (Last)**: - Layers 0–14: Light blue (low divergence, ~0.1–0.2). - Layers 15–30: Gradual darkening (increasing divergence, ~0.2–0.4). - **Trend**: Steady increase in divergence across layers. --- ### Key Observations - **"Subj."** exhibits the highest divergence in early layers (0–14), with a sharp drop afterward. - **"Attn."** shows a bimodal pattern: low divergence in early and late layers, peaking mid-layers (15–20). - **"Last."** demonstrates a consistent upward trend in divergence from layer 15 onward. - The color bar confirms that darker shades correspond to higher divergence values. --- ### Interpretation The heatmap suggests that: - **Early layers (0–14)** are dominated by subject-related divergence ("Subj."), while attention and last-layer divergence are minimal. - **Mid-layers (15–20)** show a shift: attention divergence peaks, and last-layer divergence begins to rise. - **Late layers (21–30)** revert to lower divergence for "Attn." but maintain elevated "Last." divergence. This pattern may indicate that subject-related features dominate early processing, while attention and final-layer representations become more significant in later layers. The divergence trends could reflect hierarchical processing in a neural network or similar system, where early layers focus on raw subject features, and later layers integrate attention and final outputs. No explicit textual data or tables are present beyond the axis labels and legend. All values are inferred from color intensity and spatial positioning.