## Bar Chart: Percentage of Image/Text Tokens by Expert and Layer ### Overview The image presents three bar charts, each representing a different layer (0, 16, and 23). Each chart shows the percentage of Image and Text tokens for different "Experts" (numbered 0-7). The y-axis represents the percentage of Image/Text tokens, ranging from 0% to 100%. The x-axis represents the different experts. ### Components/Axes * **Titles:** * Top-left chart: "Layer 0" * Top-middle chart: "Layer 16" * Top-right chart: "Layer 23" * **Y-axis:** "% of I/T tokens" (ranging from 0 to 100 in increments of 25) * **X-axis:** "Experts" (categorical, numbered differently for each layer) * **Legend:** Located within each chart. * Orange: "Text" * Teal: "Image" ### Detailed Analysis **Layer 0** * Experts (x-axis): 7, 2, 4, 3, 0, 5, 1, 6 * Trend: Experts 7, 2, 4, 3, and 0 are dominated by Text tokens. Experts 5, 1, and 6 are dominated by Image tokens. * Data Points: * Expert 7: Text ~100%, Image ~0% * Expert 2: Text ~100%, Image ~0% * Expert 4: Text ~100%, Image ~0% * Expert 3: Text ~100%, Image ~0% * Expert 0: Text ~90%, Image ~10% * Expert 5: Text ~30%, Image ~70% * Expert 1: Text ~0%, Image ~100% * Expert 6: Text ~0%, Image ~100% **Layer 16** * Experts (x-axis): 2, 5, 1, 7, 0, 3, 6, 4 * Trend: Experts 2, 5, 1, 7, 0, 3, 6, and 4 have a mix of Text and Image tokens. * Data Points: * Expert 2: Text ~80%, Image ~20% * Expert 5: Text ~80%, Image ~20% * Expert 1: Text ~75%, Image ~25% * Expert 7: Text ~70%, Image ~30% * Expert 0: Text ~65%, Image ~35% * Expert 3: Text ~40%, Image ~60% * Expert 6: Text ~30%, Image ~70% * Expert 4: Text ~20%, Image ~80% **Layer 23** * Experts (x-axis): 7, 2, 0, 4, 5, 6, 1, 3 * Trend: Experts 7, 2, 0, 4, 5, 6, 1, and 3 have a mix of Text and Image tokens. * Data Points: * Expert 7: Text ~95%, Image ~5% * Expert 2: Text ~90%, Image ~10% * Expert 0: Text ~80%, Image ~20% * Expert 4: Text ~60%, Image ~40% * Expert 5: Text ~55%, Image ~45% * Expert 6: Text ~30%, Image ~70% * Expert 1: Text ~20%, Image ~80% * Expert 3: Text ~20%, Image ~80% ### Key Observations * In Layer 0, some experts are highly specialized in either Text or Image tokens. * In Layers 16 and 23, the distribution of Text and Image tokens is more balanced across experts. * The expert order on the x-axis changes between layers. ### Interpretation The charts illustrate how different "experts" within a model (likely a neural network) process image and text tokens at different layers. Layer 0 shows a clear specialization, with some experts focusing almost exclusively on text and others on images. As the data progresses through the network (Layers 16 and 23), the experts become more balanced in their processing of both types of tokens. This suggests that the model is integrating information from both modalities as it goes deeper. The changing order of experts on the x-axis between layers might indicate a re-organization or re-weighting of expert contributions as the model processes the data.

\n ## Stacked Bar Charts: Percentage of L/T Tokens by Expert and Layer ### Overview The image presents three stacked bar charts, each representing a different layer (0, 16, and 23) of a model. Each chart displays the percentage of L/T (likely Language/Text) tokens attributed to either "Image" or "Text" as assessed by different "Experts" (numbered 0 through 7). The charts visually compare the distribution of these token types across layers and expert opinions. ### Components/Axes * **X-axis:** "Experts" - numbered 0 to 7. * **Y-axis:** "% of L/T tokens" - ranging from 0% to 100%. * **Stacked Bars:** Each bar represents an expert's assessment. The bars are divided into two segments: * "Image" (represented by teal/green color) * "Text" (represented by orange/yellow color) * **Titles:** Each chart has a title indicating the layer number: "Layer 0", "Layer 16", "Layer 23". * **Legend:** A small box in each chart identifies the colors corresponding to "Text" and "Image". ### Detailed Analysis or Content Details **Layer 0:** The chart for Layer 0 shows a relatively even distribution between "Image" and "Text" for most experts. * Expert 7: Approximately 10% Image, 90% Text. * Expert 2: Approximately 20% Image, 80% Text. * Expert 4: Approximately 30% Image, 70% Text. * Expert 3: Approximately 35% Image, 65% Text. * Expert 0: Approximately 40% Image, 60% Text. * Expert 5: Approximately 45% Image, 55% Text. * Expert 1: Approximately 50% Image, 50% Text. * Expert 6: Approximately 55% Image, 45% Text. **Layer 16:** The chart for Layer 16 shows a shift towards "Text" being more dominant for most experts. * Expert 2: Approximately 0% Image, 100% Text. * Expert 5: Approximately 10% Image, 90% Text. * Expert 1: Approximately 20% Image, 80% Text. * Expert 7: Approximately 25% Image, 75% Text. * Expert 0: Approximately 30% Image, 70% Text. * Expert 3: Approximately 40% Image, 60% Text. * Expert 6: Approximately 50% Image, 50% Text. * Expert 4: Approximately 60% Image, 40% Text. **Layer 23:** The chart for Layer 23 continues the trend of "Text" dominance, with even higher percentages for most experts. * Expert 7: Approximately 0% Image, 100% Text. * Expert 2: Approximately 10% Image, 90% Text. * Expert 0: Approximately 20% Image, 80% Text. * Expert 4: Approximately 30% Image, 70% Text. * Expert 5: Approximately 35% Image, 65% Text. * Expert 1: Approximately 40% Image, 60% Text. * Expert 3: Approximately 45% Image, 55% Text. * Expert 6: Approximately 50% Image, 50% Text. ### Key Observations * **Trend:** As the layer number increases (0 -> 16 -> 23), the percentage of tokens attributed to "Text" generally increases, while the percentage attributed to "Image" decreases. * **Expert Variability:** There is some variability in the assessments made by different experts, particularly in Layer 0. * **Layer 23 Dominance:** In Layer 23, most experts assign a very high percentage (80% or more) of tokens to "Text". ### Interpretation The data suggests that as the model progresses through deeper layers (from Layer 0 to Layer 23), it increasingly interprets the input as "Text" rather than "Image". This could indicate that the model is learning to abstract away from the raw visual features of the image and focus more on the textual information associated with it. The initial layers (Layer 0) show a more balanced view, likely reflecting the initial processing of the image's visual components. The increasing dominance of "Text" in later layers might be a consequence of the model's training objective or architecture, which may prioritize textual understanding. The variability among experts suggests that the distinction between "Image" and "Text" tokens is not always clear-cut and can be subject to interpretation. The charts provide a visual representation of how the model's internal representation of the input data changes as it processes information through different layers.

## Stacked Bar Chart: Token Allocation by Expert Across Layers ### Overview The image displays three stacked bar charts arranged horizontally, each representing a different layer (Layer 0, Layer 16, Layer 23) of a neural network or similar model. Each chart shows the percentage of "TT tokens" (likely "Text-Text" or a specific token type) allocated to "Text" versus "Image" processing across a set of numbered experts. The overall trend shows a shift from text-dominant processing in early layers to a more balanced or image-heavy allocation in deeper layers. ### Components/Axes * **Chart Type:** Stacked Bar Charts (3 panels). * **Panel Titles (Top Center):** "Layer 0", "Layer 16", "Layer 23". * **Y-Axis (Left Side):** Label: "% of TT tokens". Scale: 0, 25, 50, 75, 100 (percentages). * **X-Axis (Bottom):** Label: "Experts". Each bar is labeled with an expert number below it. * **Legend (Bottom Right of each panel):** A two-color key. * Orange/Gold box: "Text" * Teal/Green box: "Image" * **Data Series:** Each bar is a stack of two segments: the lower orange segment represents the percentage of tokens for "Text", and the upper teal segment represents the percentage for "Image". ### Detailed Analysis **Layer 0 (Left Panel):** * **Experts (Left to Right):** 7, 1, 4, 3, 5, 2, 1, 6. *(Note: Expert "1" appears twice).* * **Trend:** The "Text" (orange) segment dominates all bars, generally occupying 70-95% of the token allocation. The "Image" (teal) segment is a small portion at the top. * **Approximate Data Points (Text %, Image %):** * Expert 7: ~95% Text, ~5% Image. * Expert 1 (first): ~90% Text, ~10% Image. * Expert 4: ~85% Text, ~15% Image. * Expert 3: ~80% Text, ~20% Image. * Expert 5: ~75% Text, ~25% Image. * Expert 2: ~70% Text, ~30% Image. * Expert 1 (second): ~65% Text, ~35% Image. * Expert 6: ~60% Text, ~40% Image. **Layer 16 (Middle Panel):** * **Experts (Left to Right):** 2, 5, 7, 0, 3, 6, 4. * **Trend:** The "Image" (teal) segment is significantly larger than in Layer 0. The allocation is more varied, with some experts still text-heavy and others becoming image-heavy. * **Approximate Data Points (Text %, Image %):** * Expert 2: ~80% Text, ~20% Image. * Expert 5: ~75% Text, ~25% Image. * Expert 7: ~70% Text, ~30% Image. * Expert 0: ~55% Text, ~45% Image. * Expert 3: ~50% Text, ~50% Image. * Expert 6: ~45% Text, ~55% Image. * Expert 4: ~40% Text, ~60% Image. **Layer 23 (Right Panel):** * **Experts (Left to Right):** 7, 1, 0, 2, 5, 6, 1, 3. *(Note: Expert "1" appears twice again).* * **Trend:** The "Image" (teal) segment is now dominant or co-dominant for most experts. The shift towards image processing is most pronounced here. * **Approximate Data Points (Text %, Image %):** * Expert 7: ~70% Text, ~30% Image. * Expert 1 (first): ~65% Text, ~35% Image. * Expert 0: ~50% Text, ~50% Image. * Expert 2: ~45% Text, ~55% Image. * Expert 5: ~40% Text, ~60% Image. * Expert 6: ~35% Text, ~65% Image. * Expert 1 (second): ~30% Text, ~70% Image. * Expert 3: ~25% Text, ~75% Image. ### Key Observations 1. **Layer-Dependent Specialization:** There is a clear progression from Layer 0 to Layer 23. Early layers are heavily specialized for text token processing, while deeper layers show a much higher allocation to image tokens. 2. **Expert Heterogeneity:** Within each layer, different experts show different allocation ratios, indicating functional specialization among experts even at the same depth. 3. **Duplicate Expert Labels:** The expert number "1" appears twice in both Layer 0 and Layer 23. This could indicate two distinct experts with the same ID, a labeling error, or a representation of different attention heads or sub-modules within the same expert. 4. **Inversion of Dominance:** The expert with the highest text allocation in Layer 0 (Expert 7, ~95%) still has a relatively high text allocation in Layer 23 (~70%), but the expert with the highest image allocation in Layer 23 (Expert 3, ~75% Image) had a moderate text allocation in Layer 0 (~80% Text). This suggests the ranking of experts by function changes across layers. ### Interpretation This visualization demonstrates the hierarchical processing within a multimodal model. The data suggests that: * **Early Layers (e.g., Layer 0)** are primarily engaged in processing textual information, likely extracting basic linguistic features. The minimal image token allocation here might represent initial visual feature grounding or cross-modal alignment. * **Intermediate Layers (e.g., Layer 16)** show a transition where visual information becomes increasingly important. This could correspond to stages where the model integrates textual and visual features for more complex understanding. * **Deep Layers (e.g., Layer 23)** are heavily involved in processing image-derived tokens. This implies that high-level reasoning, scene understanding, or generation tasks in the model rely more on visual representations, which have been built up from the earlier layers. The variation among experts within a layer indicates a **"Mixture of Experts" (MoE)** architecture, where different sub-networks (experts) specialize in different types of data or tasks. The shift in specialization across layers is a key finding, showing that the model's processing strategy is not static but evolves with depth, moving from text-centric to a more balanced or image-centric focus for higher-level abstraction. The duplicate expert IDs warrant further investigation to understand the model's exact architecture.

## Grouped Bar Chart: Distribution of Text and Image Token Usage Across Experts in Different Layers ### Overview The image displays three grouped bar charts representing the distribution of text (orange) and image (teal) token usage across experts in three transformer layers (Layer 0, Layer 16, Layer 23). Each chart shows the percentage of text/image tokens processed by individual experts within their respective layers. ### Components/Axes - **X-axis**: Labeled "Experts," listing expert IDs (0–7). Layer 0 includes experts 0–6 (7 experts), while Layers 16 and 23 include experts 0–7 (8 experts). - **Y-axis**: Labeled "% of L/T tokens," scaled from 0% to 100% in 25% increments. - **Legends**: Positioned at the bottom-left of each chart. Orange represents "Text," teal represents "Image." - **Charts**: Three separate bar charts, one per layer, arranged horizontally. ### Detailed Analysis #### Layer 0 - **Experts 0–6**: - Text tokens dominate, with most bars exceeding 75% (e.g., Expert 0: ~80%, Expert 6: ~95%). - Image tokens are minimal, with only Expert 5 showing ~20% image usage. - **Trend**: Nearly uniform text dominance across all experts. #### Layer 16 - **Experts 0–7**: - Mixed distribution: Text ranges from ~40% (Expert 6) to ~80% (Expert 0). - Image tokens increase in mid-to-high experts (e.g., Expert 5: ~40%, Expert 7: ~60%). - **Trend**: Gradual shift toward image token usage in higher-numbered experts. #### Layer 23 - **Experts 0–7**: - Text tokens decline significantly (e.g., Expert 0: ~70%, Expert 7: ~50%). - Image tokens dominate, with Experts 1, 3, and 6 showing ~70–90% image usage. - Outlier: Expert 4 has ~30% image usage, the lowest in the layer. - **Trend**: Strong shift toward image token processing, with experts 1, 3, and 6 specializing in image handling. ### Key Observations 1. **Layer 0**: Text tokens overwhelmingly dominate (80–95% range), with minimal image processing. 2. **Layer 16**: Balanced but uneven distribution, with mid-experts (5–7) handling more image tokens. 3. **Layer 23**: Image tokens dominate (50–90% range), with experts 1, 3, and 6 as primary image processors. 4. **Expert Specialization**: Experts 1, 3, and 6 in Layer 23 exhibit unique roles in image token processing. ### Interpretation The data suggests a hierarchical processing strategy: - **Early Layers (Layer 0)**: Focus on text token extraction, likely for foundational language understanding. - **Mid Layers (Layer 16)**: Begin integrating multimodal data, with some experts specializing in image-text alignment. - **Late Layers (Layer 23)**: Prioritize image token processing, indicating a shift toward visual-semantic synthesis. Experts 1, 3, and 6 in Layer 23 may act as specialized "image gatekeepers," filtering or refining visual information for higher-level tasks. Notable anomalies include Expert 4 in Layer 23 (low image usage) and Expert 5 in Layer 16 (highest image usage in that layer), suggesting potential architectural or functional diversity among experts. The increasing image token reliance in deeper layers aligns with transformer architectures' tendency to handle multimodal integration in later stages.