# Technical Data Extraction: Principal Component Analysis (PCA) of Experts This image consists of three side-by-side scatter plots representing a Principal Component Analysis (PCA) of "Experts" in a high-dimensional space, likely related to Mixture-of-Experts (MoE) model pruning or merging. ## General Chart Configuration - **X-Axis:** PC1 (Principal Component 1). Scale ranges from approximately -180 to 100. - **Y-Axis:** PC2 (Principal Component 2). Scale ranges from -60 to 40. - **Grid:** Light gray grid lines are present at intervals of 100 units on the X-axis and 20 units on the Y-axis. - **Layout:** Three panels showing the evolution of expert states: "Original Experts," "Surviving," and "Merged." --- ## 1. Original Experts (Left Panel) This panel shows the initial distribution of all experts. - **Legend/Label:** Original Experts (represented by light red/pink circles). - **Data Points:** There are 10 distinct points. - **Cluster A (Top Right):** 6 points located roughly between PC1 [20, 90] and PC2 [0, 30]. - **Cluster B (Bottom Right):** 1 point located at approximately PC1 [25], PC2 [-58]. - **Cluster C (Bottom Left):** 1 point located at approximately PC1 [-75], PC2 [-48]. - **Cluster D (Top Left):** 2 points located at approximately PC1 [-160 to -170], PC2 [30 to 40]. ## 2. Surviving (Middle Panel) This panel highlights which of the original experts were retained (survived) without being merged. - **Legend/Label:** Surviving (represented by blue circles). - **Context:** The original points are shown in a faded light gray background. - **Data Points:** 4 points are highlighted in blue. - 1 point at PC1 [-85], PC2 [-48]. - 1 point at PC1 [-20], PC2 [-5]. - 1 point at PC1 [-5], PC2 [18]. - 1 point at PC1 [15], PC2 [2]. ## 3. Merged (Right Panel) This panel shows the result of merging experts. - **Legend/Label:** Merged (represented by green 'x' markers). - **Context:** The original points are shown in a faded light gray background. - **Data Points:** 4 points are highlighted with green 'x' markers, representing the new positions of merged expert entities. - 1 point at PC1 [-75], PC2 [1]. - 1 point at PC1 [-25], PC2 [-42]. - 1 point at PC1 [10], PC2 [8]. - 1 point at PC1 [25], PC2 [15]. --- ## Summary of Data Trends 1. **Dimensionality Reduction:** The visualization uses PCA to project expert parameters into a 2D space to observe their relationships. 2. **Sparsification/Consolidation:** The process moves from 10 "Original Experts" to a combination of "Surviving" and "Merged" entities. 3. **Spatial Shift:** The "Merged" experts (green 'x') appear to be positioned at centroids or intermediate locations relative to the original clusters, particularly visible in the top-right and bottom-left regions of the plot.

## Principal Component Analysis (PCA) of Expert Data ### Overview The image displays a Principal Component Analysis (PCA) plot, which is a dimensionality reduction technique used to visualize high-dimensional data in a lower-dimensional space. The plot is divided into three sections, each representing a different group of experts: Original Experts, Surviving, and Merged. ### Components/Axes - **X-axis (PC1)**: Represents the first principal component, which captures the most variance in the data. - **Y-axis (PC2)**: Represents the second principal component, which captures the second most variance in the data. - **Legend**: The legend indicates the color coding for each group of experts. - **Original Experts**: Red dots - **Surviving**: Blue dots - **Merged**: Green crosses ### Detailed Analysis or ### Content Details - **Original Experts**: The red dots are scattered across the plot, indicating a wide range of values for PC1 and PC2. There is no clear clustering or pattern. - **Surviving**: The blue dots are more concentrated towards the upper right quadrant of the plot, suggesting that these experts have higher values for both PC1 and PC2. - **Merged**: The green crosses are also scattered, but they are more towards the lower left quadrant, indicating lower values for both PC1 and PC2. ### Key Observations - **Original Experts**: No clear trend or pattern, suggesting that the original experts have diverse characteristics. - **Surviving**: The concentration of blue dots in the upper right quadrant suggests that these experts have similar characteristics, possibly indicating a commonality or a group. - **Merged**: The green crosses in the lower left quadrant suggest that the merged group has different characteristics compared to the original and surviving groups. ### Interpretation The PCA plot suggests that the original experts have a wide range of characteristics, while the surviving and merged groups have more similar characteristics. The surviving group is more concentrated, indicating a commonality or a group, while the merged group has different characteristics. This could imply that the merging process has resulted in a group with distinct characteristics, possibly due to the merging of different expert groups with varying characteristics.

Here is a detailed technical description of the image. ### **Overview** The image presents a series of three scatter plots arranged horizontally, sharing common X and Y axes. The plots illustrate the distribution of data points in a two-dimensional principal component space, labeled "PC1" and "PC2". Each plot shows a different subset or transformation of the data, distinguished by title, marker color, and marker type. ### **Common Chart Elements** * **Y-axis:** Labeled "**PC2**". The axis has major tick marks and labels at **-60**, **-40**, **-20**, **0**, and **20**. * **X-axis:** Labeled "**PC1**". The axis has major tick marks and labels at **-100**, **0**, and **100**. * **Grid:** A light gray grid is overlaid on all three plots, corresponding to the major tick marks of both axes. --- ### **Plot 1: Original Experts** * **Title:** "**Original Experts**" is displayed at the top center of the plot. * **Legend:** A red circle marker is shown to the left of the title. * **Data Points:** The plot contains several red circle markers representing the "Original Experts". * **Estimated Coordinates (PC1, PC2):** * (-170, 32) * (-75, -48) * (20, 22) * (28, 2) * (35, 6) * (22, -57) * (60, 25) * (90, 15) --- ### **Plot 2: Surviving** * **Title:** "**Surviving**" is displayed at the top center of the plot. * **Legend:** A blue circle marker is shown to the left of the title. * **Data Points:** The plot features blue circle markers representing the "Surviving" points. Faint gray circle markers are also visible in the background, corresponding to the positions of the "Original Experts" from the first plot, providing context. * **Estimated Coordinates of "Surviving" Points (Blue Circles) (PC1, PC2):** * (-85, -47) * (-20, -5) * (-5, 17) * (15, 2) --- ### **Plot 3: Merged** * **Title:** "**Merged**" is displayed at the top center of the plot. * **Legend:** A green 'x' marker is shown to the left of the title. * **Data Points:** The plot displays green 'x' markers representing the "Merged" points. Similar to the second plot, faint gray circle markers are visible in the background, showing the positions of the "Original Experts". * **Estimated Coordinates of "Merged" Points (Green 'x's) (PC1, PC2):** * (-75, 0) * (-25, -41) * (8, 7) * (25, 14)

# Technical Document Extraction: Scatter Plot Analysis ## Overview The image contains three scatter plots arranged horizontally, each representing different data categories. The plots share identical axis labels (PC1 and PC2) and ranges (-100 to 100), but differ in data point distribution and color coding. --- ### **1. Original Experts** - **Legend**: Pink dots labeled "Original Experts" - **Axis Labels**: - X-axis: PC1 (range: -100 to 100) - Y-axis: PC2 (range: -100 to 100) - **Data Distribution**: - Points are scattered across the plot with no clear clustering. - Notable coordinates: - (-100, 20), (0, 20), (50, 20), (100, 20) - (-50, -40), (0, -60) - **Trend**: Dispersed distribution with points concentrated in the upper half (PC2 > 0). --- ### **2. Surviving** - **Legend**: Blue dots labeled "Surviving" - **Axis Labels**: - X-axis: PC1 (range: -100 to 100) - Y-axis: PC2 (range: -100 to 100) - **Data Distribution**: - Points cluster in the positive PC1 and PC2 quadrant. - Notable coordinates: - (-50, -40), (0, 0), (50, 20), (100, 40) - **Trend**: Concentrated in the upper-right quadrant (PC1 > 0, PC2 > 0), suggesting a survival bias toward higher PC1/PC2 values. --- ### **3. Merged** - **Legend**: Green crosses labeled "Merged" - **Axis Labels**: - X-axis: PC1 (range: -100 to 100) - Y-axis: PC2 (range: -100 to 100) - **Data Distribution**: - Fewer points compared to the other plots, indicating data reduction post-merging. - Notable coordinates: - (-50, -40), (0, 0), (50, 20), (100, 40) - **Trend**: Sparse distribution with points aligned along a diagonal from lower-left to upper-right (PC1 ≈ PC2). --- ### **Legend Spatial Grounding** - **Original Experts**: Top-left of the first plot (pink). - **Surviving**: Top-left of the second plot (blue). - **Merged**: Top-left of the third plot (green). --- ### **Key Observations** 1. **Data Reduction**: The "Merged" plot contains significantly fewer points than the original datasets, suggesting a filtering or consolidation process. 2. **Survival Bias**: The "Surviving" category exhibits a clear preference for higher PC1 and PC2 values compared to "Original Experts." 3. **Dimensionality**: All plots use the same principal components (PC1 and PC2), indicating consistent feature extraction across categories. --- ### **Data Table Reconstruction** | Category | Color | Data Points (PC1, PC2) | Count | |-------------------|-------|--------------------------------------------|-------| | Original Experts | Pink | (-100, 20), (0, 20), (50, 20), (100, 20), (-50, -40), (0, -60) | 6 | | Surviving | Blue | (-50, -40), (0, 0), (50, 20), (100, 40) | 4 | | Merged | Green | (-50, -40), (0, 0), (50, 20), (100, 40) | 4 | --- ### **Conclusion** The plots illustrate a dimensionality reduction process where "Original Experts" are dispersed, "Surviving" data points cluster in high-PC1/PC2 regions, and "Merged" data shows reduced dimensionality with aligned points. The consistent axis ranges and legend placement ensure comparability across categories.