## Scatter Plot: AC Performance Gain vs. Cosine Similarity ### Overview The image is a scatter plot showing the relationship between AC (Accuracy) performance gain on biographies and the cosine similarity (normalized squared Frobenius norm) of activation spaces A and B. Each point represents a different model configuration, labeled with the model names. The plot visualizes how the similarity of activation spaces correlates with the performance gain. ### Components/Axes * **Title:** AC performance gain vs. "cosine similarity" (normalized squared Frobenius norm) of A, B's activation spaces * **X-axis:** ||Y^T X||_F^2 / (||X||_F^2 ||Y||_F^2) - Represents the cosine similarity between activation spaces. The scale ranges from approximately 0.35 to 0.65, with tick marks at intervals of 0.05. * **Y-axis:** AC performance gain on Biographies (AC perf. - avg(A, B) perf.) - Represents the gain in AC performance. The scale ranges from -0.025 to 0.175, with tick marks at intervals of 0.025. * **Data Points:** Each data point is a blue circle, labeled with the corresponding model names. ### Detailed Analysis The data points, from left to right, are approximately: * **gemma2b,gemma9b:** X: 0.34, Y: -0.01 * **llama3b,llama8b:** X: 0.51, Y: 0.03 * **llama3b,gemma2b:** X: 0.57, Y: 0.02 * **qwen1.5b,gemma2b:** X: 0.57, Y: 0.02 * **qwen1.5b,llama3b:** X: 0.57, Y: 0.10 * **qwen1.5b,qwen3b:** X: 0.63, Y: 0.17 ### Key Observations * There appears to be a positive correlation between cosine similarity and AC performance gain. As the cosine similarity increases, the AC performance gain tends to increase as well. * The models "llama3b,gemma2b" and "qwen1.5b,gemma2b" have similar cosine similarity and AC performance gain values, clustering together. * The model "gemma2b,gemma9b" has the lowest cosine similarity and the lowest AC performance gain. * The model "qwen1.5b,qwen3b" has the highest cosine similarity and the highest AC performance gain. ### Interpretation The scatter plot suggests that a higher cosine similarity between the activation spaces of models A and B is associated with a greater AC performance gain on biographies. This could indicate that models with more similar internal representations tend to perform better in this specific task. The clustering of certain models suggests that specific model combinations yield similar performance characteristics. The outlier "gemma2b,gemma9b" indicates that this model combination may have significantly different internal representations or is less effective for the biography task.

\n ## Scatter Plot: AC Performance Gain vs. Cosine Similarity ### Overview This image presents a scatter plot visualizing the relationship between AC (Accuracy Consistency) performance gain on Biographies and a normalized squared Frobenius norm representing the cosine similarity of activation spaces between models. Each point on the plot represents a pair of models. ### Components/Axes * **X-axis Title:** ||X^TX||/||X||²||Y||² (Represents the normalized squared Frobenius norm) * Scale: Approximately 0.3 to 0.65 * **Y-axis Title:** AC performance gain on Biographies (AC perf. - avg(A, B) perf.) * Scale: Approximately -0.025 to 0.175 * **Data Points:** Each point is labeled with a pair of model names. * **No Legend:** The model pairs are directly labeled on the plot. ### Detailed Analysis The scatter plot displays the following data points (model pairs) and their approximate coordinates: 1. **gemma2b,gemma9b:** (0.35, -0.015) - Located in the bottom-left corner of the plot. 2. **qwen1.5b,llama3b:** (0.55, 0.10) - Located in the upper-middle of the plot. 3. **llama3b,llama8b:** (0.5, 0.025) - Located in the lower-middle of the plot. 4. **llama3b,gemma2b:** (0.58, 0.025) - Located in the lower-middle of the plot, slightly to the right of the previous point. 5. **qwen1.5b,gemma2b:** (0.6, 0.025) - Located in the lower-middle of the plot, slightly to the right of the previous point. 6. **qwen1.5b,qwen3b:** (0.65, 0.17) - Located in the top-right corner of the plot. **Trends:** * There is a general upward trend, suggesting that as the normalized squared Frobenius norm increases, the AC performance gain on Biographies also tends to increase. However, this trend is not strictly linear, and there is considerable scatter. * The points are not clustered, indicating a wide range of performance gains for different model pairs at similar cosine similarity values. ### Key Observations * The model pair `qwen1.5b,qwen3b` exhibits the highest AC performance gain on Biographies (approximately 0.17) and the highest normalized squared Frobenius norm (approximately 0.65). * The model pair `gemma2b,gemma9b` exhibits the lowest AC performance gain on Biographies (approximately -0.015) and the lowest normalized squared Frobenius norm (approximately 0.35). * Several model pairs (llama3b,gemma2b, qwen1.5b,gemma2b) have similar values for both the x and y axes, clustering around (0.58-0.6, 0.025). ### Interpretation The plot suggests a correlation between the cosine similarity of activation spaces and the AC performance gain on Biographies. Higher cosine similarity (as measured by the normalized squared Frobenius norm) appears to be associated with higher AC performance gains. However, the scatter in the data indicates that cosine similarity is not the sole determinant of AC performance. Other factors, such as model architecture, training data, and hyperparameters, likely play a significant role. The negative AC performance gain for the `gemma2b,gemma9b` pair suggests that combining or comparing these models might lead to a decrease in accuracy consistency on Biographies. This could be due to differences in how these models represent biographical information in their activation spaces. The clustering of points around similar values suggests that certain model combinations exhibit similar performance characteristics. Further investigation could explore the reasons for this clustering and identify the underlying factors that contribute to it. The plot provides a visual representation of the relationship between activation space similarity and performance, which can be useful for understanding and improving model behavior.

## Scatter Plot: AC Performance Gain vs. Activation Space Similarity ### Overview This image is a scatter plot comparing the performance gain of an "AC" method against a measure of similarity between the activation spaces of pairs of language models. The plot contains six data points, each labeled with a pair of model names (e.g., "qwen1.5b,qwen3b"). The overall trend suggests a positive correlation: as the activation space similarity increases, the performance gain from AC also tends to increase. ### Components/Axes * **Chart Title:** "AC performance gain vs. "cosine similarity" (normalized squared Frobenius norm) of A,B's activation spaces" * **Y-Axis:** * **Label:** "AC performance gain on Biographies (AC perf. - avg(A, B) perf.)" * **Scale:** Linear, ranging from approximately -0.025 to 0.175. Major tick marks are at intervals of 0.025 (e.g., 0.000, 0.025, 0.050, ..., 0.175). * **Interpretation:** This axis represents the improvement in performance on a "Biographies" task when using the AC method, calculated as the AC performance minus the average performance of the two individual models (A and B). A positive value indicates AC outperforms the average of the pair. * **X-Axis:** * **Label:** `||Y^T X||_F^2 / (||X||_F^2 ||Y||_F^2)` * **Scale:** Linear, ranging from approximately 0.35 to 0.65. Major tick marks are at 0.35, 0.40, 0.45, 0.50, 0.55, 0.60, and 0.65. * **Interpretation:** This is the mathematical formula for the "cosine similarity" metric referenced in the title. It is the normalized squared Frobenius norm of the product of the transposed activation matrix Y and activation matrix X. It quantifies the similarity between the activation spaces of model A (represented by X) and model B (represented by Y). A value closer to 1 indicates higher similarity. * **Data Points:** Six blue circular markers, each annotated with a text label identifying a pair of models. There is no separate legend; labels are placed directly adjacent to their corresponding points. * **Grid:** A light gray dashed grid is present for both major x and y ticks. ### Detailed Analysis The following table reconstructs the data from the six labeled points. Values are approximate based on visual inspection of the plot. | Data Point Label (Model Pair) | Approx. X-Value (Similarity) | Approx. Y-Value (AC Gain) | Spatial Position (Relative) | | :--- | :--- | :--- | :--- | | `qwen1.5b,qwen3b` | 0.63 | 0.17 | Top-right corner | | `qwen1.5b,llama3b` | 0.57 | 0.10 | Upper-middle right | | `llama3b,gemma2b` | 0.57 | 0.03 | Center-right, below the previous point | | `qwen1.5b,gemma2b` | 0.57 | 0.02 | Center-right, just below `llama3b,gemma2b` | | `llama3b,llama8b` | 0.52 | 0.03 | Center | | `gemma2b,gemma9b` | 0.35 | -0.01 | Bottom-left corner | **Trend Verification:** * The data series does not form a line but shows a clear upward trend from left to right. * The point with the lowest similarity (`gemma2b,gemma9b`, x≈0.35) has a slightly negative performance gain. * The point with the highest similarity (`qwen1.5b,qwen3b`, x≈0.63) has the highest performance gain by a significant margin. * Three points cluster around a similarity of 0.57, with performance gains ranging from 0.02 to 0.10. ### Key Observations 1. **Positive Correlation:** There is a strong visual suggestion that higher activation space similarity between two models is associated with a greater performance gain from the AC method. 2. **Highest Performing Pair:** The pair `qwen1.5b,qwen3b` is a clear outlier in terms of gain, achieving nearly double the gain of the next highest point (`qwen1.5b,llama3b`). This pair also exhibits the highest measured similarity. 3. **Negative Gain:** The pair `gemma2b,gemma9b` is the only one showing a negative AC gain (approx. -0.01), indicating the AC method performed slightly worse than the average of the two individual models for this pair. This pair also has the lowest similarity score. 4. **Clustering at Mid-Range Similarity:** Three data points (`qwen1.5b,llama3b`, `llama3b,gemma2b`, `qwen1.5b,gemma2b`) share a very similar x-value (~0.57) but show a wide spread in y-values (0.02 to 0.10). This indicates that while similarity is a strong factor, other model-pair characteristics also significantly influence the AC gain. ### Interpretation The chart presents empirical evidence for a hypothesis in machine learning research: the effectiveness of a model merging or ensemble technique (here called "AC") is related to the representational similarity of the constituent models. * **What the data suggests:** The positive trend implies that the AC method is most beneficial when applied to models whose internal activation spaces are already highly aligned (high cosine similarity). When models are dissimilar (low similarity), the method may offer little benefit or even be detrimental, as seen with the `gemma2b,gemma9b` pair. * **Relationship between elements:** The x-axis (similarity) acts as the independent variable or predictor, while the y-axis (performance gain) is the dependent outcome. The plot tests how well the former predicts the latter. * **Notable anomalies and insights:** The cluster at x≈0.57 is particularly interesting. It shows that for a given level of similarity, the gain can vary substantially. This suggests that similarity is a necessary but not sufficient condition for high gain. The specific nature of the model pair (e.g., architecture family, size difference) likely plays a crucial secondary role. For instance, `qwen1.5b,llama3b` (cross-family) yields a much higher gain than `llama3b,gemma2b` (also cross-family) at the same similarity level, hinting at other underlying factors. * **Peircean investigative reading:** The chart is an indexical sign pointing to a causal relationship. The physical arrangement of points (their correlation) indexes an underlying principle about model compatibility. The symbolic labels (model names) allow us to hypothesize about *why* certain pairs are more compatible. For example, the high-performing `qwen1.5b,qwen3b` pair consists of two models from the same family (Qwen) at different scales, suggesting intra-family merging might be particularly effective with this AC method. Conversely, the negative-gain `gemma2b,gemma9b` pair is also intra-family (Gemma), indicating that family alone is not a guarantee of success; the specific similarity metric captured here is a more direct indicator.

## Scatter Plot: AC Performance Gain vs. Cosine Similarity ### Overview The image is a scatter plot comparing AC performance gain (on the y-axis) against cosine similarity (normalized squared Frobenius norm of activation spaces) on the x-axis. Data points are labeled with pairs of model names (e.g., "qwen1.5b,qwen3b"), and their positions reflect their respective metric values. ### Components/Axes - **Title**: "AC performance gain vs. 'cosine similarity' (normalized squared Frobenius norm) of A,B's activation spaces" - **X-axis**: - Label: `||Y^T X||_F²/(||X||_F² ||Y||_F²)` (normalized squared Frobenius norm) - Scale: 0.35 to 0.65 (approximate) - **Y-axis**: - Label: "AC performance gain on Biographies (AC perf. - avg(A, B) perf.)" - Scale: -0.025 to 0.175 (approximate) - **Data Points**: Blue dots with labels combining model names (e.g., "qwen1.5b,qwen3b"). No explicit legend is present, but all points are blue. ### Detailed Analysis 1. **Data Points**: - **qwen1.5b,qwen3b**: X ≈ 0.63, Y ≈ 0.17 (top-right corner). - **qwen1.5b,llama3b**: X ≈ 0.57, Y ≈ 0.10 (upper-middle-right). - **llama3b,llama8b**: X ≈ 0.52, Y ≈ 0.025 (middle-center). - **llama3b,gemma2b**: X ≈ 0.57, Y ≈ 0.02 (middle-right). - **qwen1.5b,gemma2b**: X ≈ 0.57, Y ≈ 0.015 (middle-right, slightly lower than "llama3b,gemma2b"). - **gemma2b,gemma9b**: X ≈ 0.35, Y ≈ -0.01 (bottom-left). 2. **Trends**: - Higher cosine similarity (x-axis) generally correlates with higher AC performance gain (y-axis), but exceptions exist. - The highest gain (0.17) occurs at the maximum x-value (0.63), suggesting a strong relationship between similarity and performance for "qwen1.5b,qwen3b". - Lower gains (e.g., -0.01 for "gemma2b,gemma9b") occur at lower x-values, indicating poorer performance for dissimilar models. ### Key Observations - **Outlier**: "qwen1.5b,gemma2b" has a high x-value (0.57) but a relatively low y-value (0.015), suggesting that high similarity does not always guarantee high performance. - **Consistency**: "qwen1.5b" pairs (e.g., with "qwen3b" or "llama3b") consistently show higher gains than "llama3b" or "gemma2b" pairs. - **Negative Gain**: "gemma2b,gemma9b" is the only point below the zero line, indicating a performance loss compared to the average. ### Interpretation The data suggests that models with higher cosine similarity in their activation spaces tend to exhibit better AC performance gains. However, the presence of outliers (e.g., "qwen1.5b,gemma2b") implies that similarity alone is not the sole determinant of performance. Architectural differences, training data, or other latent factors may influence results. The negative gain for "gemma2b,gemma9b" highlights cases where dissimilar models underperform, possibly due to conflicting activation patterns. This scatter plot underscores the importance of activation space alignment in model performance but also signals the need for further investigation into confounding variables.