## Scatter Plot Matrix: Brain Alignment vs. Model Performance ### Overview The image presents a matrix of scatter plots analyzing the relationship between brain alignment and model performance for different Pythia models. Each column represents a different Pythia model size (70M, 160M, 2.8B, and 8 Models). Each model has two scatter plots, one plotting "NWP (Perplexity)" against "Brain Alignment", and the other plotting "Behavioral Alignment" against "Brain Alignment". The data points are color-coded to indicate the training stage (Early vs. Late). Each plot includes a regression line with a confidence interval and the Pearson correlation coefficient (r-value) with significance markers. ### Components/Axes * **Titles:** The plots are arranged in a 2x4 grid, with titles above each column indicating the model: (a) Pythia-70M, (b) Pythia-160M, (c) Pythia-2.8B, (d) Pythia (8 Models). * **Y-Axes (Left Column):** The left column has a shared y-axis labeled "NWP (Perplexity)" for the top row and "Behavior" for the bottom row. * **Y-Axes (All Plots):** All plots have a y-axis labeled "Brain Alignment". The scale ranges from approximately 0.2 to 0.5 for the top row and varies slightly for the bottom row. * **X-Axes (Top Row):** The top row has an x-axis labeled "Log(NWP Perplexity)". The scale ranges from approximately 4 to 10. * **X-Axes (Bottom Row):** The bottom row has an x-axis labeled "Behavioral Alignment". The scale ranges from approximately 0.38 to 0.44. * **Legend:** Each plot contains a legend in the top-left corner indicating the "Training Stage": "Early" (represented by circles) and "Late" (represented by squares). The "Early" data points are colored in shades of purple/blue, while the "Late" data points are colored in shades of orange/yellow/red. * **Correlation Coefficient (r):** Each plot displays the Pearson correlation coefficient (r) with significance markers (*, **, ****). "n.s." indicates a non-significant correlation. * **Regression Line:** Each plot includes a regression line with a shaded confidence interval. ### Detailed Analysis **Row 1: NWP (Perplexity) vs. Brain Alignment** * **(a) Pythia-70M:** * Trend: The "Early" data points (blue/purple circles) show a positive trend. * r = 0.92**** * "Late" data points (orange/yellow/red squares) are clustered near x=4. * r = 0.60* * **(b) Pythia-160M:** * Trend: The "Early" data points (blue/purple circles) show a positive trend. * r = 0.89**** * "Late" data points (orange/yellow/red squares) are clustered near x=4. * r = n.s. * **(c) Pythia-2.8B:** * Trend: The "Early" data points (blue/purple circles) show a positive trend. * r = 0.63* * "Late" data points (orange/yellow/red squares) are clustered near x=4. * r = n.s. * **(d) Pythia (8 Models):** * Trend: The "Early" data points (blue/purple circles) show a positive trend. * r = 0.81**** * "Late" data points (orange/yellow/red squares) are clustered near x=4. * r = 0.26** **Row 2: Behavioral Alignment vs. Brain Alignment** * **(a) Pythia-70M:** * Trend: The "Early" data points (blue/purple circles) show a positive trend. * r = 0.97**** * The "Late" data points (orange/yellow/red squares) show a slight negative trend. * r = n.s. * **(b) Pythia-160M:** * Trend: The "Early" data points (blue/purple circles) show a positive trend. * r = 0.90**** * The "Late" data points (orange/yellow/red squares) show a slight negative trend. * r = n.s. * **(c) Pythia-2.8B:** * Trend: The "Early" data points (blue/purple circles) show a positive trend. * r = 0.89**** * The "Late" data points (orange/yellow/red squares) show a negative trend. * r = -0.54* * **(d) Pythia (8 Models):** * Trend: The "Early" data points (blue/purple circles) show a positive trend. * r = 0.84**** * The "Late" data points (orange/yellow/red squares) show a slight negative trend. * r = n.s. ### Key Observations * **NWP (Perplexity) vs. Brain Alignment:** There is a generally positive correlation between Log(NWP Perplexity) and Brain Alignment for the "Early" training stage across all models. The "Late" training stage data points are clustered at low Log(NWP Perplexity) values. * **Behavioral Alignment vs. Brain Alignment:** There is a strong positive correlation between Behavioral Alignment and Brain Alignment for the "Early" training stage across all models. The "Late" training stage shows a weaker or negative correlation. * **Significance:** The correlation coefficients for the "Early" training stage are generally statistically significant (p < 0.0001), while the "Late" training stage correlations are often not significant. ### Interpretation The data suggests that brain alignment, as measured in these models, is strongly correlated with both NWP (Perplexity) and Behavioral Alignment during the early stages of training. The strong positive correlations indicate that as the models learn and their internal representations become more aligned with the human brain, their performance on language tasks (as measured by perplexity) and their behavioral alignment also improve. The clustering of "Late" training stage data points at low Log(NWP Perplexity) values in the top row suggests that the models reach a certain level of performance beyond which further training does not significantly improve perplexity. The weaker or negative correlations observed for the "Late" training stage in the bottom row could indicate that after a certain point, improvements in behavioral alignment do not necessarily translate to improvements in brain alignment, or vice versa. This could be due to overfitting or the models developing different strategies for solving the tasks. The differences in correlation strength and significance between the "Early" and "Late" training stages highlight the dynamic relationship between brain alignment and model performance during the learning process. The plots provide evidence that brain alignment is a useful metric for understanding and potentially improving the performance of language models, particularly during the initial stages of training.

## Scatter Plots: Alignment vs. Perplexity & Behavior ### Overview The image presents four scatter plots, labeled (a) through (d), each representing a different model: Pythia-70M, Pythia-160M, Pythia-2.8B, and Pythia (8 Models). Each plot contains two scatter plots stacked vertically. The top plot shows the relationship between NWP (Perplexity) and Brain Alignment, while the bottom plot shows the relationship between Behavior and Behavioral Alignment. Each plot uses two colors to differentiate between "Early" and "Late" training stages, and includes correlation coefficients (r) with significance indicators. ### Components/Axes Each plot shares the following components: * **X-axis:** Log(NWP Perplexity) in plots (a) and (b), and Behavioral Alignment in plots (c) and (d). Scales range approximately from 3.8 to 10 in (a) and (b), and 0.38 to 0.46 in (c) and (d). * **Y-axis:** Brain Alignment in the top plots (a-d), and Behavior in the bottom plots (a-d). Scales range approximately from 0.2 to 0.5 in all plots. * **Legend:** Located in the top-left corner of each plot, distinguishing between "Early" (green) and "Late" (red) training stages. * **Correlation Coefficient (r):** Displayed in each plot, indicating the strength and direction of the linear relationship between the variables. Significance is indicated by asterisks: * (p < 0.05), ** (p < 0.01), *** (p < 0.001), and n.s. (not significant). * **Regression Line:** A black line representing the linear regression fit for each training stage (Early and Late). * **Shaded Area:** A grey shaded area around each regression line, representing the 95% confidence interval. ### Detailed Analysis or Content Details **Plot (a): Pythia-70M** * **Top Plot (NWP Perplexity vs. Brain Alignment):** * Early (Green): Line slopes downward. r = 0.80*, indicating a strong positive correlation. Approximately 15 data points. * Late (Red): Line slopes downward. r = 0.92***, indicating a very strong positive correlation. Approximately 15 data points. * **Bottom Plot (Behavior vs. Behavioral Alignment):** * Early (Green): Line slopes upward. r = n.s., indicating no significant correlation. Approximately 15 data points. * Late (Red): Line slopes upward. r = 0.97***, indicating a very strong positive correlation. Approximately 15 data points. **Plot (b): Pythia-160M** * **Top Plot (NWP Perplexity vs. Brain Alignment):** * Early (Green): Line is approximately horizontal. r = n.s., indicating no significant correlation. Approximately 15 data points. * Late (Red): Line slopes downward. r = 0.89***, indicating a very strong positive correlation. Approximately 15 data points. * **Bottom Plot (Behavior vs. Behavioral Alignment):** * Early (Green): Line slopes upward. r = n.s., indicating no significant correlation. Approximately 15 data points. * Late (Red): Line slopes upward. r = 0.90***, indicating a very strong positive correlation. Approximately 15 data points. **Plot (c): Pythia-2.8B** * **Top Plot (NWP Perplexity vs. Brain Alignment):** * Early (Green): Line is approximately horizontal. r = n.s., indicating no significant correlation. Approximately 15 data points. * Late (Red): Line slopes downward. r = 0.83***, indicating a very strong positive correlation. Approximately 15 data points. * **Bottom Plot (Behavior vs. Behavioral Alignment):** * Early (Green): Line slopes upward. r = 0.45*, indicating a weak positive correlation. Approximately 15 data points. * Late (Red): Line slopes upward. r = 0.89***, indicating a very strong positive correlation. Approximately 15 data points. **Plot (d): Pythia (8 Models)** * **Top Plot (NWP Perplexity vs. Brain Alignment):** * Early (Green): Line slopes downward. r = 0.28*, indicating a weak positive correlation. Approximately 15 data points. * Late (Red): Line slopes downward. r = 0.81***, indicating a very strong positive correlation. Approximately 15 data points. * **Bottom Plot (Behavior vs. Behavioral Alignment):** * Early (Green): Line slopes upward. r = n.s., indicating no significant correlation. Approximately 15 data points. * Late (Red): Line slopes upward. r = 0.84***, indicating a very strong positive correlation. Approximately 15 data points. ### Key Observations * The "Late" training stage consistently shows a strong negative correlation between NWP Perplexity and Brain Alignment across all models. * The "Late" training stage consistently shows a strong positive correlation between Behavior and Behavioral Alignment across all models. * The "Early" training stage often shows no significant correlation or a weak correlation in both the top and bottom plots. * The strength of the correlation generally increases with model size (from 70M to 2.8B). * The Pythia (8 Models) plot shows a weaker correlation in the top plot for the "Early" stage compared to the other models. ### Interpretation The data suggests that as models are trained ("Late" stage), they exhibit a stronger alignment between their internal representations (Brain Alignment) and their performance on language tasks (NWP Perplexity). Lower perplexity (better performance) is associated with higher brain alignment. Similarly, the "Late" stage shows a strong alignment between the model's behavior and its behavioral alignment, indicating that the model is learning to behave in a more consistent and predictable manner. The lack of significant correlation in the "Early" stage suggests that the models are still in a phase of exploration and haven't yet converged on stable representations or behaviors. The increasing correlation with model size indicates that larger models are better able to learn and maintain these alignments. The outlier in plot (d) for the "Early" stage in the top plot suggests that there might be some variability in the initial stages of training, or that the relationship between NWP Perplexity and Brain Alignment is more complex for larger models. The consistent strong correlations in the "Late" stage across all models suggest a fundamental principle of language model training: as models learn, they develop more coherent internal representations and behaviors.

\n ## Scatter Plot Matrix: Brain Alignment vs. NWP Perplexity and Behavioral Alignment Across Pythia Model Sizes ### Overview The image displays an 8-panel scatter plot matrix arranged in a 2x4 grid. The top row analyzes the relationship between "Brain Alignment" and "Log(NWP Perplexity)". The bottom row analyzes the relationship between "Brain Alignment" and "Behavioral Alignment". Each column corresponds to a different model or set of models from the Pythia family: (a) Pythia-70M, (b) Pythia-160M, (c) Pythia-2.8B, and (d) an aggregate of 8 Pythia models. Data points are categorized by "Training Stage": "Early" (circles) and "Late" (squares). Each panel includes a regression line with a shaded confidence interval and a reported Pearson correlation coefficient (r) with significance levels. ### Components/Axes * **Overall Structure:** 2 rows x 4 columns grid of scatter plots. * **Row Labels (Left Side):** * Top Row: "NWP (Perplexity)" * Bottom Row: "Behavior" * **Column Titles (Top):** * (a) Pythia-70M * (b) Pythia-160M * (c) Pythia-2.8B * (d) Pythia (8 Models) * **Y-Axis (All Panels):** "Brain Alignment". Scale varies slightly per panel but generally ranges from ~0.15 to 0.55. * **X-Axis (Top Row Panels):** "Log(NWP Perplexity)". Scale is inverted, decreasing from left to right (e.g., 10 to 4). * **X-Axis (Bottom Row Panels):** "Behavioral Alignment". Scale is linear and increases from left to right (e.g., 0.39 to 0.44 for panel a). * **Legend (Present in all panels):** "Training Stage" with two categories: * "Early": Represented by circle markers (●). Color varies by panel (shades of blue/purple). * "Late": Represented by square markers (■). Color varies by panel (shades of orange/red/green). * **Statistical Annotations:** Each panel contains one or two text boxes reporting the Pearson correlation coefficient (r) for the respective training stage data, along with significance asterisks (* p<0.05, ** p<0.01, *** p<0.001, **** p<0.0001) or "n.s." for not significant. ### Detailed Analysis **Top Row: NWP (Perplexity) vs. Brain Alignment** * **Trend Verification:** In all panels, the "Early" stage data (blue/purple circles) shows a clear positive trend: as Log(NWP Perplexity) decreases (moving right on the x-axis), Brain Alignment increases. The "Late" stage data (green/yellow squares) is clustered in the top-right corner (low perplexity, high alignment) and shows a weaker or non-significant trend. * **Panel (a) Pythia-70M:** * Early Stage: Strong positive correlation, r = 0.92****. Data points range from approx. (LogP=10.5, BA=0.22) to (LogP=5.5, BA=0.42). * Late Stage: Moderate positive correlation, r = 0.60*. Data points cluster tightly around (LogP=4.5, BA=0.48-0.52). * **Panel (b) Pythia-160M:** * Early Stage: Strong positive correlation, r = 0.89****. Data points range from approx. (LogP=11, BA=0.20) to (LogP=5.5, BA=0.48). * Late Stage: Correlation is not significant (r = n.s.). Data points cluster around (LogP=4.5, BA=0.45-0.50). * **Panel (c) Pythia-2.8B:** * Early Stage: Moderate positive correlation, r = 0.63*. Data points range from approx. (LogP=11, BA=0.20) to (LogP=5.5, BA=0.40). * Late Stage: Correlation is not significant (r = n.s.). Data points cluster around (LogP=4.5, BA=0.38-0.45). * **Panel (d) Pythia (8 Models):** * Early Stage: Strong positive correlation, r = 0.81****. Data points show a clear upward trend from left to right. * Late Stage: Weak positive correlation, r = 0.26**. Data points are densely clustered in the top-right. **Bottom Row: Behavioral Alignment vs. Brain Alignment** * **Trend Verification:** The "Early" stage data (purple circles) consistently shows a strong positive trend: as Behavioral Alignment increases, Brain Alignment increases. The "Late" stage data (orange/red squares) shows a flat or negative trend. * **Panel (a) Pythia-70M:** * Early Stage: Very strong positive correlation, r = 0.97****. Data points form a tight line from approx. (BA=0.39, BrainA=0.20) to (BA=0.44, BrainA=0.42). * Late Stage: Correlation is not significant (r = n.s.). Data points form a horizontal cluster around BrainA=0.50. * **Panel (b) Pythia-160M:** * Early Stage: Strong positive correlation, r = 0.90****. Data points range from approx. (BA=0.38, BrainA=0.19) to (BA=0.44, BrainA=0.42). * Late Stage: Correlation is not significant (r = n.s.). Data points cluster around BrainA=0.48. * **Panel (c) Pythia-2.8B:** * Early Stage: Strong positive correlation, r = 0.89****. Data points range from approx. (BA=0.36, BrainA=0.20) to (BA=0.44, BrainA=0.40). * Late Stage: Moderate *negative* correlation, r = -0.54*. Data points show a slight downward trend. * **Panel (d) Pythia (8 Models):** * Early Stage: Strong positive correlation, r = 0.84****. Data points show a clear upward trend. * Late Stage: Correlation is not significant (r = n.s.). Data points form a dense, horizontal cloud around BrainA=0.50. ### Key Observations 1. **Training Stage Dichotomy:** There is a stark contrast between "Early" and "Late" training stages across all models and metrics. Early stages show strong, significant correlations, while late stages often show non-significant or weak correlations. 2. **Metric Relationship:** For early training, both NWP Perplexity (lower is better) and Behavioral Alignment (higher is better) are strongly positively correlated with Brain Alignment. 3. **Model Size Effect:** The strength of the correlation for the Early stage in the NWP row appears to decrease with model size (r=0.92 for 70M, r=0.89 for 160M, r=0.63 for 2.8B). This pattern is less clear in the Behavior row. 4. **Late-Stage Clustering:** Late-stage data points consistently cluster in regions of high Brain Alignment (>0.4) and high Behavioral Alignment/Low NWP Perplexity, but show little variance, leading to weak correlations. 5. **Negative Correlation Anomaly:** Panel (c) bottom row is the only instance showing a significant negative correlation (r = -0.54*) for the Late stage, suggesting that for the 2.8B model, later training might decouple or inversely relate behavioral and brain alignment. ### Interpretation This data suggests a fundamental shift in the relationship between a language model's internal representations (proxied by "Brain Alignment") and its performance metrics (NWP Perplexity, Behavioral Alignment) over the course of training. * **Early Training Phase:** The model is in a rapid learning phase where improvements in language modeling (lower perplexity) and behavioral mimicry are tightly coupled with the development of brain-like representations. All metrics improve in lockstep. * **Late Training Phase:** The model enters a refinement or specialization phase. Brain Alignment plateaus at a high level, and further improvements in perplexity or behavioral alignment become marginal and decoupled from changes in brain alignment. The model's internal representations stabilize, even as surface-level performance metrics might still see small gains. * **Implication for Alignment:** The strong early correlation suggests that training objectives which improve brain alignment might also naturally lead to better behavioral alignment and language modeling performance, particularly in early stages. However, the decoupling in late stages indicates that achieving the final few percentage points of behavioral alignment may require different techniques, as they are no longer strongly linked to the brain-alignment of the model's representations. The negative correlation in the largest model (2.8B) is a notable outlier that warrants further investigation into the dynamics of very large model training.

## Scatter Plots: Brain Alignment vs. Model Parameters ### Overview The image contains eight scatter plots comparing brain alignment (NWP and Behavioral) with model parameters across different Pythia architectures and training stages. Each plot includes correlation coefficients (r), training stage indicators (Early/Late), and confidence intervals. Data points are color-coded by training stage, with Early (circles) and Late (squares) stages. --- ### Components/Axes 1. **Top Row (NWP Perplexity vs. Brain Alignment)**: - **X-axis**: Log(NWP Perplexity) (logarithmic scale, 4–10) - **Y-axis**: Brain Alignment (Perplexity) (linear scale, 0.15–0.55) - **Legends**: Early (blue circles), Late (orange squares) - **Correlation Labels**: r-values (e.g., r=0.92, r=0.60) with asterisks for significance. 2. **Bottom Row (Behavioral Alignment vs. Brain Alignment)**: - **X-axis**: Behavioral Alignment (linear scale, 0.36–0.46) - **Y-axis**: Brain Alignment (linear scale, 0.15–0.55) - **Legends**: Early (blue circles), Late (orange squares) - **Correlation Labels**: r-values (e.g., r=0.97, r=-0.54) with asterisks. --- ### Detailed Analysis #### Top Row (NWP Perplexity vs. Brain Alignment) 1. **(a) Pythia-70M**: - Early: r=0.92 (strong positive correlation), Late: r=0.60 (moderate positive). - Trend: Early data points cluster tightly along the line; Late points show wider dispersion. - Confidence Interval: Shaded gray band indicates variability. 2. **(b) Pythia-160M**: - Early: r=0.89 (strong positive), Late: r=n.s. (no significant correlation). - Trend: Early points align closely; Late points scatter broadly, especially at high perplexity. 3. **(c) Pythia-2.8B**: - Early: r=0.63 (moderate positive), Late: r=n.s. - Trend: Early points show a gradual increase; Late points cluster at lower brain alignment. 4. **(d) Pythia (8 Models)**: - Early: r=0.81 (strong positive), Late: r=0.26 (weak positive). - Trend: Early points follow a steep upward slope; Late points are more dispersed. #### Bottom Row (Behavioral Alignment vs. Brain Alignment) 1. **(a) Pythia-70M**: - Early: r=0.97 (very strong positive), Late: r=n.s. - Trend: Early points align almost perfectly; Late points scatter widely. 2. **(b) Pythia-160M**: - Early: r=0.89 (strong positive), Late: r=n.s. - Trend: Early points cluster tightly; Late points show no clear pattern. 3. **(c) Pythia-2.8B**: - Early: r=0.89 (strong positive), Late: r=-0.54 (negative correlation). - Trend: Early points follow a steep upward slope; Late points show a downward trend. 4. **(d) Pythia (8 Models)**: - Early: r=0.84 (strong positive), Late: r=0.84 (strong positive). - Trend: Both stages show a consistent upward slope, with Late points slightly more dispersed. --- ### Key Observations 1. **Early Training Dominance**: Early-stage models consistently show stronger correlations (r > 0.8) between brain alignment and model parameters, suggesting better alignment during initial training. 2. **Late-Stage Variability**: Late-stage correlations are weaker or non-significant (r=n.s.) in most cases, except Pythia (8 Models), where Late retains r=0.84. 3. **Negative Correlation Anomaly**: Pythia-2.8B Late stage exhibits a negative correlation (r=-0.54), indicating an inverse relationship between behavioral alignment and brain alignment. 4. **Confidence Intervals**: Shaded regions highlight uncertainty, with wider bands in Late stages, reflecting higher variability. --- ### Interpretation 1. **Training Dynamics**: Early training stages likely capture foundational patterns in brain alignment, while Late stages may overfit or diverge due to optimization pressures. 2. **Model Complexity**: Larger models (e.g., Pythia-160M) show reduced Late-stage correlations, possibly due to increased capacity leading to overfitting. 3. **Behavioral Alignment**: Strong positive correlations in Early stages suggest that behavioral alignment is a robust proxy for brain alignment during initial learning. 4. **Anomaly in Pythia-2.8B Late**: The negative correlation may indicate a shift in learning dynamics, such as prioritizing different features or data artifacts. This analysis underscores the importance of early training stages in aligning model parameters with brain and behavioral data, while Late stages require careful regularization to maintain alignment.