## Chart: Pythia Model Performance ### Overview The image presents four line charts comparing the performance of Pythia language models of varying sizes (1B, 2.8B, 6.9B, and an average of 5 models) on formal and functional competence tasks. The charts display normalized accuracy against the number of tokens processed during training. ### Components/Axes * **Titles:** * (a) Pythia-1B * (b) Pythia-2.8B * (c) Pythia-6.9B * (d) Pythia (5 Models) * **Y-Axis (Left):** * Label: "Formal Competence" (Top Row), "Functional Competence" (Bottom Row) * Sub-Label: "Normalized Accuracy" * Scale: 0.0 to 0.8 (Top Row), -0.1 to 0.5 (Bottom Row), with increments of 0.1. * **X-Axis (Bottom):** * Label: "Number of Tokens" * Scale: 0 to 256B, with markers at intervals of 2, 4, 8, 12, 16, 24, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256 (all in Billions - 'B'). * **Legend (Bottom):** * **Formal Competence:** * BLiMP (Light Blue, Circle Marker) * SyntaxGym (Light Blue, X Marker) * **Functional Competence:** * ARC-Easy (Dark Blue, Circle Marker) * PIQA (Light Blue, X Marker) * Social-IQA (Dark Blue, Square Marker) * ARC Challenge (Dark Blue, Diamond Marker) * HellaSwag (Dark Blue, Triangle Marker) * WinoGrande (Dark Blue, Plus Marker) ### Detailed Analysis **Chart (a) Pythia-1B:** * **Formal Competence:** * BLiMP (Light Blue, Circle): Starts at approximately 0.05, remains relatively flat until 32B tokens, then increases to approximately 0.65 and plateaus. * SyntaxGym (Light Blue, X): Starts at approximately 0.25, dips slightly around 16B tokens, then increases sharply to approximately 0.78 and plateaus. * **Functional Competence:** * ARC-Easy (Dark Blue, Circle): Starts at approximately 0.0, increases to approximately 0.42 by 256B tokens. * PIQA (Light Blue, X): Starts at approximately 0.0, increases to approximately 0.40 by 256B tokens. * Social-IQA (Dark Blue, Square): Starts at approximately 0.0, increases to approximately 0.10 by 256B tokens. * ARC Challenge (Dark Blue, Diamond): Starts at approximately -0.05, increases to approximately 0.15 by 256B tokens. * HellaSwag (Dark Blue, Triangle): Starts at approximately 0.0, increases to approximately 0.10 by 256B tokens. * WinoGrande (Dark Blue, Plus): Starts at approximately 0.0, increases to approximately 0.10 by 256B tokens. **Chart (b) Pythia-2.8B:** * **Formal Competence:** * BLiMP (Light Blue, Circle): Starts at approximately 0.0, remains relatively flat until 32B tokens, then increases to approximately 0.70 and plateaus. * SyntaxGym (Light Blue, X): Starts at approximately 0.25, dips slightly around 16B tokens, then increases sharply to approximately 0.80 and plateaus. * **Functional Competence:** * ARC-Easy (Dark Blue, Circle): Starts at approximately 0.0, increases to approximately 0.50 by 256B tokens. * PIQA (Light Blue, X): Starts at approximately 0.0, increases to approximately 0.45 by 256B tokens. * Social-IQA (Dark Blue, Square): Starts at approximately 0.0, increases to approximately 0.20 by 256B tokens. * ARC Challenge (Dark Blue, Diamond): Starts at approximately -0.05, increases to approximately 0.30 by 256B tokens. * HellaSwag (Dark Blue, Triangle): Starts at approximately 0.0, increases to approximately 0.20 by 256B tokens. * WinoGrande (Dark Blue, Plus): Starts at approximately 0.0, increases to approximately 0.20 by 256B tokens. **Chart (c) Pythia-6.9B:** * **Formal Competence:** * BLiMP (Light Blue, Circle): Starts at approximately 0.15, remains relatively flat until 32B tokens, then increases to approximately 0.65 and plateaus. * SyntaxGym (Light Blue, X): Starts at approximately 0.25, dips slightly around 16B tokens, then increases sharply to approximately 0.82 and plateaus. * **Functional Competence:** * ARC-Easy (Dark Blue, Circle): Starts at approximately 0.0, increases to approximately 0.52 by 256B tokens. * PIQA (Light Blue, X): Starts at approximately 0.0, increases to approximately 0.48 by 256B tokens. * Social-IQA (Dark Blue, Square): Starts at approximately 0.0, increases to approximately 0.22 by 256B tokens. * ARC Challenge (Dark Blue, Diamond): Starts at approximately -0.05, increases to approximately 0.32 by 256B tokens. * HellaSwag (Dark Blue, Triangle): Starts at approximately 0.0, increases to approximately 0.22 by 256B tokens. * WinoGrande (Dark Blue, Plus): Starts at approximately 0.0, increases to approximately 0.22 by 256B tokens. **Chart (d) Pythia (5 Models):** * **Formal Competence:** * BLiMP (Light Blue, Circle): Starts at approximately 0.05, remains relatively flat until 32B tokens, then increases to approximately 0.65 and plateaus. * SyntaxGym (Light Blue, X): Starts at approximately 0.25, dips slightly around 16B tokens, then increases sharply to approximately 0.82 and plateaus. * **Functional Competence:** * ARC-Easy (Dark Blue, Circle): The shaded region represents the range of performance across the 5 models. The average performance starts at approximately 0.0, increases to approximately 0.45 by 256B tokens. * PIQA (Light Blue, X): The shaded region represents the range of performance across the 5 models. The average performance starts at approximately 0.0, increases to approximately 0.42 by 256B tokens. * Social-IQA (Dark Blue, Square): The shaded region represents the range of performance across the 5 models. The average performance starts at approximately 0.0, increases to approximately 0.15 by 256B tokens. * ARC Challenge (Dark Blue, Diamond): The shaded region represents the range of performance across the 5 models. The average performance starts at approximately -0.05, increases to approximately 0.20 by 256B tokens. * HellaSwag (Dark Blue, Triangle): The shaded region represents the range of performance across the 5 models. The average performance starts at approximately 0.0, increases to approximately 0.15 by 256B tokens. * WinoGrande (Dark Blue, Plus): The shaded region represents the range of performance across the 5 models. The average performance starts at approximately 0.0, increases to approximately 0.15 by 256B tokens. * **Annotation:** * "5.6% of training time" is indicated by a bracket above the x-axis, spanning from 0 to approximately 16B tokens. ### Key Observations * **Formal Competence:** BLiMP and SyntaxGym consistently show a sharp increase in normalized accuracy after processing 32B tokens across all model sizes. SyntaxGym generally achieves higher accuracy than BLiMP. * **Functional Competence:** The performance on functional competence tasks varies significantly. ARC-Easy and PIQA generally outperform Social-IQA, ARC Challenge, HellaSwag, and WinoGrande. * **Model Size:** Increasing the model size (from 1B to 6.9B) generally improves the performance on functional competence tasks, as evidenced by the higher normalized accuracy achieved by larger models. * **Training Time:** The annotation on chart (d) indicates that the initial 5.6% of training time (up to 16B tokens) corresponds to a period of relatively low performance, especially for formal competence tasks. ### Interpretation The charts demonstrate the impact of model size and training progress on the performance of Pythia language models. The sharp increase in formal competence after 32B tokens suggests a critical learning phase. The varying performance across different functional competence tasks highlights the models' strengths and weaknesses in specific areas of reasoning and understanding. The shaded regions in chart (d) provide insight into the variability of performance across different models within the Pythia family. Overall, the data suggests that larger models and longer training times lead to improved performance, but the specific task significantly influences the achieved accuracy.

\n ## Chart: Model Performance vs. Number of Tokens ### Overview The image presents a series of four charts comparing the performance of different Pythia models (1B, 2.8B, 6.9B, and an ensemble of 5 models) on two types of competence tests: Formal and Functional. Performance is measured as Normalized Accuracy against the Number of Tokens processed. Each chart displays multiple data series representing different benchmark datasets. ### Components/Axes * **X-axis:** Number of Tokens (Scale: 0 to approximately 2000, with no specific markings) * **Y-axis (Top Charts - Formal Competence):** Normalized Accuracy (Scale: 0 to 0.8, with markings at 0, 0.2, 0.4, 0.6, and 0.8) * **Y-axis (Bottom Charts - Functional Competence):** Normalized Accuracy (Scale: -0.1 to 0.4, with markings at -0.1, 0, 0.1, 0.2, 0.3, and 0.4) * **Legend:** Located at the bottom-center of the image. * **Formal Competence:** BLIMP (Light Blue), SyntaxGym (Light Green) * **Functional Competence:** ARC-Easy (Dark Green), PIQA (Orange), Social-IQA (Dark Blue), ARC Challenge (Purple), HellaSwag (Teal), and Winogrande (Red) * **Sub-Titles:** Each chart is labeled (a) Pythia-1B, (b) Pythia-2.8B, (c) Pythia-6.9B, (d) Pythia (5 Models) * **Annotation:** Chart (d) includes the annotation "5.6% of training time" ### Detailed Analysis or Content Details **Chart (a) Pythia-1B:** * **Formal Competence:** * BLIMP: Line slopes upward, starting around 0.05 and reaching approximately 0.25. * SyntaxGym: Line is relatively flat, fluctuating around 0.1. * **Functional Competence:** * ARC-Easy: Line starts near 0 and increases to approximately 0.15. * PIQA: Line starts near 0 and increases to approximately 0.2. * Social-IQA: Line starts near 0 and increases to approximately 0.1. * ARC Challenge: Line is relatively flat, fluctuating around 0. * HellaSwag: Line starts near 0 and increases to approximately 0.1. * Winogrande: Line is relatively flat, fluctuating around 0. **Chart (b) Pythia-2.8B:** * **Formal Competence:** * BLIMP: Line slopes upward, starting around 0.1 and reaching approximately 0.4. * SyntaxGym: Line is relatively flat, fluctuating around 0.15. * **Functional Competence:** * ARC-Easy: Line starts near 0 and increases to approximately 0.3. * PIQA: Line starts near 0 and increases to approximately 0.35. * Social-IQA: Line starts near 0 and increases to approximately 0.2. * ARC Challenge: Line is relatively flat, fluctuating around 0. * HellaSwag: Line starts near 0 and increases to approximately 0.2. * Winogrande: Line is relatively flat, fluctuating around 0. **Chart (c) Pythia-6.9B:** * **Formal Competence:** * BLIMP: Line slopes upward, starting around 0.15 and reaching approximately 0.6. * SyntaxGym: Line is relatively flat, fluctuating around 0.2. * **Functional Competence:** * ARC-Easy: Line starts near 0 and increases to approximately 0.4. * PIQA: Line starts near 0 and increases to approximately 0.45. * Social-IQA: Line starts near 0 and increases to approximately 0.3. * ARC Challenge: Line is relatively flat, fluctuating around 0. * HellaSwag: Line starts near 0 and increases to approximately 0.3. * Winogrande: Line is relatively flat, fluctuating around 0. **Chart (d) Pythia (5 Models):** * **Formal Competence:** * BLIMP: Line slopes upward, starting around 0.2 and reaching approximately 0.7. * SyntaxGym: Line is relatively flat, fluctuating around 0.25. * **Functional Competence:** * ARC-Easy: Line starts near 0 and increases to approximately 0.3. * PIQA: Line starts near 0 and increases to approximately 0.35. * Social-IQA: Line starts near 0 and increases to approximately 0.25. * ARC Challenge: Line is relatively flat, fluctuating around 0. * HellaSwag: Line starts near 0 and increases to approximately 0.25. * Winogrande: Line is relatively flat, fluctuating around 0. ### Key Observations * Generally, performance on both Formal and Functional Competence tasks increases with model size (1B to 6.9B to 5 Models). * BLIMP consistently shows the highest performance among the Formal Competence benchmarks. * PIQA consistently shows the highest performance among the Functional Competence benchmarks. * ARC Challenge and Winogrande consistently show the lowest performance across all models. * The 5-model ensemble shows the highest overall performance, particularly on Formal Competence. * The shaded areas around the lines represent the variance in performance. ### Interpretation The charts demonstrate a clear positive correlation between model size and performance on both Formal and Functional competence benchmarks. Larger models (Pythia-6.9B and the 5-model ensemble) consistently outperform smaller models (Pythia-1B and Pythia-2.8B). This suggests that increasing model capacity leads to improved ability to process and understand language. The divergence in performance between Formal and Functional Competence tasks indicates that the models may be better at tasks requiring strict grammatical understanding (Formal) than those requiring real-world reasoning and common sense (Functional). The consistently low performance on ARC Challenge and Winogrande suggests these tasks are particularly challenging for the models, potentially due to their reliance on complex reasoning or nuanced understanding of context. The annotation "5.6% of training time" on chart (d) suggests that the 5-model ensemble achieves its superior performance at a computational cost, requiring significantly more training time than the individual models. This highlights the trade-off between performance and efficiency in model development. The shaded areas around the lines indicate the variability in performance, which could be due to factors such as data sampling or model initialization.

## Multi-Panel Line Chart: Pythia Model Performance Across Training Tokens ### Overview This image contains eight line charts arranged in a 2x4 grid, displaying the performance of different-sized Pythia language models on various benchmarks as a function of training data (number of tokens). The charts are grouped into two rows representing two broad categories of model capability: "Formal Competence" (top row) and "Functional Competence" (bottom row). The columns correspond to different model sizes: (a) Pythia-1B, (b) Pythia-2.8B, (c) Pythia-6.9B, and (d) an aggregate of 5 Pythia models. A comprehensive legend is provided at the bottom. ### Components/Axes * **Chart Titles (Column Headers):** * (a) Pythia-1B * (b) Pythia-2.8B * (c) Pythia-6.9B * (d) Pythia (5 Models) * **Row Labels (Y-axis Titles for each row):** * Top Row: "Formal Competence" * Bottom Row: "Functional Competence" * **Axes (for all 8 charts):** * **X-axis:** "Number of Tokens". Scale is logarithmic, with major tick marks at: 0, 2M, 4M, 8M, 16M, 32M, 64M, 128M, 256M, 512M, 1B, 2B, 4B, 8B, 16B, 32B, 64B, 100B, 128B, 144B, 160B, 176B, 192B, 208B, 224B, 256B, 288B. * **Y-axis:** "Normalized Accuracy". Scale is linear, ranging from approximately -0.1 to 0.8 or 0.9 depending on the chart. * **Legend (Bottom of image):** * **Formal Competence:** * Light blue circle marker: **BLiMP** * Light blue 'x' marker: **SyntaxGym** * **Functional Competence:** * Medium blue circle marker: **ARC-Easy** * Medium blue 'x' marker: **PIQA** * Medium blue square marker: **Social-IQA** * Dark blue diamond marker: **ARC Challenge** * Dark blue star/asterisk marker: **HellaSwag** * Dark blue plus '+' marker: **WinoGrande** * **Annotation (Panel d, bottom row):** A bracket spanning from ~0 to ~16B tokens with the text "5.6% of training time". ### Detailed Analysis **Top Row - Formal Competence (BLiMP & SyntaxGym):** * **Trend:** For all model sizes (1B, 2.8B, 6.9B, and the 5-model aggregate), both BLiMP and SyntaxGym show a very similar pattern. Performance remains low and relatively flat (around 0.1-0.25 normalized accuracy) for training token counts up to approximately 512M-1B tokens. * **Key Transition:** Between 1B and 4B tokens, there is a sharp, near-vertical increase in accuracy for both benchmarks. * **Plateau:** After ~4B tokens, performance plateaus. BLiMP plateaus at a higher level (~0.65-0.7) than SyntaxGym (~0.8-0.85). This plateau is consistent across all model sizes. * **Data Points (Approximate Plateau Values):** * **BLiMP:** ~0.65 (1B), ~0.66 (2.8B), ~0.65 (6.9B), ~0.65 (5 Models). * **SyntaxGym:** ~0.82 (1B), ~0.83 (2.8B), ~0.84 (6.9B), ~0.83 (5 Models). **Bottom Row - Functional Competence (ARC-Easy, PIQA, Social-IQA, ARC Challenge, HellaSwag, WinoGrande):** * **General Trend:** All six benchmarks show a more gradual and varied learning curve compared to the formal competence tasks. Performance generally improves with more training tokens, but the rate and final level differ significantly by task. * **Task Hierarchy (by final performance):** 1. **ARC-Easy & PIQA (Top Performers):** These two tasks (medium blue circle and 'x') show the strongest and most consistent improvement. They start near 0, begin a steady rise around 256M-512M tokens, and continue climbing, reaching ~0.4-0.5 normalized accuracy by 288B tokens. Their curves are closely aligned. 2. **Social-IQA (Mid-tier):** The medium blue square line shows moderate improvement, starting near 0 and rising to approximately 0.1-0.2 by 288B tokens. 3. **ARC Challenge, HellaSwag, WinoGrande (Lower Performers):** These three tasks (dark blue diamond, star, plus) show the slowest growth. They often start at or below zero normalized accuracy. They begin to rise noticeably only after 1B-2B tokens and reach final values between ~0.0 and ~0.25, with significant variance between tasks and model sizes. HellaSwag (star) often shows the lowest performance. * **Model Size Comparison:** Larger models (2.8B, 6.9B) generally achieve slightly higher final accuracy on these functional tasks than the 1B model, but the overall shape of the learning curves is consistent. * **5-Model Aggregate (Panel d):** This chart includes shaded error bands, indicating variance across the five models. The "5.6% of training time" annotation highlights that the initial, low-performance phase constitutes a small fraction of the total training budget before significant gains are observed. ### Key Observations 1. **Phase Transition:** A dramatic, synchronized phase transition occurs for *all* benchmarks (both formal and functional) between 512M and 4B training tokens. This suggests a critical point in training where fundamental capabilities are acquired. 2. **Competence Dichotomy:** There is a clear separation between "Formal Competence" (linguistic syntax/grammar tasks like BLiMP, SyntaxGym) and "Functional Competence" (reasoning/knowledge tasks like ARC, PIQA). Formal competence is mastered quickly and to a high level after the phase transition, while functional competence improves more gradually and plateaus at lower levels. 3. **Task Difficulty Spectrum:** Within functional competence, a clear hierarchy of difficulty is evident, with ARC-Easy/PIQA being "easier" than Social-IQA, which is easier than ARC Challenge/HellaSwag/WinoGrande. 4. **Scalability:** The patterns are remarkably consistent across model sizes (1B to 6.9B parameters), indicating that these learning dynamics are a property of the training process and data, not just model scale. Larger models show modest performance gains but follow the same trajectory. ### Interpretation The data demonstrates a fundamental characteristic of large language model training: **capability acquisition is not linear**. Models spend a significant portion of early training (the first ~5.6% of tokens, up to ~16B) in a low-competence state, building basic statistical regularities. Then, a rapid phase transition occurs where core linguistic (formal) and reasoning (functional) abilities emerge almost simultaneously across a wide range of benchmarks. The stark difference between the high, flat plateaus of formal competence and the lower, still-rising curves of functional competence suggests that mastering syntactic structure is a prerequisite that is achieved relatively "easily" once sufficient data is seen. In contrast, the knowledge and complex reasoning required for functional tasks are harder to acquire and may continue to improve with even more data or different training approaches. The consistency across model sizes implies these are robust phenomena in the scaling of transformer-based LMs trained on natural language corpora. The charts effectively argue that "more data" leads to a predictable, non-linear unlocking of capabilities, with different skill sets emerging on different timelines.

## Line Graphs: Model Performance Across Formal and Functional Competence ### Overview The image contains four grouped line graphs comparing model performance across formal and functional competence tasks. Each graph represents a different Pythia model size (1B, 2.8B, 6.9B, and 5 Models) with two subplots per graph: formal competence (top) and functional competence (bottom). The graphs show normalized accuracy against the number of tokens, with shaded regions indicating 95% confidence intervals. ### Components/Axes - **X-axis**: Number of Tokens (0 to 256, logarithmic scale) - **Y-axis**: Normalized Accuracy (-0.1 to 0.8) - **Legends**: - **Formal Competence**: - BLiMP (light blue circles) - SyntaxGym (light blue crosses) - **Functional Competence**: - ARC-Easy (dark blue circles) - PIQA (dark blue crosses) - Social-IQA (dark blue squares) - ARC Challenge (dark blue diamonds) - HellaSwag (dark blue stars) - WinoGrande (dark blue plus signs) - **Shading**: Light blue regions represent 95% confidence intervals. ### Detailed Analysis #### Formal Competence (Top Subplots) - **Pythia-1B (a)**: - BLiMP and SyntaxGym start at ~0.1 accuracy, plateauing at ~0.7-0.8 after ~100 tokens. - Confidence intervals narrow significantly after 100 tokens. - **Pythia-2.8B (b)**: - Similar trend to 1B but with slightly higher initial accuracy (~0.2 vs. 0.1). - Plateaus at ~0.7-0.8 with tighter confidence intervals. - **Pythia-6.9B (c)**: - Rapid rise to ~0.7 accuracy by ~50 tokens, plateauing at ~0.8. - Confidence intervals remain narrow throughout. - **Pythia (5 Models) (d)**: - Combines results from multiple models, showing ~0.7 accuracy by ~100 tokens. - Confidence intervals widen slightly compared to single-model graphs. #### Functional Competence (Bottom Subplots) - **Pythia-1B (a)**: - ARC-Easy and PIQA start near 0, peaking at ~0.3-0.4 after ~200 tokens. - Social-IQA and ARC Challenge show negative accuracy (-0.1 to 0.1) initially. - HellaSwag and WinoGrande plateau at ~0.2-0.3. - **Pythia-2.8B (b)**: - ARC-Easy and PIQA reach ~0.4-0.5, with Social-IQA and ARC Challenge improving to ~0.1-0.2. - HellaSwag and WinoGrande plateau at ~0.3-0.4. - **Pythia-6.9B (c)**: - ARC-Easy and PIQA peak at ~0.5-0.6, with Social-IQA and ARC Challenge reaching ~0.3-0.4. - HellaSwag and WinoGrande plateau at ~0.4-0.5. - **Pythia (5 Models) (d)**: - Combines results, showing ~0.4-0.5 accuracy for most tasks. - Confidence intervals are wider, indicating higher variability. ### Key Observations 1. **Formal vs. Functional Tasks**: - Formal tasks (BLiMP, SyntaxGym) consistently outperform functional tasks across all models. - Functional tasks show greater variability, with some models (e.g., Social-IQA, ARC Challenge) performing poorly in smaller models. 2. **Model Size Impact**: - Larger models (6.9B) achieve higher accuracy in both task types compared to smaller models (1B, 2.8B). - The "5 Models" graph (d) suggests ensemble approaches improve functional task performance but with increased computational cost (5.6% training time noted). 3. **Confidence Intervals**: - Functional tasks exhibit wider confidence intervals, indicating less reliable performance estimates. - Formal tasks show tighter intervals, suggesting more stable results. ### Interpretation The data demonstrates that larger language models (e.g., Pythia-6.9B) excel in formal linguistic tasks (e.g., syntax, grammar) but struggle with functional reasoning (e.g., commonsense, logic). Functional tasks require more tokens to reach stable performance, and smaller models often fail to achieve meaningful accuracy. The "5 Models" graph highlights a trade-off between performance gains and training efficiency, as combining models improves functional task results but increases computational overhead. The shaded confidence intervals emphasize the uncertainty in functional task evaluations, suggesting these tasks may require more robust evaluation frameworks.