## Line Chart: Model Accuracy vs. Training Progress for Various Datasets and Model Sizes ### Overview The image presents a series of line charts comparing the RM@8 accuracy of different model sizes (0.5b, 1.5b, 3b, 7b, 14b, 32b) across various training progress levels (0.0 to 1.0) for different datasets. Each chart corresponds to a specific dataset: GSM8K, Math 500, Minerva Math, Gaokao2023EN, Olympiad Bench, College Math, MMLU STEM, and an average across all datasets. The charts share a common legend indicating model sizes by color. ### Components/Axes * **X-axis (Horizontal):** Training Progress, ranging from 0.0 to 1.0 in increments of 0.2. * **Y-axis (Vertical):** RM@8 Accuracy, with varying scales depending on the dataset. * **Titles:** Each chart has a title indicating the dataset name (e.g., "GSM8K", "Math 500"). * **Legend (Right):** A vertical color bar indicating model sizes: 0.5b (dark purple), 1.5b (purple), 3b (dark blue), 7b (teal), 14b (light green), and 32b (yellow). * **Gridlines:** Each chart has faint grey gridlines to aid in reading values. ### Detailed Analysis **1. GSM8K** * Y-axis ranges from 94.5 to 97.0. * **0.5b (dark purple):** Starts at approximately 94.6, dips slightly, then increases to around 95.3. * **1.5b (purple):** Starts at approximately 95.3, dips slightly, then increases to around 95.4. * **3b (dark blue):** Relatively flat at approximately 96.0. * **7b (teal):** Relatively flat at approximately 96.4. * **14b (light green):** Relatively flat at approximately 96.7. * **32b (yellow):** Relatively flat at approximately 97.1. **2. Math 500** * Y-axis ranges from 79 to 87. * **0.5b (dark purple):** Starts at approximately 81.5, dips to 79, then increases to around 82. * **1.5b (purple):** Starts at approximately 82, dips to 81, then increases to around 82.5. * **3b (dark blue):** Relatively flat at approximately 83.5. * **7b (teal):** Relatively flat at approximately 84. * **14b (light green):** Relatively flat at approximately 85. * **32b (yellow):** Relatively flat at approximately 86.5. **3. Minerva Math** * Y-axis ranges from 36 to 44. * **0.5b (dark purple):** Starts at approximately 37.5, dips slightly, then increases to around 37. * **1.5b (purple):** Starts at approximately 37, dips slightly, then increases to around 37. * **3b (dark blue):** Starts at approximately 38, increases to 40, then decreases to around 39. * **7b (teal):** Relatively flat at approximately 41. * **14b (light green):** Relatively flat at approximately 42.5. * **32b (yellow):** Relatively flat at approximately 44. **4. Gaokao2023EN** * Y-axis ranges from 66 to 74. * **0.5b (dark purple):** Starts at approximately 67.5, increases to 68.5, then decreases to around 68. * **1.5b (purple):** Starts at approximately 68.5, increases to 69, then decreases to around 68. * **3b (dark blue):** Starts at approximately 69.5, increases to 70.5, then decreases to around 70. * **7b (teal):** Starts at approximately 72.5, increases to 73, then decreases to around 72. * **14b (light green):** Starts at approximately 73, increases to 73.5, then decreases to around 73. * **32b (yellow):** Relatively flat at approximately 74. **5. Olympiad Bench** * Y-axis ranges from 40 to 48. * **0.5b (dark purple):** Starts at approximately 41.5, dips to 40, then increases to around 41.5. * **1.5b (purple):** Starts at approximately 43, increases to 44, then decreases to around 44. * **3b (dark blue):** Relatively flat at approximately 44.5. * **7b (teal):** Relatively flat at approximately 46. * **14b (light green):** Relatively flat at approximately 46.5. * **32b (yellow):** Relatively flat at approximately 48. **6. College Math** * Y-axis ranges from 42.5 to 46.5. * **0.5b (dark purple):** Starts at approximately 43.5, dips to 43, then increases to around 44.5. * **1.5b (purple):** Starts at approximately 44.5, increases to 45, then decreases to around 45. * **3b (dark blue):** Relatively flat at approximately 45.5. * **7b (teal):** Relatively flat at approximately 46. * **14b (light green):** Relatively flat at approximately 46. * **32b (yellow):** Relatively flat at approximately 46.5. **7. MMLU STEM** * Y-axis ranges from 75 to 87. * **0.5b (dark purple):** Starts at approximately 78, dips to 76, then increases to around 77. * **1.5b (purple):** Starts at approximately 78, dips to 76, then increases to around 77. * **3b (dark blue):** Starts at approximately 78, increases to 80, then decreases to around 80. * **7b (teal):** Starts at approximately 82, increases to 83, then decreases to around 83. * **14b (light green):** Starts at approximately 84, increases to 85, then decreases to around 85. * **32b (yellow):** Starts at approximately 85, increases to 86, then decreases to around 86. **8. Average** * Y-axis ranges from 62 to 70. * **0.5b (dark purple):** Starts at approximately 63, dips to 62.5, then increases to around 63.5. * **1.5b (purple):** Starts at approximately 63.5, dips to 63, then increases to around 64. * **3b (dark blue):** Starts at approximately 65, increases to 66, then decreases to around 65.5. * **7b (teal):** Starts at approximately 67, increases to 68, then decreases to around 67.5. * **14b (light green):** Starts at approximately 68, increases to 68.5, then decreases to around 68. * **32b (yellow):** Starts at approximately 68.5, increases to 69, then decreases to around 69. ### Key Observations * Generally, larger model sizes (32b, 14b) achieve higher RM@8 accuracy compared to smaller models (0.5b, 1.5b) across all datasets. * The "Average" chart shows a clear separation between model sizes, with larger models consistently outperforming smaller ones. * The performance difference between model sizes is more pronounced in some datasets (e.g., MMLU STEM) than others (e.g., College Math). * The initial training progress (0.0 to 0.2) often shows a significant increase in accuracy for smaller models, while larger models tend to be more stable. * Some datasets (e.g., Math 500, Olympiad Bench) exhibit a dip in accuracy for smaller models around the 0.2 to 0.4 training progress mark. ### Interpretation The data suggests that increasing model size generally leads to improved performance, as measured by RM@8 accuracy, across a variety of datasets. However, the extent of improvement varies depending on the specific dataset. The initial training phase appears to be more critical for smaller models, as they exhibit more significant fluctuations in accuracy compared to larger models. The observed dips in accuracy for smaller models in certain datasets might indicate overfitting or instability during the early stages of training. The "Average" chart provides a consolidated view, highlighting the overall trend of larger models achieving higher accuracy.

## Charts: Model Performance vs. Training Progress ### Overview The image presents a 3x4 grid of line charts, each depicting the relationship between RM@8 Accuracy and Training Progress for different datasets and model sizes. The charts compare the performance of models with varying numbers of parameters (0.5b, 1.5b, 3b, 7b, 14b, and 32b) as they are trained. The datasets include GSM8K, Math 500, Minerva Math, Gaokao2023EN, Olympiad Bench, College Math, MMLU STEM, and an Average across all datasets. ### Components/Axes * **X-axis:** Training Progress, ranging from 0.0 to 1.0. * **Y-axis:** RM@8 Accuracy, with varying scales depending on the dataset. * GSM8K: 94.5 to 97.0 * Math 500: 79 to 87 * Minerva Math: 36 to 44 * Gaokao2023EN: 67 to 74 * Olympiad Bench: 40 to 48 * College Math: 43 to 46.5 * MMLU STEM: 76 to 86 * Average: 63 to 69 * **Legend:** Located on the right side of the image, indicating model sizes with corresponding colors: * 32b (Dark Blue) * 14b (Light Blue) * 7b (Green) * 3b (Red) * 1.5b (Orange) * 0.5b (Purple) ### Detailed Analysis or Content Details **GSM8K:** * The 32b model (Dark Blue) shows a slight upward trend, starting around 96.2 and ending around 96.8. * The 14b model (Light Blue) starts around 95.8 and ends around 96.5, showing a moderate upward trend. * The 7b model (Green) starts around 95.5 and ends around 96.2, with a slight upward trend. * The 3b model (Red) starts around 95.2 and ends around 95.8, with a minimal upward trend. * The 1.5b model (Orange) starts around 94.8 and ends around 95.3, with a slight upward trend. * The 0.5b model (Purple) starts around 94.5 and ends around 95.0, with a minimal upward trend. **Math 500:** * The 32b model (Dark Blue) shows a slight downward trend, starting around 86.5 and ending around 85.5. * The 14b model (Light Blue) shows a downward trend, starting around 85.0 and ending around 83.5. * The 7b model (Green) shows a downward trend, starting around 84.0 and ending around 82.0. * The 3b model (Red) shows a downward trend, starting around 83.0 and ending around 80.5. * The 1.5b model (Orange) shows a downward trend, starting around 82.0 and ending around 79.5. * The 0.5b model (Purple) shows a downward trend, starting around 81.0 and ending around 79.0. **Minerva Math:** * The 32b model (Dark Blue) shows a slight upward trend, starting around 41.5 and ending around 42.5. * The 14b model (Light Blue) shows a slight upward trend, starting around 40.5 and ending around 41.5. * The 7b model (Green) shows a slight upward trend, starting around 39.5 and ending around 40.5. * The 3b model (Red) shows a slight upward trend, starting around 38.5 and ending around 39.5. * The 1.5b model (Orange) shows a slight upward trend, starting around 37.5 and ending around 38.5. * The 0.5b model (Purple) shows a slight upward trend, starting around 36.5 and ending around 37.5. **Gaokao2023EN:** * The 32b model (Dark Blue) shows a slight upward trend, starting around 72.5 and ending around 73.5. * The 14b model (Light Blue) shows a slight upward trend, starting around 71.5 and ending around 72.5. * The 7b model (Green) shows a slight upward trend, starting around 70.5 and ending around 71.5. * The 3b model (Red) shows a slight upward trend, starting around 69.5 and ending around 70.5. * The 1.5b model (Orange) shows a slight upward trend, starting around 68.5 and ending around 69.5. * The 0.5b model (Purple) shows a slight upward trend, starting around 67.5 and ending around 68.5. **Olympiad Bench:** * The 32b model (Dark Blue) shows a slight upward trend, starting around 45.5 and ending around 46.5. * The 14b model (Light Blue) shows a slight upward trend, starting around 44.5 and ending around 45.5. * The 7b model (Green) shows a slight upward trend, starting around 43.5 and ending around 44.5. * The 3b model (Red) shows a slight upward trend, starting around 42.5 and ending around 43.5. * The 1.5b model (Orange) shows a slight upward trend, starting around 41.5 and ending around 42.5. * The 0.5b model (Purple) shows a slight upward trend, starting around 40.5 and ending around 41.5. **College Math:** * The 32b model (Dark Blue) shows a slight upward trend, starting around 45.5 and ending around 46.0. * The 14b model (Light Blue) shows a slight upward trend, starting around 44.5 and ending around 45.0. * The 7b model (Green) shows a slight upward trend, starting around 43.5 and ending around 44.0. * The 3b model (Red) shows a slight upward trend, starting around 42.5 and ending around 43.0. * The 1.5b model (Orange) shows a slight upward trend, starting around 41.5 and ending around 42.0. * The 0.5b model (Purple) shows a slight upward trend, starting around 40.5 and ending around 41.0. **MMLU STEM:** * The 32b model (Dark Blue) shows a slight upward trend, starting around 82.5 and ending around 83.5. * The 14b model (Light Blue) shows a slight upward trend, starting around 81.5 and ending around 82.5. * The 7b model (Green) shows a slight upward trend, starting around 80.5 and ending around 81.5. * The 3b model (Red) shows a slight upward trend, starting around 79.5 and ending around 80.5. * The 1.5b model (Orange) shows a slight upward trend, starting around 78.5 and ending around 79.5. * The 0.5b model (Purple) shows a slight upward trend, starting around 76.5 and ending around 77.5. **Average:** * The 32b model (Dark Blue) shows a slight upward trend, starting around 66.5 and ending around 67.5. * The 14b model (Light Blue) shows a slight upward trend, starting around 65.5 and ending around 66.5. * The 7b model (Green) shows a slight upward trend, starting around 64.5 and ending around 65.5. * The 3b model (Red) shows a slight upward trend, starting around 63.5 and ending around 64.5. * The 1.5b model (Orange) shows a slight upward trend, starting around 63.0 and ending around 64.0. * The 0.5b model (Purple) shows a slight upward trend, starting around 62.5 and ending around 63.5. ### Key Observations * Generally, larger models (32b, 14b) exhibit slightly higher accuracy than smaller models (0.5b, 1.5b) across most datasets. * The Math 500 dataset shows a consistent downward trend in accuracy for all model sizes, suggesting potential difficulties in this domain. * The Minerva Math dataset shows the lowest overall accuracy levels, indicating a challenging task. * The GSM8K dataset shows the highest overall accuracy levels. * The accuracy improvements across training progress are generally small, suggesting diminishing returns. ### Interpretation The charts demonstrate the performance of language models of varying sizes on a range of tasks. The consistent trend of larger models outperforming smaller models highlights the benefits of increased model capacity. However, the relatively small accuracy gains with increased training progress suggest that the models may be approaching their performance limits or that the training data is not sufficiently diverse or challenging. The downward trend observed in the Math 500 dataset indicates that this task is particularly difficult for these models, potentially requiring specialized architectures or training techniques. The differences in performance across datasets suggest that the models' capabilities vary depending on the nature of the task. The average chart provides a consolidated view of performance, showing that even the largest models have room for improvement. The data suggests that scaling model size is a viable strategy for improving performance, but it is not a panacea, and further research is needed to address the challenges posed by specific tasks like Math 500.

## Multi-Chart Performance Analysis: Model Size vs. Training Progress ### Overview The image displays a grid of eight line charts arranged in two rows of four. Each chart plots the performance of different-sized language models on a specific benchmark as a function of training progress. The overall visualization compares how model scale (from 0.5 billion to 32 billion parameters) affects learning trajectories across diverse mathematical and reasoning tasks. ### Components/Axes * **Chart Grid:** 8 individual line charts. * **Chart Titles (Top Row, Left to Right):** `GSM8K`, `Math 500`, `Minerva Math`, `Gaokao2023EN`. * **Chart Titles (Bottom Row, Left to Right):** `Olympiad Bench`, `College Math`, `MMLU STEM`, `Average`. * **X-Axis (All Charts):** Labeled `Training Progress`. Scale ranges from `0.0` to `1.0` with major ticks at 0.0, 0.2, 0.4, 0.6, 0.8, 1.0. * **Y-Axis (All Charts):** Labeled `RM@8 Accuracy`. The scale varies per chart to fit the data range. * **Legend (Right Side, Vertical):** A color bar titled `Model Sizes`. It maps colors to model parameter counts: * Yellow: `32b` * Light Green: `14b` * Teal/Green: `7b` * Blue: `3b` * Dark Blue: `1.5b` * Dark Purple: `0.5b` ### Detailed Analysis **Trend Verification & Data Points (Approximate Values):** **Top Row Charts:** 1. **GSM8K:** * **Y-Axis Range:** ~94.5 to 97.0. * **Trends:** All models show a general upward trend. The 32b model (yellow) starts highest (~97.0) and ends slightly higher. The 0.5b model (purple) starts lowest (~94.5) and shows the most volatility, with a dip around progress 0.3 before recovering. 2. **Math 500:** * **Y-Axis Range:** ~79 to 87. * **Trends:** Clear stratification by model size. The 32b model leads (~86.5 start, ~87.0 end). The 0.5b model shows a significant dip to ~79.0 at progress 0.4 before recovering to ~81.5. 3. **Minerva Math:** * **Y-Axis Range:** ~36 to 44. * **Trends:** High volatility, especially for smaller models. The 32b model is relatively stable at the top (~44.0). The 0.5b model fluctuates between ~36.0 and ~38.0. 4. **Gaokao2023EN:** * **Y-Axis Range:** ~67 to 74. * **Trends:** Steady, stratified improvement. The 32b model starts at ~73.5 and ends near ~74.0. The 0.5b model starts at ~67.5 and ends near ~68.0. **Bottom Row Charts:** 5. **Olympiad Bench:** * **Y-Axis Range:** ~40 to 48. * **Trends:** The 32b model shows a strong upward trend from ~47.0 to ~48.0. The 0.5b model is volatile, dipping to ~40.0 at progress 0.4 before rising to ~41.5. 6. **College Math:** * **Y-Axis Range:** ~43.0 to 46.5. * **Trends:** Tight clustering in the middle. The 32b model leads (~46.3 start, ~46.4 end). The 0.5b model dips to ~42.8 at progress 0.2. 7. **MMLU STEM:** * **Y-Axis Range:** ~76 to 86. * **Trends:** Clear, consistent stratification. The 32b model improves from ~84.5 to ~86.0. The 0.5b model fluctuates between ~76.0 and ~77.5. 8. **Average:** * **Y-Axis Range:** ~63 to 69. * **Trends:** Summarizes the overall pattern. Performance improves with both training progress and model size. The 32b model averages from ~68.5 to ~69.0. The 0.5b model averages from ~63.0 to ~63.5. ### Key Observations 1. **Consistent Hierarchy:** In every single chart, the performance order by model size is perfectly maintained throughout training: 32b > 14b > 7b > 3b > 1.5b > 0.5b. There are no crossovers. 2. **Volatility vs. Scale:** Smaller models (0.5b, 1.5b) exhibit significantly more performance volatility (sharp dips and recoveries) during training compared to larger models, which show smoother, more monotonic improvement curves. 3. **Benchmark Difficulty:** The absolute accuracy ranges reveal benchmark difficulty. `Minerva Math` and `Olympiad Bench` are the most challenging (lowest max accuracy ~44 and ~48), while `GSM8K` and `MMLU STEM` are the easiest (highest max accuracy ~97 and ~86). 4. **Training Progress Impact:** All models show net positive improvement from progress 0.0 to 1.0 across all benchmarks, confirming effective learning. ### Interpretation This visualization provides strong empirical evidence for two fundamental principles in modern AI scaling: 1. **The Scaling Law in Action:** The perfect, persistent stratification of performance by model size across eight diverse benchmarks demonstrates that increasing parameter count is a reliable, predictable method for improving task performance. The "Average" chart crystallizes this, showing a clean, tiered progression. 2. **Training Stability Scales with Size:** The pronounced volatility in smaller models suggests they are more sensitive to the training process—perhaps more susceptible to optimization challenges, data ordering, or temporary overfitting. Larger models appear more robust, learning in a smoother, more stable manner. This implies that scaling not only improves final performance but also the reliability of the training trajectory. 3. **Benchmark-Specific Learning Profiles:** While the hierarchy is constant, the shape of the learning curves differs. Benchmarks like `Minerva Math` show high volatility for all but the largest models, indicating a complex task where smaller models struggle to find a stable solution. In contrast, `Gaokao2023EN` shows smooth, parallel learning curves, suggesting a more straightforward skill acquisition process. **Conclusion:** The data argues that for these mathematical and reasoning tasks, model scale is the dominant factor determining both the absolute performance ceiling and the stability of the learning process. The charts serve as a technical reference for predicting model behavior and selecting an appropriate scale for a desired performance level and training stability requirement.

## Heatmap: Model Accuracy Over Training Progress ### Overview The heatmap displays the accuracy of various models (GSM8K, Math 500, Minerva Math, Gaokao2023EN, Olympiad Bench, College Math, MMMLU STEM, Average) over different training progress stages. The color gradient represents the model size, with darker shades indicating larger models. ### Components/Axes - **X-axis**: Training Progress (ranging from 0 to 1) - **Y-axis**: Model Size (ranging from 0.5b to 32b) - **Color Gradient**: Darker shades indicate larger models - **Legend**: Color legend on the right side, indicating model sizes - **Data Series**: Each row represents a different model, with columns showing accuracy at various training progress stages ### Detailed Analysis or ### Content Details - **GSM8K**: Shows a general trend of increasing accuracy as training progresses, with the largest model (32b) maintaining the highest accuracy throughout. - **Math 500**: Displays a similar trend, with the largest model consistently achieving the highest accuracy. - **Minerva Math**: Shows a slight decrease in accuracy at the beginning of training, followed by an increase as training progresses. The largest model maintains a high accuracy level. - **Gaokao2023EN**: Exhibits a steady increase in accuracy with training, with the largest model maintaining the highest accuracy. - **Olympiad Bench**: Shows a gradual increase in accuracy, with the largest model maintaining the highest accuracy. - **College Math**: Displays a slight decrease in accuracy at the beginning of training, followed by an increase as training progresses. The largest model maintains a high accuracy level. - **MMMLU STEM**: Shows a steady increase in accuracy with training, with the largest model maintaining the highest accuracy. - **Average**: Represents the average accuracy across all models and training progress stages. The largest model maintains the highest average accuracy. ### Key Observations - The largest model consistently achieves the highest accuracy across all models and training progress stages. - There is a general trend of increasing accuracy as training progresses for most models. - The accuracy of smaller models (0.5b to 1.5b) tends to be lower than that of larger models (32b). ### Interpretation The data suggests that larger models tend to maintain higher accuracy throughout the training process. This could be due to the increased computational resources and capacity of larger models, allowing them to learn more complex patterns and representations. The slight decrease in accuracy for smaller models at the beginning of training could be attributed to the initial learning phase where smaller models may not have enough capacity to learn the underlying patterns. However, as training progresses, the larger models are able to maintain and even improve their accuracy, indicating that they are better suited for the task at hand.

## Line Graphs: Model Accuracy Across Datasets and Model Sizes ### Overview The image contains eight line graphs comparing model accuracy (RM@8) across different datasets (GSM8K, Math 500, Minerva Math, Gaokao2023EN, Olympiad Bench, College Math, MMLU STEM, and Average) as training progresses from 0.0 to 1.0. Each graph includes four data series representing model sizes (0.5b, 1.5b, 3b, 7b, 14b, 32b), with colors mapped to sizes via a legend. Both y-axes are labeled "RM@8 Accuracy," though the right axis appears redundant. --- ### Components/Axes - **X-Axis**: Training Progress (0.0 to 1.0 in increments of 0.2). - **Y-Axes**: - Left: RM@8 Accuracy (ranges vary by dataset, e.g., 94.5–97.5 for GSM8K). - Right: RM@8 Accuracy (same scale as left, likely a duplication). - **Legend**: Located on the right, with colors corresponding to model sizes: - Yellow: 32b - Green: 14b - Blue: 3b - Purple: 0.5b - **Datasets**: Each graph is titled with a dataset name (e.g., "GSM8K," "Math 500"). --- ### Detailed Analysis #### GSM8K - **32b (Yellow)**: Starts at ~97.0, increases steadily to ~97.5. - **14b (Green)**: Starts at ~96.5, rises to ~97.0. - **3b (Blue)**: Starts at ~95.8, increases to ~96.0. - **0.5b (Purple)**: Starts at ~94.5, dips to ~94.8 at 0.4, then rises to ~95.3. #### Math 500 - **32b (Yellow)**: Peaks at ~86.5, then stabilizes. - **14b (Green)**: Starts at ~85.0, rises to ~85.5. - **3b (Blue)**: Fluctuates between ~82.5 and ~83.5. - **0.5b (Purple)**: Drops to ~79.0 at 0.2, recovers to ~81.0. #### Minerva Math - **32b (Yellow)**: Starts at ~44.0, rises to ~44.5. - **14b (Green)**: Starts at ~42.0, increases to ~42.5. - **3b (Blue)**: Starts at ~40.0, rises to ~40.5. - **0.5b (Purple)**: Dips to ~36.0 at 0.2, recovers to ~38.0. #### Gaokao2023EN - **32b (Yellow)**: Starts at ~73.0, rises to ~74.0. - **14b (Green)**: Starts at ~72.0, increases to ~72.5. - **3b (Blue)**: Starts at ~70.0, rises to ~70.5. - **0.5b (Purple)**: Dips to ~67.0 at 0.2, recovers to ~68.0. #### Olympiad Bench - **32b (Yellow)**: Starts at ~47.0, rises to ~48.0. - **14b (Green)**: Starts at ~46.0, increases to ~46.5. - **3b (Blue)**: Starts at ~43.0, rises to ~44.0. - **0.5b (Purple)**: Dips to ~40.0 at 0.2, recovers to ~41.0. #### College Math - **32b (Yellow)**: Starts at ~46.0, rises to ~46.5. - **14b (Green)**: Starts at ~45.0, increases to ~45.5. - **3b (Blue)**: Starts at ~44.0, rises to ~44.5. - **0.5b (Purple)**: Dips to ~43.0 at 0.2, recovers to ~44.0. #### MMLU STEM - **32b (Yellow)**: Starts at ~68.0, rises to ~69.0. - **14b (Green)**: Starts at ~67.0, increases to ~67.5. - **3b (Blue)**: Starts at ~65.0, rises to ~65.5. - **0.5b (Purple)**: Dips to ~63.0 at 0.2, recovers to ~64.0. #### Average - **32b (Yellow)**: Starts at ~68.0, rises to ~69.0. - **14b (Green)**: Starts at ~67.0, increases to ~67.5. - **3b (Blue)**: Starts at ~65.0, rises to ~65.5. - **0.5b (Purple)**: Dips to ~63.0 at 0.2, recovers to ~64.0. --- ### Key Observations 1. **Model Size Correlation**: Larger models (32b, 14b) consistently achieve higher RM@8 accuracy across all datasets. 2. **Training Instability**: Smaller models (0.5b, 1.5b) exhibit dips in accuracy during early training (e.g., 0.2–0.4 progress). 3. **Dataset Variability**: - GSM8K and Olympiad Bench show the highest accuracy gains. - Math 500 and Gaokao2023EN exhibit more volatility in smaller models. 4. **Average Trend**: The "Average" graph mirrors individual dataset trends, confirming consistency across tasks. --- ### Interpretation - **Performance Scaling**: Larger models (32b, 14b) demonstrate superior accuracy, suggesting that parameter count correlates with task performance. - **Training Dynamics**: Smaller models (0.5b, 1.5b) experience instability during early training, potentially due to optimization challenges or data scarcity. - **Dataset-Specific Behavior**: Tasks like GSM8K and Olympiad Bench may require higher model capacity, while others (e.g., Math 500) show mixed results, indicating task-dependent scaling laws. - **Redundant Y-Axis**: The duplicated "RM@8 Accuracy" labels on both axes suggest a possible visualization error, though the data remains interpretable. This analysis highlights the importance of model size in achieving accuracy and underscores the need for stable training protocols, particularly for smaller models.