## Line Chart: Model Accuracy vs. Training Progress for Various Datasets and Model Sizes ### Overview The image presents a series of line charts comparing the RM@8 accuracy of different model sizes (0.5b, 1.5b, 3b, 7b, 14b, 32b) across various training progress levels (0.0 to 1.0) for different datasets. Each chart corresponds to a specific dataset: GSM8K, Math 500, Minerva Math, Gaokao2023EN, Olympiad Bench, College Math, MMLU STEM, and an average across all datasets. The charts share a common legend indicating model sizes by color. ### Components/Axes * **X-axis (Horizontal):** Training Progress, ranging from 0.0 to 1.0 in increments of 0.2. * **Y-axis (Vertical):** RM@8 Accuracy, with varying scales depending on the dataset. * **Titles:** Each chart has a title indicating the dataset name (e.g., "GSM8K", "Math 500"). * **Legend (Right):** A vertical color bar indicating model sizes: 0.5b (dark purple), 1.5b (purple), 3b (dark blue), 7b (teal), 14b (light green), and 32b (yellow). * **Gridlines:** Each chart has faint grey gridlines to aid in reading values. ### Detailed Analysis **1. GSM8K** * Y-axis ranges from 94.5 to 97.0. * **0.5b (dark purple):** Starts at approximately 94.6, dips slightly, then increases to around 95.3. * **1.5b (purple):** Starts at approximately 95.3, dips slightly, then increases to around 95.4. * **3b (dark blue):** Relatively flat at approximately 96.0. * **7b (teal):** Relatively flat at approximately 96.4. * **14b (light green):** Relatively flat at approximately 96.7. * **32b (yellow):** Relatively flat at approximately 97.1. **2. Math 500** * Y-axis ranges from 79 to 87. * **0.5b (dark purple):** Starts at approximately 81.5, dips to 79, then increases to around 82. * **1.5b (purple):** Starts at approximately 82, dips to 81, then increases to around 82.5. * **3b (dark blue):** Relatively flat at approximately 83.5. * **7b (teal):** Relatively flat at approximately 84. * **14b (light green):** Relatively flat at approximately 85. * **32b (yellow):** Relatively flat at approximately 86.5. **3. Minerva Math** * Y-axis ranges from 36 to 44. * **0.5b (dark purple):** Starts at approximately 37.5, dips slightly, then increases to around 37. * **1.5b (purple):** Starts at approximately 37, dips slightly, then increases to around 37. * **3b (dark blue):** Starts at approximately 38, increases to 40, then decreases to around 39. * **7b (teal):** Relatively flat at approximately 41. * **14b (light green):** Relatively flat at approximately 42.5. * **32b (yellow):** Relatively flat at approximately 44. **4. Gaokao2023EN** * Y-axis ranges from 66 to 74. * **0.5b (dark purple):** Starts at approximately 67.5, increases to 68.5, then decreases to around 68. * **1.5b (purple):** Starts at approximately 68.5, increases to 69, then decreases to around 68. * **3b (dark blue):** Starts at approximately 69.5, increases to 70.5, then decreases to around 70. * **7b (teal):** Starts at approximately 72.5, increases to 73, then decreases to around 72. * **14b (light green):** Starts at approximately 73, increases to 73.5, then decreases to around 73. * **32b (yellow):** Relatively flat at approximately 74. **5. Olympiad Bench** * Y-axis ranges from 40 to 48. * **0.5b (dark purple):** Starts at approximately 41.5, dips to 40, then increases to around 41.5. * **1.5b (purple):** Starts at approximately 43, increases to 44, then decreases to around 44. * **3b (dark blue):** Relatively flat at approximately 44.5. * **7b (teal):** Relatively flat at approximately 46. * **14b (light green):** Relatively flat at approximately 46.5. * **32b (yellow):** Relatively flat at approximately 48. **6. College Math** * Y-axis ranges from 42.5 to 46.5. * **0.5b (dark purple):** Starts at approximately 43.5, dips to 43, then increases to around 44.5. * **1.5b (purple):** Starts at approximately 44.5, increases to 45, then decreases to around 45. * **3b (dark blue):** Relatively flat at approximately 45.5. * **7b (teal):** Relatively flat at approximately 46. * **14b (light green):** Relatively flat at approximately 46. * **32b (yellow):** Relatively flat at approximately 46.5. **7. MMLU STEM** * Y-axis ranges from 75 to 87. * **0.5b (dark purple):** Starts at approximately 78, dips to 76, then increases to around 77. * **1.5b (purple):** Starts at approximately 78, dips to 76, then increases to around 77. * **3b (dark blue):** Starts at approximately 78, increases to 80, then decreases to around 80. * **7b (teal):** Starts at approximately 82, increases to 83, then decreases to around 83. * **14b (light green):** Starts at approximately 84, increases to 85, then decreases to around 85. * **32b (yellow):** Starts at approximately 85, increases to 86, then decreases to around 86. **8. Average** * Y-axis ranges from 62 to 70. * **0.5b (dark purple):** Starts at approximately 63, dips to 62.5, then increases to around 63.5. * **1.5b (purple):** Starts at approximately 63.5, dips to 63, then increases to around 64. * **3b (dark blue):** Starts at approximately 65, increases to 66, then decreases to around 65.5. * **7b (teal):** Starts at approximately 67, increases to 68, then decreases to around 67.5. * **14b (light green):** Starts at approximately 68, increases to 68.5, then decreases to around 68. * **32b (yellow):** Starts at approximately 68.5, increases to 69, then decreases to around 69. ### Key Observations * Generally, larger model sizes (32b, 14b) achieve higher RM@8 accuracy compared to smaller models (0.5b, 1.5b) across all datasets. * The "Average" chart shows a clear separation between model sizes, with larger models consistently outperforming smaller ones. * The performance difference between model sizes is more pronounced in some datasets (e.g., MMLU STEM) than others (e.g., College Math). * The initial training progress (0.0 to 0.2) often shows a significant increase in accuracy for smaller models, while larger models tend to be more stable. * Some datasets (e.g., Math 500, Olympiad Bench) exhibit a dip in accuracy for smaller models around the 0.2 to 0.4 training progress mark. ### Interpretation The data suggests that increasing model size generally leads to improved performance, as measured by RM@8 accuracy, across a variety of datasets. However, the extent of improvement varies depending on the specific dataset. The initial training phase appears to be more critical for smaller models, as they exhibit more significant fluctuations in accuracy compared to larger models. The observed dips in accuracy for smaller models in certain datasets might indicate overfitting or instability during the early stages of training. The "Average" chart provides a consolidated view, highlighting the overall trend of larger models achieving higher accuracy.