## Scatter Plot: Accuracy on the MATH test set ### Overview The image is a scatter plot comparing the accuracy of various language models on the MATH test set against their model size. The plot displays model size on the x-axis and accuracy percentage on the y-axis. Each point represents a different model, with its name displayed next to the point. Some models are marked with a red triangle, while others are marked with a green, blue, purple, or orange circle. Horizontal dashed lines indicate the accuracy of some prominent models for reference. ### Components/Axes * **Title:** Accuracy on the MATH test set * **X-axis:** Model size, with markers at 7B, 32B, 47B, 57B, 70B, and 72B. * **Y-axis:** Accuracy (%), with markers from 42.5% to 72.5% in 2.5% increments. * **Horizontal Dashed Lines (Top to Bottom):** * Gemini-1.5-Pro (at approximately 67.5% accuracy) * GPT-4-1106 (at approximately 64.5% accuracy) * Claude-3-Opus (at approximately 60.5% accuracy) * **Data Points:** Each point represents a model, with its name displayed next to it. The points are colored differently, but there is no explicit legend provided to explain the color coding. ### Detailed Analysis or Content Details Here's a breakdown of the models and their approximate positions on the plot: * **Models with Red Triangle Markers:** * Qwen2-7B-Step-DPO: Located at approximately (7B, 55.5%) * Qwen1.5-32B-Step-DPO: Located at approximately (32B, 57.0%) * Qwen2-57B-A14B-Step-DPO: Located at approximately (57B, 56.0%) * Llama3-70B-Step-DPO: Located at approximately (70B, 59.5%) * Qwen2-72B-Step-DPO: Located at approximately (72B, 64.0%) * Qwen2-72B-Instruct-Step-DPO: Located at approximately (72B, 71.0%) * **Models with Green Circle Markers:** * DeepSeekMath-RL: Located at approximately (7B, 52.0%) * Qwen2-7B-Instruct: Located at approximately (7B, 48.5%) * Qwen2-72B-Instruct†: Located at approximately (72B, 69.0%) * **Models with Blue Circle Markers:** * Qwen1.5-32B-SFT: Located at approximately (32B, 54.0%) * Qwen2-57B-A14B-Instruct: Located at approximately (57B, 48.5%) * **Models with Orange Circle Markers:** * Llama3-70B-Instruct: Located at approximately (70B, 49.5%) * **Models with Purple Circle Markers:** * MathGenieLM-Mistral: Located at approximately (7B, 45.0%) * **Models with Dark Blue Circle Markers:** * MAmmoTH2-Mixtral-8x7B: Located at approximately (47B, 47.0%) ### Key Observations * The accuracy generally tends to increase with model size, but there are exceptions. * Models with "Step-DPO" in their name tend to have higher accuracy than their counterparts without it, especially for larger models like Qwen2-72B. * The Gemini-1.5-Pro model has the highest accuracy among the models indicated by horizontal lines. * The Qwen2-72B-Instruct-Step-DPO model achieves the highest accuracy among all models plotted. * The smallest models (7B) show a wide range of accuracy, suggesting that factors other than size significantly impact performance at this scale. ### Interpretation The scatter plot illustrates the relationship between model size and accuracy on the MATH test set for various language models. The general trend suggests that larger models tend to perform better, but the specific architecture, training method (e.g., Step-DPO), and other factors play a crucial role in determining the final accuracy. The horizontal lines provide a benchmark against well-known models like Gemini-1.5-Pro and GPT-4-1106. The clustering of points indicates that certain model families (e.g., Qwen, Llama) have different performance characteristics. The presence of outliers suggests that some models are either particularly effective or ineffective for their size. The plot highlights the importance of both model size and training techniques in achieving high accuracy on complex tasks like the MATH test set.

## Scatter Plot: Accuracy on the MATH test set ### Overview This scatter plot visualizes the accuracy of various language models on the MATH test set, plotted against their model size. The accuracy is measured in percentage (%), and the model size is measured in billions of parameters (B). Different models are represented by different colored markers (red triangles, green circles, and black squares). ### Components/Axes * **Title:** Accuracy on the MATH test set * **X-axis:** Model size (B). Scale ranges from approximately 5B to 75B. Markers are placed at 7B, 32B, 47B, 57B, 70B, and 72B. * **Y-axis:** Accuracy (%). Scale ranges from approximately 42% to 72.5%. Markers are placed at 42.5%, 45%, 47.5%, 50%, 52.5%, 55%, 57.5%, 60%, 62.5%, 65%, 67.5%, 70%, and 72.5%. * **Legend:** Implicitly defined by marker shape and color. * Red Triangles: Models ending in "-DPO" * Green Circles: Models ending in "-Instruct" or "-Instruct-SFT" * Black Squares: Models without "-DPO" or "-Instruct" in their name. ### Detailed Analysis The plot shows a general trend of increasing accuracy with increasing model size, but with significant variation between models. * **Gemini-1.5-Pro:** Approximately 70.0% accuracy at 72B. (Red Triangle) * **Qwen2-72B-Instruct-Step-DPO:** Approximately 70.5% accuracy at 72B. (Red Triangle) * **Qwen2-72B-Instruct:** Approximately 69.5% accuracy at 72B. (Red Triangle) * **Qwen2-72B-Step-DPO:** Approximately 68.5% accuracy at 72B. (Red Triangle) * **GPT-4-1106:** Approximately 65.0% accuracy at 72B. (Red Triangle) * **Claude-3-Opus:** Approximately 60.0% accuracy at 72B. (Red Triangle) * **Llama3-70B-DPO:** Approximately 58.0% accuracy at 70B. (Red Triangle) * **Qwen2-57B-A14B-Step-DPO:** Approximately 57.5% accuracy at 57B. (Red Triangle) * **Qwen2-57B-A14B-Instruct:** Approximately 50.0% accuracy at 57B. (Green Circle) * **Qwen2-32B-Step-DPO:** Approximately 57.5% accuracy at 32B. (Red Triangle) * **Qwen1.5-32B-SFT:** Approximately 52.5% accuracy at 32B. (Green Circle) * **Qwen2-7B-Step-DPO:** Approximately 55.0% accuracy at 7B. (Red Triangle) * **Qwen2-7B-Instruct:** Approximately 50.0% accuracy at 7B. (Green Circle) * **DeepSeekMath-RL:** Approximately 52.5% accuracy at 7B. (Green Circle) * **DeepSeekMath-Instruct:** Approximately 47.5% accuracy at 7B. (Green Circle) * **MathGenieLM-Mistral:** Approximately 45.0% accuracy at 7B. (Green Circle) * **MAmmoTH2-Mixtral-8x7B:** Approximately 47.5% accuracy at 47B. (Black Square) * **Llama3-70B-Instruct:** Approximately 50.0% accuracy at 70B. (Green Circle) ### Key Observations * Models with "-DPO" in their name (red triangles) generally exhibit higher accuracy than those without, especially at larger model sizes. * Qwen2-72B-Instruct-Step-DPO achieves the highest accuracy among the models shown. * There's a noticeable gap in accuracy between models around the 32B and 57B size ranges. * The models ending in "-Instruct" (green circles) show a relatively consistent accuracy across different model sizes. * MAmmoTH2-Mixtral-8x7B is an outlier, showing lower accuracy than other models of similar size. ### Interpretation The data suggests that model size is a significant factor in achieving higher accuracy on the MATH test set, but it is not the only one. The training methodology, as indicated by the "-DPO" suffix, appears to play a crucial role. Models trained with Direct Preference Optimization (DPO) consistently outperform others. The variation in accuracy among models of similar size highlights the importance of architectural choices and training data. The outlier, MAmmoTH2-Mixtral-8x7B, may indicate that its architecture or training process is less effective for this specific task. The plot demonstrates the ongoing progress in language model development, with newer models like Gemini-1.5-Pro and Qwen2-72B-Instruct-Step-DPO achieving state-of-the-art results. The consistent performance of the "-Instruct" models suggests that instruction tuning is a valuable technique for improving performance on tasks requiring following instructions.

\n ## Scatter Plot: Accuracy on the MATH test set ### Overview This is a scatter plot comparing the accuracy of various large language models (LLMs) on the MATH test set, plotted against their model size (parameter count). The chart highlights the performance of models from the Qwen family, particularly those using "Step-DPO" fine-tuning, against other prominent models and baselines. ### Components/Axes * **Chart Title:** "Accuracy on the MATH test set" (centered at the top). * **Y-Axis:** Labeled "Accuracy (%)". Scale ranges from 42.5 to 72.5, with major ticks every 2.5 units (42.5, 45.0, 47.5, ..., 72.5). * **X-Axis:** Labeled "Model size". It is a categorical axis with the following discrete labels from left to right: "7B", "32B", "47B", "57B", "70B", "72B". * **Legend (Left side, vertically aligned):** * Green circle: `DeepSeekMath-RL` * Olive circle: `Qwen2-7B-Instruct` * Dark red circle: `DeepSeekMath-Instruct` * Purple circle: `MathGenieLM-Mistral` * **Reference Baselines (Horizontal dashed lines with labels on the left):** * `Gemini-1.5-Pro` at approximately 68.0% accuracy. * `GPT-4-1106` at approximately 64.5% accuracy. * `Claude-3-Opus` at approximately 60.0% accuracy. ### Detailed Analysis **Data Points (Approximate values, identified by marker shape/color and label):** * **7B Size:** * Red Triangle: `Qwen2-7B-Step-DPO` at ~56.0%. * **32B Size:** * Red Triangle: `Qwen1.5-32B-Step-DPO` at ~57.0%. * Cyan Circle: `Qwen1.5-32B-SFT` at ~55.0%. * **47B Size:** * Blue Circle: `MAmmoTH2-Mixtral-8x7B` at ~47.0%. * **57B Size:** * Red Triangle: `Qwen2-57B-A14B-Step-DPO` at ~56.5%. * Olive Circle: `Qwen2-57B-A14B-Instruct` at ~49.0%. * **70B Size:** * Red Triangle: `Llama3-70B-Step-DPO` at ~59.5%. * Orange Circle: `Llama3-70B-Instruct` at ~50.5%. * **72B Size:** * Red Triangle: `Qwen2-72B-Step-DPO` at ~65.0%. * Olive Circle (with dagger †): `Qwen2-72B-Instruct†` at ~69.5%. * Olive Circle (with arrow): `Qwen2-72B-Instruct` at ~59.5%. ### Key Observations 1. **Step-DPO Performance Trend:** Models labeled with "Step-DPO" (red triangles) consistently outperform their corresponding "Instruct" or "SFT" base models at the same parameter size. This is visible at 32B, 57B, 70B, and 72B. 2. **Scale vs. Accuracy:** For the Qwen2 Step-DPO series (red triangles), there is a clear positive correlation between model size and accuracy, rising from ~56% at 7B to ~65% at 72B. 3. **Top Performers:** The highest accuracy points on the chart are the Qwen2-72B variants. `Qwen2-72B-Instruct†` (~69.5%) surpasses the `Gemini-1.5-Pro` baseline. `Qwen2-72B-Step-DPO` (~65.0%) surpasses the `GPT-4-1106` baseline. 4. **Architectural Outlier:** `MAmmoTH2-Mixtral-8x7B` (blue circle at 47B) has significantly lower accuracy (~47%) compared to other models in a similar size range, suggesting different training or architecture. 5. **Llama3 Comparison:** At 70B, `Llama3-70B-Step-DPO` (~59.5%) shows a substantial improvement over `Llama3-70B-Instruct` (~50.5%), mirroring the Step-DPO benefit seen in Qwen models. ### Interpretation The data demonstrates the effectiveness of the "Step-DPO" fine-tuning technique for improving mathematical reasoning performance in LLMs. Across multiple model families (Qwen, Llama) and sizes, Step-DPO yields a consistent and often significant accuracy boost over standard instruction-tuned (Instruct) or supervised fine-tuned (SFT) versions. The plot suggests that while base model scale is a primary driver of performance (the upward trend of red triangles), advanced alignment techniques like Step-DPO can provide a critical performance leap, allowing smaller Step-DPO models to rival or exceed larger, differently tuned models. For instance, the 72B Step-DPO model outperforms the much larger reference baseline `GPT-4-1106`. The presence of two `Qwen2-72B-Instruct` points (one with a dagger † at ~69.5% and one at ~59.5%) implies a potential variant or different evaluation condition for the same base model, with the dagger version performing exceptionally well, even surpassing the Gemini baseline. This highlights that model performance is not solely a function of size and tuning method, but also of specific implementation details or training checkpoints. **Language Note:** All text in the image is in English.

## Scatter Plot: Accuracy on the MATH test set ### Overview The image is a scatter plot visualizing the accuracy of various AI models on a MATH test set. The x-axis represents model size (in billions of parameters), and the y-axis represents accuracy (in percentage). Data points are color-coded and shaped to distinguish between models and their configurations (e.g., "Step-DPO" vs. "Instruct"). The plot includes labeled data points, a legend, and axis markers. --- ### Components/Axes - **Title**: "Accuracy on the MATH test set" - **X-axis**: "Model size" (ranging from 7B to 72B, with markers at 7B, 32B, 47B, 57B, 70B, 72B) - **Y-axis**: "Accuracy (%)" (ranging from 42.5% to 72.5%, with markers at 42.5%, 45%, 47.5%, 50%, 52.5%, 55%, 57.5%, 60%, 62.5%, 65%, 67.5%, 70%) - **Legend**: - **Colors/Symbols**: - **Red triangle**: Qwen2-7B-Step-DPO, Qwen2-32B-Step-DPO, Qwen2-57B-A14B-Step-DPO, Qwen2-72B-Step-DPO - **Green circle**: DeepSeekMath-RL, Qwen2-7B-Instruct, Qwen2-72B-Instruct - **Blue circle**: Qwen1.5-32B-SFT, MAmmoTH2-Mixtral-8x7B - **Yellow circle**: Llama3-70B-Instruct - **Purple circle**: MathGenieLM-Mistral - **Orange circle**: Llama3-70B-Step-DPO - **Brown circle**: Qwen2-72B-Instruct† - **Red triangle with black outline**: Qwen2-72B-Instruct-Step-DPO - **Red triangle with white outline**: Qwen2-72B-Step-DPO - **Red triangle with black outline and white outline**: Qwen2-72B-Instruct-Step-DPO (duplicate label?) --- ### Detailed Analysis #### Data Points and Trends 1. **Model Size vs. Accuracy**: - **Largest models** (70B–72B) generally show higher accuracy: - **Qwen2-72B-Instruct-Step-DPO** (72B, 70%) is the highest. - **Qwen2-72B-Instruct** (72B, 69%) follows closely. - **Llama3-70B-Step-DPO** (70B, 60%) and **Llama3-70B-Instruct** (70B, 50%) are lower but still among the top. - **Mid-sized models** (32B–57B): - **Qwen2-32B-Step-DPO** (32B, 58%) and **Qwen2-57B-A14B-Step-DPO** (57B, 57%) show moderate accuracy. - **Qwen1.5-32B-SFT** (32B, 54%) and **MAmmoTH2-Mixtral-8x7B** (47B, 47%) are lower. - **Smallest models** (7B): - **Qwen2-7B-Step-DPO** (7B, 55%) and **DeepSeekMath-Instruct** (7B, 45%) are the lowest. 2. **Step-DPO vs. Instruct**: - **Step-DPO** (red triangles) generally outperforms **Instruct** (green circles) across similar model sizes: - Qwen2-72B-Step-DPO (65%) > Qwen2-72B-Instruct (69%). - Qwen2-57B-A14B-Step-DPO (57%) > Qwen2-57B-A14B-Instruct (49%). - Exceptions: - **Gemini-1.5-Pro** (68%) and **GPT-4-1106** (65%) are high-performing but not labeled with Step-DPO. 3. **Notable Outliers**: - **MathGenieLM-Mistral** (45%) is the lowest accuracy. - **Qwen2-72B-Instruct†** (69%) is the second-highest accuracy but lacks a Step-DPO label. --- ### Key Observations 1. **Model Size Correlation**: - Larger models (70B–72B) dominate the top accuracy range (65%–70%). - Smaller models (7B–32B) cluster below 60%, with Step-DPO configurations showing marginal improvements. 2. **Step-DPO Effectiveness**: - Step-DPO (red triangles) consistently outperforms Instruct (green circles) for the same model size (e.g., Qwen2-72B-Step-DPO vs. Qwen2-72B-Instruct). 3. **Exceptions**: - **Gemini-1.5-Pro** (68%) and **GPT-4-1106** (65%) achieve high accuracy without Step-DPO, suggesting alternative architectures or training methods. 4. **Llama3 Variants**: - Llama3-70B-Step-DPO (60%) outperforms Llama3-70B-Instruct (50%), but both lag behind Qwen2 models. --- ### Interpretation The data demonstrates a **positive correlation between model size and accuracy**, with larger models (70B–72B) achieving the highest performance. The **Step-DPO method** (red triangles) consistently improves accuracy compared to standard Instruct configurations (green circles) for the same model size. However, exceptions like **Gemini-1.5-Pro** and **GPT-4-1106** suggest that architectural innovations or specialized training data can also drive high performance. The **Qwen2-72B-Instruct-Step-DPO** (70%) stands out as the most accurate model, while **MathGenieLM-Mistral** (45%) highlights the limitations of smaller, less optimized models. The plot underscores the importance of both model scale and training methodology in achieving high accuracy on mathematical tasks.