## Chart: Scaling Training Data for MATH-500 ### Overview The image is a line chart comparing the accuracy (%) of different models (ThinkPRM-1.5B@4 (1K), ThinkPRM-1.5B@4 (65K), DiscPRM-1.5B, and RLHFFlow-8B-Deepseek) against the number of solutions (2^1, 2^3, 2^5, 2^7) used for scaling training data. The data is for the MATH-500 dataset, and the generator used was Llama 3.2-3B-Instruct. ### Components/Axes * **Title:** Scaling training data: MATH-500 * **Subtitle:** Generator: Llama 3.2-3B-Instruct * **Y-axis:** Accuracy (%) * Scale ranges from 45 to 70, with tick marks at intervals of 5. * **X-axis:** Number of solutions * Values: 2^1, 2^3, 2^5, 2^7 * **Legend:** Located at the bottom of the chart. * ThinkPRM-1.5B@4 (1K) (Orange with triangle markers) * ThinkPRM-1.5B@4 (65K) (Pink with triangle markers, dashed line) * DiscPRM-1.5B (Teal with circle markers) * RLHFFlow-8B-Deepseek (Yellow with circle markers) ### Detailed Analysis * **ThinkPRM-1.5B@4 (1K) (Orange, triangle markers):** * Trend: Generally increasing. * Data Points: * 2^1: ~46% * 2^3: ~58% * 2^5: ~62% * 2^7: ~68% * **ThinkPRM-1.5B@4 (65K) (Pink, triangle markers, dashed line):** * Trend: Increasing, plateaus, then increases again. * Data Points: * 2^1: ~55% * 2^3: ~64% * 2^5: ~64% * 2^7: ~70% * **DiscPRM-1.5B (Teal, circle markers):** * Trend: Increasing, plateaus, then slightly increases. * Data Points: * 2^1: ~53% * 2^3: ~60% * 2^5: ~65% * 2^7: ~63% * **RLHFFlow-8B-Deepseek (Yellow, circle markers):** * Trend: Increasing, plateaus. * Data Points: * 2^1: ~50% * 2^3: ~62% * 2^5: ~60% * 2^7: ~63% ### Key Observations * ThinkPRM-1.5B@4 (65K) achieves the highest accuracy at 2^7 solutions. * DiscPRM-1.5B and RLHFFlow-8B-Deepseek plateau in accuracy after 2^3 solutions. * ThinkPRM-1.5B@4 (1K) shows a steady increase in accuracy across all solution counts. ### Interpretation The chart illustrates the impact of scaling training data on the accuracy of different models for the MATH-500 dataset. ThinkPRM-1.5B@4 (65K) benefits the most from increased training data, achieving the highest accuracy. DiscPRM-1.5B and RLHFFlow-8B-Deepseek show diminishing returns after a certain point, suggesting that increasing the number of solutions beyond 2^3 does not significantly improve their performance. ThinkPRM-1.5B@4 (1K) shows a consistent improvement with more data, indicating it may benefit from even further scaling. The choice of model and the optimal amount of training data depend on the specific performance requirements and computational resources available.

## Line Chart: Scaling Training Data - MATH-500 ### Overview This line chart illustrates the relationship between the number of solutions used for training and the resulting accuracy on the MATH-500 dataset. Four different models are compared: ThinkPRM-1.5B@4 (1K), ThinkPRM-1.5B@4 (65K), DiscPRM-1.5B, and RLHFflow-8B-Deepseek. The generator used for all models is Llama 3.2-3B-Instruct. ### Components/Axes * **Title:** Scaling training data: MATH-500 * **X-axis:** Number of solutions. Scale is logarithmic, with markers at 2¹, 2³, 2⁵, and 2⁷. * **Y-axis:** Accuracy (%). Scale ranges from 45% to 70%. * **Legend:** Located at the bottom of the chart. * ThinkPRM-1.5B@4 (1K) - Orange dashed line with triangle markers. * ThinkPRM-1.5B@4 (65K) - Purple dashed line with triangle markers. * DiscPRM-1.5B - Teal solid line with circle markers. * RLHFflow-8B-Deepseek - Yellow solid line with circle markers. ### Detailed Analysis * **ThinkPRM-1.5B@4 (1K):** The line starts at approximately 48% accuracy at 2¹ solutions, increases to around 58% at 2³ solutions, then rises to approximately 65% at 2⁵ solutions, and finally reaches about 68% at 2⁷ solutions. The trend is generally upward, with diminishing returns as the number of solutions increases. * **ThinkPRM-1.5B@4 (65K):** This line exhibits the steepest upward trend. It begins at approximately 50% accuracy at 2¹ solutions, jumps to around 63% at 2³ solutions, continues to approximately 69% at 2⁵ solutions, and reaches a peak of around 70% at 2⁷ solutions. * **DiscPRM-1.5B:** The line starts at approximately 52% accuracy at 2¹ solutions, increases to around 60% at 2³ solutions, then rises to approximately 65% at 2⁵ solutions, and plateaus at around 65% at 2⁷ solutions. * **RLHFflow-8B-Deepseek:** This line shows a moderate upward trend. It begins at approximately 47% accuracy at 2¹ solutions, increases to around 57% at 2³ solutions, then rises to approximately 61% at 2⁵ solutions, and finally reaches about 63% at 2⁷ solutions. ### Key Observations * The model ThinkPRM-1.5B@4 (65K) consistently outperforms the other models across all training data sizes. * Increasing the training data size generally leads to improved accuracy for all models, but the rate of improvement diminishes as the number of solutions increases. * DiscPRM-1.5B shows a plateau in accuracy after 2⁵ solutions. * RLHFflow-8B-Deepseek consistently has the lowest accuracy among the four models. ### Interpretation The data suggests that scaling the training data size significantly improves the accuracy of these models on the MATH-500 dataset. The model ThinkPRM-1.5B@4 (65K) demonstrates the most substantial gains from increased training data, indicating that it benefits the most from a larger dataset. The logarithmic scale on the x-axis highlights the diminishing returns of adding more data; while initial increases in data size lead to large accuracy improvements, the gains become smaller as the dataset grows. The plateau observed in DiscPRM-1.5B suggests that this model may have reached its capacity to learn from the MATH-500 dataset, or that further improvements require architectural changes rather than simply more data. The consistent lower performance of RLHFflow-8B-Deepseek could be due to its architecture, training methodology, or other factors. Overall, the chart emphasizes the importance of data scaling in improving the performance of language models on mathematical reasoning tasks.

## Line Chart: Scaling training data: MATH-500 ### Overview This is a line chart illustrating the relationship between the amount of training data (measured in "Number of solutions") and model performance (measured in "Accuracy (%)") on the MATH-500 benchmark. The chart compares four different model training methods or configurations. The overall trend for all methods is that accuracy increases as the number of training solutions increases, following a logarithmic scale on the x-axis. ### Components/Axes * **Chart Title:** "Scaling training data: MATH-500" * **Subtitle/Generator:** "Generator: Llama 3.2-3B-Instruct" * **Y-Axis:** * **Label:** "Accuracy (%)" * **Scale:** Linear, ranging from 45 to 70, with major tick marks at 5-unit intervals (45, 50, 55, 60, 65, 70). * **X-Axis:** * **Label:** "Number of solutions" * **Scale:** Logarithmic (base 2), with labeled tick marks at 2¹ (2), 2³ (8), 2⁵ (32), and 2⁷ (128). The axis starts at approximately 2⁰ (1). * **Legend:** Located at the bottom of the chart, outside the plot area. It contains four entries, each with a unique color, line style, and marker shape. 1. **ThinkPRM-1.5B@4 (1K):** Orange dashed line with upward-pointing triangle markers. 2. **ThinkPRM-1.5B@4 (65K):** Pink (light purple) dashed line with upward-pointing triangle markers. 3. **DiscPRM-1.5B:** Green solid line with circle markers. 4. **RLHFFlow-8B-Deepseek:** Yellow solid line with circle markers. * **Grid:** A light gray grid is present in the background of the plot area. ### Detailed Analysis The chart plots four data series. Below is a breakdown of each series, its visual trend, and approximate data points extracted by aligning markers with the grid. **1. ThinkPRM-1.5B@4 (65K) - Pink Dashed Line, Triangle Markers** * **Trend:** Shows the steepest and highest overall growth. It starts as the second-highest performer at the lowest data point and becomes the clear top performer from 2³ solutions onward, maintaining a significant lead. * **Data Points (Approximate):** * At ~2⁰ (1) solutions: ~46% * At 2¹ (2) solutions: ~55% * At 2² (4) solutions: ~58% * At 2³ (8) solutions: ~64% * At 2⁴ (16) solutions: ~64.5% * At 2⁵ (32) solutions: ~67% * At 2⁶ (64) solutions: ~70% (Peak) * At 2⁷ (128) solutions: ~69% **2. ThinkPRM-1.5B@4 (1K) - Orange Dashed Line, Triangle Markers** * **Trend:** Shows strong, steady growth. It starts as the lowest performer but catches up to and eventually surpasses the DiscPRM-1.5B model at higher data volumes. * **Data Points (Approximate):** * At ~2⁰ (1) solutions: ~46% * At 2¹ (2) solutions: ~53% * At 2² (4) solutions: ~58% * At 2³ (8) solutions: ~58% * At 2⁴ (16) solutions: ~62% * At 2⁵ (32) solutions: ~63% * At 2⁶ (64) solutions: ~65% * At 2⁷ (128) solutions: ~68% **3. DiscPRM-1.5B - Green Solid Line, Circle Markers** * **Trend:** Shows consistent growth but at a slightly lower rate than the ThinkPRM variants at higher data scales. It starts as the second-lowest performer and is overtaken by ThinkPRM-1.5B@4 (1K) after 2⁵ solutions. * **Data Points (Approximate):** * At ~2⁰ (1) solutions: ~46% * At 2¹ (2) solutions: ~53% * At 2² (4) solutions: ~56% * At 2³ (8) solutions: ~60% * At 2⁴ (16) solutions: ~60% * At 2⁵ (32) solutions: ~63% * At 2⁶ (64) solutions: ~65% * At 2⁷ (128) solutions: ~63% (Note: This point appears to dip slightly from the previous point.) **4. RLHFFlow-8B-Deepseek - Yellow Solid Line, Circle Markers** * **Trend:** Shows the slowest rate of improvement. It starts as the lowest performer and remains the lowest-performing method across the entire range of training data, though its accuracy does increase. * **Data Points (Approximate):** * At ~2⁰ (1) solutions: ~46% * At 2¹ (2) solutions: ~50% * At 2² (4) solutions: ~53% * At 2³ (8) solutions: ~58% * At 2⁴ (16) solutions: ~60% * At 2⁵ (32) solutions: ~60% * At 2⁶ (64) solutions: ~62% * At 2⁷ (128) solutions: ~63% ### Key Observations 1. **Universal Scaling Law:** All four methods demonstrate that increasing the volume of training data (solutions) leads to improved accuracy on the MATH-500 benchmark, confirming a positive scaling relationship. 2. **Performance Hierarchy:** A clear performance hierarchy is established and maintained as data scales: `ThinkPRM-1.5B@4 (65K)` > `ThinkPRM-1.5B@4 (1K)` ≈ `DiscPRM-1.5B` > `RLHFFlow-8B-Deepseek`. The gap between the top and bottom methods widens significantly with more data. 3. **Diminishing Returns:** The curves for all methods begin to flatten between 2⁵ (32) and 2⁷ (128) solutions, suggesting diminishing returns from adding more data beyond a certain point. The `ThinkPRM-1.5B@4 (65K)` model even shows a slight decrease at the final data point. 4. **Crossover Point:** The `ThinkPRM-1.5B@4 (1K)` (orange) line crosses above the `DiscPRM-1.5B` (green) line between 2⁵ (32) and 2⁶ (64) solutions, indicating it becomes more data-efficient at larger scales. 5. **Initial Convergence:** All models start at nearly the same accuracy (~46%) when trained on the smallest dataset (~1 solution), but their performance diverges rapidly as data increases. ### Interpretation This chart provides a comparative analysis of different training methodologies (likely involving different reward models or training objectives like "ThinkPRM," "DiscPRM," and "RLHFFlow") when applied to a base generator model (Llama 3.2-3B-Instruct) for mathematical reasoning. The data suggests that the **`ThinkPRM-1.5B@4` method, especially when scaled to 65K solutions, is the most effective and data-efficient approach** among those tested for improving mathematical accuracy. Its superior performance implies that its underlying training strategy (possibly involving more sophisticated process supervision or thinking-step reward modeling) extracts more learning signal per data point. The fact that the much larger `RLHFFlow-8B-Deepseek` model (8B parameters vs. 1.5B for the others) performs the worst is a critical finding. It indicates that **model size alone is not the primary driver of performance on this task**; the training methodology and data quality/quantity are more decisive factors. This challenges the simple "bigger is better" paradigm and highlights the importance of algorithmic innovation in training. The flattening of the curves suggests that for this specific task and generator, simply adding more solution data of the same type may yield limited future gains. Further improvements might require higher-quality data, more advanced training techniques, or changes to the base generator model itself. The slight dip for the top model at 128 solutions could be noise or an early sign of overfitting to the training data distribution.

## Line Chart: Scaling training data: MATH-500 ### Overview The chart illustrates the relationship between the number of training solutions (x-axis) and model accuracy (y-axis) for four different AI models. The x-axis uses exponential scaling (2¹, 2³, 2⁵, 2⁷), while the y-axis shows accuracy percentages from 45% to 70%. Four distinct data series are plotted with unique colors and markers. ### Components/Axes - **X-axis**: "Number of solutions" with logarithmic spacing (2¹, 2³, 2⁵, 2⁷) - **Y-axis**: "Accuracy (%)" ranging from 45% to 70% - **Legend**: Located at the bottom, with four entries: - Orange triangle: ThinkPRM-1.5B@4 (1K) - Teal circle: DiscPRM-1.5B - Purple diamond: ThinkPRM-1.5B@4 (65K) - Yellow square: RLHFFlow-8B-Deepseek - **Title**: "Scaling training data: MATH-500" (top center) - **Subtitle**: "Generator: Llama 3.2-3B-Instruct" (top left) ### Detailed Analysis 1. **ThinkPRM-1.5B@4 (1K)** (Orange triangle): - Starts at ~45% accuracy at 2¹ - Gradual increase to ~55% at 2³ - Sharp rise to ~60% at 2⁵ - Plateaus near 60% at 2⁷ 2. **DiscPRM-1.5B** (Teal circle): - Begins at ~45% at 2¹ - Steady climb to ~55% at 2³ - Accelerated growth to ~60% at 2⁵ - Continues to ~62% at 2⁷ 3. **ThinkPRM-1.5B@4 (65K)** (Purple diamond): - Starts at ~50% at 2¹ - Rapid ascent to ~60% at 2³ - Sustained growth to ~65% at 2⁵ - Peaks at ~68% at 2⁷ 4. **RLHFFlow-8B-Deepseek** (Yellow square): - Initial value ~45% at 2¹ - Moderate increase to ~52% at 2³ - Steep rise to ~63% at 2⁵ - Levels off near 63% at 2⁷ ### Key Observations - **Performance Gaps**: ThinkPRM-1.5B@4 (65K) consistently outperforms other models, achieving ~68% accuracy at 2⁷ vs. ~63% for RLHFFlow-8B-Deepseek. - **Scaling Efficiency**: ThinkPRM-1.5B@4 (65K) shows the steepest improvement curve, suggesting superior data utilization. - **Baseline Comparison**: The 1K solution variant (ThinkPRM-1.5B@4) underperforms across all metrics, highlighting the importance of training data volume. - **Model Specialization**: RLHFFlow-8B-Deepseek demonstrates strong scaling despite being a different architecture, indicating robust design. ### Interpretation The data demonstrates that increasing training data volume (solutions) correlates with improved accuracy across all models, with diminishing returns at higher solution counts. The ThinkPRM-1.5B@4 (65K) model's superior performance suggests that: 1. **Data Quality Matters**: The 65K solution variant likely incorporates more diverse or higher-quality examples. 2. **Architectural Synergy**: The Llama 3.2-3B-Instruct generator may be particularly well-suited to this scaling approach. 3. **Efficiency Tradeoffs**: While RLHFFlow-8B-Deepseek shows strong scaling, its lower final accuracy suggests potential tradeoffs between model size and data efficiency. Notably, the chart reveals that even modest increases in training data (e.g., 2³ to 2⁵ solutions) yield significant accuracy gains, emphasizing the value of data scaling in LLM development. The performance gap between 1K and 65K solution variants underscores the critical role of training data quantity in achieving state-of-the-art results.