## Chart: Compute-matched analysis: MATH-500 ### Overview The image is a line chart comparing the accuracy of two methods, "ThinkPRM-14B" and "Majority voting", against the estimated FLOPs (Floating Point Operations per Second) on a logarithmic scale. The chart is titled "Compute-matched analysis: MATH-500" and indicates the generator used is "Qwen2.5-14B". ### Components/Axes * **Title:** Compute-matched analysis: MATH-500 * **Subtitle:** Generator: Qwen2.5-14B * **Y-axis:** Accuracy (%) * Scale ranges from 50 to 85, with tick marks at intervals of 5. * **X-axis:** Estimated FLOPs (log scale) * Scale ranges from 1 x 10^15 to 1 x 10^17. * **Legend:** Located in the bottom-right corner. * ThinkPRM-14B (represented by an orange line) * Majority voting (represented by a tan line) ### Detailed Analysis * **ThinkPRM-14B (Orange Line):** * Trend: Generally slopes upward, indicating increasing accuracy with higher FLOPs. * Data Points: * At 1 x 10^15 FLOPs, accuracy is approximately 51%. * At approximately 1.5 x 10^15 FLOPs, accuracy is approximately 62%. * At approximately 2.5 x 10^15 FLOPs, accuracy is approximately 69%. * At approximately 5 x 10^15 FLOPs, accuracy is approximately 74%. * At 1 x 10^16 FLOPs, accuracy is approximately 76%. * At approximately 3 x 10^16 FLOPs, accuracy is approximately 79%. * At approximately 6 x 10^16 FLOPs, accuracy is approximately 83%. * At 1 x 10^17 FLOPs, accuracy is approximately 86%. * **Majority voting (Tan Line):** * Trend: Generally slopes upward, but plateaus towards the higher FLOPs. * Data Points: * At 1 x 10^15 FLOPs, accuracy is approximately 51%. * At approximately 1.5 x 10^15 FLOPs, accuracy is approximately 67%. * At approximately 2.5 x 10^15 FLOPs, accuracy is approximately 74%. * At approximately 5 x 10^15 FLOPs, accuracy is approximately 74%. * At 1 x 10^16 FLOPs, accuracy is approximately 73%. * At approximately 3 x 10^16 FLOPs, accuracy is approximately 78%. * At approximately 6 x 10^16 FLOPs, accuracy is approximately 79%. ### Key Observations * Both methods start with similar accuracy at lower FLOPs (around 51% at 1 x 10^15 FLOPs). * ThinkPRM-14B consistently outperforms Majority voting as FLOPs increase, especially at higher FLOPs. * Majority voting shows a plateau in accuracy improvement beyond 1 x 10^16 FLOPs. ### Interpretation The data suggests that ThinkPRM-14B scales more effectively with increased computational resources (FLOPs) compared to Majority voting for the MATH-500 task. The plateau in Majority voting's accuracy indicates a potential limitation in its ability to leverage additional computational power, while ThinkPRM-14B continues to improve. This implies that ThinkPRM-14B is a more efficient or better-suited method for this particular task when computational resources are abundant. The "Compute-matched analysis" title suggests that the comparison is controlled for computational cost, making the accuracy difference more meaningful. The generator "Qwen2.5-14B" likely refers to the model used to generate or evaluate the solutions.

## Line Chart: Compute-matched analysis: MATH-500 ### Overview This image presents a line chart comparing the accuracy of two methods, "ThinkPRM-14B" and "Majority voting", as a function of estimated FLOPS (Floating Point Operations Per Second) on the MATH-500 dataset. The chart is designed to show how performance scales with computational resources. ### Components/Axes * **Title:** Compute-matched analysis: MATH-500 * **Subtitle:** Generator: Qwen2.5-14B * **X-axis Label:** Estimated FLOPS (log scale) * **X-axis Scale:** Logarithmic, ranging from approximately 1 x 10¹⁵ to 1 x 10¹⁷. Markers are at 1 x 10¹⁵, 5 x 10¹⁵, 1 x 10¹⁶, 5 x 10¹⁶, and 1 x 10¹⁷. * **Y-axis Label:** Accuracy (%) * **Y-axis Scale:** Linear, ranging from 50% to 85%. Markers are at 50%, 55%, 60%, 65%, 70%, 75%, 80%, and 85%. * **Legend:** Located in the bottom-right corner. * "ThinkPRM-14B" – Represented by a solid orange line with circular markers. * "Majority voting" – Represented by a dashed orange line with circular markers. ### Detailed Analysis **ThinkPRM-14B (Solid Orange Line):** The line slopes upward, indicating increasing accuracy with increasing FLOPS. * At 1 x 10¹⁵ FLOPS: Approximately 51% accuracy. * At 5 x 10¹⁵ FLOPS: Approximately 62% accuracy. * At 1 x 10¹⁶ FLOPS: Approximately 72% accuracy. * At 5 x 10¹⁶ FLOPS: Approximately 78% accuracy. * At 1 x 10¹⁷ FLOPS: Approximately 85% accuracy. **Majority Voting (Dashed Orange Line):** The line also slopes upward, but with a different trajectory than ThinkPRM-14B. * At 1 x 10¹⁵ FLOPS: Approximately 51% accuracy. * At 5 x 10¹⁵ FLOPS: Approximately 52% accuracy. * At 1 x 10¹⁶ FLOPS: Approximately 75% accuracy. * At 5 x 10¹⁶ FLOPS: Approximately 78% accuracy. * At 1 x 10¹⁷ FLOPS: Approximately 79% accuracy. ### Key Observations * Both methods show an increase in accuracy with increasing FLOPS. * ThinkPRM-14B consistently outperforms Majority voting across all FLOPS levels. * The performance gap between the two methods widens as FLOPS increase, particularly between 1 x 10¹⁵ and 1 x 10¹⁶ FLOPS. * The accuracy of Majority voting plateaus at higher FLOPS levels (above 1 x 10¹⁶ FLOPS). ### Interpretation The data suggests that ThinkPRM-14B is a more effective method for solving MATH-500 problems than Majority voting, especially when significant computational resources are available. The increasing accuracy with FLOPS indicates that both methods benefit from increased computational power, but ThinkPRM-14B demonstrates a stronger scaling effect. The plateau in Majority voting's accuracy suggests that its performance is limited by factors other than computational resources, such as the inherent limitations of the voting mechanism itself. The generator used, Qwen2.5-14B, likely influences the performance of ThinkPRM-14B. The chart highlights the trade-off between computational cost and accuracy, and suggests that investing in more FLOPS can lead to substantial improvements in performance for ThinkPRM-14B. The initial low accuracy for both methods at the lowest FLOPS level suggests that a minimum level of computation is required to achieve meaningful results.

## Line Chart: Compute-Matched Analysis of MATH-500 Accuracy ### Overview The image is a line chart comparing the performance of two methods, "ThinkPRM-14B" and "Majority voting," on the MATH-500 benchmark. The analysis plots accuracy against estimated computational cost (FLOPs) on a logarithmic scale. The chart demonstrates how the accuracy of each method scales with increased computational resources. ### Components/Axes * **Chart Title:** "Compute-matched analysis: MATH-500" * **Subtitle/Generator:** "Generator: Qwen2.5-14B" * **Y-Axis:** Labeled "Accuracy (%)". The scale runs from 50 to 85, with major tick marks at 50, 55, 60, 65, 70, 75, 80, and 85. * **X-Axis:** Labeled "Estimated FLOPs (log scale)". The scale is logarithmic, with major labeled tick marks at `1 x 10^15`, `1 x 10^16`, and `1 x 10^17`. * **Legend:** Located in the bottom-right quadrant of the chart area. * **ThinkPRM-14B:** Represented by an orange line with circular markers. * **Majority voting:** Represented by a light brown/tan line with circular markers. ### Detailed Analysis **Data Series 1: ThinkPRM-14B (Orange Line)** * **Trend:** The line shows a consistent, strong upward slope across the entire range of compute, indicating that accuracy improves steadily as more FLOPs are allocated. * **Approximate Data Points:** * At ~1 x 10^15 FLOPs: Accuracy ≈ 51% * At ~3 x 10^15 FLOPs: Accuracy ≈ 62% * At ~1 x 10^16 FLOPs: Accuracy ≈ 74% * At ~3 x 10^16 FLOPs: Accuracy ≈ 79% * At ~1 x 10^17 FLOPs: Accuracy ≈ 85% **Data Series 2: Majority voting (Light Brown Line)** * **Trend:** The line shows a steep initial increase in accuracy at lower compute levels, but the rate of improvement slows significantly (plateaus) after approximately 1 x 10^16 FLOPs. * **Approximate Data Points:** * At ~1 x 10^15 FLOPs: Accuracy ≈ 51% (similar starting point to ThinkPRM-14B). * At ~3 x 10^15 FLOPs: Accuracy ≈ 67% (notably higher than ThinkPRM-14B at this point). * At ~1 x 10^16 FLOPs: Accuracy ≈ 74% (intersects with ThinkPRM-14B). * At ~3 x 10^16 FLOPs: Accuracy ≈ 73% (slight dip or plateau). * At ~1 x 10^17 FLOPs: Accuracy ≈ 79% (ends lower than ThinkPRM-14B). ### Key Observations 1. **Crossover Point:** The two methods achieve approximately equal accuracy (~74%) at an estimated compute level of 1 x 10^16 FLOPs. 2. **Diverging Scaling:** After the crossover point, the performance trajectories diverge. ThinkPRM-14B continues to scale efficiently, while Majority voting exhibits diminishing returns. 3. **Initial Advantage:** Majority voting provides a significant accuracy advantage at lower compute budgets (between ~2 x 10^15 and 8 x 10^15 FLOPs). 4. **Final Outcome:** At the highest compute level shown (~1 x 10^17 FLOPs), ThinkPRM-14B outperforms Majority voting by approximately 6 percentage points (85% vs. 79%). ### Interpretation This chart illustrates a classic trade-off in machine learning between a method that is highly efficient at low compute (Majority voting) and one that scales more effectively with abundant resources (ThinkPRM-14B). * **What the data suggests:** The "ThinkPRM-14B" method appears to be a more scalable architecture or technique for this task. Its consistent upward trend implies it can effectively utilize additional computational power to improve performance without hitting an early plateau. In contrast, "Majority voting" likely represents an ensemble or sampling technique that provides quick gains but has a fundamental performance ceiling that is reached relatively quickly. * **How elements relate:** The x-axis (compute) is the independent variable being controlled, and the y-axis (accuracy) is the dependent outcome. The two lines represent different strategies for converting compute into performance. The crossover point is critical, as it defines the computational budget at which one should switch from using Majority voting to ThinkPRM-14B for optimal results. * **Notable implications:** For projects with constrained computational budgets (below 1 x 10^16 FLOPs), Majority voting is the more effective choice. For state-of-the-art results where maximum accuracy is the goal and compute is less constrained, ThinkPRM-14B is the superior approach. The chart provides a clear, data-driven rationale for selecting a method based on available resources.

## Line Graph: Compute-matched analysis: MATH-500 ### Overview The image is a line graph comparing the accuracy of two computational models, **ThinkPRM-14B** and **Majority voting**, across varying computational costs (FLOPs) on the MATH-500 benchmark. The x-axis uses a logarithmic scale for FLOPs, while the y-axis represents accuracy in percentage. ### Components/Axes - **X-axis**: "Estimated FLOPs (log scale)" with markers at `1×10¹⁵`, `1×10¹⁶`, and `1×10¹⁷`. - **Y-axis**: "Accuracy (%)" ranging from 50% to 85% in 5% increments. - **Legend**: Located at the bottom-right corner, with: - **Orange line**: "ThinkPRM-14B" - **Beige line**: "Majority voting" - **Title**: "Compute-matched analysis: MATH-500" at the top. - **Subtitle**: "Generator: Qwen2.5-14B" in the top-left corner. ### Detailed Analysis #### ThinkPRM-14B (Orange Line) - **Trend**: Steadily increases from ~50% at `1×10¹⁵` FLOPs to ~85% at `1×10¹⁷` FLOPs. - **Key Data Points**: - `1×10¹⁵` FLOPs: ~50% accuracy. - `1×10¹⁶` FLOPs: ~70% accuracy. - `1×10¹⁷` FLOPs: ~85% accuracy. #### Majority Voting (Beige Line) - **Trend**: Gradual increase from ~50% at `1×10¹⁵` FLOPs to ~78% at `1×10¹⁷` FLOPs. - **Key Data Points**: - `1×10¹⁵` FLOPs: ~50% accuracy. - `1×10¹⁶` FLOPs: ~72% accuracy. - `1×10¹⁷` FLOPs: ~78% accuracy. ### Key Observations 1. **Performance Gap**: ThinkPRM-14B consistently outperforms Majority voting across all FLOP levels, with the gap widening at higher computational costs. 2. **Scalability**: ThinkPRM-14B shows a steeper improvement curve, suggesting better utilization of increased computational resources. 3. **Majority Voting Plateau**: Majority voting’s accuracy plateaus near 78% despite further FLOP increases, indicating diminishing returns. ### Interpretation The data demonstrates that **ThinkPRM-14B** achieves significantly higher accuracy than **Majority voting** as computational resources scale. This suggests that ThinkPRM-14B’s architecture or training is more effective at leveraging computational power for the MATH-500 task. Majority voting, while simpler, shows limited scalability, implying it may rely on heuristic aggregation rather than model-specific optimization. The results highlight the importance of model design in computational efficiency for complex reasoning tasks.