## Chart: Compute-matched analysis: GPQA-Physics ### Overview The image is a line chart comparing the accuracy of "ThinkPRM-14B" and "Majority voting" methods against the estimated FLOPS (log scale) for a "GPQA-Physics" task. The chart includes a title, axis labels, a legend, and data points for each method. The generator used is "Qwen2.5-32B-Instruct". ### Components/Axes * **Title:** Compute-matched analysis: GPQA-Physics * **Subtitle:** Generator: Qwen2.5-32B-Instruct * **X-axis:** Estimated FLOPS (log scale) * Axis markers: 2 x 10^15, 5 x 10^15, 1 x 10^16, 2 x 10^16, 5 x 10^16 * **Y-axis:** Accuracy (%) * Axis markers: 55, 60, 65, 70 * **Legend:** Located in the bottom-right corner. * ThinkPRM-14B (brown line) * Majority voting (tan line) ### Detailed Analysis * **ThinkPRM-14B (brown line):** * Trend: Generally increasing with some fluctuations. * Data points: * At 2 x 10^15 FLOPS, Accuracy ≈ 54.7% * At 5 x 10^15 FLOPS, Accuracy ≈ 55.9% * At 1 x 10^16 FLOPS, Accuracy ≈ 54.6% * At 2 x 10^16 FLOPS, Accuracy ≈ 64.0% * At 5 x 10^16 FLOPS, Accuracy ≈ 68.7% * At 5 x 10^16 FLOPS, Accuracy ≈ 72.3% * **Majority voting (tan line):** * Trend: Increases, plateaus, then remains relatively constant. * Data points: * At 2 x 10^15 FLOPS, Accuracy ≈ 53.7% * At 5 x 10^15 FLOPS, Accuracy ≈ 58.2% * At 1 x 10^16 FLOPS, Accuracy ≈ 61.8% * At 2 x 10^16 FLOPS, Accuracy ≈ 61.8% * At 5 x 10^16 FLOPS, Accuracy ≈ 61.8% ### Key Observations * ThinkPRM-14B generally outperforms Majority voting, especially at higher FLOPS. * Majority voting plateaus in accuracy after 1 x 10^16 FLOPS. * ThinkPRM-14B shows a more significant increase in accuracy as FLOPS increase. ### Interpretation The chart demonstrates that for the GPQA-Physics task, the "ThinkPRM-14B" method achieves higher accuracy compared to "Majority voting" as the computational resources (estimated FLOPS) increase. The "Majority voting" method plateaus in performance, suggesting it may have reached its limit in terms of accuracy for this task, while "ThinkPRM-14B" continues to improve with more computational power. This suggests that "ThinkPRM-14B" is more scalable and can better leverage increased computational resources for this specific task. The generator used, "Qwen2.5-32B-Instruct", provides the foundation for both methods, and the comparison highlights the effectiveness of different approaches in utilizing the generator's capabilities.

## Line Chart: Compute-matched analysis: GPQA-Physics ### Overview This image presents a line chart illustrating the relationship between Estimated FLOPS (on a logarithmic scale) and Accuracy (%) for two different methods: ThinkPRM-14B and Majority voting. The chart focuses on the GPQA-Physics dataset and uses data generated by Qwen2.5-32B-Instruct. ### Components/Axes * **Title:** Compute-matched analysis: GPQA-Physics * **Subtitle:** Generator: Qwen2.5-32B-Instruct * **X-axis:** Estimated FLOPS (log scale). Markers are at 2 x 10¹⁵, 5 x 10¹⁵, 1 x 10¹⁶, 2 x 10¹⁶, and 5 x 10¹⁶. * **Y-axis:** Accuracy (%). Scale ranges from approximately 54% to 72%. * **Legend:** Located in the bottom-right corner. * ThinkPRM-14B (represented by a solid orange line) * Majority voting (represented by a dashed gray line) ### Detailed Analysis **ThinkPRM-14B (Orange Line):** The line generally slopes upward, indicating increasing accuracy with increasing FLOPS. * At 2 x 10¹⁵ FLOPS, accuracy is approximately 55%. * At 5 x 10¹⁵ FLOPS, accuracy dips to approximately 53%. * At 1 x 10¹⁶ FLOPS, accuracy rises to approximately 57%. * At 2 x 10¹⁶ FLOPS, accuracy is approximately 65%. * At 5 x 10¹⁶ FLOPS, accuracy reaches approximately 71%. **Majority Voting (Gray Dashed Line):** The line shows a more moderate increase in accuracy with increasing FLOPS. * At 2 x 10¹⁵ FLOPS, accuracy is approximately 55%. * At 5 x 10¹⁵ FLOPS, accuracy decreases to approximately 52%. * At 1 x 10¹⁶ FLOPS, accuracy rises to approximately 55%. * At 2 x 10¹⁶ FLOPS, accuracy is approximately 62%. * At 5 x 10¹⁶ FLOPS, accuracy is approximately 62%. ### Key Observations * ThinkPRM-14B consistently outperforms Majority voting across all FLOPS levels. * Both methods show a dip in accuracy at 5 x 10¹⁵ FLOPS. * The accuracy of Majority voting plateaus at approximately 62% after 2 x 10¹⁶ FLOPS, while ThinkPRM-14B continues to improve. * The largest gains in accuracy for ThinkPRM-14B occur between 2 x 10¹⁶ and 5 x 10¹⁶ FLOPS. ### Interpretation The data suggests that ThinkPRM-14B benefits significantly from increased computational resources (FLOPS) in solving GPQA-Physics problems, demonstrating a clear positive correlation between compute and performance. Majority voting, while providing a baseline level of accuracy, shows diminishing returns with increased FLOPS. The initial dip in accuracy for both methods at 5 x 10¹⁵ FLOPS could be due to noise in the data or a specific characteristic of the GPQA-Physics dataset at that computational scale. The plateauing of Majority voting suggests it reaches a performance limit, while ThinkPRM-14B continues to leverage additional compute for improved accuracy. This indicates that ThinkPRM-14B is a more scalable approach for this task. The use of Qwen2.5-32B-Instruct as the generator implies that the quality of the generated data also plays a role in the overall performance of both methods.

## Line Chart: Compute-matched analysis: GPQA-Physics ### Overview The image is a line chart comparing the performance (accuracy) of two methods, "ThinkPRM-14B" and "Majority voting," as a function of increasing computational resources (estimated FLOPs). The analysis is performed on the GPQA-Physics benchmark using the Qwen2.5-32B-Instruct model as the generator. ### Components/Axes * **Chart Title:** "Compute-matched analysis: GPQA-Physics" * **Subtitle/Generator Label:** "Generator: Qwen2.5-32B-Instruct" * **Y-Axis:** * **Label:** "Accuracy (%)" * **Scale:** Linear, ranging from 55 to 70, with major tick marks at 55, 60, 65, and 70. * **X-Axis:** * **Label:** "Estimated FLOPs (log₁₀ scale)" * **Scale:** Logarithmic (base 10). Major tick marks are labeled: `2 x 10¹⁵`, `5 x 10¹⁵`, `1 x 10¹⁶`, `2 x 10¹⁶`, `5 x 10¹⁶`. * **Legend:** Located in the bottom-right quadrant of the chart area. * **Orange line with circle markers:** "ThinkPRM-14B" * **Light brown/tan line with circle markers:** "Majority voting" ### Detailed Analysis **Data Series 1: ThinkPRM-14B (Orange Line)** * **Trend:** The line shows a relatively flat or slightly increasing trend at lower compute levels, followed by a steep, consistent upward slope at higher compute levels. * **Data Points (Approximate):** * At ~2 x 10¹⁵ FLOPs: Accuracy ≈ 55% * At ~5 x 10¹⁵ FLOPs: Accuracy ≈ 55.5% * At ~1 x 10¹⁶ FLOPs: Accuracy ≈ 55% * At ~2 x 10¹⁶ FLOPs: Accuracy ≈ 64% * At ~5 x 10¹⁶ FLOPs: Accuracy ≈ 68% * At the final point (estimated >5 x 10¹⁶ FLOPs): Accuracy ≈ 72% **Data Series 2: Majority voting (Light Brown Line)** * **Trend:** The line shows an initial dip, followed by a steady, moderate upward trend that appears to plateau at the highest compute levels shown. * **Data Points (Approximate):** * At ~2 x 10¹⁵ FLOPs: Accuracy ≈ 55% * At ~5 x 10¹⁵ FLOPs: Accuracy ≈ 52% (This is a notable dip) * At ~1 x 10¹⁶ FLOPs: Accuracy ≈ 58% * At ~2 x 10¹⁶ FLOPs: Accuracy ≈ 61.5% * At ~5 x 10¹⁶ FLOPs: Accuracy ≈ 61.5% (Plateau) ### Key Observations 1. **Crossover Point:** The two methods have similar accuracy at the lowest compute point (~2 x 10¹⁵ FLOPs). ThinkPRM-14B dips below Majority voting at ~5 x 10¹⁵ FLOPs but then surpasses it decisively at ~1 x 10¹⁶ FLOPs and maintains a significant lead thereafter. 2. **Scaling Behavior:** ThinkPRM-14B demonstrates superior scaling with increased compute. Its accuracy continues to climb steeply across the entire range, especially after 1 x 10¹⁶ FLOPs. Majority voting shows more modest gains and appears to saturate. 3. **Anomaly:** The Majority voting series shows a distinct performance drop at ~5 x 10¹⁵ FLOPs before recovering. This could indicate a specific compute regime where the voting mechanism is less effective or a potential measurement outlier. ### Interpretation This chart illustrates a **compute-performance scaling law comparison** for two reasoning or inference techniques on a physics QA task. The key finding is that the "ThinkPRM-14B" method (likely a process reward model or a specific reasoning framework) is significantly more **compute-efficient at higher scales** than the simpler "Majority voting" baseline. * **What it suggests:** Investing more computational resources (FLOPs) yields substantially greater accuracy improvements when using ThinkPRM-14B compared to majority voting. The diverging trends imply that for large-scale, high-performance applications, advanced methods like ThinkPRM are necessary to fully leverage available compute. * **Relationship between elements:** The x-axis (compute) is the independent variable being increased. The y-axis (accuracy) is the dependent outcome. The two lines represent different algorithms attempting to convert the same "budget" of compute into performance. The widening gap between the lines visually quantifies the growing advantage of the more sophisticated method. * **Notable patterns:** The plateau in Majority voting suggests it hits a performance ceiling, while ThinkPRM-14B shows no such saturation within the tested range, hinting at a higher potential ceiling. The initial dip for Majority voting is curious and might warrant investigation into the stability of that method at specific compute points.

## Line Graph: Compute-matched analysis: GPQA-Physics ### Overview The image is a line graph comparing the accuracy of two models (ThinkPRM-14B and Majority voting) across varying computational budgets (FLOPs) on the GPQA-Physics benchmark. The x-axis uses a logarithmic scale for FLOPs, while the y-axis represents accuracy in percentage. Two distinct trends are visible: ThinkPRM-14B shows a steep upward trajectory, while Majority voting exhibits a plateau after initial growth. ### Components/Axes - **Title**: "Compute-matched analysis: GPQA-Physics" (top-center) - **X-axis**: "Estimated FLOPs (log scale)" with markers at 2×10¹⁵, 5×10¹⁵, 1×10¹⁶, 2×10¹⁶, and 5×10¹⁶. - **Y-axis**: "Accuracy (%)" ranging from 55% to 70% in 5% increments. - **Legend**: Located at the bottom-right corner, with: - **Orange line**: ThinkPRM-14B - **Beige line**: Majority voting ### Detailed Analysis #### ThinkPRM-14B (Orange Line) - **Trend**: Starts at ~55% accuracy at 2×10¹⁵ FLOPs, dips slightly to ~54% at 5×10¹⁵, then rises sharply to ~72% at 5×10¹⁶ FLOPs. - **Key Data Points**: - 2×10¹⁵ FLOPs: ~55% (±1%) - 5×10¹⁵ FLOPs: ~56% (±1%) - 1×10¹⁶ FLOPs: ~55% (±1%) - 2×10¹⁶ FLOPs: ~64% (±1%) - 5×10¹⁶ FLOPs: ~67% (±1%) - 7×10¹⁶ FLOPs: ~72% (±1%) #### Majority Voting (Beige Line) - **Trend**: Begins at ~55% at 2×10¹⁵ FLOPs, drops to ~52% at 5×10¹⁵, then rises to ~62% at 2×10¹⁶ FLOPs and plateaus at ~62% for higher FLOPs. - **Key Data Points**: - 2×10¹⁵ FLOPs: ~55% (±1%) - 5×10¹⁵ FLOPs: ~52% (±1%) - 1×10¹⁶ FLOPs: ~58% (±1%) - 2×10¹⁶ FLOPs: ~62% (±1%) - 5×10¹⁶ FLOPs: ~62% (±1%) ### Key Observations 1. **Compute Efficiency**: ThinkPRM-14B demonstrates a strong positive correlation between FLOPs and accuracy, outperforming Majority voting by ~10% at 5×10¹⁶ FLOPs. 2. **Diminishing Returns**: Majority voting plateaus at ~62% accuracy despite increased compute, suggesting limited scalability. 3. **Initial Dip**: Both models show a minor accuracy drop between 2×10¹⁵ and 5×10¹⁵ FLOPs, potentially indicating optimization challenges at mid-scale compute. ### Interpretation The data suggests that ThinkPRM-14B leverages compute more effectively than Majority voting for GPQA-Physics tasks. The steep rise in ThinkPRM-14B’s accuracy at higher FLOPs implies that larger models or optimized architectures can achieve significant performance gains. In contrast, Majority voting’s plateau highlights the limitations of ensemble methods without architectural improvements. The initial dip in both models may reflect transitional phases where increased compute does not yet translate to better performance, possibly due to training instability or suboptimal hyperparameter tuning at mid-scale budgets. This analysis underscores the importance of model design over raw compute in achieving high accuracy on physics-based reasoning tasks.