## Line Chart: Parallel scaling of verifier compute: MATH-500 ### Overview The image is a line chart comparing the accuracy (%) of different models (ThinkPRM-14B, ThinkPRM-14B@4, ThinkPRM-14B@8, DiscPRM-14B, and Majority) against the number of solutions (2^0 to 2^5) for a verifier compute task on MATH-500. The chart illustrates how the accuracy of each model scales with an increasing number of solutions. ### Components/Axes * **Title:** Parallel scaling of verifier compute: MATH-500 * **X-axis:** Number of solutions (2^0, 2^1, 2^2, 2^3, 2^4, 2^5) * **Y-axis:** Accuracy (%) (50, 55, 60, 65, 70, 75, 80, 85) * **Legend:** Located at the bottom of the chart. * ThinkPRM-14B (brown line with star markers) * ThinkPRM-14B@4 (light blue dashed line with triangle markers) * ThinkPRM-14B@8 (yellow dash-dot line with square markers) * DiscPRM-14B (green line with circle markers) * Majority (tan line with circle markers) ### Detailed Analysis * **X-Axis Values:** The x-axis represents the number of solutions, with values at powers of 2: 2^0 (1), 2^1 (2), 2^2 (4), 2^3 (8), 2^4 (16), and 2^5 (32). * **Y-Axis Values:** The y-axis represents the accuracy in percentage, ranging from 50% to 85% in increments of 5%. **Data Series Analysis:** * **ThinkPRM-14B (brown line with star markers):** * Trend: Generally increasing. * Data Points: * 2^0: ~51% * 2^1: ~51% * 2^2: ~69% * 2^3: ~77% * 2^4: ~79% * 2^5: ~83% * **ThinkPRM-14B@4 (light blue dashed line with triangle markers):** * Trend: Increasing. * Data Points: * 2^0: ~51% * 2^1: ~62% * 2^2: ~69% * 2^3: ~81% * 2^4: ~82% * 2^5: ~84% * **ThinkPRM-14B@8 (yellow dash-dot line with square markers):** * Trend: Increasing. * Data Points: * 2^0: ~51% * 2^1: ~61% * 2^2: ~69% * 2^3: ~77% * 2^4: ~80% * 2^5: ~83% * **DiscPRM-14B (green line with circle markers):** * Trend: Increasing, then plateaus. * Data Points: * 2^0: ~51% * 2^1: ~62% * 2^2: ~70% * 2^3: ~73% * 2^4: ~73% * 2^5: ~73% * **Majority (tan line with circle markers):** * Trend: Stagnant, then increasing. * Data Points: * 2^0: ~51% * 2^1: ~51% * 2^2: ~69% * 2^3: ~73% * 2^4: ~73% * 2^5: ~73% ### Key Observations * ThinkPRM-14B@4 generally achieves the highest accuracy across all solution counts. * DiscPRM-14B and Majority models plateau in accuracy after 2^3 solutions. * All models start with similar accuracy at 2^0 solutions. * The accuracy of all models generally increases with the number of solutions, except for DiscPRM-14B and Majority, which plateau. ### Interpretation The chart demonstrates the parallel scaling performance of different verifier compute models on the MATH-500 dataset. The ThinkPRM-14B variants, especially ThinkPRM-14B@4, show the best scaling behavior, achieving higher accuracy as the number of solutions increases. The DiscPRM-14B and Majority models, however, reach a performance ceiling, suggesting that increasing the number of solutions beyond a certain point does not improve their accuracy. This indicates that the ThinkPRM-14B models are better suited for tasks where a larger number of solutions can be explored to improve verification accuracy. The "@4" and "@8" likely refer to different configurations or parameters within the ThinkPRM-14B model, with "@4" being the most effective in this scenario.

## Line Chart: Parallel Scaling of Verifier Compute - MATH-500 ### Overview This line chart illustrates the relationship between the number of solutions generated and the accuracy achieved by different verification methods on the MATH-500 dataset. The x-axis represents the number of solutions (on a logarithmic scale), and the y-axis represents the accuracy in percentage. The chart compares the performance of several models: ThinkPRM-14B, ThinkPRM-14B@4, ThinkPRM-14B@8, DiscPRM-14B, and Majority voting. ### Components/Axes * **Title:** Parallel scaling of verifier compute: MATH-500 * **X-axis Label:** Number of solutions * **X-axis Scale:** Logarithmic scale, with markers at 2⁰, 2¹, 2², 2³, 2⁴, and 2⁵. * **Y-axis Label:** Accuracy (%) * **Y-axis Scale:** Linear scale, ranging from 50% to 85%. * **Legend:** Located at the bottom of the chart. * ThinkPRM-14B (Orange) * ThinkPRM-14B@4 (Light Blue, dashed) * ThinkPRM-14B@8 (Yellow) * DiscPRM-14B (Teal) * Majority (Brown) ### Detailed Analysis The chart displays five distinct lines, each representing a different verification method. * **ThinkPRM-14B (Orange):** This line starts at approximately 52% accuracy at 2⁰ solutions, steadily increases to around 78% at 2³ solutions, then continues to rise to approximately 83% at 2⁵ solutions. * **ThinkPRM-14B@4 (Light Blue, dashed):** This line begins at roughly 52% accuracy at 2⁰ solutions, rapidly increases to approximately 81% at 2² solutions, plateaus around 82% at 2³ and 2⁴ solutions, and then slightly decreases to around 81% at 2⁵ solutions. * **ThinkPRM-14B@8 (Yellow):** This line starts at approximately 52% accuracy at 2⁰ solutions, increases to around 78% at 2³ solutions, and continues to rise to approximately 84% at 2⁵ solutions. * **DiscPRM-14B (Teal):** This line begins at approximately 52% accuracy at 2⁰ solutions, increases to around 72% at 2³ solutions, and remains relatively stable at around 73% at 2⁴ and 2⁵ solutions. * **Majority (Brown):** This line starts at approximately 52% accuracy at 2⁰ solutions, sharply increases to around 68% at 2² solutions, then rises to approximately 73% at 2³ solutions, and decreases to around 71% at 2⁵ solutions. ### Key Observations * **Performance Improvement with More Solutions:** All methods demonstrate an increase in accuracy as the number of solutions increases, indicating that generating more potential solutions improves verification performance. * **ThinkPRM-14B@4 Outperforms:** The ThinkPRM-14B@4 model consistently achieves the highest accuracy across most of the solution range, peaking at approximately 82%. * **Diminishing Returns:** The rate of accuracy improvement appears to diminish as the number of solutions increases, particularly for ThinkPRM-14B@4. * **Majority Voting is Lowest:** The Majority voting method consistently exhibits the lowest accuracy among the tested models. * **ThinkPRM-14B and ThinkPRM-14B@8 are similar:** These two lines are very close to each other. ### Interpretation The data suggests that parallelizing the verification process (as demonstrated by ThinkPRM-14B@4 and ThinkPRM-14B@8) can significantly improve accuracy, especially when a moderate number of solutions are considered. The ThinkPRM-14B@4 model appears to strike a balance between computational cost and accuracy, achieving high performance without requiring a large number of solutions. The diminishing returns observed at higher solution counts suggest that there may be a point where the computational cost of generating additional solutions outweighs the marginal gains in accuracy. The lower performance of the Majority voting method indicates that a more sophisticated verification strategy is necessary for achieving high accuracy on the MATH-500 dataset. The logarithmic scale on the x-axis highlights the importance of scaling the number of solutions to achieve substantial accuracy improvements. The consistent starting point of all lines at approximately 52% suggests a baseline accuracy level inherent to the problem or the initial solution generation process.

## Line Chart: Parallel scaling of verifier compute: MATH-500 ### Overview This image is a line chart titled "Parallel scaling of verifier compute: MATH-500". It plots the accuracy percentage of five different models or methods against an increasing number of solutions, presented on a logarithmic scale (base 2). The chart demonstrates how the performance of these verifiers scales as more parallel solutions are considered. ### Components/Axes * **Title:** "Parallel scaling of verifier compute: MATH-500" (Top center). * **Y-Axis:** Labeled "Accuracy (%)". The scale runs from 50 to 85, with major tick marks every 5 units (50, 55, 60, 65, 70, 75, 80, 85). * **X-Axis:** Labeled "Number of solutions". The scale is logarithmic with base 2, showing tick marks at 2⁰ (1), 2¹ (2), 2² (4), 2³ (8), 2⁴ (16), and 2⁵ (32). * **Legend:** Positioned at the bottom of the chart, centered. It contains five entries, each with a colored line, marker symbol, and label: 1. **ThinkPRM-14B:** Orange line with star (★) markers. 2. **DiscPRM-14B:** Teal/green line with circle (●) markers. 3. **ThinkPRM-14B@4:** Light blue dashed line with triangle (▲) markers. 4. **ThinkPRM-14B@8:** Yellow line with square (■) markers. 5. **Majority:** Brown/tan line with circle (●) markers. ### Detailed Analysis The chart tracks five data series. All series begin at the same point (1 solution, 50% accuracy). Below is the approximate data extracted for each series, with trends noted. **1. ThinkPRM-14B (Orange, ★)** * **Trend:** Consistent, strong upward slope. * **Data Points (Approx.):** * 2⁰ (1): 50% * 2¹ (2): ~61% * 2² (4): ~68% * 2³ (8): ~75% * 2⁴ (16): ~79% * 2⁵ (32): ~83% **2. DiscPRM-14B (Teal, ●)** * **Trend:** Increases initially, then plateaus after 8 solutions. * **Data Points (Approx.):** * 2⁰ (1): 50% * 2¹ (2): ~61% * 2² (4): ~67% * 2³ (8): ~73% * 2⁴ (16): ~73% (plateau) * 2⁵ (32): ~75% **3. ThinkPRM-14B@4 (Light Blue, ▲, Dashed Line)** * **Trend:** Very steep upward slope, one of the top performers. * **Data Points (Approx.):** * 2⁰ (1): 50% * 2¹ (2): ~65% * 2² (4): ~71% * 2³ (8): ~81% * 2⁴ (16): ~82% * 2⁵ (32): ~84% **4. ThinkPRM-14B@8 (Yellow, ■)** * **Trend:** Strong upward slope, ends as the highest-performing series. * **Data Points (Approx.):** * 2⁰ (1): 50% * 2¹ (2): ~65% * 2² (4): ~71% * 2³ (8): ~77% * 2⁴ (16): ~80% * 2⁵ (32): ~85% **5. Majority (Brown, ●)** * **Trend:** Increases to a point, then flatlines completely. * **Data Points (Approx.):** * 2⁰ (1): 50% * 2¹ (2): 50% (no initial gain) * 2² (4): ~67% * 2³ (8): ~73% * 2⁴ (16): ~73% (plateau) * 2⁵ (32): ~73% (plateau) ### Key Observations 1. **Universal Starting Point:** All methods start at 50% accuracy with a single solution, which is the baseline. 2. **Clear Performance Tiers:** At the maximum measured point (32 solutions), a clear hierarchy emerges: * **Top Tier:** ThinkPRM-14B@8 (~85%) and ThinkPRM-14B@4 (~84%). * **Middle Tier:** ThinkPRM-14B (~83%). * **Lower Tier:** DiscPRM-14B (~75%) and Majority (~73%). 3. **Scaling Behavior:** The "ThinkPRM" family of models (all variants) shows continuous improvement across the entire range of solutions. In contrast, both "DiscPRM-14B" and "Majority" show diminishing returns, plateauing after 8 solutions (2³). 4. **Initial Jump:** The "@4" and "@8" variants of ThinkPRM show a larger initial performance jump from 1 to 2 solutions compared to the base ThinkPRM-14B. 5. **Majority Baseline:** The "Majority" method, likely a simple voting baseline, fails to gain any benefit from 1 to 2 solutions and is ultimately outperformed by all neural verifier models. ### Interpretation This chart provides a performance benchmark for different AI verification strategies on the MATH-500 dataset, specifically measuring how accuracy improves when the system is allowed to generate and evaluate multiple solution candidates in parallel. The data suggests that the **ThinkPRM architecture scales more effectively with increased parallel compute** than the DiscPRM architecture or a simple majority vote. The continuous upward trend of the ThinkPRM lines indicates that its method for verifying or ranking solutions continues to extract useful signal even from a large pool of 32 candidates. The plateau of the **Majority and DiscPRM methods** implies a fundamental limit to their approach. For Majority, this is intuitive—beyond a certain point, adding more random guesses doesn't improve the consensus. For DiscPRM, it may indicate that its discrimination capability saturates, and it cannot effectively differentiate between correct and incorrect solutions in a larger, noisier set. The superior performance of **ThinkPRM-14B@8 and @4** over the base ThinkPRM-14B suggests that techniques like increased sampling (@8 likely means 8 samples per problem) or other ensemble methods provide a significant boost, making them the most compute-efficient strategies for achieving high accuracy in this parallel scaling regime. The chart makes a strong case for investing in verifier models that are designed to leverage parallelism, as the accuracy gains are substantial and sustained.

## Line Chart: Parallel scaling of verifier compute: MATH-500 ### Overview The chart illustrates the relationship between the number of solutions (x-axis) and accuracy percentage (y-axis) for different computational configurations. Five data series are plotted, showing how accuracy improves as the number of solutions increases exponentially (2⁰ to 2⁵). The chart emphasizes parallel scaling efficiency across different model variants. ### Components/Axes - **X-axis**: "Number of solutions" (logarithmic scale: 2⁰, 2¹, 2², 2³, 2⁴, 2⁵) - **Y-axis**: "Accuracy (%)" (linear scale: 50% to 85%) - **Legend**: Located at the bottom, with five entries: - Orange: ThinkPRM-14B - Green: DiscPRM-14B - Blue: ThinkPRM-14B@4 - Brown: Majority - Yellow: ThinkPRM-14B@8 - **Line styles**: Solid lines for all series, with markers (star, triangle, square) for data points ### Detailed Analysis 1. **ThinkPRM-14B (orange)**: - Starts at ~50% accuracy at 2⁰ - Reaches ~80% at 2⁵ - Steady upward slope with moderate curvature 2. **DiscPRM-14B (green)**: - Begins at ~50% at 2⁰ - Peaks at ~75% at 2⁵ - Slower growth compared to ThinkPRM variants 3. **ThinkPRM-14B@4 (blue)**: - Starts at ~50% at 2⁰ - Reaches ~82% at 2⁵ - Sharpest initial increase, then plateaus 4. **Majority (brown)**: - Flat line at ~50% until 2² - Rises to ~73% at 2⁵ - Least effective scaling 5. **ThinkPRM-14B@8 (yellow)**: - Highest performance across all points - ~85% accuracy at 2⁵ - Most aggressive upward trajectory ### Key Observations - All models show improved accuracy with more solutions, but scaling efficiency varies - ThinkPRM-14B@8 consistently outperforms others by 5-10% at higher solution counts - Majority method lags significantly until 2³, then improves slowly - ThinkPRM-14B@4 shows the steepest initial improvement (50%→70% between 2¹→2²) - DiscPRM-14B demonstrates the most stable but slower growth pattern ### Interpretation The data suggests that: 1. **Model configuration impacts scaling efficiency**: Higher configurations (e.g., @8) achieve better accuracy gains per additional solution 2. **Parallel compute benefits are non-linear**: Most models show accelerating returns up to 2³ solutions, then plateau 3. **Majority method limitations**: Its flat initial performance indicates it may not leverage parallel compute effectively 4. **ThinkPRM variants outperform DiscPRM**: Suggests architectural differences in verifier compute optimization The chart demonstrates that increasing parallel compute resources improves verification accuracy, with model architecture and configuration playing critical roles in scaling efficiency. The ThinkPRM-14B@8 configuration appears optimal for this benchmark, achieving near-85% accuracy at maximum solution count.