## Line Chart: Accuracy vs. Thinking Compute ### Overview The image is a line chart comparing the accuracy of four different methods ("pass@k (Oracle)", "majority@k", "short-1@k (Ours)", and "short-3@k (Ours)") against the "Thinking Compute" measured in thousands of thinking tokens. The chart displays how accuracy changes as the thinking compute increases for each method. ### Components/Axes * **X-axis:** "Thinking Compute (thinking tokens in thousands)". The scale ranges from approximately 25 to 175, with tick marks at intervals of 25. * **Y-axis:** "Accuracy". The scale ranges from 0.84 to 0.92, with tick marks at intervals of 0.02. * **Legend:** Located in the bottom-right corner of the chart. * Black dotted line with triangle markers: "pass@k (Oracle)" * Brown solid line with circle markers: "majority@k" * Light blue solid line with square markers: "short-1@k (Ours)" * Teal solid line with diamond markers: "short-3@k (Ours)" ### Detailed Analysis * **pass@k (Oracle):** (Black dotted line with triangle markers) * Trend: The line slopes sharply upward initially, then flattens out as the thinking compute increases. * Data Points: * At 25k tokens, accuracy is approximately 0.88. * At 50k tokens, accuracy is approximately 0.91. * At 75k tokens, accuracy is approximately 0.925. * At 100k tokens, accuracy is approximately 0.93. * At 125k tokens, accuracy is approximately 0.93. * At 150k tokens, accuracy is approximately 0.93. * At 175k tokens, accuracy is approximately 0.93. * **majority@k:** (Brown solid line with circle markers) * Trend: The line slopes upward consistently. * Data Points: * At 25k tokens, accuracy is approximately 0.84. * At 50k tokens, accuracy is approximately 0.87. * At 75k tokens, accuracy is approximately 0.89. * At 100k tokens, accuracy is approximately 0.905. * At 125k tokens, accuracy is approximately 0.915. * At 150k tokens, accuracy is approximately 0.92. * At 175k tokens, accuracy is approximately 0.925. * **short-1@k (Ours):** (Light blue solid line with square markers) * Trend: The line slopes upward initially, reaches a peak, and then slopes downward. * Data Points: * At 25k tokens, accuracy is approximately 0.84. * At 50k tokens, accuracy is approximately 0.88. * At 75k tokens, accuracy is approximately 0.882. * At 100k tokens, accuracy is approximately 0.88. * At 125k tokens, accuracy is approximately 0.87. * **short-3@k (Ours):** (Teal solid line with diamond markers) * Trend: The line slopes upward initially, then flattens out. * Data Points: * At 25k tokens, accuracy is approximately 0.84. * At 50k tokens, accuracy is approximately 0.89. * At 75k tokens, accuracy is approximately 0.91. * At 100k tokens, accuracy is approximately 0.92. * At 125k tokens, accuracy is approximately 0.922. * At 150k tokens, accuracy is approximately 0.922. * At 175k tokens, accuracy is approximately 0.922. ### Key Observations * "pass@k (Oracle)" achieves the highest accuracy overall. * "majority@k" shows a steady increase in accuracy with increasing thinking compute, but it consistently underperforms compared to "pass@k (Oracle)" and "short-3@k (Ours)". * "short-1@k (Ours)" reaches a peak accuracy and then declines, suggesting that increasing thinking compute beyond a certain point may be detrimental to its performance. * "short-3@k (Ours)" performs well, approaching the accuracy of "pass@k (Oracle)" as thinking compute increases. ### Interpretation The chart demonstrates the relationship between thinking compute and accuracy for different methods. The "pass@k (Oracle)" method serves as an upper bound or ideal performance, while the other methods show varying degrees of improvement as thinking compute increases. The "short-1@k (Ours)" method's decline in accuracy after a certain point suggests a potential overfitting or diminishing returns effect. The "short-3@k (Ours)" method appears to be a promising approach, achieving relatively high accuracy with increasing thinking compute. The "majority@k" method shows consistent improvement but lags behind the others, indicating it may not be as effective in leveraging increased thinking compute. The data suggests that the choice of method and the amount of thinking compute should be carefully considered to optimize accuracy.

\n ## Line Chart: Accuracy vs. Thinking Compute ### Overview This image presents a line chart illustrating the relationship between "Thinking Compute" (measured in thousands of tokens) and "Accuracy" for several different methods. The chart compares the performance of "pass@k (Oracle)", "majority@k", "short-1@k (Ours)", and "short-3@k (Ours)". ### Components/Axes * **X-axis:** "Thinking Compute (thinking tokens in thousands)". The scale ranges from approximately 20 to 175, with markers at 25, 50, 75, 100, 125, 150, and 175. * **Y-axis:** "Accuracy". The scale ranges from approximately 0.84 to 0.93, with markers at 0.84, 0.86, 0.88, 0.90, and 0.92. * **Legend:** Located in the bottom-right corner of the chart. It identifies the following data series: * "pass@k (Oracle)" - represented by a dotted black line. * "majority@k" - represented by a dotted purple line. * "short-1@k (Ours)" - represented by a solid red line. * "short-3@k (Ours)" - represented by a solid cyan line. * **Gridlines:** A light gray grid is present to aid in reading values. ### Detailed Analysis * **pass@k (Oracle):** This line starts at approximately 0.84 at a compute of 20, rises sharply to approximately 0.93 at a compute of 75, and then plateaus, remaining around 0.93 for the rest of the range. * **majority@k:** This line begins at approximately 0.84 at a compute of 20, increases steadily to approximately 0.91 at a compute of 75, and then continues to increase, reaching approximately 0.925 at a compute of 175. * **short-1@k (Ours):** This line starts at approximately 0.84 at a compute of 20, increases steadily to approximately 0.91 at a compute of 150, and then plateaus. * **short-3@k (Ours):** This line begins at approximately 0.84 at a compute of 20, rises rapidly to approximately 0.89 at a compute of 50, then plateaus around 0.88-0.89 for the remainder of the range. Here's a more detailed breakdown of approximate data points: | Thinking Compute (thousands) | pass@k (Oracle) | majority@k | short-1@k (Ours) | short-3@k (Ours) | |---|---|---|---|---| | 25 | 0.89 | 0.87 | 0.86 | 0.87 | | 50 | 0.92 | 0.89 | 0.88 | 0.89 | | 75 | 0.93 | 0.91 | 0.90 | 0.88 | | 100 | 0.93 | 0.91 | 0.91 | 0.88 | | 125 | 0.93 | 0.91 | 0.91 | 0.87 | | 150 | 0.93 | 0.92 | 0.91 | 0.87 | | 175 | 0.93 | 0.925 | 0.91 | 0.87 | ### Key Observations * "pass@k (Oracle)" achieves the highest accuracy and plateaus quickly. * "short-3@k (Ours)" has the lowest accuracy and also plateaus quickly. * "majority@k" and "short-1@k (Ours)" show a more gradual increase in accuracy. * The performance gap between "pass@k (Oracle)" and the other methods widens as compute increases. ### Interpretation The chart demonstrates the impact of "Thinking Compute" on the accuracy of different methods. "pass@k (Oracle)" benefits significantly from even a small increase in compute, quickly reaching a high level of accuracy and then stabilizing. This suggests that the "Oracle" method is highly efficient in utilizing computational resources. The "short-3@k (Ours)" method shows limited improvement with increased compute, indicating it may be constrained by its design or require significantly more compute to achieve comparable accuracy. The "majority@k" and "short-1@k (Ours)" methods fall in between, showing a more gradual improvement with increasing compute. The "Ours" designation suggests these are methods developed by the authors of the study. The data suggests that while increasing compute generally improves accuracy, the effectiveness of that increase varies significantly depending on the method used. The plateauing of the lines indicates diminishing returns – beyond a certain point, adding more compute does not yield substantial gains in accuracy.

## Line Chart: Accuracy vs. Thinking Compute for Different Methods ### Overview The image is a line chart comparing the performance of four different methods or models. The chart plots "Accuracy" on the vertical axis against "Thinking Compute" (measured in thousands of thinking tokens) on the horizontal axis. The primary purpose is to show how the accuracy of each method scales with increased computational resources (thinking tokens). The chart contains four distinct data series, each represented by a unique line style, color, and marker. ### Components/Axes * **Y-Axis (Vertical):** * **Label:** "Accuracy" * **Scale:** Linear scale ranging from approximately 0.84 to 0.93. * **Major Ticks:** 0.84, 0.86, 0.88, 0.90, 0.92. * **X-Axis (Horizontal):** * **Label:** "Thinking Compute (thinking tokens in thousands)" * **Scale:** Linear scale ranging from approximately 20 to 175. * **Major Ticks:** 25, 50, 75, 100, 125, 150, 175. * **Legend:** * **Position:** Bottom-right quadrant of the chart area. * **Entries (from top to bottom as listed):** 1. `pass@k (Oracle)`: Represented by a black, dotted line with upward-pointing triangle markers (▲). 2. `majority@k`: Represented by a solid, dark red (maroon) line with circle markers (●). 3. `short-1@k (Ours)`: Represented by a solid, light blue (cyan) line with square markers (■). 4. `short-3@k (Ours)`: Represented by a solid, teal (blue-green) line with diamond markers (◆). ### Detailed Analysis **1. `pass@k (Oracle)` (Black Dotted Line, ▲):** * **Trend:** Shows a steep, concave-down increase in accuracy with compute, exhibiting strong diminishing returns. It is the highest-performing series across the entire range. * **Approximate Data Points:** * At ~20k tokens: Accuracy ≈ 0.84 * At 50k tokens: Accuracy ≈ 0.90 * At 75k tokens: Accuracy ≈ 0.92 * At 100k tokens: Accuracy ≈ 0.925 * At 125k tokens: Accuracy ≈ 0.928 (appears to plateau near this value). **2. `majority@k` (Dark Red Solid Line, ●):** * **Trend:** Shows a steady, nearly linear increase in accuracy with compute. It starts as the lowest-performing method but eventually surpasses `short-1@k`. * **Approximate Data Points:** * At ~20k tokens: Accuracy ≈ 0.84 * At 50k tokens: Accuracy ≈ 0.863 * At 75k tokens: Accuracy ≈ 0.885 * At 100k tokens: Accuracy ≈ 0.896 * At 125k tokens: Accuracy ≈ 0.905 * At 150k tokens: Accuracy ≈ 0.913 * At ~170k tokens: Accuracy ≈ 0.924 **3. `short-1@k (Ours)` (Light Blue Solid Line, ■):** * **Trend:** Shows an initial increase, peaks, and then begins to decline. This suggests a potential overfitting or efficiency loss at higher compute levels for this specific method. * **Approximate Data Points:** * At ~20k tokens: Accuracy ≈ 0.84 * At 35k tokens: Accuracy ≈ 0.874 * At 50k tokens: Accuracy ≈ 0.879 * At 65k tokens: Accuracy ≈ 0.881 (peak) * At 80k tokens: Accuracy ≈ 0.880 * At 100k tokens: Accuracy ≈ 0.877 * At 120k tokens: Accuracy ≈ 0.870 **4. `short-3@k (Ours)` (Teal Solid Line, ◆):** * **Trend:** Shows a strong, concave-down increase similar to the Oracle but at a lower absolute accuracy. It consistently outperforms `short-1@k` and `majority@k` for most of the range, plateauing at higher compute. * **Approximate Data Points:** * At ~20k tokens: Accuracy ≈ 0.84 * At 35k tokens: Accuracy ≈ 0.864 * At 50k tokens: Accuracy ≈ 0.894 * At 65k tokens: Accuracy ≈ 0.906 * At 80k tokens: Accuracy ≈ 0.913 * At 100k tokens: Accuracy ≈ 0.920 * At 125k tokens: Accuracy ≈ 0.922 * At 140k tokens: Accuracy ≈ 0.922 (plateau). ### Key Observations 1. **Performance Hierarchy:** The Oracle (`pass@k`) sets the upper bound. Among the non-oracle methods, `short-3@k (Ours)` is the top performer for compute budgets above ~40k tokens. `majority@k` shows the most consistent scaling without degradation. 2. **Divergent Scaling:** The two "Ours" methods (`short-1@k` and `short-3@k`) exhibit fundamentally different scaling behaviors. `short-3@k` scales well, while `short-1@k` peaks and regresses, indicating that the "short-3" variant is more robust to increased compute. 3. **Crossover Point:** The `majority@k` line crosses above the `short-1@k` line at approximately 80k thinking tokens. Before this point, `short-1@k` is more accurate; after, `majority@k` is superior. 4. **Convergence at Low Compute:** All four methods start at nearly the same accuracy point (~0.84) when thinking compute is very low (~20k tokens), suggesting a common baseline performance. ### Interpretation This chart likely comes from research on scaling inference-time compute ("thinking tokens") for language models or reasoning systems. The data suggests several key insights: * **Value of Increased Compute:** For most methods, allocating more thinking tokens leads to higher accuracy, validating the core hypothesis that "thinking more" can improve performance. * **Method Efficiency Matters:** The stark difference between `short-1@k` and `short-3@k` demonstrates that not all methods benefit equally from extra compute. The "short-3" approach is architecturally or algorithmically better at converting additional tokens into accuracy gains. The decline of `short-1@k` could indicate it starts generating redundant or counterproductive reasoning steps at high token counts. * **Oracle as a Benchmark:** The `pass@k (Oracle)` line represents an idealized upper bound (perhaps using ground-truth selection). The gap between it and `short-3@k` shows the remaining potential for improvement in the proposed method. * **Practical Trade-offs:** The choice of method depends on the available compute budget. For very low budgets (<40k tokens), the methods are similar. For medium budgets (40k-80k), `short-3@k` is best. For very high budgets where `short-3@k` plateaus, `majority@k` continues to improve slowly and might eventually catch up, though it requires significantly more tokens to reach the same accuracy level that `short-3@k` achieves earlier. The chart effectively argues for the superiority of the `short-3@k (Ours)` method in the mid-to-high compute regime, while honestly showing the limitations of its `short-1@k` counterpart.

## Line Chart: Model Accuracy vs. Thinking Compute ### Overview The chart compares the accuracy of four different models (pass@k, majority@k, short-1@k, short-3@k) across varying levels of thinking compute (measured in thousands of tokens). Accuracy is plotted on the y-axis (0.84–0.92), while thinking compute is on the x-axis (25–175k tokens). The Oracle (pass@k) serves as the benchmark, with other models showing varying performance trends. ### Components/Axes - **X-axis**: Thinking Compute (thinking tokens in thousands) – Range: 25 to 175k - **Y-axis**: Accuracy – Range: 0.84 to 0.92 - **Legend**: Located in the bottom-right corner, with four entries: - **pass@k (Oracle)**: Dashed line with triangle markers (black) - **majority@k**: Solid red line with circle markers - **short-1@k (Ours)**: Solid blue line with square markers - **short-3@k (Ours)**: Solid green line with diamond markers ### Detailed Analysis 1. **pass@k (Oracle)**: - Starts at 0.84 accuracy at 25k tokens. - Increases steadily to 0.92 accuracy at 175k tokens. - Linear upward trend with no plateaus. 2. **majority@k**: - Begins at 0.84 accuracy at 25k tokens. - Slower, gradual increase compared to Oracle. - Reaches 0.92 accuracy at 150k tokens. - Linear upward trend but lags behind Oracle. 3. **short-1@k (Ours)**: - Starts at 0.84 accuracy at 25k tokens. - Peaks at 0.88 accuracy around 75k tokens. - Declines slightly to 0.87 accuracy at 175k tokens. - Non-linear: Rises sharply, then plateaus/declines. 4. **short-3@k (Ours)**: - Starts at 0.84 accuracy at 25k tokens. - Peaks at 0.92 accuracy around 100k tokens. - Plateaus at 0.92 accuracy from 100k to 175k tokens. - Non-linear: Rapid rise followed by stabilization. ### Key Observations - **Oracle Dominance**: The pass@k (Oracle) consistently outperforms all other models across all compute levels. - **majority@k Trade-off**: Requires significantly more compute (150k tokens) to match Oracle’s 175k-token performance. - **short-1@k Efficiency**: Achieves moderate accuracy (0.88) with fewer tokens (75k) but degrades at higher compute. - **short-3@k Efficiency**: Matches Oracle’s accuracy (0.92) at 100k tokens but plateaus, suggesting diminishing returns beyond this point. ### Interpretation The chart highlights the relationship between compute efficiency and accuracy for different models. The Oracle (pass@k) represents the ideal performance, while majority@k demonstrates a compute-heavy approach. The short-1@k and short-3@k models (labeled "Ours") show trade-offs: short-1@k sacrifices accuracy at higher compute, while short-3@k achieves Oracle-level accuracy at 100k tokens but offers no further gains. This suggests that optimizing compute allocation is critical for balancing efficiency and performance, with short-3@k potentially offering the best cost-accuracy ratio up to 100k tokens. The Oracle’s linear scalability underscores the theoretical upper bound for these models.