## Chart: Accuracy vs. Thinking Compute ### Overview The image is a line chart comparing the accuracy of different methods (pass@k (Oracle), majority@k, short-1@k (Ours), and short-3@k (Ours)) against the thinking compute, measured in thousands of thinking tokens. The chart shows how accuracy improves with increased thinking compute for each method. ### Components/Axes * **X-axis:** Thinking Compute (thinking tokens in thousands). Scale ranges from 0 to 150, with tick marks at 50, 100, and 150. * **Y-axis:** Accuracy. Scale ranges from 0.78 to 0.90, with tick marks at 0.78, 0.80, 0.82, 0.84, 0.86, 0.88, and 0.90. * **Legend:** Located in the bottom-right corner of the chart. * `pass@k (Oracle)`: Black dotted line with triangle markers. * `majority@k`: Brown/Red solid line with circle markers. * `short-1@k (Ours)`: Blue solid line with square markers. * `short-3@k (Ours)`: Cyan solid line with diamond markers. ### Detailed Analysis * **pass@k (Oracle):** (Black dotted line with triangle markers) * Trend: Rapidly increases initially, then the rate of increase slows down. * Data Points: * At x=20, y ≈ 0.82 * At x=50, y ≈ 0.86 * At x=100, y ≈ 0.89 * At x=150, y ≈ 0.905 * **majority@k:** (Brown/Red solid line with circle markers) * Trend: Increases almost linearly. * Data Points: * At x=20, y ≈ 0.78 * At x=50, y ≈ 0.81 * At x=75, y ≈ 0.83 * At x=100, y ≈ 0.845 * At x=125, y ≈ 0.855 * At x=150, y ≈ 0.87 * **short-1@k (Ours):** (Blue solid line with square markers) * Trend: Increases, then plateaus. * Data Points: * At x=20, y ≈ 0.78 * At x=50, y ≈ 0.83 * At x=75, y ≈ 0.845 * At x=100, y ≈ 0.847 * At x=125, y ≈ 0.848 * **short-3@k (Ours):** (Cyan solid line with diamond markers) * Trend: Increases, then the rate of increase slows down. * Data Points: * At x=20, y ≈ 0.78 * At x=50, y ≈ 0.82 * At x=75, y ≈ 0.86 * At x=100, y ≈ 0.875 * At x=125, y ≈ 0.882 * At x=150, y ≈ 0.89 ### Key Observations * `pass@k (Oracle)` consistently outperforms the other methods across all thinking compute values. * `majority@k` shows a steady, linear increase in accuracy as thinking compute increases. * `short-1@k (Ours)` plateaus in accuracy after a certain point. * `short-3@k (Ours)` performs better than `short-1@k (Ours)` and `majority@k`, but worse than `pass@k (Oracle)`. ### Interpretation The chart illustrates the relationship between thinking compute and accuracy for different methods. The `pass@k (Oracle)` method achieves the highest accuracy, suggesting it is the most effective approach. The `majority@k` method shows a consistent improvement with increased compute, while the `short-1@k (Ours)` method reaches a point of diminishing returns. The `short-3@k (Ours)` method provides a balance between performance and compute efficiency. The data suggests that increasing thinking compute generally improves accuracy, but the extent of improvement varies depending on the method used. The "Oracle" method likely represents an upper bound on performance, while the other methods represent practical implementations.

\n ## Line Chart: Accuracy vs. Thinking Compute ### Overview This image presents a line chart comparing the accuracy of different methods (pass@k, majority@k, short-1@k, and short-3@k) as a function of "Thinking Compute" measured in thousands of tokens. The chart illustrates how accuracy improves with increased computational effort for each method. ### Components/Axes * **X-axis:** "Thinking Compute (thinking tokens in thousands)". Scale ranges from approximately 0 to 160, with markers at 0, 50, 100, and 150. * **Y-axis:** "Accuracy". Scale ranges from approximately 0.78 to 0.91, with markers at 0.78, 0.80, 0.82, 0.84, 0.86, 0.88, and 0.90. * **Legend:** Located in the top-right corner, listing the following data series with corresponding colors: * pass@k (Oracle) - Black dotted line with triangle markers. * majority@k - Brown solid line with circle markers. * short-1@k (Ours) - Red solid line with circle markers. * short-3@k (Ours) - Cyan solid line with diamond markers. ### Detailed Analysis * **pass@k (Oracle):** This line starts at approximately 0.79 at a Thinking Compute of 0, rises steeply to approximately 0.87 at a Thinking Compute of 50, continues to rise but at a decreasing rate, reaching approximately 0.90 at a Thinking Compute of 150. The trend is upward and leveling off. * **majority@k:** This line begins at approximately 0.79 at a Thinking Compute of 0, increases steadily to approximately 0.84 at a Thinking Compute of 50, and continues to rise, reaching approximately 0.87 at a Thinking Compute of 150. The trend is consistently upward, but less steep than pass@k. * **short-1@k (Ours):** This line starts at approximately 0.78 at a Thinking Compute of 0, increases rapidly to approximately 0.84 at a Thinking Compute of 50, and then plateaus, reaching approximately 0.85 at a Thinking Compute of 150. The trend is initially steep, then flattens. * **short-3@k (Ours):** This line begins at approximately 0.79 at a Thinking Compute of 0, increases rapidly to approximately 0.86 at a Thinking Compute of 50, and then rises more slowly, reaching approximately 0.88 at a Thinking Compute of 150. The trend is upward, with a decreasing rate of increase after a Thinking Compute of 50. ### Key Observations * "pass@k (Oracle)" consistently achieves the highest accuracy across all levels of "Thinking Compute". * "short-3@k (Ours)" outperforms "short-1@k (Ours)" at all levels of "Thinking Compute". * The rate of accuracy improvement diminishes for all methods as "Thinking Compute" increases, suggesting a point of diminishing returns. * "short-1@k (Ours)" shows the least improvement in accuracy with increasing "Thinking Compute". ### Interpretation The chart demonstrates the relationship between computational effort (measured as "Thinking Compute") and the accuracy of different methods for a task. The "pass@k (Oracle)" method, likely representing an ideal or upper-bound performance, serves as a benchmark. The "short-1@k" and "short-3@k" methods, labeled as "Ours," represent approaches developed by the authors. The data suggests that increasing "Thinking Compute" generally improves accuracy, but the benefit is not linear. The diminishing returns observed for all methods indicate that there's a trade-off between computational cost and performance gains. The superior performance of "short-3@k" over "short-1@k" suggests that incorporating more information or complexity into the model (represented by the '3' in short-3@k) leads to better results, but with increased computational cost. The gap between the "Ours" methods and the "Oracle" method highlights the potential for further improvement in the developed approaches. The chart provides evidence for the effectiveness of the "Ours" methods, particularly "short-3@k," while also indicating areas for future research and optimization.

## Line Chart: Accuracy vs. Thinking Compute for Different Methods ### Overview The image displays a line chart comparing the performance of four different methods or models. The chart plots "Accuracy" on the vertical axis against "Thinking Compute" (measured in thousands of thinking tokens) on the horizontal axis. The primary trend for all series is that accuracy increases with increased thinking compute, but the rate of improvement and the final accuracy achieved vary significantly between methods. ### Components/Axes * **X-Axis (Horizontal):** * **Label:** "Thinking Compute (thinking tokens in thousands)" * **Scale:** Linear scale ranging from approximately 0 to 180 (in thousands of tokens). * **Major Ticks:** Labeled at 50, 100, and 150. * **Y-Axis (Vertical):** * **Label:** "Accuracy" * **Scale:** Linear scale ranging from 0.78 to 0.90. * **Major Ticks:** Labeled at 0.78, 0.80, 0.82, 0.84, 0.86, 0.88, and 0.90. * **Legend:** Located in the bottom-right quadrant of the chart area. It contains four entries: 1. `pass@k (Oracle)`: Represented by a black, dotted line with upward-pointing triangle markers. 2. `majority@k`: Represented by a solid, dark red (maroon) line with circular markers. 3. `short-1@k (Ours)`: Represented by a solid, light blue (cyan) line with square markers. 4. `short-3@k (Ours)`: Represented by a solid, teal (darker cyan) line with diamond markers. * **Grid:** A light gray grid is present, with vertical lines at the major x-axis ticks and horizontal lines at the major y-axis ticks. ### Detailed Analysis **Trend Verification & Data Point Extraction (Approximate Values):** 1. **pass@k (Oracle) [Black Dotted Line, Triangle Markers]:** * **Trend:** This line shows the steepest initial ascent and achieves the highest overall accuracy. It appears to be the upper-bound or ideal performance. * **Data Points:** * At ~10k tokens: Accuracy ≈ 0.78 * At ~25k tokens: Accuracy ≈ 0.835 * At ~50k tokens: Accuracy ≈ 0.855 * At ~75k tokens: Accuracy ≈ 0.875 * At ~100k tokens: Accuracy ≈ 0.885 * At ~125k tokens: Accuracy ≈ 0.895 * At ~150k tokens: Accuracy ≈ 0.902 (highest point on the chart) 2. **short-3@k (Ours) [Teal Line, Diamond Markers]:** * **Trend:** This is the second-best performing method. It follows a similar curve to `pass@k (Oracle)` but consistently below it. The gap between this line and the oracle line narrows slightly as compute increases. * **Data Points:** * At ~10k tokens: Accuracy ≈ 0.78 * At ~25k tokens: Accuracy ≈ 0.818 * At ~50k tokens: Accuracy ≈ 0.848 * At ~75k tokens: Accuracy ≈ 0.870 * At ~100k tokens: Accuracy ≈ 0.878 * At ~125k tokens: Accuracy ≈ 0.885 * At ~150k tokens: Accuracy ≈ 0.890 3. **short-1@k (Ours) [Light Blue Line, Square Markers]:** * **Trend:** This method improves rapidly at very low compute but then plateaus much earlier than the others. After approximately 75k tokens, its accuracy gains become negligible, and it is overtaken by `majority@k`. * **Data Points:** * At ~10k tokens: Accuracy ≈ 0.78 * At ~25k tokens: Accuracy ≈ 0.818 (similar to `short-3@k` at this point) * At ~50k tokens: Accuracy ≈ 0.838 * At ~75k tokens: Accuracy ≈ 0.845 * At ~100k tokens: Accuracy ≈ 0.848 * At ~125k tokens: Accuracy ≈ 0.848 (plateau) * At ~150k tokens: Accuracy ≈ 0.848 (plateau) 4. **majority@k [Dark Red Line, Circle Markers]:** * **Trend:** This method starts with the lowest accuracy at low compute but shows steady, nearly linear improvement. It surpasses the plateaued `short-1@k` method at around 110k tokens and continues to climb. * **Data Points:** * At ~10k tokens: Accuracy ≈ 0.78 * At ~25k tokens: Accuracy ≈ 0.795 * At ~50k tokens: Accuracy ≈ 0.815 * At ~75k tokens: Accuracy ≈ 0.828 * At ~100k tokens: Accuracy ≈ 0.838 * At ~125k tokens: Accuracy ≈ 0.854 * At ~150k tokens: Accuracy ≈ 0.860 * At ~180k tokens (estimated): Accuracy ≈ 0.870 ### Key Observations 1. **Performance Hierarchy:** A clear performance hierarchy is established: `pass@k (Oracle)` > `short-3@k (Ours)` > `majority@k` > `short-1@k (Ours)` at high compute levels (>110k tokens). 2. **Diminishing Returns:** All methods show diminishing returns (the slope of the curve decreases), but the point of severe plateauing varies. `short-1@k` plateaus earliest and most dramatically. 3. **Crossover Point:** A significant crossover occurs between `majority@k` and `short-1@k` at approximately 110k thinking tokens, where `majority@k` becomes the more accurate method despite starting lower. 4. **Oracle Gap:** The gap between the best proposed method (`short-3@k`) and the oracle (`pass@k`) remains relatively constant (≈0.01-0.015 accuracy points) across most of the compute range, suggesting a consistent performance ceiling. 5. **Low-Compute Similarity:** At the lowest compute point (~10k tokens), all four methods start at nearly the same accuracy (≈0.78), indicating that with minimal "thinking," the method choice is less impactful. ### Interpretation This chart demonstrates the trade-off between computational cost ("Thinking Compute") and performance (Accuracy) for different reasoning or generation strategies in an AI system. * **What the data suggests:** The `pass@k (Oracle)` line likely represents an idealized upper bound, perhaps achieved by having perfect knowledge of which of `k` generated samples is correct. The proposed methods, `short-1@k` and `short-3@k`, are practical attempts to approach this oracle performance. `short-3@k` is significantly more effective than `short-1@k`, suggesting that allowing for or considering more diverse or longer "short" reasoning paths (3 vs. 1) yields better results. * **Relationship between elements:** The `majority@k` method, which likely selects the most common answer among `k` samples, serves as a strong baseline. Its steady climb shows that simple aggregation benefits consistently from more compute. The fact that `short-3@k` outperforms it indicates that the "short" methods are doing more than just aggregation; they are likely leveraging the compute to generate higher-quality individual samples. * **Notable Anomalies/Insights:** The early plateau of `short-1@k` is critical. It implies that this method exhausts its ability to improve with more compute relatively quickly. In contrast, `short-3@k` and `majority@k` continue to scale, making them more suitable for scenarios where high compute budgets are available. The chart argues for the efficacy of the `short-3@k` approach, as it provides the best practical performance, closest to the oracle, across a wide range of compute budgets.

## Line Chart: Accuracy vs. Thinking Compute ### Overview The chart compares the accuracy of four different methods (pass@k, majority@k, short-1@k, short-3@k) across varying levels of thinking compute (measured in thousands of tokens). Accuracy is plotted on the y-axis (0.78–0.90), while thinking compute is on the x-axis (50–150k tokens). All methods show upward trends, with pass@k (Oracle) achieving the highest accuracy. ### Components/Axes - **Y-Axis**: Accuracy (0.78–0.90, increments of 0.02). - **X-Axis**: Thinking Compute (50–150k tokens, increments of 50k). - **Legend**: Located in the bottom-right corner, with four entries: - **pass@k (Oracle)**: Black triangles (▲). - **majority@k**: Red circles (●). - **short-1@k (Ours)**: Blue squares (■). - **short-3@k (Ours)**: Green diamonds (◇). ### Detailed Analysis 1. **pass@k (Oracle)**: - Starts at 0.78 (50k tokens). - Rises sharply to 0.90 by 100k tokens. - Plateaus at 0.90 beyond 100k tokens. - *Key data points*: 0.78 (50k), 0.84 (75k), 0.90 (100k+). 2. **majority@k**: - Starts at 0.78 (50k tokens). - Gradually increases to 0.86 by 150k tokens. - *Key data points*: 0.78 (50k), 0.82 (100k), 0.86 (150k). 3. **short-1@k (Ours)**: - Starts at 0.78 (50k tokens). - Reaches 0.84 by 100k tokens. - Plateaus at 0.84 beyond 100k tokens. - *Key data points*: 0.78 (50k), 0.84 (100k+). 4. **short-3@k (Ours)**: - Starts at 0.78 (50k tokens). - Rises to 0.88 by 150k tokens. - *Key data points*: 0.78 (50k), 0.86 (125k), 0.88 (150k). ### Key Observations - **pass@k (Oracle)** dominates in accuracy, achieving 0.90 at 100k tokens. - **short-3@k (Ours)** outperforms other methods, reaching 0.88 at 150k tokens. - **majority@k** shows the slowest improvement, ending at 0.86. - All methods plateau after 100k tokens, suggesting diminishing returns. ### Interpretation The data demonstrates that increasing thinking compute improves accuracy across all methods, with **pass@k (Oracle)** being the most effective. The proposed methods (**short-1@k** and **short-3@k**) achieve competitive results, with **short-3@k** closing the gap to the Oracle. The plateauing trends indicate that beyond 100k tokens, additional compute yields minimal accuracy gains. This suggests a trade-off between computational cost and marginal performance improvements. The **majority@k** method’s lower performance highlights its inefficiency compared to the other approaches.