## Line Chart: Accuracy vs. Thinking Compute ### Overview This chart illustrates the relationship between "Thinking Compute" (measured in thousands of thinking tokens) and model "Accuracy." It compares four different strategies for generating or selecting reasoning paths: an Oracle baseline, a majority-voting method, and two variations of a proposed method labeled "short-k." The chart demonstrates how accuracy scales as the computational budget increases for each method. ### Components/Axes * **Y-Axis:** Labeled "Accuracy." The scale ranges from 0.78 to 0.90, with grid lines at 0.02 intervals. * **X-Axis:** Labeled "Thinking Compute (thinking tokens in thousands)." The scale ranges from approximately 20 to 200, with major grid lines at 50, 100, and 150. * **Legend:** Located in the bottom-right quadrant of the chart. * **pass@k (Oracle):** Black dotted line with triangle markers. * **majority@k:** Dark red solid line with circle markers. * **short-1@k (Ours):** Light blue solid line with square markers. * **short-3@k (Ours):** Cyan/Teal solid line with diamond markers. ### Detailed Analysis All four data series originate from a common starting point at approximately (20k tokens, 0.78 accuracy). 1. **pass@k (Oracle) [Black Dotted Line]:** * **Trend:** This line exhibits the steepest upward slope, indicating the highest efficiency in gaining accuracy per token. * **Data Points:** It rises consistently, reaching approximately 0.835 at 30k tokens, 0.855 at 50k, 0.878 at 80k, and terminates at approximately 0.903 at 140k tokens. 2. **short-3@k (Ours) [Cyan/Teal Solid Line]:** * **Trend:** This line shows a strong, steady upward slope, second only to the Oracle. * **Data Points:** It tracks closely with the Oracle initially. It reaches approximately 0.817 at 30k tokens, 0.848 at 55k, 0.877 at 100k, and terminates at approximately 0.89 at 150k tokens. 3. **short-1@k (Ours) [Light Blue Solid Line]:** * **Trend:** This line rises sharply at the beginning but exhibits significant diminishing returns, plateauing after approximately 100k tokens. * **Data Points:** It reaches approximately 0.817 at 30k tokens, 0.839 at 75k, and plateaus around 0.848 between 110k and 140k tokens. 4. **majority@k [Dark Red Solid Line]:** * **Trend:** This line has the shallowest slope but maintains a consistent upward trajectory without the plateauing effect seen in the *short-1@k* method. * **Data Points:** It reaches approximately 0.815 at 55k tokens, 0.838 at 100k, 0.86 at 150k, and terminates at approximately 0.87 at 190k tokens. ### Key Observations * **Crossover Point:** The *majority@k* method (dark red) crosses the *short-1@k* method (light blue) at approximately 110k tokens. Before this point, *short-1@k* is more accurate; after this point, *majority@k* is more accurate. * **Diminishing Returns:** The *short-1@k* method shows a clear "elbow" or plateau, suggesting that adding more thinking tokens beyond 100k provides negligible accuracy gains for this specific strategy. * **Performance Gap:** The Oracle (*pass@k*) maintains a significant performance lead over all other methods throughout the entire range of compute. ### Interpretation This chart likely originates from research into Large Language Model (LLM) reasoning, specifically regarding "Chain of Thought" or "Test-time Compute" optimization. * **The "Oracle" Baseline:** The *pass@k (Oracle)* represents the theoretical upper bound of performance—essentially, if the model generates *k* solutions, the Oracle assumes the model is "correct" if the right answer appears in *any* of those *k* solutions. * **"Ours" Methods:** The *short-k* methods likely represent a filtering or selection mechanism designed to identify the "best" reasoning path more efficiently than standard majority voting. * **Strategic Trade-offs:** The data suggests that while *short-1@k* is highly efficient at low compute budgets (up to ~100k tokens), it fails to scale effectively. Conversely, *majority@k* is less efficient at low compute but scales better as the budget increases. *short-3@k* appears to be the most robust "Ours" method, maintaining high accuracy without the plateauing issues of *short-1@k*.