## Line Chart: Accuracy vs. Thinking Compute
### Overview
The chart compares the accuracy of four different methods (pass@k, majority@k, short-1@k, short-3@k) across varying levels of thinking compute (measured in thousands of tokens). Accuracy is plotted on the y-axis (0.78–0.90), while thinking compute is on the x-axis (50–150k tokens). All methods show upward trends, with pass@k (Oracle) achieving the highest accuracy.
### Components/Axes
- **Y-Axis**: Accuracy (0.78–0.90, increments of 0.02).
- **X-Axis**: Thinking Compute (50–150k tokens, increments of 50k).
- **Legend**: Located in the bottom-right corner, with four entries:
- **pass@k (Oracle)**: Black triangles (▲).
- **majority@k**: Red circles (●).
- **short-1@k (Ours)**: Blue squares (■).
- **short-3@k (Ours)**: Green diamonds (◇).
### Detailed Analysis
1. **pass@k (Oracle)**:
- Starts at 0.78 (50k tokens).
- Rises sharply to 0.90 by 100k tokens.
- Plateaus at 0.90 beyond 100k tokens.
- *Key data points*: 0.78 (50k), 0.84 (75k), 0.90 (100k+).
2. **majority@k**:
- Starts at 0.78 (50k tokens).
- Gradually increases to 0.86 by 150k tokens.
- *Key data points*: 0.78 (50k), 0.82 (100k), 0.86 (150k).
3. **short-1@k (Ours)**:
- Starts at 0.78 (50k tokens).
- Reaches 0.84 by 100k tokens.
- Plateaus at 0.84 beyond 100k tokens.
- *Key data points*: 0.78 (50k), 0.84 (100k+).
4. **short-3@k (Ours)**:
- Starts at 0.78 (50k tokens).
- Rises to 0.88 by 150k tokens.
- *Key data points*: 0.78 (50k), 0.86 (125k), 0.88 (150k).
### Key Observations
- **pass@k (Oracle)** dominates in accuracy, achieving 0.90 at 100k tokens.
- **short-3@k (Ours)** outperforms other methods, reaching 0.88 at 150k tokens.
- **majority@k** shows the slowest improvement, ending at 0.86.
- All methods plateau after 100k tokens, suggesting diminishing returns.
### Interpretation
The data demonstrates that increasing thinking compute improves accuracy across all methods, with **pass@k (Oracle)** being the most effective. The proposed methods (**short-1@k** and **short-3@k**) achieve competitive results, with **short-3@k** closing the gap to the Oracle. The plateauing trends indicate that beyond 100k tokens, additional compute yields minimal accuracy gains. This suggests a trade-off between computational cost and marginal performance improvements. The **majority@k** method’s lower performance highlights its inefficiency compared to the other approaches.
