## Chart: Accuracy vs. Thinking Compute
### Overview
The image is a line chart comparing the accuracy of different models (pass@k (Oracle), majority@k, short-1@k (Ours), and short-3@k (Ours)) against the thinking compute (thinking tokens in thousands). The chart shows how accuracy increases with thinking compute for each model.
### Components/Axes
* **X-axis:** Thinking Compute (thinking tokens in thousands). Scale ranges from 20 to 140, with tick marks at 20, 40, 60, 80, 100, 120, and 140.
* **Y-axis:** Accuracy. Scale ranges from 0.40 to 0.65, with tick marks at 0.40, 0.45, 0.50, 0.55, 0.60, and 0.65.
* **Legend:** Located in the bottom-right corner of the chart.
* `pass@k (Oracle)`: Black dotted line with triangle markers.
* `majority@k`: Brown solid line with circle markers.
* `short-1@k (Ours)`: Blue solid line with square markers.
* `short-3@k (Ours)`: Teal solid line with diamond markers.
### Detailed Analysis
* **pass@k (Oracle):** The black dotted line with triangle markers shows a steep upward trend, indicating a rapid increase in accuracy with increasing thinking compute.
* At 20k tokens, accuracy is approximately 0.40.
* At 40k tokens, accuracy is approximately 0.50.
* At 60k tokens, accuracy is approximately 0.58.
* At 80k tokens, accuracy is approximately 0.63.
* At 85k tokens, accuracy is approximately 0.65.
* **majority@k:** The brown solid line with circle markers shows a gradual upward trend, indicating a slower increase in accuracy with increasing thinking compute.
* At 20k tokens, accuracy is approximately 0.40.
* At 40k tokens, accuracy is approximately 0.43.
* At 60k tokens, accuracy is approximately 0.47.
* At 80k tokens, accuracy is approximately 0.50.
* At 100k tokens, accuracy is approximately 0.51.
* At 120k tokens, accuracy is approximately 0.515.
* At 140k tokens, accuracy is approximately 0.52.
* **short-1@k (Ours):** The blue solid line with square markers shows an upward trend, with accuracy increasing with thinking compute.
* At 20k tokens, accuracy is approximately 0.40.
* At 40k tokens, accuracy is approximately 0.49.
* At 60k tokens, accuracy is approximately 0.52.
* At 80k tokens, accuracy is approximately 0.54.
* **short-3@k (Ours):** The teal solid line with diamond markers shows an upward trend, with accuracy increasing with thinking compute.
* At 20k tokens, accuracy is approximately 0.40.
* At 40k tokens, accuracy is approximately 0.48.
* At 60k tokens, accuracy is approximately 0.51.
* At 80k tokens, accuracy is approximately 0.54.
### Key Observations
* The `pass@k (Oracle)` model achieves the highest accuracy for a given thinking compute value.
* The `majority@k` model has the lowest accuracy compared to the other models.
* The `short-1@k (Ours)` and `short-3@k (Ours)` models perform similarly, with `short-1@k` slightly outperforming `short-3@k`.
* All models show an increase in accuracy with increasing thinking compute, but the rate of increase varies.
### Interpretation
The chart demonstrates the relationship between thinking compute and accuracy for different models. The `pass@k (Oracle)` model serves as an upper bound or ideal performance, while the `majority@k` model represents a baseline. The `short-1@k (Ours)` and `short-3@k (Ours)` models show improved performance compared to the baseline, suggesting that the "Ours" models are effective in leveraging thinking compute to improve accuracy. The diminishing returns observed in the `majority@k` model suggest that simply increasing compute may not always lead to significant gains in accuracy, and more sophisticated models like `pass@k` and the "Ours" models are needed to effectively utilize higher compute budgets.
