## Line Charts: Accuracy vs. #Sampled Reasoning Paths
### Overview
The image presents four side-by-side line charts comparing the performance of two decoding strategies—"Greedy Decode (Single-path)" and "Self Consistency (Multi-path)"—across four distinct datasets: **MultiArith**, **SVAMP**, **Commonsense QA**, and **ARC (Challenge)**. The charts illustrate how increasing the number of sampled reasoning paths affects the accuracy of the model.
### Components/Axes
* **X-Axis (All Charts):** Labeled "#Sampled Reasoning Paths". The scale is consistent across all four charts, ranging from 0 to 40, with major ticks at 0, 5, 10, 15, 20, 25, 30, 35, and 40.
* **Y-Axis (All Charts):** Labeled "Accuracy (%)". The scale varies per chart to accommodate the specific performance range of each dataset.
* **Legend (Located in the bottom-right of the ARC (Challenge) chart):**
* **Orange line with circular markers:** "Greedy Decode (Single-path)"
* **Blue line with vertical error bars:** "Self Consistency (Multi-path)"
### Detailed Analysis
#### 1. MultiArith
* **Greedy Decode:** Remains constant at approximately **52%**.
* **Self Consistency:** Shows a steep upward trend.
* 1 path: ~48%
* 5 paths: ~63%
* 10 paths: ~70%
* 20 paths: ~73%
* 40 paths: ~76%
#### 2. SVAMP
* **Greedy Decode:** Remains constant at approximately **39%**.
* **Self Consistency:** Shows a steep upward trend, flattening after 20 paths.
* 1 path: ~34%
* 5 paths: ~44%
* 10 paths: ~50%
* 20 paths: ~52%
* 40 paths: ~53%
#### 3. Commonsense QA
* **Greedy Decode:** Remains constant at approximately **58%**.
* **Self Consistency:** Shows a rapid initial increase, plateauing quickly.
* 1 path: ~56%
* 5 paths: ~61%
* 10 paths: ~62%
* 20 paths: ~62.5%
* 40 paths: ~62.8%
#### 4. ARC (Challenge)
* **Greedy Decode:** Remains constant at approximately **55%**.
* **Self Consistency:** Shows a steep upward trend, plateauing after 20 paths.
* 1 path: ~51%
* 5 paths: ~57%
* 10 paths: ~59%
* 20 paths: ~60%
* 40 paths: ~60%
### Key Observations
* **Performance Gap:** In all four datasets, "Self Consistency" significantly outperforms "Greedy Decode" once the number of sampled paths exceeds 5.
* **Diminishing Returns:** The "Self Consistency" curves exhibit a logarithmic growth pattern. The most significant accuracy gains occur between 1 and 10 paths. Beyond 20 paths, the accuracy gains become marginal (plateauing).
* **Variance/Uncertainty:** The blue error bars on the "Self Consistency" lines are largest at the 1-path mark, indicating higher variance or uncertainty in the model's output when sampling is low. As the number of paths increases, the error bars shrink, indicating increased stability and confidence in the accuracy metric.
* **Greedy Decode Stability:** The "Greedy Decode" strategy is represented as a perfectly flat line, confirming that it does not utilize multiple paths and thus its accuracy remains static regardless of the "#Sampled Reasoning Paths" axis.
### Interpretation
The data demonstrates that "Self Consistency" is a superior decoding strategy compared to "Greedy Decode" for these specific reasoning tasks. By sampling multiple reasoning paths, the model can effectively "vote" or aggregate results, which filters out noise and incorrect reasoning chains, leading to higher overall accuracy.
The consistent plateauing effect observed across all four charts suggests that there is an optimal computational budget for sampling. Increasing the number of paths beyond 20 yields very little additional accuracy, suggesting that the marginal utility of extra computation decreases rapidly. This allows practitioners to balance performance and computational cost effectively by selecting a sampling number in the 10–20 range.
