## Line Charts: Accuracy vs. Sampled Reasoning Paths for Different Datasets ### Overview The image presents three line charts comparing the accuracy of different reasoning methods across three datasets: GSM8K, MultiArith, and ARC (Challenge). The x-axis represents the number of sampled reasoning paths, and the y-axis represents the accuracy in percentage. Three methods are compared: Self Consistency (Multi-path), Sample & Rank (Multi-path), and Greedy Decode (Single-path). ### Components/Axes * **X-axis (all charts):** "#Sampled Reasoning Paths" with markers at 0, 5, 10, 15, 20, 25, 30, 35, and 40. * **Y-axis (GSM8K):** "Accuracy (%)" with markers at 12, 14, 16, 18, 20, 22, and 24. * **Y-axis (MultiArith):** "Accuracy (%)" with markers at 50, 55, 60, 65, 70, 75, and 80. * **Y-axis (ARC (Challenge):** "Accuracy (%)" with markers at 30, 35, 40, 45, 50, and 55. * **Legend (bottom-right):** * Blue line with star markers: "Self Consistency (Multi-path)" * Green line with square markers: "Sample & Rank (Multi-path)" * Orange line with circle markers: "Greedy Decode (Single-path)" ### Detailed Analysis **1. GSM8K Chart (left)** * **Self Consistency (Multi-path) - Blue:** The line starts at approximately 12% accuracy with 0 sampled paths and increases to approximately 23% accuracy with 40 sampled paths. The trend is upward. * (0, 12%), (5, 16%), (10, 19%), (20, 21%), (40, 23%) * **Sample & Rank (Multi-path) - Green:** The line starts at approximately 13% accuracy with 0 sampled paths, increases to approximately 17% accuracy with 20 sampled paths, and remains relatively flat until 40 sampled paths. * (0, 13%), (5, 16%), (10, 17%), (20, 17%), (40, 17%) * **Greedy Decode (Single-path) - Orange:** The line remains relatively flat at approximately 14-15% accuracy across all sampled reasoning paths. * (0, 14%), (5, 15%), (10, 15%), (20, 15%), (40, 15%) **2. MultiArith Chart (center)** * **Self Consistency (Multi-path) - Blue:** The line starts at approximately 48% accuracy with 0 sampled paths and increases to approximately 82% accuracy with 40 sampled paths. The trend is upward. * (0, 48%), (5, 73%), (10, 77%), (20, 80%), (40, 82%) * **Sample & Rank (Multi-path) - Green:** The line starts at approximately 50% accuracy with 0 sampled paths, increases to approximately 68% accuracy with 20 sampled paths, and remains relatively flat until 40 sampled paths. * (0, 50%), (5, 62%), (10, 65%), (20, 68%), (40, 68%) * **Greedy Decode (Single-path) - Orange:** The line remains relatively flat at approximately 60% accuracy across all sampled reasoning paths. * (0, 60%), (5, 60%), (10, 60%), (20, 60%), (40, 60%) **3. ARC (Challenge) Chart (right)** * **Self Consistency (Multi-path) - Blue:** The line starts at approximately 36% accuracy with 0 sampled paths and increases to approximately 54% accuracy with 40 sampled paths. The trend is upward. * (0, 36%), (5, 48%), (10, 51%), (20, 52%), (40, 54%) * **Sample & Rank (Multi-path) - Green:** The line starts at approximately 34% accuracy with 0 sampled paths, increases to approximately 42% accuracy with 20 sampled paths, and remains relatively flat until 40 sampled paths. * (0, 34%), (5, 39%), (10, 41%), (20, 42%), (40, 42%) * **Greedy Decode (Single-path) - Orange:** The line remains relatively flat at approximately 43% accuracy across all sampled reasoning paths. * (0, 43%), (5, 43%), (10, 43%), (20, 43%), (40, 43%) ### Key Observations * **Self Consistency (Multi-path)** consistently shows the highest accuracy and the most significant improvement with an increasing number of sampled reasoning paths across all three datasets. * **Sample & Rank (Multi-path)** shows some improvement with an increasing number of sampled reasoning paths, but the improvement plateaus after a certain point. * **Greedy Decode (Single-path)** consistently shows the lowest accuracy and minimal improvement with an increasing number of sampled reasoning paths. ### Interpretation The data suggests that using multiple reasoning paths (as implemented in Self Consistency and Sample & Rank) generally leads to higher accuracy compared to using a single reasoning path (Greedy Decode). The Self Consistency method, which likely aggregates information from multiple paths more effectively, consistently outperforms the other methods. The diminishing returns observed with Sample & Rank suggest that simply sampling and ranking paths has limitations, and a more sophisticated aggregation method like Self Consistency is needed to fully leverage the benefits of multiple reasoning paths. The Greedy Decode method's flat performance indicates that its single-path approach is insufficient for these complex reasoning tasks. The performance differences across datasets highlight the varying difficulty levels and the suitability of different methods for specific problem types.

## Chart: Accuracy vs. Number of Sampled Reasoning Paths ### Overview This image presents three line charts, each displaying the relationship between "Accuracy (%)" and "#Sampled Reasoning Paths" for different datasets: GSM8K, MultiArith, and ARC (Challenge). Each chart includes three data series representing different decoding methods: "Self Consistency (Multi-path)", "Sample & Rank (Multi-path)", and "Greedy Decode (Single-path)". ### Components/Axes * **X-axis:** "#Sampled Reasoning Paths" - ranging from 0 to 40, with markers at 0, 5, 10, 15, 20, 25, 30, 35, and 40. * **Y-axis:** "Accuracy (%)" - ranging from approximately 12% to 80%, depending on the chart. * **Datasets (Charts):** GSM8K, MultiArith, ARC (Challenge) - each chart represents one dataset. * **Legend:** Located in the top-left corner of each chart, identifying the data series by color and name. * Blue: "Self Consistency (Multi-path)" * Green: "Sample & Rank (Multi-path)" * Orange: "Greedy Decode (Single-path)" ### Detailed Analysis or Content Details **GSM8K Chart:** * **Self Consistency (Blue):** The line slopes sharply upward from approximately 14% at 0 paths to approximately 23% at 35 paths. Data points (approximate): (0, 14%), (5, 17%), (10, 19%), (15, 21%), (20, 22%), (25, 22.5%), (30, 23%), (35, 23%). * **Sample & Rank (Green):** The line shows a moderate upward trend, leveling off after 15 paths. Data points (approximate): (0, 12%), (5, 15%), (10, 16%), (15, 17%), (20, 17.5%), (25, 17.5%), (30, 17.5%), (35, 17.5%). * **Greedy Decode (Orange):** The line is relatively flat, fluctuating around 14-15%. Data points (approximate): (0, 14%), (5, 14%), (10, 14.5%), (15, 14.5%), (20, 14.5%), (25, 14.5%), (30, 14.5%), (35, 14.5%). **MultiArith Chart:** * **Self Consistency (Blue):** The line exhibits a strong upward trend, increasing rapidly from approximately 50% to approximately 82% as the number of paths increases. Data points (approximate): (0, 50%), (5, 62%), (10, 70%), (15, 75%), (20, 78%), (25, 80%), (30, 81%), (35, 82%). * **Sample & Rank (Green):** The line shows a moderate upward trend, but remains significantly below the "Self Consistency" line. Data points (approximate): (0, 50%), (5, 58%), (10, 63%), (15, 65%), (20, 66%), (25, 67%), (30, 67%), (35, 67%). * **Greedy Decode (Orange):** The line is relatively flat, fluctuating around 54-56%. Data points (approximate): (0, 54%), (5, 55%), (10, 55%), (15, 55.5%), (20, 55.5%), (25, 55.5%), (30, 55.5%), (35, 55.5%). **ARC (Challenge) Chart:** * **Self Consistency (Blue):** The line shows a strong upward trend, increasing from approximately 32% to approximately 53% as the number of paths increases. Data points (approximate): (0, 32%), (5, 40%), (10, 45%), (15, 47%), (20, 49%), (25, 50%), (30, 52%), (35, 53%). * **Sample & Rank (Green):** The line shows a moderate upward trend, leveling off after 20 paths. Data points (approximate): (0, 30%), (5, 36%), (10, 40%), (15, 42%), (20, 42.5%), (25, 42.5%), (30, 42.5%), (35, 42.5%). * **Greedy Decode (Orange):** The line is relatively flat, fluctuating around 41-42%. Data points (approximate): (0, 41%), (5, 41%), (10, 41.5%), (15, 41.5%), (20, 41.5%), (25, 41.5%), (30, 41.5%), (35, 41.5%). ### Key Observations * "Self Consistency" consistently outperforms both "Sample & Rank" and "Greedy Decode" across all three datasets. * The benefit of increasing the number of sampled reasoning paths diminishes after a certain point, particularly for "Sample & Rank" and "Greedy Decode". * "Greedy Decode" shows minimal improvement with increased reasoning paths, suggesting it is not effectively utilizing additional information. * The MultiArith dataset shows the most significant improvement in accuracy with increased reasoning paths, especially for "Self Consistency". ### Interpretation The charts demonstrate the effectiveness of multi-path reasoning techniques, specifically "Self Consistency", in improving accuracy on mathematical and reasoning tasks. The superior performance of "Self Consistency" suggests that sampling multiple reasoning paths and aggregating their results leads to more robust and accurate solutions. The diminishing returns observed with increasing paths indicate that there is a point where the computational cost outweighs the marginal gains in accuracy. The relatively flat performance of "Greedy Decode" highlights the limitations of single-path reasoning, which may be susceptible to errors or suboptimal solutions. The differences in performance across datasets suggest that the effectiveness of these techniques may vary depending on the complexity and characteristics of the task. The data suggests that for complex reasoning tasks, exploring multiple reasoning paths is crucial for achieving high accuracy, and "Self Consistency" is a particularly promising approach.

## Line Charts: Performance of Reasoning Methods Across Datasets ### Overview The image displays three horizontally arranged line charts comparing the accuracy of three different reasoning methods as the number of sampled reasoning paths increases. The charts are titled "GSM8K", "MultiArith", and "ARC (Challenge)". Each chart plots "Accuracy (%)" on the y-axis against "#Sampled Reasoning Paths" on the x-axis. ### Components/Axes * **Titles (Top-Center of each chart):** GSM8K, MultiArith, ARC (Challenge). * **Y-Axis Label (Left side, rotated):** "Accuracy (%)". * **X-Axis Label (Bottom center):** "#Sampled Reasoning Paths". * **X-Axis Scale (All charts):** Linear scale with major ticks at 0, 5, 10, 20, 30, and 40. * **Y-Axis Scales:** * **GSM8K:** Ranges from 12 to 24, with major ticks every 2 units (12, 14, 16, 18, 20, 22, 24). * **MultiArith:** Ranges from 50 to 80, with major ticks every 5 units (50, 55, 60, 65, 70, 75, 80). * **ARC (Challenge):** Ranges from 30 to 55, with major ticks every 5 units (30, 35, 40, 45, 50, 55). * **Legend (Bottom-right of each chart):** * **Blue line with star markers:** "Self Consistency (Multi-path)" * **Green line with square markers:** "Sample & Rank (Multi-path)" * **Orange line with circle markers:** "Greedy Decode (Single-path)" ### Detailed Analysis **Chart 1: GSM8K** * **Trend Verification:** * **Self Consistency (Blue):** Shows a strong, consistent upward slope from left to right. * **Sample & Rank (Green):** Shows a moderate upward slope, flattening after 10 paths. * **Greedy Decode (Orange):** Appears as a perfectly horizontal line. * **Data Points (Approximate):** * **Self Consistency:** (0, ~12%), (5, ~16%), (10, ~19%), (20, ~21%), (40, ~24%). * **Sample & Rank:** (0, ~14%), (5, ~16%), (10, ~17%), (20, ~17.5%), (40, ~18%). * **Greedy Decode:** Constant at ~15% for all x-values. **Chart 2: MultiArith** * **Trend Verification:** * **Self Consistency (Blue):** Steep upward slope initially, then continues rising steadily. * **Sample & Rank (Green):** Moderate upward slope, plateauing after 20 paths. * **Greedy Decode (Orange):** Horizontal line. * **Data Points (Approximate):** * **Self Consistency:** (0, ~50%), (5, ~68%), (10, ~76%), (20, ~80%), (40, ~82%). * **Sample & Rank:** (0, ~50%), (5, ~60%), (10, ~64%), (20, ~68%), (40, ~67%). * **Greedy Decode:** Constant at ~60% for all x-values. **Chart 3: ARC (Challenge)** * **Trend Verification:** * **Self Consistency (Blue):** Steady upward slope. * **Sample & Rank (Green):** Moderate upward slope. * **Greedy Decode (Orange):** Nearly horizontal, with a very slight upward drift. * **Data Points (Approximate):** * **Self Consistency:** (0, ~35%), (5, ~47%), (10, ~50%), (20, ~53%), (40, ~55%). * **Sample & Rank:** (0, ~35%), (5, ~40%), (10, ~42%), (20, ~44%), (40, ~45%). * **Greedy Decode:** Starts at ~42% (x=0), ends at ~43% (x=40). ### Key Observations 1. **Consistent Hierarchy:** Across all three datasets, the "Self Consistency (Multi-path)" method (blue) achieves the highest final accuracy, followed by "Sample & Rank (Multi-path)" (green), with "Greedy Decode (Single-path)" (orange) performing the worst or being static. 2. **Impact of Multi-Path Sampling:** Both multi-path methods (blue and green lines) show clear improvement as the number of sampled reasoning paths increases from 0 to 40. The improvement is most dramatic for "Self Consistency." 3. **Plateau Effect:** The "Sample & Rank" method shows diminishing returns, with its accuracy curve flattening significantly after 10-20 sampled paths in all charts. 4. **Baseline Performance:** The "Greedy Decode" line serves as a flat baseline, as it is a single-path method and its performance does not change with the x-axis variable (#Sampled Reasoning Paths). Its starting point (x=0) represents the single-path accuracy. 5. **Dataset Difficulty:** The absolute accuracy values suggest the relative difficulty of the tasks. GSM8K has the lowest overall accuracy range (12-24%), ARC (Challenge) is in the middle (35-55%), and MultiArith has the highest (50-82%). ### Interpretation The data demonstrates the significant benefit of **multi-path reasoning and aggregation strategies** over a single greedy decode for improving accuracy on mathematical and reasoning benchmarks. The "Self Consistency" method, which likely involves sampling multiple reasoning paths and selecting the most frequent answer, shows the strongest scaling law with increased sampling compute. The "Sample & Rank" method also benefits from sampling but appears less effective at leveraging additional paths beyond a certain point (around 20), suggesting its ranking mechanism may have a lower ceiling or be more sensitive to the quality of the initial sample pool. The flat "Greedy Decode" line is a critical control, confirming that the observed improvements are due to the multi-path sampling and aggregation process, not an artifact of the evaluation. The charts collectively argue that investing computational resources into generating and evaluating multiple reasoning traces is a highly effective strategy for boosting the performance of language models on complex tasks, with "Self Consistency" being the most efficient method among those compared.

## Line Chart: Accuracy vs. #Sampled Reasoning Paths Across Tasks ### Overview The image displays three line charts comparing the accuracy of three reasoning strategies ("Self Consistency (Multi-path)", "Sample & Rank (Multi-path)", and "Greedy Decode (Single-path)") across three tasks: GSM8K, MultiArith, and ARC (Challenge). Accuracy (%) is plotted on the y-axis, while the number of sampled reasoning paths (0–40) is on the x-axis. Each sub-chart uses distinct colors for data series, with legends positioned to the right of each sub-chart. --- ### Components/Axes - **Y-Axis**: "Accuracy (%)" with ranges: - GSM8K: 12–24% - MultiArith: 50–80% - ARC (Challenge): 30–55% - **X-Axis**: "#Sampled Reasoning Paths" (0–40) for all sub-charts. - **Legends**: - Blue stars: "Self Consistency (Multi-path)" - Green squares: "Sample & Rank (Multi-path)" - Orange circles: "Greedy Decode (Single-path)" - **Sub-chart Titles**: - Top-left: "GSM8K" - Center: "MultiArith" - Top-right: "ARC (Challenge)" --- ### Detailed Analysis #### GSM8K Sub-Chart - **Self Consistency (Multi-path)**: Blue stars show a steep upward trend, starting at ~12% (0 paths) and reaching ~24% (40 paths). - **Sample & Rank (Multi-path)**: Green squares increase gradually from ~14% (0 paths) to ~18% (40 paths). - **Greedy Decode (Single-path)**: Orange circles remain flat at ~14% across all paths. #### MultiArith Sub-Chart - **Self Consistency (Multi-path)**: Blue stars rise sharply from ~50% (0 paths) to ~80% (40 paths). - **Sample & Rank (Multi-path)**: Green squares increase from ~55% (0 paths) to ~70% (40 paths). - **Greedy Decode (Single-path)**: Orange circles stay flat at ~60% across all paths. #### ARC (Challenge) Sub-Chart - **Self Consistency (Multi-path)**: Blue stars increase from ~30% (0 paths) to ~55% (40 paths). - **Sample & Rank (Multi-path)**: Green squares rise from ~35% (0 paths) to ~45% (40 paths). - **Greedy Decode (Single-path)**: Orange circles remain flat at ~40% across all paths. --- ### Key Observations 1. **Self Consistency (Multi-path)** consistently outperforms other strategies in all tasks, with the steepest improvement in MultiArith. 2. **Sample & Rank (Multi-path)** shows moderate gains but plateaus at higher path counts. 3. **Greedy Decode (Single-path)** demonstrates no improvement with additional paths in any task. 4. **ARC (Challenge)** has the lowest baseline accuracy (~30–40%) compared to GSM8K (~12–14%) and MultiArith (~50–60%). --- ### Interpretation The data suggests that **multi-path reasoning strategies** (Self Consistency and Sample & Rank) significantly improve accuracy compared to single-path methods (Greedy Decode). The steepest gains are observed in MultiArith, where multi-path methods achieve near-human-level performance (~80%). The ARC (Challenge) task, with lower overall accuracy, may reflect higher complexity or ambiguity in its dataset. The flat performance of Greedy Decode highlights the limitations of single-path reasoning in capturing nuanced problem-solving. These trends align with prior research emphasizing the value of iterative, multi-step reasoning in AI systems.