## Line Chart: Accuracy vs. #Sampled Reasoning Paths Across Tasks ### Overview The image displays three line charts comparing the accuracy of three reasoning strategies ("Self Consistency (Multi-path)", "Sample & Rank (Multi-path)", and "Greedy Decode (Single-path)") across three tasks: GSM8K, MultiArith, and ARC (Challenge). Accuracy (%) is plotted on the y-axis, while the number of sampled reasoning paths (0–40) is on the x-axis. Each sub-chart uses distinct colors for data series, with legends positioned to the right of each sub-chart. --- ### Components/Axes - **Y-Axis**: "Accuracy (%)" with ranges: - GSM8K: 12–24% - MultiArith: 50–80% - ARC (Challenge): 30–55% - **X-Axis**: "#Sampled Reasoning Paths" (0–40) for all sub-charts. - **Legends**: - Blue stars: "Self Consistency (Multi-path)" - Green squares: "Sample & Rank (Multi-path)" - Orange circles: "Greedy Decode (Single-path)" - **Sub-chart Titles**: - Top-left: "GSM8K" - Center: "MultiArith" - Top-right: "ARC (Challenge)" --- ### Detailed Analysis #### GSM8K Sub-Chart - **Self Consistency (Multi-path)**: Blue stars show a steep upward trend, starting at ~12% (0 paths) and reaching ~24% (40 paths). - **Sample & Rank (Multi-path)**: Green squares increase gradually from ~14% (0 paths) to ~18% (40 paths). - **Greedy Decode (Single-path)**: Orange circles remain flat at ~14% across all paths. #### MultiArith Sub-Chart - **Self Consistency (Multi-path)**: Blue stars rise sharply from ~50% (0 paths) to ~80% (40 paths). - **Sample & Rank (Multi-path)**: Green squares increase from ~55% (0 paths) to ~70% (40 paths). - **Greedy Decode (Single-path)**: Orange circles stay flat at ~60% across all paths. #### ARC (Challenge) Sub-Chart - **Self Consistency (Multi-path)**: Blue stars increase from ~30% (0 paths) to ~55% (40 paths). - **Sample & Rank (Multi-path)**: Green squares rise from ~35% (0 paths) to ~45% (40 paths). - **Greedy Decode (Single-path)**: Orange circles remain flat at ~40% across all paths. --- ### Key Observations 1. **Self Consistency (Multi-path)** consistently outperforms other strategies in all tasks, with the steepest improvement in MultiArith. 2. **Sample & Rank (Multi-path)** shows moderate gains but plateaus at higher path counts. 3. **Greedy Decode (Single-path)** demonstrates no improvement with additional paths in any task. 4. **ARC (Challenge)** has the lowest baseline accuracy (~30–40%) compared to GSM8K (~12–14%) and MultiArith (~50–60%). --- ### Interpretation The data suggests that **multi-path reasoning strategies** (Self Consistency and Sample & Rank) significantly improve accuracy compared to single-path methods (Greedy Decode). The steepest gains are observed in MultiArith, where multi-path methods achieve near-human-level performance (~80%). The ARC (Challenge) task, with lower overall accuracy, may reflect higher complexity or ambiguity in its dataset. The flat performance of Greedy Decode highlights the limitations of single-path reasoning in capturing nuanced problem-solving. These trends align with prior research emphasizing the value of iterative, multi-step reasoning in AI systems.