## Bar Chart: Solve Rate Comparison ### Overview The image presents a bar chart comparing the solve rates (%) of different prompting methods on two datasets: GSM8K and MAWPS. The chart includes a legend that identifies each prompting method with a unique color or pattern. The solve rates are displayed on the y-axis, ranging from 0% to 20% for GSM8K and 0% to 60% for MAWPS. ### Components/Axes * **Y-axis:** "Solve rate (%)", ranging from 0 to 20 in increments of 5 for GSM8K, and 0 to 60 in increments of 20 for MAWPS. * **X-axis:** Categorical, with two categories: "GSM8K" and "MAWPS". * **Legend (Top-Left):** * Yellow: "Standard prompting" * Brown: "Chain-of-thought prompting" * Brown with diagonal lines from top-left to bottom-right: "different annotator (B)" * Brown with diagonal lines from top-right to bottom-left: "different annotator (C)" * Brown with dots: "intentionally concise style" * Brown with horizontal lines: "exemplars from GSM8K (α)" * Brown with vertical lines: "exemplars from GSM8K (β)" * Brown with small squares: "exemplars from GSM8K (γ)" ### Detailed Analysis **GSM8K Dataset:** * **Standard prompting (Yellow):** Solve rate is approximately 6%. * **Chain-of-thought prompting (Brown):** Solve rate is approximately 14%. * **Different annotator (B) (Brown with diagonal lines from top-left to bottom-right):** Solve rate is approximately 16%. * **Different annotator (C) (Brown with diagonal lines from top-right to bottom-left):** Solve rate is approximately 18%. * **Intentionally concise style (Brown with dots):** Solve rate is approximately 11%. * **Exemplars from GSM8K (α) (Brown with horizontal lines):** Solve rate is approximately 13%. * **Exemplars from GSM8K (β) (Brown with vertical lines):** Solve rate is approximately 13%. * **Exemplars from GSM8K (γ) (Brown with small squares):** Solve rate is approximately 13%. **MAWPS Dataset:** * **Standard prompting (Yellow):** Solve rate is approximately 43%. * **Chain-of-thought prompting (Brown):** Solve rate is approximately 58%. * **Different annotator (B) (Brown with diagonal lines from top-left to bottom-right):** Solve rate is approximately 59%. * **Different annotator (C) (Brown with diagonal lines from top-right to bottom-left):** Solve rate is approximately 61%. * **Intentionally concise style (Brown with dots):** Solve rate is approximately 51%. * **Exemplars from GSM8K (α) (Brown with horizontal lines):** Solve rate is approximately 58%. * **Exemplars from GSM8K (β) (Brown with vertical lines):** Solve rate is approximately 62%. * **Exemplars from GSM8K (γ) (Brown with small squares):** Solve rate is approximately 56%. ### Key Observations * For both datasets, "Standard prompting" yields the lowest solve rate. * "Different annotator (C)" generally results in a higher solve rate compared to other methods. * The solve rates for "exemplars from GSM8K (α)", "exemplars from GSM8K (β)", and "exemplars from GSM8K (γ)" are relatively similar within each dataset. * The solve rates are significantly higher for MAWPS compared to GSM8K across all prompting methods. ### Interpretation The chart demonstrates the impact of different prompting methods on the solve rate of models across two datasets, GSM8K and MAWPS. The "Chain-of-thought prompting" and its variations ("different annotator (B)", "different annotator (C)") consistently outperform "Standard prompting", suggesting that providing more context or structure to the model's input can improve its problem-solving ability. The "exemplars from GSM8K" methods show a moderate improvement over standard prompting. The higher solve rates on MAWPS compared to GSM8K indicate that the MAWPS dataset may be inherently easier or more suitable for these prompting techniques. The variations in solve rates among the different annotators and exemplars suggest that the specific phrasing and content of the prompts can influence performance.

\n ## Bar Chart: Solve Rate Comparison for Different Prompting Strategies ### Overview This image presents a comparative bar chart illustrating the solve rates achieved by various prompting strategies on two datasets: GSM8K and MAWPS. The chart consists of two sub-charts, one for each dataset, with bars representing the solve rate (%) for each prompting method. ### Components/Axes * **X-axis:** Dataset names - GSM8K and MAWPS. * **Y-axis:** Solve rate (%), ranging from 0 to 20 for GSM8K and 0 to 60 for MAWPS. * **Legend (Top-Left):** * Yellow: Standard prompting * Orange: Chain-of-thought prompting * Light Brown (Striped): different annotator (B) * Darker Brown (Striped): different annotator (C) * Black: intentionally concise style * Dark Grey (Striped): exemplars from GSM8K (α) * Medium Grey (Striped): exemplars from GSM8K (β) * Light Grey (Striped): exemplars from GSM8K (γ) ### Detailed Analysis or Content Details **GSM8K Sub-Chart:** * **Standard prompting (Yellow):** Solve rate approximately 6%. * **Chain-of-thought prompting (Orange):** Solve rate approximately 16%. * **different annotator (B) (Light Brown):** Solve rate approximately 14%. * **different annotator (C) (Darker Brown):** Solve rate approximately 11%. * **intentionally concise style (Black):** Solve rate approximately 13%. * **exemplars from GSM8K (α) (Dark Grey):** Solve rate approximately 13%. * **exemplars from GSM8K (β) (Medium Grey):** Solve rate approximately 12%. * **exemplars from GSM8K (γ) (Light Grey):** Solve rate approximately 12%. **MAWPS Sub-Chart:** * **Standard prompting (Yellow):** Solve rate approximately 45%. * **Chain-of-thought prompting (Orange):** Solve rate approximately 57%. * **different annotator (B) (Light Brown):** Solve rate approximately 57%. * **different annotator (C) (Darker Brown):** Solve rate approximately 56%. * **intentionally concise style (Black):** Solve rate approximately 56%. * **exemplars from GSM8K (α) (Dark Grey):** Solve rate approximately 56%. * **exemplars from GSM8K (β) (Medium Grey):** Solve rate approximately 55%. * **exemplars from GSM8K (γ) (Light Grey):** Solve rate approximately 55%. ### Key Observations * Chain-of-thought prompting consistently outperforms standard prompting across both datasets. * The solve rates are significantly higher for the MAWPS dataset compared to the GSM8K dataset, regardless of the prompting strategy. * Variations in annotators (B and C) and style (concise) have a relatively small impact on solve rates, especially for MAWPS. * Using exemplars from GSM8K also yields high solve rates on MAWPS, suggesting transfer learning benefits. ### Interpretation The data suggests that chain-of-thought prompting is a highly effective technique for improving the performance of language models on mathematical reasoning tasks. The substantial difference in solve rates between GSM8K and MAWPS indicates that the difficulty of the problems varies significantly between the two datasets. MAWPS appears to be easier to solve, as evidenced by the higher solve rates across all prompting strategies. The relatively minor impact of annotator variations and stylistic choices suggests that the core prompting strategy (chain-of-thought) is more important than these factors. The successful transfer of exemplars from GSM8K to MAWPS highlights the potential for leveraging knowledge from one dataset to improve performance on another. The chart demonstrates the importance of prompt engineering in enhancing the capabilities of language models for complex problem-solving.

## Bar Chart: Solve Rate Comparison Across Prompting Techniques ### Overview The image is a grouped bar chart comparing solve rates (%) for two models (GSM8K and MAWPS) across six prompting techniques. The chart uses color-coded bars with distinct patterns to differentiate methods, with a legend on the right. Solve rates range from 0% to 60% on the y-axis, and models are labeled on the x-axis. ### Components/Axes - **X-axis**: Model names ("GSM8K" and "MAWPS"), positioned at the bottom. - **Y-axis**: "Solve rate (%)", scaled from 0 to 60 in increments of 20. - **Legend**: Located on the right, with six entries: 1. **Standard prompting** (solid yellow) 2. **Chain-of-thought prompting** (solid orange) 3. **Different annotator (B)** (orange with diagonal stripes) 4. **Different annotator (C)** (orange with dots) 5. **Intentionally concise style** (beige with diagonal stripes) 6. **Exemplars from GSM8K (α/β/γ)** (beige with dots, grouped under one legend entry). ### Detailed Analysis #### GSM8K Bars (Left Group) - **Standard prompting** (yellow): ~6% solve rate. - **Chain-of-thought prompting** (orange): ~15%. - **Different annotator (B)** (striped orange): ~18%. - **Different annotator (C)** (dotted orange): ~12%. - **Intentionally concise style** (striped beige): ~13%. - **Exemplars from GSM8K (α/β/γ)** (dotted beige): ~14%. #### MAWPS Bars (Right Group) - **Standard prompting** (yellow): ~42%. - **Chain-of-thought prompting** (orange): ~58%. - **Different annotator (B)** (striped orange): ~59%. - **Different annotator (C)** (dotted orange): ~57%. - **Intentionally concise style** (striped beige): ~55%. - **Exemplars from GSM8K (α/β/γ)** (dotted beige): ~56%. ### Key Observations 1. **Model Performance**: MAWPS consistently outperforms GSM8K across all prompting techniques (e.g., MAWPS's "Chain-of-thought" at 58% vs. GSM8K's 15%). 2. **Prompting Technique Impact**: - **Chain-of-thought prompting** and **exemplars from GSM8K** yield the highest solve rates for both models. - **Different annotators (B/C)** show mixed results, with B slightly outperforming C in GSM8K but underperforming in MAWPS. - **Intentionally concise style** performs poorly in GSM8K (~13%) but moderately in MAWPS (~55%). 3. **Data Trends**: - GSM8K bars are shorter and more variable (6–18%). - MAWPS bars are taller and more consistent (42–59%). ### Interpretation The data suggests that **MAWPS is inherently more capable** than GSM8K, as evidenced by its higher baseline solve rates. Prompting techniques like **Chain-of-thought** and **GSM8K exemplars** significantly enhance performance for both models, with MAWPS benefiting more from these methods. The use of **different annotators** introduces variability, with annotator B generally performing better than C. The **intentionally concise style** underperforms in GSM8K but remains viable for MAWPS, indicating that model architecture may mitigate the drawbacks of overly simplified prompts. The legend's grouping of α/β/γ exemplars under one category implies they are treated as a unified approach, despite potential differences in their individual contributions.