## Bar Chart: GSM8K Accuracy Comparison ### Overview The image is a bar chart comparing the accuracy of different Large Language Models (LLMs) on the GSM8K dataset. The chart compares fine-tuned LLMs against versions enhanced with "+SHEPHERD". The y-axis represents accuracy in percentage, and the x-axis represents different LLM configurations. ### Components/Axes * **Title:** GSM8K * **Y-axis:** Accuracy (%) * Scale: 70 to 95, with tick marks at 70, 75, 80, 85, 90, and 95. * **X-axis:** LLM configurations * Categories: LLaMA2-70B MAmmoTH, LLaMA2-70B WizardMATH, LLaMA2-70B MetaMATH, LLemma-34B MetaMATH*, DeepSeek-67B MetaMATH* * **Legend:** Located at the top of the chart. * Blue: Fine-tuned LLMs * Orange: +SHEPHERD * **Horizontal Lines:** * GPT-4-0613*: 94.4 (Green line) * GPT-4 (early): 92.0 (Red line) ### Detailed Analysis The chart presents accuracy data for various LLMs, both fine-tuned and enhanced with "+SHEPHERD". * **LLaMA2-70B MAmmoTH:** * Fine-tuned LLMs (Blue): 72.4% * **LLaMA2-70B WizardMATH:** * Fine-tuned LLMs (Blue): 81.6% * **LLaMA2-70B MetaMATH:** * Fine-tuned LLMs (Blue): 80.4% * +SHEPHERD (Orange): 93.2% (Total height of the bar) * **LLemma-34B MetaMATH*:** * Fine-tuned LLMs (Blue): 75.8% * +SHEPHERD (Orange): 90.9% (Total height of the bar) * **DeepSeek-67B MetaMATH*:** * Fine-tuned LLMs (Blue): 82.8% * +SHEPHERD (Orange): 93.3% (Total height of the bar) ### Key Observations * The "+SHEPHERD" enhancement consistently improves the accuracy of the LLMs. * The DeepSeek-67B MetaMATH* with +SHEPHERD achieves the highest accuracy among the models tested, closely followed by LLaMA2-70B MetaMATH with +SHEPHERD. * The GPT-4 models (early and 0613*) serve as benchmarks, with the +SHEPHERD enhanced models approaching or exceeding their performance. ### Interpretation The data suggests that the "+SHEPHERD" enhancement is effective in improving the accuracy of LLMs on the GSM8K dataset. The comparison with GPT-4 models indicates that these enhanced models are competitive with state-of-the-art LLMs. The chart highlights the potential of combining fine-tuning with additional techniques like "+SHEPHERD" to achieve higher accuracy in mathematical reasoning tasks. The performance of DeepSeek-67B MetaMATH* and LLaMA2-70B MetaMATH* with +SHEPHERD is particularly noteworthy.

## Bar Chart: LLM Accuracy on GSM8K Dataset ### Overview This bar chart compares the accuracy of several Large Language Models (LLMs) on the GSM8K dataset, a benchmark for mathematical problem-solving. The chart shows the accuracy of the models in their fine-tuned state and when augmented with the "+SHEPHERD" method. Two horizontal lines indicate the accuracy of GPT-4 models (early and a later version). ### Components/Axes * **X-axis:** LLM Models - LLaMA2-70B (AMMO TH), LLaMA2-70B (WizardMATH), LLaMA2-70B (MetaMATH), LLemma-34B (MetaMATH), DeepSeek-67B (MetaMATH). * **Y-axis:** Accuracy (%) - Scale ranges from 70% to 95%. Gridlines are present at 5% intervals. * **Legend:** * Gray: Fine-tuned LLMs * Orange: +SHEPHERD * **Horizontal Lines:** * GPT-4-0613*: 94.4% * GPT-4 (early): 92.0% * **Title:** GSM8K (located along the x-axis) ### Detailed Analysis The chart consists of five sets of paired bars, each representing a different LLM. The left bar in each pair represents the accuracy of the fine-tuned LLM, and the right bar represents the accuracy when using the "+SHEPHERD" method. * **LLaMA2-70B (AMMO TH):** * Fine-tuned: Approximately 72.4% * +SHEPHERD: Not applicable (only one bar present) * **LLaMA2-70B (WizardMATH):** * Fine-tuned: Approximately 81.6% * +SHEPHERD: Not applicable (only one bar present) * **LLaMA2-70B (MetaMATH):** * Fine-tuned: Approximately 80.4% * +SHEPHERD: Approximately 93.2% * **LLemma-34B (MetaMATH):** * Fine-tuned: Approximately 75.8% * +SHEPHERD: Approximately 90.9% * **DeepSeek-67B (MetaMATH):** * Fine-tuned: Approximately 82.8% * +SHEPHERD: Approximately 93.3% The horizontal lines representing GPT-4's accuracy are positioned at 92.0% (GPT-4 early) and 94.4% (GPT-4-0613*). ### Key Observations * The "+SHEPHERD" method consistently improves the accuracy of the LLMs. * LLaMA2-70B (MetaMATH), LLemma-34B (MetaMATH), and DeepSeek-67B (MetaMATH) with "+SHEPHERD" achieve accuracy levels comparable to or exceeding the earlier version of GPT-4. * LLaMA2-70B (AMMO TH) and LLaMA2-70B (WizardMATH) have significantly lower accuracy compared to the other models, even with fine-tuning. * The largest performance gain from "+SHEPHERD" is observed with LLaMA2-70B (MetaMATH), increasing accuracy by approximately 12.8 percentage points. ### Interpretation The data suggests that the "+SHEPHERD" method is a highly effective technique for improving the mathematical problem-solving capabilities of LLMs. The substantial accuracy gains observed across multiple models indicate that "+SHEPHERD" provides a significant boost to performance on the GSM8K dataset. The fact that some models, when combined with "+SHEPHERD", reach or surpass the accuracy of GPT-4 (early) is particularly noteworthy, suggesting that these models can be competitive with state-of-the-art performance with the right augmentation. The varying levels of improvement across different models suggest that the effectiveness of "+SHEPHERD" may depend on the underlying architecture and training data of the LLM. The lower performance of LLaMA2-70B (AMMO TH) and LLaMA2-70B (WizardMATH) may indicate that these models are less well-suited for mathematical reasoning tasks, or that their fine-tuning process was less effective. The consistent improvement across the other models suggests a generalizable benefit from the "+SHEPHERD" approach.

## Bar Chart: Model Accuracy Comparison on GSM8K Dataset ### Overview This is a grouped bar chart comparing the accuracy percentages of five different large language models (LLMs) on the GSM8K mathematical reasoning benchmark. Each model is evaluated under two conditions: as a "Fine-tuned LLM" (blue bars) and with the addition of "+SHEPHERD" (orange bars). Two horizontal red lines provide performance benchmarks for GPT-4 models. ### Components/Axes * **Chart Type:** Grouped vertical bar chart. * **Y-Axis:** Labeled "Accuracy (%)". Scale ranges from 70 to 95, with major tick marks every 5 units (70, 75, 80, 85, 90, 95). * **X-Axis:** Labeled "GSM8K". It lists five distinct model configurations: 1. `LLaMA2-70b MAMoTH` 2. `LLaMA2-70b WizardMATH` 3. `LLaMA2-70b MetaMath*` 4. `LLaMA-34b MetaMath*` 5. `DeepSeek-67b MetaMath*` * **Legend:** Positioned at the top center of the chart area. * A blue square corresponds to the label "Fine-tuned LLMs". * An orange square corresponds to the label "+SHEPHERD". * **Benchmark Lines:** Two solid red horizontal lines span the chart width. * The upper line is labeled `GPT-4-0613*: 94.4` at its left end. * The lower line is labeled `GPT-4 (early): 92.0` at its left end. ### Detailed Analysis The chart presents paired data for each of the five models. The blue bar represents the baseline fine-tuned model's accuracy, and the orange bar represents the accuracy after applying the SHEPHERD method. 1. **LLaMA2-70b MAMoTH:** * Fine-tuned LLMs (Blue): 72.4% * +SHEPHERD (Orange): 81.6% * **Trend:** A significant upward increase of approximately 9.2 percentage points. 2. **LLaMA2-70b WizardMATH:** * Fine-tuned LLMs (Blue): 81.6% * +SHEPHERD (Orange): 93.2% * **Trend:** A substantial upward increase of approximately 11.6 percentage points. 3. **LLaMA2-70b MetaMath*:** * Fine-tuned LLMs (Blue): 80.4% * +SHEPHERD (Orange): 90.9% * **Trend:** A strong upward increase of approximately 10.5 percentage points. 4. **LLaMA-34b MetaMath*:** * Fine-tuned LLMs (Blue): 75.8% * +SHEPHERD (Orange): 90.9% * **Trend:** A very large upward increase of approximately 15.1 percentage points. 5. **DeepSeek-67b MetaMath*:** * Fine-tuned LLMs (Blue): 82.8% * +SHEPHERD (Orange): 93.3% * **Trend:** A strong upward increase of approximately 10.5 percentage points. **Benchmark Comparison:** * The `GPT-4 (early)` benchmark is at 92.0%. * The `GPT-4-0613*` benchmark is at 94.4%. * With SHEPHERD, three models (LLaMA2-70b WizardMATH, LLaMA2-70b MetaMath*, DeepSeek-67b MetaMath*) achieve accuracy at or above the `GPT-4 (early)` benchmark. * The highest performing model configuration is `DeepSeek-67b MetaMath* +SHEPHERD` at 93.3%, which is 1.1 percentage points below the `GPT-4-0613*` benchmark. ### Key Observations 1. **Universal Improvement:** The addition of SHEPHERD results in a consistent and substantial accuracy improvement for every single model tested. The improvement ranges from ~9.2 to ~15.1 percentage points. 2. **Performance Tier:** The SHEPHERD-enhanced models cluster in a high-performance tier between 90.9% and 93.3%, while the baseline fine-tuned models are more spread out between 72.4% and 82.8%. 3. **Model Comparison:** Among the baseline fine-tuned models, `DeepSeek-67b MetaMath*` (82.8%) and `LLaMA2-70b WizardMATH` (81.6%) perform best. With SHEPHERD, `DeepSeek-67b MetaMath*` (93.3%) and `LLaMA2-70b WizardMATH` (93.2%) remain the top performers. 4. **Benchmark Proximity:** The SHEPHERD method enables several open-weight models to reach performance levels competitive with early and mid-2023 versions of a leading proprietary model (GPT-4). ### Interpretation This chart demonstrates the potent effect of the SHEPHERD technique as a performance multiplier for fine-tuned LLMs on mathematical reasoning tasks. The data suggests that SHEPHERD is not merely an incremental improvement but a transformative one, consistently bridging a significant portion of the gap between various fine-tuned models and the GPT-4 benchmark. The fact that models of different sizes (34b, 67b, 70b) and from different fine-tuning families (MAMoTH, WizardMATH, MetaMath*) all show dramatic gains indicates that SHEPHERD likely addresses a fundamental limitation in the reasoning or problem-solving process of the base fine-tuned models, rather than being a narrow fix for a specific model architecture. The narrowing of the performance spread among the SHEPHERD-enhanced models (90.9% - 93.3%) compared to the baselines (72.4% - 82.8%) implies that the technique may help models converge toward a higher performance ceiling on this benchmark, potentially by instilling a more robust and generalizable reasoning methodology. The results position SHEPHERD as a highly valuable method for elevating the capabilities of open-weight models to compete with state-of-the-art proprietary systems on complex reasoning benchmarks.

## Bar Chart: Model Accuracy Comparison on GSM8K Benchmark ### Overview The chart compares the accuracy of different large language models (LLMs) on the GSM8K benchmark, evaluating two approaches: "Fine-tuned LLMs" (blue) and "+SHEPHERD" (orange). The y-axis represents accuracy in percentage, while the x-axis lists five models: LLama2-70B MAmmoTH, LLama2-70B WizardMATH, LLama2-70B MetaMATH*, LLaMA-34B MetaMATH*, and DeepSeek-67B MetaMATH*. Two reference lines are included: "GPT-4 (early): 92.0" (red) and "GPT-4-0613*: 94.4" (gold). ### Components/Axes - **X-axis (Models)**: - LLama2-70B MAmmoTH - LLama2-70B WizardMATH - LLama2-70B MetaMATH* - LLaMA-34B MetaMATH* - DeepSeek-67B MetaMATH* - **Y-axis (Accuracy %)**: Ranges from 70% to 95%, with gridlines at 5% intervals. - **Legend**: - Blue: Fine-tuned LLMs - Orange: +SHEPHERD - **Title**: "GSM8K" (bottom center). ### Detailed Analysis - **LLama2-70B MAmmoTH**: - Fine-tuned LLMs: 72.4% (blue) - +SHEPHERD: 93.2% (orange) - **LLama2-70B WizardMATH**: - Fine-tuned LLMs: 81.6% (blue) - +SHEPHERD: 80.4% (orange) - **LLama2-70B MetaMATH***: - Fine-tuned LLMs: 75.8% (blue) - +SHEPHERD: 90.9% (orange) - **LLaMA-34B MetaMATH***: - Fine-tuned LLMs: 82.8% (blue) - +SHEPHERD: Not explicitly labeled (implied by bar height). - **DeepSeek-67B MetaMATH***: - Fine-tuned LLMs: 82.8% (blue) - +SHEPHERD: 93.3% (orange) ### Key Observations 1. **+SHEPHERD consistently improves accuracy** across all models, with gains ranging from +17.4% (LLama2-70B MAmmoTH) to +7.5% (LLama2-70B WizardMATH). 2. **DeepSeek-67B MetaMATH*** achieves the highest accuracy (93.3%) with +SHEPHERD, surpassing all other models. 3. **LLama2-70B MAmmoTH** shows the largest improvement (+20.8%) when +SHEPHERD is applied. 4. **GPT-4 benchmarks** (92.0% and 94.4%) are not part of the chart but are referenced as external standards. ### Interpretation The data demonstrates that the +SHEPHERD method significantly enhances the performance of LLMs on the GSM8K benchmark, particularly for models with lower baseline accuracy (e.g., LLama2-70B MAmmoTH). This suggests that +SHEPHERD may address limitations in fine-tuned models, such as reasoning gaps or domain-specific knowledge deficits. The consistent improvement across models implies that +SHEPHERD is a robust augmentation technique, though its effectiveness varies slightly depending on the base model. The absence of explicit values for LLaMA-34B MetaMATH* with +SHEPHERD introduces minor uncertainty, but its bar height aligns with the trend of higher accuracy for +SHEPHERD. The GPT-4 benchmarks highlight the gap between current LLMs and state-of-the-art performance, emphasizing the need for further optimization.