## Bar Chart: Ablation study of problem-distiller ### Overview The image is a bar chart comparing the accuracy (%) of different models (BoT+Llama-3-70B with/without problem-distiller and BoT+GPT-4 with/without problem-distiller) across four tasks: Game of 24, Word list sorting, Checkmate-in-One, and MGSM. The chart visually represents the performance of each model on each task, allowing for a direct comparison of their effectiveness. ### Components/Axes * **Title:** Ablation study of problem-distiller * **Y-axis:** * Label: Accuracy (%) * Scale: 0 to 100, with tick marks at intervals of 10. * **X-axis:** * Categories: Game of 24, Word list sorting, Checkmate-in-One, MGSM * **Legend:** Located at the top of the chart. * Blue: BoT+Llama-3-70B (w/o problem-distiller) * Orange: BoT+Llama-3-70B * Gray: BoT+GPT-4 (w/o problem-distiller) * Yellow: BoT+GPT-4 ### Detailed Analysis **Game of 24:** * BoT+Llama-3-70B (w/o problem-distiller) (Blue): 71.2% * BoT+Llama-3-70B (Orange): 78.4% * BoT+GPT-4 (w/o problem-distiller) (Gray): 76.5% * BoT+GPT-4 (Yellow): 82.4% **Word list sorting:** * BoT+Llama-3-70B (w/o problem-distiller) (Blue): 89.5% * BoT+Llama-3-70B (Orange): 92.3% * BoT+GPT-4 (w/o problem-distiller) (Gray): 97.3% * BoT+GPT-4 (Yellow): 99.6% **Checkmate-in-One:** * BoT+Llama-3-70B (w/o problem-distiller) (Blue): 64.3% * BoT+Llama-3-70B (Orange): 75.6% * BoT+GPT-4 (w/o problem-distiller) (Gray): 78.9% * BoT+GPT-4 (Yellow): 86.4% **MGSM:** * BoT+Llama-3-70B (w/o problem-distiller) (Blue): 85.6% * BoT+Llama-3-70B (Orange): 86.8% * BoT+GPT-4 (w/o problem-distiller) (Gray): 87.4% * BoT+GPT-4 (Yellow): 89.2% ### Key Observations * The "Word list sorting" task has the highest accuracy across all models. * The "Checkmate-in-One" task generally has the lowest accuracy across all models. * BoT+GPT-4 (Yellow) generally outperforms BoT+Llama-3-70B (Blue) on all tasks. * The problem-distiller seems to improve performance in most cases, as the orange and yellow bars are generally higher than the blue and gray bars, respectively. ### Interpretation The chart presents an ablation study, which aims to understand the impact of removing a specific component (the "problem-distiller") from the models. The data suggests that the problem-distiller generally improves the accuracy of both BoT+Llama-3-70B and BoT+GPT-4 models across the tested tasks. The BoT+GPT-4 model consistently achieves higher accuracy compared to BoT+Llama-3-70B, indicating that GPT-4 is a more effective base model for these tasks. The "Word list sorting" task appears to be relatively easier for these models, while "Checkmate-in-One" poses a greater challenge. The performance difference between models with and without the problem-distiller highlights the importance of this component in achieving optimal accuracy.

\n ## Bar Chart: Ablation study of problem-distiller ### Overview This bar chart presents a comparative analysis of model performance (Accuracy in percentage) across four different tasks: Game of 24, Word list sorting, Checkmate-in-One, and MGSM. The chart compares the performance of models with and without the "problem-distiller" component. The models tested are BoT+Llama-3-70B and BoT+GPT-4. ### Components/Axes * **Title:** "Ablation study of problem-distiller" (positioned at the top-center) * **X-axis:** Task names: "Game of 24", "Word list sorting", "Checkmate-in-One", "MGSM" (placed at the bottom) * **Y-axis:** Accuracy (%) - Scale ranges from 0 to 100 (placed on the left) * **Legend:** Located at the top of the chart, indicating the data series: * Blue: BoT+Llama-3-70B (w/o problem-distiller) * Orange: BoT+Llama-3-70B (w/ problem-distiller) * Red: BoT+GPT-4 (w/o problem-distiller) * Yellow: BoT+GPT-4 (w/ problem-distiller) ### Detailed Analysis The chart consists of four groups of bars, one for each task. Each group contains four bars representing the accuracy of each model configuration. **Game of 24:** * BoT+Llama-3-70B (w/o problem-distiller): Approximately 71.2% accuracy. * BoT+Llama-3-70B (w/ problem-distiller): Approximately 78.4% accuracy. * BoT+GPT-4 (w/o problem-distiller): Approximately 76.5% accuracy. * BoT+GPT-4 (w/ problem-distiller): Approximately 82.4% accuracy. **Word list sorting:** * BoT+Llama-3-70B (w/o problem-distiller): Approximately 89.5% accuracy. * BoT+Llama-3-70B (w/ problem-distiller): Approximately 92.3% accuracy. * BoT+GPT-4 (w/o problem-distiller): Approximately 97.3% accuracy. * BoT+GPT-4 (w/ problem-distiller): Approximately 99.6% accuracy. **Checkmate-in-One:** * BoT+Llama-3-70B (w/o problem-distiller): Approximately 64.3% accuracy. * BoT+Llama-3-70B (w/ problem-distiller): Approximately 75.6% accuracy. * BoT+GPT-4 (w/o problem-distiller): Approximately 78.9% accuracy. * BoT+GPT-4 (w/ problem-distiller): Approximately 86.4% accuracy. **MGSM:** * BoT+Llama-3-70B (w/o problem-distiller): Approximately 85.6% accuracy. * BoT+Llama-3-70B (w/ problem-distiller): Approximately 86.8% accuracy. * BoT+GPT-4 (w/o problem-distiller): Approximately 87.4% accuracy. * BoT+GPT-4 (w/ problem-distiller): Approximately 89.2% accuracy. ### Key Observations * The "problem-distiller" consistently improves the accuracy of both BoT+Llama-3-70B and BoT+GPT-4 across all tasks. * BoT+GPT-4 generally outperforms BoT+Llama-3-70B, both with and without the problem-distiller. * The largest performance gains from the problem-distiller are observed in the "Checkmate-in-One" task for BoT+Llama-3-70B (an increase of approximately 11.3 percentage points). * The smallest performance gains from the problem-distiller are observed in the "MGSM" task for BoT+Llama-3-70B (an increase of approximately 1.2 percentage points). ### Interpretation The data strongly suggests that the "problem-distiller" is an effective component for improving the performance of both model architectures (Llama-3-70B and GPT-4) across a variety of reasoning tasks. The consistent improvement across all tasks indicates that the problem-distiller is not task-specific but rather provides a general benefit to the models' reasoning capabilities. The larger gains observed in "Checkmate-in-One" might indicate that this task benefits more from the problem-distiller's ability to refine or structure the problem representation. The fact that GPT-4 consistently achieves higher accuracy, even without the problem-distiller, highlights its superior inherent reasoning abilities. The ablation study demonstrates the value added by the problem-distiller, quantifying its impact on model performance. The chart provides empirical evidence supporting the integration of the problem-distiller into these models to enhance their problem-solving skills.

## Bar Chart: Ablation study of problem-distiller ### Overview This is a grouped bar chart titled "Ablation study of problem-distiller." It compares the accuracy (in percentage) of four different model configurations across four distinct reasoning tasks. The chart evaluates the impact of a "problem-distiller" component by showing performance both with and without it for two base models (Llama-3-70B and GPT-4). ### Components/Axes * **Chart Title:** "Ablation study of problem-distiller" (centered at the top). * **Y-Axis:** Labeled "Accuracy (%)". The scale runs from 0 to 100 in increments of 10. * **X-Axis:** Lists four task categories: "Game of 24", "Word list sorting", "Checkmate-in-One", and "MGSM". * **Legend:** Positioned at the top of the chart, below the title. It defines four data series by color: * **Blue:** BoT+Llama-3-70B (w/o problem-distiller) * **Orange:** BoT+Llama-3-70B * **Grey:** BoT+GPT-4 (w/o problem-distiller) * **Yellow:** BoT+GPT-4 * *Note: "w/o" is an abbreviation for "without".* ### Detailed Analysis The chart presents accuracy data for each of the four tasks. The values are extracted directly from the labels above each bar. | Task | BoT+Llama-3-70B (w/o problem-distiller) [Blue] | BoT+Llama-3-70B [Orange] | BoT+GPT-4 (w/o problem-distiller) [Grey] | BoT+GPT-4 [Yellow] | | :--- | :--- | :--- | :--- | :--- | | **Game of 24** | 71.2% | 78.4% | 76.5% | 82.4% | | **Word list sorting** | 89.5% | 92.3% | 97.3% | 99.6% | | **Checkmate-in-One** | 64.3% | 75.6% | 78.9% | 86.4% | | **MGSM** | 85.6% | 86.8% | 87.4% | 89.2% | ### Key Observations * **Consistent Improvement:** For every task and both base models (Llama-3-70B and GPT-4), the configuration *with* the problem-distiller (Orange and Yellow bars) achieves higher accuracy than the configuration *without* it (Blue and Grey bars). * **Model Performance Hierarchy:** The BoT+GPT-4 configuration (Yellow) consistently achieves the highest accuracy across all four tasks. The BoT+Llama-3-70B (w/o problem-distiller) (Blue) consistently has the lowest accuracy. * **Task Variability:** The magnitude of improvement from adding the problem-distiller varies by task. The largest absolute gains are seen in "Checkmate-in-One" (+11.3% for Llama-3, +7.5% for GPT-4) and "Game of 24" (+7.2% for Llama-3, +5.9% for GPT-4). The smallest gains are in "MGSM" (+1.2% for Llama-3, +1.8% for GPT-4). * **Near-Perfect Performance:** The BoT+GPT-4 model achieves 99.6% accuracy on the "Word list sorting" task, which is the highest value on the chart. ### Interpretation This ablation study provides strong empirical evidence for the efficacy of the "problem-distiller" component. The data suggests that integrating this component systematically improves the reasoning accuracy of large language models (LLaMA-3-70B and GPT-4) when they are used within the "BoT" (likely "Brain of Thought" or similar) framework. The consistent positive delta across diverse tasks—from mathematical puzzles (Game of 24, MGSM) to logical sequencing (Word list sorting) and strategic planning (Checkmate-in-One)—indicates that the problem-distiller is a generally beneficial module, not one specialized for a single type of problem. The fact that it boosts both a smaller open-source model (Llama-3-70B) and a larger proprietary model (GPT-4) suggests it addresses a fundamental challenge in problem representation or decomposition that is common across model architectures. The varying degree of improvement implies that some tasks (like Checkmate-in-One) benefit more from the problem-distiller's intervention than others (like MGSM). This could be because MGSM tasks are already well-structured for the base models, or because the problem-distiller's specific methodology is particularly adept at clarifying the kind of multi-step, state-based reasoning required for chess puzzles. The near-ceiling performance on "Word list sorting" (99.6%) suggests this task may be approaching saturation for the GPT-4-based system with this augmentation.

## Bar Chart: Ablation study of problem-distiller ### Overview The chart compares the accuracy of four model configurations across four tasks: Game of 24, Word list sorting, Checkmate-in-One, and MGSM. Each task is represented by a group of four bars, with colors corresponding to specific model combinations. The y-axis measures accuracy in percentage, ranging from 0 to 100. ### Components/Axes - **X-axis (Tasks)**: - Game of 24 - Word list sorting - Checkmate-in-One - MGSM - **Y-axis (Accuracy)**: - Scale: 0% to 100% in 10% increments - Label: "Accuracy (%)" - **Legend (Top-left)**: - Blue: BoT+Llama-3-70B (w/o problem-distiller) - Orange: BoT+Llama-3-70B - Gray: BoT+GPT-4 (w/o problem-distiller) - Yellow: BoT+GPT-4 ### Detailed Analysis #### Task: Game of 24 - Blue (BoT+Llama-3-70B w/o): 71.2% - Orange (BoT+Llama-3-70B): 78.4% - Gray (BoT+GPT-4 w/o): 76.5% - Yellow (BoT+GPT-4): 82.4% #### Task: Word list sorting - Blue (BoT+Llama-3-70B w/o): 89.5% - Orange (BoT+Llama-3-70B): 92.3% - Gray (BoT+GPT-4 w/o): 97.3% - Yellow (BoT+GPT-4): 99.6% #### Task: Checkmate-in-One - Blue (BoT+Llama-3-70B w/o): 64.3% - Orange (BoT+Llama-3-70B): 75.6% - Gray (BoT+GPT-4 w/o): 78.9% - Yellow (BoT+GPT-4): 86.4% #### Task: MGSM - Blue (BoT+Llama-3-70B w/o): 85.6% - Orange (BoT+Llama-3-70B): 86.8% - Gray (BoT+GPT-4 w/o): 87.4% - Yellow (BoT+GPT-4): 89.2% ### Key Observations 1. **Problem-distiller impact**: - Orange/yellow bars (with problem-distiller) consistently outperform blue/gray bars (without) across all tasks. - Largest improvement: Word list sorting (BoT+GPT-4: +2.3% with problem-distiller). 2. **Model performance**: - BoT+GPT-4 models (yellow/gray) generally outperform BoT+Llama-3-70B (orange/blue). - Checkmate-in-One shows the lowest accuracy overall (64.3% baseline). 3. **Task difficulty**: - Word list sorting achieves near-perfect accuracy (99.6% peak). - Checkmate-in-One has the largest performance gap between models (+22.1% between lowest and highest). ### Interpretation The data demonstrates that the problem-distiller significantly enhances model performance across all tasks, with the most dramatic improvement observed in Word list sorting. BoT+GPT-4 models consistently outperform BoT+Llama-3-70B, suggesting GPT-4's superior base capabilities. Checkmate-in-One's lower accuracy highlights its complexity compared to other tasks. The ablation study confirms that problem-distillation is critical for optimizing model performance, particularly for GPT-4-based systems.