## Bar Chart: Accuracy and Trial Numbers across Difficulty Level ### Overview The image is a bar chart comparing the accuracy and trial numbers of a model (Llama3.1-8B-Instruct) across different difficulty levels. The chart displays two accuracy metrics (SFT Accuracy and SFT+RL Accuracy) and two trial number metrics (SFT Trials and SFT+RL Trials) for each difficulty level (Level 1 to Level 5). ### Components/Axes * **Title:** Accuracy and Trial Numbers across Difficulty Level (Base Model: Llama3.1-8B-Instruct) * **X-axis:** Difficulty Level (Levels 1, 2, 3, 4, 5) * **Left Y-axis:** Accuracy (ranging from 0.2 to 1.0, with gridlines at intervals of 0.1) * **Right Y-axis:** Trial Numbers (ranging from 0 to 6, with gridlines at intervals of 1) * **Legend:** Located at the top-right of the chart. * SFT Accuracy (light green) * SFT+RL Accuracy (dark green) * SFT Trials (light red) * SFT+RL Trials (dark red) ### Detailed Analysis The chart presents four data series across five difficulty levels. **Accuracy Data Series:** * **SFT Accuracy (light green):** * Level 1: 0.814 * Level 2: 0.733 * Level 3: 0.610 * Level 4: 0.367 * Level 5: 0.239 * Trend: Decreasing accuracy as difficulty level increases. * **SFT+RL Accuracy (dark green):** * Level 1: 0.930 * Level 2: 0.722 * Level 3: 0.638 * Level 4: 0.445 * Level 5: 0.276 * Trend: Decreasing accuracy as difficulty level increases. **Trial Numbers Data Series:** * **SFT Trials (light red):** * Level 1: 3.279 * Level 2: 3.367 * Level 3: 3.924 * Level 4: 5.117 * Level 5: 4.104 * Trend: Increasing trial numbers from Level 1 to Level 4, then a decrease at Level 5. * **SFT+RL Trials (dark red):** * Level 1: 2.209 * Level 2: 2.844 * Level 3: 4.219 * Level 4: 4.234 * Level 5: 5.254 * Trend: Increasing trial numbers as difficulty level increases. ### Key Observations * Accuracy decreases as the difficulty level increases for both SFT and SFT+RL models. * Trial numbers generally increase with difficulty level, especially for the SFT+RL model. * The SFT+RL model consistently shows higher accuracy than the SFT model at Level 1, but the difference diminishes at higher difficulty levels. * The SFT+RL model requires fewer trials than the SFT model at Level 1 and Level 2, but more trials at Level 3, Level 4, and Level 5. ### Interpretation The data suggests that the Llama3.1-8B-Instruct model's performance degrades as the difficulty level increases. Reinforcement learning (RL) initially improves accuracy (Level 1), but this advantage diminishes with increasing difficulty. The increasing trial numbers for SFT+RL at higher difficulty levels may indicate that the model requires more attempts to achieve a comparable level of performance to the SFT model, suggesting that the RL component struggles to maintain its effectiveness as the task complexity grows. The model may benefit from further training or adjustments to the RL component to improve its robustness across varying difficulty levels.

## Bar Chart: Accuracy and Trial Numbers across Difficulty Level (Base Model: Llama3.1-8B-Instruct) ### Overview This bar chart displays the "Accuracy" and "Trial Numbers" for two different training configurations, "SFT" and "SFT+RL", across five distinct "Difficulty Levels" (Level 1 to Level 5). The chart uses a dual y-axis system: the left y-axis represents "Accuracy" ranging from 0.2 to 1.0, and the right y-axis represents "Trial Numbers" ranging from 0 to 6. The base model used for this analysis is Llama3.1-8B-Instruct. ### Components/Axes * **Title:** "Accuracy and Trial Numbers across Difficulty Level (Base Model: Llama3.1-8B-Instruct)" * **X-axis:** "Difficulty Level" with categories: "Level 1", "Level 2", "Level 3", "Level 4", "Level 5". * **Left Y-axis:** "Accuracy" with a scale from 0.2 to 1.0, marked at intervals of 0.1. * **Right Y-axis:** "Trial Numbers" with a scale from 0 to 6, marked at intervals of 1. * **Legend:** Located in the top-center of the chart. * Light Green bar: "SFT Accuracy" * Dark Green bar: "SFT+RL Accuracy" * Light Red bar: "SFT Trials" * Dark Red bar: "SFT+RL Trials" ### Detailed Analysis The chart presents grouped bars for each difficulty level, with two bars representing accuracy and two bars representing trial numbers. **Level 1:** * **SFT Accuracy:** 0.814 (light green bar, left axis) * **SFT+RL Accuracy:** 0.930 (dark green bar, left axis) * **SFT Trials:** 3.279 (light red bar, right axis) * **SFT+RL Trials:** 2.209 (dark red bar, right axis) **Level 2:** * **SFT Accuracy:** 0.733 (light green bar, left axis) * **SFT+RL Accuracy:** 0.722 (dark green bar, left axis) * **SFT Trials:** 3.367 (light red bar, right axis) * **SFT+RL Trials:** 2.844 (dark red bar, right axis) **Level 3:** * **SFT Accuracy:** 0.610 (light green bar, left axis) * **SFT+RL Accuracy:** 0.638 (dark green bar, left axis) * **SFT Trials:** 3.924 (light red bar, right axis) * **SFT+RL Trials:** 4.219 (dark red bar, right axis) **Level 4:** * **SFT Accuracy:** 0.367 (light green bar, left axis) * **SFT+RL Accuracy:** 0.445 (dark green bar, left axis) * **SFT Trials:** 5.117 (light red bar, right axis) * **SFT+RL Trials:** 4.234 (dark red bar, right axis) **Level 5:** * **SFT Accuracy:** 0.239 (light green bar, left axis) * **SFT+RL Accuracy:** 0.276 (dark green bar, left axis) * **SFT Trials:** 4.104 (light red bar, right axis) * **SFT+RL Trials:** 5.254 (dark red bar, right axis) ### Key Observations * **Accuracy Trend:** * "SFT Accuracy" generally decreases as difficulty level increases, starting at 0.814 for Level 1 and dropping to 0.239 for Level 5. * "SFT+RL Accuracy" also generally decreases with increasing difficulty, but it consistently outperforms "SFT Accuracy" for Level 1, Level 4, and Level 5. For Level 2 and Level 3, "SFT Accuracy" is slightly higher or comparable to "SFT+RL Accuracy". * **Trial Numbers Trend:** * "SFT Trials" show a general increase from Level 1 (3.279) to Level 4 (5.117), with a slight dip at Level 5 (4.104). * "SFT+RL Trials" show a decrease from Level 1 (2.209) to Level 2 (2.844), then an increase to Level 3 (4.219) and Level 4 (4.234), and finally a significant increase to Level 5 (5.254). * **Comparison of SFT vs. SFT+RL:** * "SFT+RL Accuracy" is higher than "SFT Accuracy" at Level 1 (0.930 vs 0.814), Level 4 (0.445 vs 0.367), and Level 5 (0.276 vs 0.239). * "SFT Accuracy" is slightly higher than "SFT+RL Accuracy" at Level 2 (0.733 vs 0.722) and Level 3 (0.638 vs 0.610). * "SFT Trials" are generally lower than "SFT+RL Trials" for Level 3, Level 4, and Level 5, but higher for Level 1 and Level 2. Notably, "SFT+RL Trials" are highest at Level 5 (5.254), while "SFT Trials" are highest at Level 4 (5.117). ### Interpretation This chart suggests that the "SFT+RL" training method generally leads to higher accuracy compared to "SFT" alone, particularly at lower difficulty levels (Level 1) and higher difficulty levels (Level 4 and 5). However, for intermediate difficulty levels (Level 2 and 3), the standard "SFT" method shows comparable or slightly better accuracy. The trial numbers indicate the computational effort or number of training iterations. It appears that achieving higher accuracy with "SFT+RL" might sometimes require more trials, as seen at Level 5 where "SFT+RL Trials" are the highest (5.254) and "SFT+RL Accuracy" is also higher than "SFT Accuracy". Conversely, at Level 4, "SFT Trials" are higher than "SFT+RL Trials", yet "SFT+RL Accuracy" is still superior. This implies a complex relationship between training trials and accuracy, where the RL component might be more efficient in certain scenarios or require different trial counts to reach optimal performance. The overall trend of decreasing accuracy with increasing difficulty level is expected for both training methods. The performance drop is more pronounced for "SFT Accuracy" at higher difficulty levels. The "SFT+RL" method seems to mitigate this drop to some extent, especially at Level 5, where it achieves a higher accuracy despite the overall decline. This indicates that reinforcement learning might be beneficial for improving model robustness and performance on more challenging tasks.

## Bar Chart: Accuracy and Trial Numbers across Difficulty Level ### Overview This bar chart compares the accuracy and trial numbers of a model (Llama3.1-8B-Instruct) across five difficulty levels (Level 1 to Level 5). Two training methods are compared: Supervised Fine-Tuning (SFT) and SFT combined with Reinforcement Learning (SFT+RL). Accuracy is represented on the primary y-axis (left), while trial numbers are represented on the secondary y-axis (right). ### Components/Axes * **Title:** Accuracy and Trial Numbers across Difficulty Level (Base Model: Llama3.1-8B-Instruct) - positioned at the top-center. * **X-axis:** Difficulty Level - labeled at the bottom, with markers for Level 1, Level 2, Level 3, Level 4, and Level 5. * **Primary Y-axis (left):** Accuracy - ranging from 0.2 to 1.0. * **Secondary Y-axis (right):** Trial Numbers - ranging from 0 to 6. * **Legend:** Located at the top-right. * SFT Accuracy (Green) * SFT+RL Accuracy (Dark Green) * SFT Trials (Red) * SFT+RL Trials (Dark Red) ### Detailed Analysis The chart consists of paired bars for each difficulty level, representing accuracy and trial numbers for both SFT and SFT+RL. **Level 1:** * SFT Accuracy: Approximately 0.930 (Green bar) * SFT+RL Accuracy: Approximately 0.814 (Dark Green bar) * SFT Trials: Approximately 3.279 (Red bar) * SFT+RL Trials: Approximately 2.209 (Dark Red bar) **Level 2:** * SFT Accuracy: Approximately 0.733 (Green bar) * SFT+RL Accuracy: Approximately 0.722 (Dark Green bar) * SFT Trials: Approximately 3.367 (Red bar) * SFT+RL Trials: Approximately 2.844 (Dark Red bar) **Level 3:** * SFT Accuracy: Approximately 0.4219 (Green bar) * SFT+RL Accuracy: Approximately 0.638 (Dark Green bar) * SFT Trials: Approximately 3.924 (Red bar) * SFT+RL Trials: Approximately 0.610 (Dark Red bar) **Level 4:** * SFT Accuracy: Approximately 0.445 (Green bar) * SFT+RL Accuracy: Approximately 0.367 (Dark Green bar) * SFT Trials: Approximately 5.117 (Red bar) * SFT+RL Trials: Approximately 3 (Dark Red bar) **Level 5:** * SFT Accuracy: Approximately 0.276 (Green bar) * SFT+RL Accuracy: Approximately 0.239 (Dark Green bar) * SFT Trials: Approximately 4.104 (Red bar) * SFT+RL Trials: Approximately 5.254 (Dark Red bar) **Trends:** * **SFT Accuracy:** Generally decreases as difficulty level increases. Starts high at Level 1 and declines to Level 5. * **SFT+RL Accuracy:** Shows a more complex trend. It starts lower than SFT at Level 1, but surpasses SFT at Level 3. It then declines at Level 4 and Level 5. * **SFT Trials:** Generally increases with difficulty level, with a slight dip between Level 2 and Level 3. * **SFT+RL Trials:** Increases with difficulty level, with a significant increase at Level 5. ### Key Observations * At Level 1, SFT has significantly higher accuracy than SFT+RL. * At Level 3, SFT+RL surpasses SFT in accuracy. * Trial numbers generally increase with difficulty for both methods, suggesting more attempts are needed to achieve results at higher difficulty levels. * The gap between SFT and SFT+RL trial numbers widens at higher difficulty levels. ### Interpretation The data suggests that while SFT performs better on easier tasks (Level 1), the addition of Reinforcement Learning (RL) improves performance on more challenging tasks (Level 3). However, this improvement comes at the cost of increased trial numbers, particularly at the highest difficulty levels. This indicates that while RL can enhance the model's ability to solve complex problems, it requires more training iterations. The decreasing accuracy of both methods as difficulty increases highlights the inherent limitations of the model in tackling increasingly complex tasks. The diverging trial numbers suggest that RL may be more sensitive to the difficulty of the task, requiring more exploration and refinement to achieve optimal performance. The chart provides valuable insights into the trade-offs between accuracy, training effort, and difficulty level when choosing between SFT and SFT+RL for this specific model (Llama3.1-8B-Instruct).

\n ## Grouped Bar Chart: Accuracy and Trial Numbers across Difficulty Level (Base Model: Llama3.1-8B-Instruct) ### Overview This is a grouped bar chart comparing the performance of two model training approaches—Supervised Fine-Tuning (SFT) and SFT combined with Reinforcement Learning (SFT+RL)—across five increasing difficulty levels. The chart displays two metrics for each approach: Accuracy (left y-axis) and Trial Numbers (right y-axis). The base model for all evaluations is Llama3.1-8B-Instruct. ### Components/Axes * **Title:** "Accuracy and Trial Numbers across Difficulty Level (Base Model: Llama3.1-8B-Instruct)" * **X-Axis (Horizontal):** "Difficulty Level". Categories are discrete: Level 1, Level 2, Level 3, Level 4, Level 5. * **Primary Y-Axis (Left):** "Accuracy". Scale ranges from 0.2 to 1.0, with major gridlines at 0.1 intervals. * **Secondary Y-Axis (Right):** "Trial Numbers". Scale ranges from 0 to 6, with major gridlines at integer intervals. * **Legend (Top-Right Corner):** * Light Green Bar: "SFT Accuracy" * Dark Green Bar: "SFT+RL Accuracy" * Light Red Bar: "SFT Trials" * Dark Red Bar: "SFT+RL Trials" ### Detailed Analysis Data is presented in four bars per difficulty level. Values are annotated directly on top of each bar. **Level 1:** * **Accuracy:** SFT = 0.814 (light green), SFT+RL = 0.930 (dark green). SFT+RL shows a significant accuracy improvement. * **Trials:** SFT = 3.279 (light red), SFT+RL = 2.209 (dark red). SFT+RL requires fewer trials. **Level 2:** * **Accuracy:** SFT = 0.733 (light green), SFT+RL = 0.722 (dark green). Performance is nearly identical, with SFT slightly higher. * **Trials:** SFT = 3.367 (light red), SFT+RL = 2.844 (dark red). SFT+RL requires fewer trials. **Level 3:** * **Accuracy:** SFT = 0.610 (light green), SFT+RL = 0.638 (dark green). SFT+RL shows a modest accuracy improvement. * **Trials:** SFT = 3.924 (light red), SFT+RL = 4.219 (dark red). SFT+RL requires more trials. **Level 4:** * **Accuracy:** SFT = 0.367 (light green), SFT+RL = 0.445 (dark green). SFT+RL shows a notable accuracy improvement. * **Trials:** SFT = 5.117 (light red), SFT+RL = 4.234 (dark red). SFT+RL requires fewer trials. **Level 5:** * **Accuracy:** SFT = 0.239 (light green), SFT+RL = 0.276 (dark green). SFT+RL shows a small accuracy improvement. * **Trials:** SFT = 4.104 (light red), SFT+RL = 5.254 (dark red). SFT+RL requires significantly more trials. ### Key Observations 1. **Accuracy Trend:** For both SFT and SFT+RL, accuracy consistently and sharply declines as difficulty increases from Level 1 to Level 5. 2. **Trial Trend:** The number of trials generally increases with difficulty for both methods, though the pattern is less linear than accuracy. The highest trial count for SFT is at Level 4 (5.117), and for SFT+RL is at Level 5 (5.254). 3. **SFT vs. SFT+RL Performance:** SFT+RL achieves higher accuracy than SFT at four out of five difficulty levels (Levels 1, 3, 4, 5). The exception is Level 2, where they are nearly equal. 4. **Trial Efficiency:** The relationship between trial numbers for the two methods is inconsistent. SFT+RL uses fewer trials at Levels 1, 2, and 4, but more trials at Levels 3 and 5. ### Interpretation The data demonstrates the expected inverse relationship between task difficulty and model accuracy. The core finding is that applying Reinforcement Learning (RL) on top of Supervised Fine-Tuning (SFT) generally improves the model's final accuracy across most difficulty levels for the Llama3.1-8B-Instruct base model. However, the benefit is not uniform. The most substantial accuracy gain from RL is seen at the easiest (Level 1) and mid-high (Level 4) difficulties. At the highest difficulty (Level 5), the accuracy gain from RL is marginal. The trial number data suggests a complex relationship between RL and computational cost or exploration. RL does not consistently reduce the number of trials needed. In fact, at the highest difficulty (Level 5), RL requires the most trials of any condition, suggesting the RL process may struggle to find a successful policy for very hard tasks, even if it ultimately yields a slightly better final model. The peak trial count for SFT at Level 4, followed by a drop at Level 5, might indicate a different failure mode or ceiling effect for the SFT-only approach at extreme difficulties. In summary, SFT+RL is a more accurate training paradigm than SFT alone for this model across a range of difficulties, but its efficiency in terms of trial numbers is highly dependent on the specific difficulty level.

## Bar Chart: Accuracy and Trial Numbers across Difficulty Level (Base Model: Llama3.1-8B-Instruct) ### Overview The chart compares the accuracy and trial numbers of two model configurations (SFT and SFT+RL) across five difficulty levels (1–5). Accuracy is measured on a 0–1 scale (left y-axis), while trial numbers are counted (right y-axis). The chart uses grouped bars to show performance differences between configurations and difficulty levels. ### Components/Axes - **X-axis**: Difficulty Level (1–5), labeled "Level 1" to "Level 5" - **Left Y-axis**: Accuracy (0.2–1.0), labeled "Accuracy" - **Right Y-axis**: Trial Numbers (0–6), labeled "Trial Numbers" - **Legend**: Located on the right, with four categories: - SFT Accuracy (light green) - SFT+RL Accuracy (dark green) - SFT Trials (light pink) - SFT+RL Trials (red) ### Detailed Analysis #### Accuracy Trends - **SFT Accuracy** (light green): - Level 1: 0.814 - Level 2: 0.733 - Level 3: 0.610 - Level 4: 0.367 - Level 5: 0.239 - **SFT+RL Accuracy** (dark green): - Level 1: 0.930 - Level 2: 0.722 - Level 3: 0.638 - Level 4: 0.445 - Level 5: 0.276 #### Trial Number Trends - **SFT Trials** (light pink): - Level 1: 3.279 - Level 2: 3.367 - Level 3: 3.924 - Level 4: 5.117 - Level 5: 4.104 - **SFT+RL Trials** (red): - Level 1: 2.209 - Level 2: 2.844 - Level 3: 4.219 - Level 4: 4.234 - Level 5: 5.254 ### Key Observations 1. **Accuracy Degradation**: Both configurations show declining accuracy with increasing difficulty. SFT+RL maintains higher accuracy than SFT at all levels. 2. **Trial Number Correlation**: Trial numbers generally increase with difficulty, peaking at Level 4 for SFT Trials (5.117) and Level 5 for SFT+RL Trials (5.254). 3. **Performance Gap**: The accuracy gap between SFT and SFT+RL narrows at higher difficulty levels (e.g., 0.575 difference at Level 1 vs. 0.169 at Level 5). 4. **Trial Efficiency**: SFT+RL requires fewer trials than SFT at Levels 1–3 but exceeds SFT trials at Levels 4–5. ### Interpretation The data suggests that reinforcement learning (RL) improves model robustness across difficulty levels, particularly in maintaining accuracy under challenging conditions. However, the diminishing accuracy gap at higher levels implies RL's benefits may plateau. The trial number patterns indicate that SFT+RL becomes less efficient relative to SFT as difficulty increases, requiring more attempts to achieve comparable results. This could reflect RL's computational overhead or the model's struggle to generalize to extreme difficulty thresholds. The sharp accuracy drop at Level 5 for both configurations highlights a potential limitation in the base model's capacity to handle highly complex tasks.