## Bar Chart: Evaluation on Verification and Correction (Base Model: Qwen2.5-Math-7B) ### Overview The image presents two bar charts comparing the performance of different models (SFT, SFT + Process-level RL, and SFT + Outcome-level RL) on self-verification and self-correction metrics. The left chart focuses on self-verification metrics (Verification Accuracy, Error Recall, Correct Precision), while the right chart focuses on self-correction metrics (Incorrect to Correct, Correct to Incorrect). ### Components/Axes **Left Chart (Self-verification Metrics):** * **Title:** Self-verification Metrics * **X-axis:** Categorical axis with three categories: "Verification Accuracy", "Error Recall", and "Correct Precision". * **Y-axis:** "Value (%)", ranging from 50 to 90 in increments of 10. * **Legend:** Located in the top-left corner. * Gray: SFT * Teal: SFT + Process-level RL * Peach: SFT + Outcome-level RL **Right Chart (Self-correction Metrics):** * **Title:** Self-correction Metrics * **X-axis:** Categorical axis with two categories: "Incorrect to Correct" and "Correct to Incorrect". * **Y-axis:** "Value (%)", ranging from 0 to 14 in increments of 2. * **Legend:** (Same as left chart) * Gray: SFT * Teal: SFT + Process-level RL * Peach: SFT + Outcome-level RL ### Detailed Analysis **Left Chart (Self-verification Metrics):** * **Verification Accuracy:** * SFT (Gray): 61.58% * SFT + Process-level RL (Teal): 74.61% * SFT + Outcome-level RL (Peach): 66.49% * Trend: SFT + Process-level RL performs the best, followed by SFT + Outcome-level RL, and then SFT. * **Error Recall:** * SFT (Gray): 66.83% * SFT + Process-level RL (Teal): 64.75% * SFT + Outcome-level RL (Peach): 70.11% * Trend: SFT + Outcome-level RL performs the best, followed by SFT, and then SFT + Process-level RL. * **Correct Precision:** * SFT (Gray): 84.94% * SFT + Process-level RL (Teal): 90.28% * SFT + Outcome-level RL (Peach): 87.85% * Trend: SFT + Process-level RL performs the best, followed by SFT + Outcome-level RL, and then SFT. **Right Chart (Self-correction Metrics):** * **Incorrect to Correct:** * SFT (Gray): 6.52% * SFT + Process-level RL (Teal): 12.22% * SFT + Outcome-level RL (Peach): 13.64% * Trend: SFT + Outcome-level RL performs the best, followed by SFT + Process-level RL, and then SFT. * **Correct to Incorrect:** * SFT (Gray): 1.96% * SFT + Process-level RL (Teal): 1.46% * SFT + Outcome-level RL (Peach): 0.97% * Trend: SFT performs the worst, followed by SFT + Process-level RL, and then SFT + Outcome-level RL. ### Key Observations * SFT + Process-level RL generally performs better than SFT and SFT + Outcome-level RL in self-verification metrics, especially in Verification Accuracy and Correct Precision. * SFT + Outcome-level RL shows the best performance in Error Recall and Incorrect to Correct metrics. * All models have a low percentage of "Correct to Incorrect" transitions. ### Interpretation The charts suggest that incorporating reinforcement learning (RL) into the SFT model, either at the process level or outcome level, generally improves performance in both self-verification and self-correction tasks. The specific type of RL (process-level vs. outcome-level) seems to have a varying impact depending on the metric being evaluated. SFT + Process-level RL excels in accuracy and precision, while SFT + Outcome-level RL is better at recalling errors and correcting incorrect answers. The low "Correct to Incorrect" values across all models indicate a relatively stable performance in maintaining correct answers. The base SFT model consistently underperforms compared to the RL-enhanced models, highlighting the benefits of incorporating RL techniques for these tasks.

## Bar Chart: Evaluation on Verification and Correction (Base Model: Qwen2.5-Math-7B) ### Overview This image contains two bar charts side-by-side, presenting the results of self-verification and self-correction metrics for a base model named "Qwen2.5-Math-7B". The metrics are evaluated across three different configurations: "SFT", "SFT + Process-level RL", and "SFT + Outcome-level RL". The left chart focuses on "Self-verification Metrics" and the right chart on "Self-correction Metrics". ### Components/Axes **Overall Title:** Evaluation on Verification and Correction (Base Model: Qwen2.5-Math-7B) **Left Chart:** * **Title:** Self-verification Metrics * **X-axis Title:** (Implicitly, the metric categories) * **X-axis Categories:** Verification Accuracy, Error Recall, Correct Precision * **Y-axis Title:** Value (%) * **Y-axis Scale:** 50 to 90, with major ticks at 50, 60, 70, 80, 90. * **Legend:** Located in the top-left quadrant of the left chart. * **SFT:** Represented by a grey color. * **SFT + Process-level RL:** Represented by a teal/green color. * **SFT + Outcome-level RL:** Represented by a coral/orange color. **Right Chart:** * **Title:** Self-correction Metrics * **X-axis Title:** (Implicitly, the metric categories) * **X-axis Categories:** Incorrect to Correct, Correct to Incorrect * **Y-axis Title:** Value (%) * **Y-axis Scale:** 0 to 14, with major ticks at 0, 2, 4, 6, 8, 10, 12, 14. * **Legend:** The legend from the left chart applies to both charts. ### Detailed Analysis **Left Chart: Self-verification Metrics** * **Verification Accuracy:** * SFT (Grey): 61.58% * SFT + Process-level RL (Teal): 74.61% * SFT + Outcome-level RL (Coral): 66.49% * **Error Recall:** * SFT (Grey): 66.83% * SFT + Process-level RL (Teal): 64.75% * SFT + Outcome-level RL (Coral): 70.11% * **Correct Precision:** * SFT (Grey): 84.94% * SFT + Process-level RL (Teal): 90.28% * SFT + Outcome-level RL (Coral): 87.85% **Right Chart: Self-correction Metrics** * **Incorrect to Correct:** * SFT (Grey): 6.52% * SFT + Process-level RL (Teal): 12.22% * SFT + Outcome-level RL (Coral): 13.64% * **Correct to Incorrect:** * SFT (Grey): 1.96% * SFT + Process-level RL (Teal): 1.46% * SFT + Outcome-level RL (Coral): 0.97% ### Key Observations * **Verification Accuracy:** "SFT + Process-level RL" shows the highest Verification Accuracy (74.61%), significantly outperforming both "SFT" (61.58%) and "SFT + Outcome-level RL" (66.49%). * **Error Recall:** "SFT + Outcome-level RL" has the highest Error Recall (70.11%), followed by "SFT" (66.83%). "SFT + Process-level RL" has the lowest Error Recall (64.75%). * **Correct Precision:** "SFT + Process-level RL" achieves the highest Correct Precision (90.28%), with "SFT + Outcome-level RL" (87.85%) and "SFT" (84.94%) following. * **Incorrect to Correct:** The addition of RL significantly improves the ability to correct incorrect predictions. "SFT + Outcome-level RL" shows the highest value (13.64%), followed closely by "SFT + Process-level RL" (12.22%), both substantially higher than "SFT" (6.52%). * **Correct to Incorrect:** The addition of RL appears to reduce the rate of correcting correct predictions. "SFT" has the highest rate (1.96%), while "SFT + Outcome-level RL" has the lowest (0.97%), with "SFT + Process-level RL" in between (1.46%). ### Interpretation The data suggests that incorporating Reinforcement Learning (RL) strategies, particularly "Process-level RL" and "Outcome-level RL", generally enhances the self-verification and self-correction capabilities of the "Qwen2.5-Math-7B" model compared to the base "SFT" model. Specifically, "SFT + Process-level RL" demonstrates superior performance in "Verification Accuracy" and "Correct Precision", indicating a better ability to accurately verify its own outputs and to precisely identify correct predictions. This configuration also shows a substantial improvement in correcting "Incorrect to Correct" scenarios, suggesting it is more adept at fixing its own mistakes. "SFT + Outcome-level RL" also shows significant gains in correcting "Incorrect to Correct" predictions, even surpassing "Process-level RL" in this specific metric. It also leads in "Error Recall", implying it is better at identifying errors that need attention. However, it shows a slight decrease in "Correct Precision" compared to "Process-level RL" and a notable reduction in "Correct to Incorrect" rates, which could imply a more conservative approach to corrections, potentially avoiding unnecessary changes to correct outputs. The base "SFT" model performs the lowest across most metrics, highlighting the benefit of the RL fine-tuning. The trade-off between "Incorrect to Correct" and "Correct to Incorrect" rates is also evident. While RL methods improve the correction of errors, they might also slightly increase the risk of incorrectly modifying already correct outputs, though "Outcome-level RL" appears to mitigate this risk more effectively than "Process-level RL". Overall, the results indicate that RL fine-tuning is a promising direction for improving the self-evaluation and self-correction abilities of large language models, with "SFT + Outcome-level RL" showing a strong balance of correcting errors and preserving correct outputs.

## Bar Chart: Evaluation on Verification and Correction ### Overview This image presents a bar chart comparing the performance of a base model (Qwen2.5-Math-7B) and its variations trained with different reinforcement learning (RL) techniques. The chart is split into two sections: "Self-verification Metrics" and "Self-correction Metrics". Each section displays three data series representing different training approaches: Supervised Fine-Tuning (SFT), SFT + Process-level RL, and SFT + Outcome-level RL. The y-axis represents "Value (%)". ### Components/Axes * **Title:** Evaluation on Verification and Correction (Base Model: Qwen2.5-Math-7B) * **Subtitle 1:** Self-verification Metrics * **Subtitle 2:** Self-correction Metrics * **X-axis (Self-verification):** Verification Accuracy, Error Recall, Correct Precision * **X-axis (Self-correction):** Incorrect to Correct, Correct to Incorrect * **Y-axis:** Value (%) - Scale ranges from 0 to 90 for the left chart and 0 to 14 for the right chart. * **Legend:** * SFT (Red) * SFT + Process-level RL (Green) * SFT + Outcome-level RL (Teal) ### Detailed Analysis or Content Details **Self-verification Metrics (Left Chart)** * **Verification Accuracy:** * SFT: Approximately 61.58% * SFT + Process-level RL: Approximately 66.49% * SFT + Outcome-level RL: Approximately 74.61% * Trend: The bars increase in height from SFT to SFT + Process-level RL to SFT + Outcome-level RL. * **Error Recall:** * SFT: Approximately 64.75% * SFT + Process-level RL: Approximately 66.83% * SFT + Outcome-level RL: Approximately 70.11% * Trend: The bars increase in height from SFT to SFT + Process-level RL to SFT + Outcome-level RL. * **Correct Precision:** * SFT: Approximately 84.94% * SFT + Process-level RL: Approximately 87.85% * SFT + Outcome-level RL: Approximately 90.28% * Trend: The bars increase in height from SFT to SFT + Process-level RL to SFT + Outcome-level RL. **Self-correction Metrics (Right Chart)** * **Incorrect to Correct:** * SFT: Approximately 6.52% * SFT + Process-level RL: Approximately 12.22% * SFT + Outcome-level RL: Approximately 13.64% * Trend: The bars increase in height from SFT to SFT + Process-level RL to SFT + Outcome-level RL. * **Correct to Incorrect:** * SFT: Approximately 0.97% * SFT + Process-level RL: Approximately 1.46% * SFT + Outcome-level RL: Approximately 1.96% * Trend: The bars increase in height from SFT to SFT + Process-level RL to SFT + Outcome-level RL. ### Key Observations * In all metrics, the "SFT + Outcome-level RL" consistently outperforms both "SFT" and "SFT + Process-level RL". * The "SFT + Process-level RL" generally shows improvement over the base "SFT" model. * The "Incorrect to Correct" metric shows a more substantial increase with RL training compared to the "Correct to Incorrect" metric. * The scale of the Y-axis differs between the two charts, indicating different ranges of values for self-verification and self-correction metrics. ### Interpretation The data suggests that incorporating reinforcement learning, particularly at the outcome level, significantly improves both the self-verification and self-correction capabilities of the Qwen2.5-Math-7B model. The model is better at identifying and correcting errors when trained with outcome-level RL. The larger gains observed in the "Incorrect to Correct" metric suggest that the RL training is particularly effective in enabling the model to recover from incorrect initial responses. The consistent improvement across all metrics indicates that the RL techniques are generally beneficial for enhancing the model's performance in both verifying its own work and correcting its mistakes. The difference in Y-axis scales suggests that the magnitude of improvement is greater in the self-verification metrics than in the self-correction metrics. This could indicate that the model is already relatively good at correcting its own errors, but has more room for improvement in accurately assessing the correctness of its initial responses.

## Bar Chart: Evaluation on Verification and Correction (Base Model: Qwen2.5-Math-7B) ### Overview The image is a composite bar chart comparing the performance of three different training methods on a base language model (Qwen2.5-Math-7B) across two sets of metrics: "Self-verification Metrics" and "Self-correction Metrics." The chart evaluates how well the model can verify its own outputs and correct errors. ### Components/Axes * **Title:** "Evaluation on Verification and Correction (Base Model: Qwen2.5-Math-7B)" * **Subplots:** The chart is divided into two distinct panels. * **Left Panel Title:** "Self-verification Metrics" * **Right Panel Title:** "Self-correction Metrics" * **Legend:** Located in the top-left corner of the left panel. It defines three data series: * **SFT** (Gray bar) * **SFT + Process-level RL** (Teal/Green bar) * **SFT + Outcome-level RL** (Salmon/Orange bar) * **Y-Axis (Left Panel):** Labeled "Value (%)". Scale ranges from 50 to 90, with major gridlines at intervals of 10. * **X-Axis (Left Panel):** Three categorical groups: 1. Verification Accuracy 2. Error Recall 3. Correct Precision * **Y-Axis (Right Panel):** Labeled "Value (%)". Scale ranges from 0 to 14, with major gridlines at intervals of 2. * **X-Axis (Right Panel):** Two categorical groups: 1. Incorrect to Correct 2. Correct to Incorrect ### Detailed Analysis **Self-verification Metrics (Left Panel):** 1. **Verification Accuracy:** * **SFT:** 61.58% * **SFT + Process-level RL:** 74.61% (Highest in this group) * **SFT + Outcome-level RL:** 66.49% * *Trend:* Process-level RL provides the largest improvement over the SFT baseline, followed by Outcome-level RL. 2. **Error Recall:** * **SFT:** 66.83% * **SFT + Process-level RL:** 64.75% (Lowest in this group) * **SFT + Outcome-level RL:** 70.11% (Highest in this group) * *Trend:* Outcome-level RL improves error recall over SFT, while Process-level RL slightly decreases it. 3. **Correct Precision:** * **SFT:** 84.94% * **SFT + Process-level RL:** 90.28% (Highest in this group) * **SFT + Outcome-level RL:** 87.85% * *Trend:* Both RL methods improve precision, with Process-level RL showing the greatest gain. **Self-correction Metrics (Right Panel):** 1. **Incorrect to Correct (Rate of fixing wrong answers):** * **SFT:** 6.52% * **SFT + Process-level RL:** 12.22% * **SFT + Outcome-level RL:** 13.64% (Highest in this group) * *Trend:* Both RL methods nearly double or more than double the self-correction rate compared to SFT, with Outcome-level RL performing best. 2. **Correct to Incorrect (Rate of breaking right answers):** * **SFT:** 1.96% * **SFT + Process-level RL:** 1.46% * **SFT + Outcome-level RL:** 0.97% (Lowest in this group) * *Trend:* Both RL methods reduce the rate of corrupting correct answers, with Outcome-level RL showing the most significant reduction. ### Key Observations * **Performance Trade-offs:** The two RL methods show complementary strengths. "SFT + Process-level RL" excels in Verification Accuracy and Correct Precision. "SFT + Outcome-level RL" excels in Error Recall and the "Incorrect to Correct" self-correction rate. * **Consistent Improvement in Correction:** Both RL methods dramatically improve the model's ability to correct its own errors ("Incorrect to Correct") while simultaneously reducing its tendency to alter correct answers ("Correct to Incorrect"). * **Baseline Performance:** The SFT (Supervised Fine-Tuning) baseline serves as the reference point, with all RL variants showing targeted improvements in specific metrics. ### Interpretation This data suggests that applying Reinforcement Learning (RL) to a math-capable language model enhances its metacognitive abilities—specifically, its capacity to self-verify and self-correct its reasoning. The choice of RL objective creates a specialisation: * **Process-level RL** (which likely rewards intermediate reasoning steps) appears to make the model more precise and accurate in its verification judgments, leading to higher confidence when it identifies a correct answer. * **Outcome-level RL** (which rewards only the final answer) seems to make the model more sensitive to errors, improving its recall of mistakes and its ability to transform incorrect solutions into correct ones. The most significant practical takeaway is the substantial reduction in the "Correct to Incorrect" rate alongside the increase in "Incorrect to Correct" rate. This indicates the RL-trained models are not just guessing more often but are becoming more reliable self-editors, a crucial trait for autonomous problem-solving systems. The base model, Qwen2.5-Math-7B, demonstrates a strong foundation that is meaningfully enhanced by these targeted training strategies.

## Bar Charts: Evaluation on Verification and Correction (Base Model: Qwen2.5-Math-7B) ### Overview The image contains two side-by-side bar charts comparing performance metrics for three model configurations: 1. **SFT** (Self-Taught Fine-tuning) 2. **SFT + Process-level RL** (Reinforcement Learning) 3. **SFT + Outcome-level RL** Metrics are split into **Self-verification** (left chart) and **Self-correction** (right chart). All values are percentages. --- ### Components/Axes #### Left Chart (Self-verification Metrics) - **X-axis**: - Verification Accuracy - Error Recall - Correct Precision - **Y-axis**: Value (%) from 0% to 90% - **Legend**: - Gray: SFT - Teal: SFT + Process-level RL - Orange: SFT + Outcome-level RL #### Right Chart (Self-correction Metrics) - **X-axis**: - Incorrect to Correct - Correct to Incorrect - **Y-axis**: Value (%) from 0% to 14% - **Legend**: Same color coding as left chart --- ### Detailed Analysis #### Self-verification Metrics (Left Chart) 1. **Verification Accuracy** - SFT: 61.58% - SFT + Process-level RL: 74.61% - SFT + Outcome-level RL: 66.49% 2. **Error Recall** - SFT: 66.83% - SFT + Process-level RL: 64.75% - SFT + Outcome-level RL: 70.11% 3. **Correct Precision** - SFT: 84.94% - SFT + Process-level RL: 90.28% - SFT + Outcome-level RL: 87.85% #### Self-correction Metrics (Right Chart) 1. **Incorrect to Correct** - SFT: 6.52% - SFT + Process-level RL: 12.22% - SFT + Outcome-level RL: 13.64% 2. **Correct to Incorrect** - SFT: 1.96% - SFT + Process-level RL: 1.46% - SFT + Outcome-level RL: 0.97% --- ### Key Observations 1. **Self-verification**: - **SFT + Process-level RL** outperforms SFT in all metrics, with the largest gain in **Verification Accuracy** (+13.03%). - **SFT + Outcome-level RL** shows mixed results: lower than SFT in Verification Accuracy but higher in Error Recall and Correct Precision. 2. **Self-correction**: - **SFT + Outcome-level RL** achieves the highest **Incorrect to Correct** rate (+7.12% over SFT) and the lowest **Correct to Incorrect** rate (-1.01% over SFT). - **SFT + Process-level RL** improves **Incorrect to Correct** by 5.7% over SFT but underperforms Outcome-level RL. --- ### Interpretation 1. **Process-level RL** enhances **verification robustness**, particularly in **Correct Precision** (90.28%), suggesting it improves the model's ability to identify valid solutions. 2. **Outcome-level RL** excels in **correction efficiency**, reducing errors (Correct to Incorrect drops to 0.97%) while maximizing successful corrections (Incorrect to Correct: 13.64%). 3. **Trade-offs**: - Process-level RL slightly reduces Error Recall (64.75% vs. SFT's 66.83%), possibly due to stricter validation. - Outcome-level RL sacrifices some verification accuracy (66.49% vs. SFT's 61.58%) but gains significant correction performance. The data implies that **Process-level RL** is optimal for tasks requiring high verification accuracy, while **Outcome-level RL** is better suited for error correction scenarios. Combining both approaches could balance these trade-offs.