## Bar Chart: Blind Spot summary across datasets - 95% Confidence Intervals ### Overview The image is a bar chart comparing the self-correction blind spot across different models. The chart displays the mean self-correction blind spot for each model, with error bars representing 95% confidence intervals. The models are evaluated using different self-correction methods: SCL15, GSM8K-SC (Before commit answer), GSM8K-SC (After commit answer), PRM800K-SC (Before commit answer), and PRM800K-SC (After commit answer). ### Components/Axes * **Title:** Blind Spot summary across datasets - 95% Confidence Intervals * **X-axis:** Models (DeepSeek-R1-0528, QwQ-32B, Qwen3-235B-A22B (thinking), Qwen3-30B-A3B (thinking), Qwen3-14B (thinking), gemma-3-27b-it, Qwen3-32B (thinking), gemma-3-12b-it, Phi-4-reasoning-plus) * **Y-axis:** Self-Correction Blind Spot (ranging from -0.4 to 0.4, with increments of 0.1) * **Legend:** Located in the top-right corner. * Blue: SCL15 * Orange: GSM8K-SC (Before commit answer) * Light Orange: GSM8K-SC (After commit answer) * Green: PRM800K-SC (Before commit answer) * Light Green: PRM800K-SC (After commit answer) ### Detailed Analysis The chart presents data for nine different models, each evaluated using five different self-correction methods. The height of each bar represents the mean self-correction blind spot, and the error bars indicate the 95% confidence interval. * **DeepSeek-R1-0528:** * SCL15 (Blue): Approximately -0.12, confidence interval extends from approximately -0.18 to -0.06. * GSM8K-SC (Before commit answer) (Orange): Approximately -0.01, confidence interval extends from approximately -0.07 to 0.05. * GSM8K-SC (After commit answer) (Light Orange): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * PRM800K-SC (Before commit answer) (Green): Approximately 0.07, confidence interval extends from approximately 0.01 to 0.13. * PRM800K-SC (After commit answer) (Light Green): Approximately -0.05, confidence interval extends from approximately -0.11 to 0.01. * **QwQ-32B:** * SCL15 (Blue): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * GSM8K-SC (Before commit answer) (Orange): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * GSM8K-SC (After commit answer) (Light Orange): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * PRM800K-SC (Before commit answer) (Green): Approximately -0.22, confidence interval extends from approximately -0.28 to -0.16. * PRM800K-SC (After commit answer) (Light Green): Approximately -0.15, confidence interval extends from approximately -0.21 to -0.09. * **Qwen3-235B-A22B (thinking):** * SCL15 (Blue): Approximately 0.03, confidence interval extends from approximately -0.03 to 0.09. * GSM8K-SC (Before commit answer) (Orange): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * GSM8K-SC (After commit answer) (Light Orange): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * PRM800K-SC (Before commit answer) (Green): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * PRM800K-SC (After commit answer) (Light Green): Approximately -0.19, confidence interval extends from approximately -0.3 to -0.08. * **Qwen3-30B-A3B (thinking):** * SCL15 (Blue): Approximately 0.14, confidence interval extends from approximately 0.08 to 0.2. * GSM8K-SC (Before commit answer) (Orange): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * GSM8K-SC (After commit answer) (Light Orange): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * PRM800K-SC (Before commit answer) (Green): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * PRM800K-SC (After commit answer) (Light Green): Approximately -0.17, confidence interval extends from approximately -0.28 to -0.06. * **Qwen3-14B (thinking):** * SCL15 (Blue): Approximately 0.13, confidence interval extends from approximately 0.07 to 0.19. * GSM8K-SC (Before commit answer) (Orange): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * GSM8K-SC (After commit answer) (Light Orange): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * PRM800K-SC (Before commit answer) (Green): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * PRM800K-SC (After commit answer) (Light Green): Approximately -0.05, confidence interval extends from approximately -0.16 to 0.06. * **gemma-3-27b-it:** * SCL15 (Blue): Approximately -0.08, confidence interval extends from approximately -0.14 to -0.02. * GSM8K-SC (Before commit answer) (Orange): Approximately 0.19, confidence interval extends from approximately 0.13 to 0.25. * GSM8K-SC (After commit answer) (Light Orange): Approximately 0.23, confidence interval extends from approximately 0.17 to 0.29. * PRM800K-SC (Before commit answer) (Green): Approximately 0.07, confidence interval extends from approximately 0.01 to 0.13. * PRM800K-SC (After commit answer) (Light Green): Approximately 0.03, confidence interval extends from approximately -0.03 to 0.09. * **Qwen3-32B (thinking):** * SCL15 (Blue): Approximately 0.15, confidence interval extends from approximately 0.09 to 0.21. * GSM8K-SC (Before commit answer) (Orange): Approximately 0.01, confidence interval extends from approximately -0.05 to 0.07. * GSM8K-SC (After commit answer) (Light Orange): Approximately 0.2, confidence interval extends from approximately 0.14 to 0.26. * PRM800K-SC (Before commit answer) (Green): Approximately 0.15, confidence interval extends from approximately 0.09 to 0.21. * PRM800K-SC (After commit answer) (Light Green): Approximately -0.04, confidence interval extends from approximately -0.15 to 0.07. * **gemma-3-12b-it:** * SCL15 (Blue): Approximately 0.15, confidence interval extends from approximately 0.09 to 0.21. * GSM8K-SC (Before commit answer) (Orange): Approximately 0.16, confidence interval extends from approximately 0.1 to 0.22. * GSM8K-SC (After commit answer) (Light Orange): Approximately 0.22, confidence interval extends from approximately 0.16 to 0.28. * PRM800K-SC (Before commit answer) (Green): Approximately 0.07, confidence interval extends from approximately 0.01 to 0.13. * PRM800K-SC (After commit answer) (Light Green): Approximately 0.03, confidence interval extends from approximately -0.03 to 0.09. * **Phi-4-reasoning-plus:** * SCL15 (Blue): Approximately 0.05, confidence interval extends from approximately -0.01 to 0.11. * GSM8K-SC (Before commit answer) (Orange): Approximately -0.14, confidence interval extends from approximately -0.2 to -0.08. * GSM8K-SC (After commit answer) (Light Orange): Approximately 0.15, confidence interval extends from approximately 0.09 to 0.21. * PRM800K-SC (Before commit answer) (Green): Approximately -0.14, confidence interval extends from approximately -0.2 to -0.08. * PRM800K-SC (After commit answer) (Light Green): Approximately -0.23, confidence interval extends from approximately -0.34 to -0.12. ### Key Observations * The GSM8K-SC (After commit answer) method generally shows a higher self-correction blind spot compared to the GSM8K-SC (Before commit answer) method. * The PRM800K-SC (After commit answer) method often results in a negative self-correction blind spot, indicating a potential over-correction or bias. * The confidence intervals vary across models and methods, suggesting different levels of uncertainty in the estimated blind spots. * For some models, the SCL15 method shows a relatively high self-correction blind spot compared to other methods. ### Interpretation The bar chart provides a comparative analysis of the self-correction blind spot across different models and self-correction methods. The data suggests that the choice of self-correction method can significantly impact the model's performance and bias. The positive and negative values of the self-correction blind spot indicate the degree to which the model is either under-correcting or over-correcting its errors. The confidence intervals provide a measure of the reliability of these estimates. The chart highlights the importance of carefully selecting and tuning self-correction methods to optimize model performance and minimize bias. The observed differences across models suggest that the effectiveness of a particular self-correction method may depend on the specific architecture and training data of the model.

\n ## Bar Chart: Blind Spot Summary Across Datasets - 95% Confidence Intervals ### Overview This bar chart visualizes the "Self-Correction Blind Spot" across different models, with 95% confidence intervals represented by error bars. The chart compares the blind spot for models before and after a "commit answer" step. The x-axis represents the models, and the y-axis represents the Self-Correction Blind Spot value. ### Components/Axes * **Title:** Blind Spot summary across datasets - 95% Confidence Intervals * **X-axis Label:** Models * **Y-axis Label:** Self-Correction Blind Spot * **Legend:** * SCLIS (Purple) * GSM8K-SC (Before commit answer) (Blue) * GSM8K-SC (After commit answer) (Orange) * PRM800K-SC (Before commit answer) (Green) * PRM800K-SC (After commit answer) (Gray) * **Models (X-axis categories):** DeepSeek-RL-0.5B, Qwk-32B, Owen-3-25B-A2B (thinking), Owen-3-30B-A3B (thinking), Owen-3-14B (thinking), gemma-3-27b, Owen-3-32B (thinking), gemma-3-12b, Phi-4-reasoning-plus * **Y-axis Scale:** Ranges from approximately -0.4 to 0.4. ### Detailed Analysis The chart displays bar groupings for each model, representing the Self-Correction Blind Spot for each condition (SCLIS, GSM8K-SC before/after, PRM800K-SC before/after). Each bar has an associated error bar indicating the 95% confidence interval. Here's a breakdown of the approximate values, reading from left to right: * **DeepSeek-RL-0.5B:** * SCLIS: ~0.02 * GSM8K-SC (Before): ~0.03 * GSM8K-SC (After): ~0.01 * PRM800K-SC (Before): ~0.01 * PRM800K-SC (After): ~0.00 * **Qwk-32B:** * SCLIS: ~0.01 * GSM8K-SC (Before): ~0.01 * GSM8K-SC (After): ~-0.02 * PRM800K-SC (Before): ~-0.01 * PRM800K-SC (After): ~-0.02 * **Owen-3-25B-A2B (thinking):** * SCLIS: ~0.01 * GSM8K-SC (Before): ~0.04 * GSM8K-SC (After): ~0.02 * PRM800K-SC (Before): ~0.02 * PRM800K-SC (After): ~0.01 * **Owen-3-30B-A3B (thinking):** * SCLIS: ~0.01 * GSM8K-SC (Before): ~0.06 * GSM8K-SC (After): ~0.03 * PRM800K-SC (Before): ~0.03 * PRM800K-SC (After): ~0.02 * **Owen-3-14B (thinking):** * SCLIS: ~0.00 * GSM8K-SC (Before): ~0.03 * GSM8K-SC (After): ~0.01 * PRM800K-SC (Before): ~0.01 * PRM800K-SC (After): ~0.00 * **gemma-3-27b:** * SCLIS: ~0.01 * GSM8K-SC (Before): ~0.03 * GSM8K-SC (After): ~0.01 * PRM800K-SC (Before): ~0.01 * PRM800K-SC (After): ~0.00 * **Owen-3-32B (thinking):** * SCLIS: ~0.01 * GSM8K-SC (Before): ~0.04 * GSM8K-SC (After): ~0.02 * PRM800K-SC (Before): ~0.02 * PRM800K-SC (After): ~0.01 * **gemma-3-12b:** * SCLIS: ~0.01 * GSM8K-SC (Before): ~0.05 * GSM8K-SC (After): ~0.03 * PRM800K-SC (Before): ~0.03 * PRM800K-SC (After): ~0.02 * **Phi-4-reasoning-plus:** * SCLIS: ~0.02 * GSM8K-SC (Before): ~0.06 * GSM8K-SC (After): ~0.04 * PRM800K-SC (Before): ~0.04 * PRM800K-SC (After): ~0.03 **Trends:** * For GSM8K-SC, the "After commit answer" generally shows a decrease in the Self-Correction Blind Spot compared to "Before commit answer," suggesting the commit step helps reduce blind spots. * PRM800K-SC shows a similar, but less pronounced, trend. * SCLIS values are generally low and relatively consistent across models. ### Key Observations * The models "Owen-3-30B-A3B (thinking)", "gemma-3-12b", and "Phi-4-reasoning-plus" exhibit the largest positive Self-Correction Blind Spot values for GSM8K-SC (Before commit answer). * The error bars indicate varying degrees of uncertainty in the estimates. Some models have wider confidence intervals than others. * Qwk-32B shows a negative blind spot after the commit answer for GSM8K-SC, which is an outlier. ### Interpretation The chart demonstrates the impact of a "commit answer" step on the self-correction capabilities of different language models. The reduction in the Self-Correction Blind Spot after the commit step for GSM8K-SC suggests that this process helps models identify and correct their own errors. The differences in blind spot values across models indicate varying levels of inherent self-awareness and correction ability. The negative blind spot for Qwk-32B after the commit answer is an interesting anomaly that warrants further investigation – it could indicate an overcorrection or a different interpretation of the task. The consistent, low values for SCLIS suggest it may be a different metric or operate on a different scale than the GSM8K-SC and PRM800K-SC metrics. The confidence intervals provide a measure of the reliability of these observations, highlighting the need for caution when interpreting differences between models with large intervals.

## Grouped Bar Chart with 95% Confidence Intervals: Blind Spot Summary Across Datasets ### Overview This is a grouped bar chart with vertical error bars (representing 95% confidence intervals) that quantifies the "Self-Correction Blind Spot" metric for 9 distinct large language models, evaluated across 5 dataset variants. The metric measures how self-correction (after committing to an initial answer) impacts model performance: positive values indicate self-correction worsens performance (a "blind spot"), while negative values indicate self-correction improves performance. ### Components/Axes - **Title**: "Blind Spot summary across datasets - 95% Confidence Intervals" (top-center of the chart) - **Y-axis**: Labeled *Self-Correction Blind Spot*, with a linear scale ranging from -0.4 to 0.4, in increments of 0.1. - **X-axis**: Labeled *Models*, listing 9 models (left to right): 1. DeepSeek-R1-0528 2. QwQ-32B 3. Qwen3-235B-A22B (thinking) 4. Qwen3-30B-A3B (thinking) 5. Qwen3-14B (thinking) 6. gemma-3-27b-it 7. Qwen3-32B (thinking) 8. gemma-3-12b-it 9. Phi-4-reasoning-plus - **Legend (top-right)**: 5 dataset variants, each with a unique color: 1. SCLI5 (dark blue) 2. GSM8K-SC (Before commit answer) (orange) 3. GSM8K-SC (After commit answer) (light beige) 4. PRM800K-SC (Before commit answer) (dark green) 5. PRM800K-SC (After commit answer) (light green) - **Error Bars**: Vertical bars extending above/below each data bar, representing the 95% confidence interval for each measurement. ### Detailed Analysis Below are approximate values (with uncertainty ranges from error bars) for each model-dataset pair: 1. **DeepSeek-R1-0528**: - SCLI5: ~-0.12 (range: -0.18 to -0.06) - GSM8K-SC (Before): ~-0.04 (range: -0.07 to -0.01) - GSM8K-SC (After): ~-0.02 (range: -0.05 to 0.01) - PRM800K-SC (Before): ~0.07 (range: 0.01 to 0.13) - PRM800K-SC (After): ~-0.10 (range: -0.19 to -0.01) 2. **QwQ-32B**: - SCLI5: ~-0.05 (range: -0.09 to -0.01) - GSM8K-SC (Before): ~-0.05 (range: -0.08 to -0.02) - GSM8K-SC (After): ~-0.05 (range: -0.08 to -0.02) - PRM800K-SC (Before): ~-0.22 (range: -0.33 to -0.11) - PRM800K-SC (After): ~-0.14 (range: -0.24 to -0.04) 3. **Qwen3-235B-A22B (thinking)**: - SCLI5: ~-0.01 (range: -0.07 to 0.05) - GSM8K-SC (Before): ~-0.04 (range: -0.07 to -0.01) - GSM8K-SC (After): ~-0.04 (range: -0.07 to -0.01) - PRM800K-SC (Before): ~-0.07 (range: -0.16 to 0.02) - PRM800K-SC (After): ~-0.19 (range: -0.30 to -0.08) 4. **Qwen3-30B-A3B (thinking)**: - SCLI5: ~0.14 (range: 0.09 to 0.19) - GSM8K-SC (Before): ~-0.01 (range: -0.04 to 0.02) - GSM8K-SC (After): ~-0.02 (range: -0.05 to 0.01) - PRM800K-SC (Before): ~-0.08 (range: -0.17 to 0.01) - PRM800K-SC (After): ~-0.16 (range: -0.27 to -0.05) 5. **Qwen3-14B (thinking)**: - SCLI5: ~0.13 (range: 0.09 to 0.17) - GSM8K-SC (Before): ~-0.02 (range: -0.05 to 0.01) - GSM8K-SC (After): ~-0.03 (range: -0.06 to 0.00) - PRM800K-SC (Before): ~-0.05 (range: -0.14 to 0.04) - PRM800K-SC (After): ~-0.16 (range: -0.26 to -0.06) 6. **gemma-3-27b-it**: - SCLI5: ~-0.07 (range: -0.14 to 0.00) - GSM8K-SC (Before): ~0.19 (range: 0.16 to 0.22) - GSM8K-SC (After): ~0.24 (range: 0.21 to 0.27) - PRM800K-SC (Before): ~0.07 (range: 0.01 to 0.13) - PRM800K-SC (After): ~0.15 (range: 0.08 to 0.22) 7. **Qwen3-32B (thinking)**: - SCLI5: ~0.19 (range: 0.14 to 0.24) - GSM8K-SC (Before): ~0.03 (range: 0.00 to 0.06) - GSM8K-SC (After): ~0.14 (range: 0.11 to 0.17) - PRM800K-SC (Before): ~-0.04 (range: -0.14 to 0.06) - PRM800K-SC (After): ~-0.02 (range: -0.13 to 0.09) 8. **gemma-3-12b-it**: - SCLI5: ~0.15 (range: 0.09 to 0.21) - GSM8K-SC (Before): ~0.15 (range: 0.12 to 0.18) - GSM8K-SC (After): ~0.21 (range: 0.18 to 0.24) - PRM800K-SC (Before): ~0.07 (range: 0.01 to 0.13) - PRM800K-SC (After): ~0.13 (range: 0.06 to 0.20) 9. **Phi-4-reasoning-plus**: - SCLI5: ~0.05 (range: -0.05 to 0.15) - GSM8K-SC (Before): ~-0.14 (range: -0.20 to -0.08) - GSM8K-SC (After): ~0.15 (range: 0.10 to 0.20) - PRM800K-SC (Before): ~-0.20 (range: -0.33 to -0.07) - PRM800K-SC (After): ~-0.23 (range: -0.39 to -0.07) ### Key Observations 1. **Dataset-specific trends**: - SCLI5: Most models have positive blind spots, except DeepSeek-R1-0528, QwQ-32B, Qwen3-235B-A22B (thinking), and gemma-3-27b-it. The highest positive blind spot is Qwen3-32B (thinking) (~0.19), lowest is DeepSeek-R1-0528 (~-0.12). - GSM8K-SC (Before vs After): For most models, the "After commit answer" blind spot is more positive (worse performance) than "Before", with the largest increase in gemma-3-27b-it (from ~0.19 to ~0.24). - PRM800K-SC (Before vs After): For most models, "After commit answer" blind spots are more negative (better performance) than "Before", except gemma-3-27b-it and gemma-3-12b-it. 2. **Model-specific trends**: - gemma-3-27b-it and gemma-3-12b-it have consistently positive blind spots across all datasets, with the highest values in GSM8K-SC (After). - Phi-4-reasoning-plus has the most extreme negative blind spots in PRM800K-SC (Before/After), but a positive blind spot in GSM8K-SC (After). 3. **Statistical uncertainty**: For many model-dataset pairs, error bars overlap between "Before" and "After" variants, indicating that observed differences may not be statistically significant. ### Interpretation The "Self-Correction Blind Spot" metric reveals how self-correction (post-initial answer commitment) impacts model performance: - Positive values indicate self-correction worsens performance, suggesting the model is overconfident in initial answers or has an ineffective self-correction mechanism. The gemma models show this pattern consistently, implying their self-correction does not improve (and may harm) performance across these datasets. - Negative values indicate self-correction improves performance, meaning the model can refine its initial answer effectively. Phi-4-reasoning-plus shows this strongly in PRM800K-SC, suggesting its self-correction works well for that dataset's reasoning tasks. - The difference between "Before" and "After" variants shows dataset structure impacts self-correction effectiveness: GSM8K-SC variants lead to worse self-correction performance for most models, while PRM800K-SC variants lead to better performance. This suggests the type of reasoning required (e.g., math vs. general reasoning) and answer commitment protocols shape how well models can correct their initial answers. - Overlapping error bars caution against overinterpreting small differences; larger sample sizes would be needed to confirm statistically significant trends.

## Bar Chart: Blind Spot summary across datasets - 95% Confidence Intervals ### Overview The chart compares self-correction blind spot performance across multiple AI models (e.g., DeepSeek-R1-0328, QwQ-32B, Gemma-3-27b-it) using five datasets. Values represent deviations from perfect self-correction, with error bars showing 95% confidence intervals. Positive values indicate overestimation of blind spots, while negative values suggest underestimation. ### Components/Axes - **X-axis**: Models (DeepSeek-R1-0328, QwQ-32B, Qwen-3-23B-A22B, Qwen-3-30B-A3B, Qwen-3-148, Gemma-3-27b-it, Qwen-3-328, Gemma-3-12b-it, Phi-4-reasoning-plus) - **Y-axis**: Self-Correction Blind Spot (range: -0.4 to 0.4) - **Legend**: - Blue: SCL15 - Orange: GSM8K-SC (Before commit answer) - Pink: GSM8K-SC (After commit answer) - Green: PRM800K-SC (Before commit answer) - Light Green: PRM800K-SC (After commit answer) ### Detailed Analysis 1. **DeepSeek-R1-0328**: - SCL15: -0.05 (±0.08) - GSM8K-SC (Before): -0.02 (±0.05) - GSM8K-SC (After): -0.01 (±0.04) - PRM800K-SC (Before): 0.06 (±0.07) - PRM800K-SC (After): -0.03 (±0.06) 2. **QwQ-32B**: - SCL15: -0.01 (±0.04) - GSM8K-SC (Before): -0.03 (±0.06) - GSM8K-SC (After): -0.02 (±0.05) - PRM800K-SC (Before): -0.20 (±0.12) - PRM800K-SC (After): -0.10 (±0.10) 3. **Qwen-3-23B-A22B (thinking)**: - SCL15: -0.01 (±0.03) - GSM8K-SC (Before): -0.02 (±0.04) - GSM8K-SC (After): -0.01 (±0.03) - PRM800K-SC (Before): -0.15 (±0.09) - PRM800K-SC (After): -0.05 (±0.07) 4. **Qwen-3-30B-A3B (thinking)**: - SCL15: 0.13 (±0.09) - GSM8K-SC (Before): -0.03 (±0.05) - GSM8K-SC (After): -0.02 (±0.04) - PRM800K-SC (Before): -0.05 (±0.06) - PRM800K-SC (After): -0.02 (±0.05) 5. **Qwen-3-148 (thinking)**: - SCL15: 0.12 (±0.08) - GSM8K-SC (Before): -0.02 (±0.04) - GSM8K-SC (After): -0.01 (±0.03) - PRM800K-SC (Before): -0.03 (±0.05) - PRM800K-SC (After): -0.01 (±0.04) 6. **Gemma-3-27b-it**: - SCL15: -0.05 (±0.07) - GSM8K-SC (Before): 0.18 (±0.09) - GSM8K-SC (After): 0.24 (±0.10) - PRM800K-SC (Before): 0.06 (±0.07) - PRM800K-SC (After): 0.13 (±0.08) 7. **Qwen-3-328 (thinking)**: - SCL15: 0.19 (±0.08) - GSM8K-SC (Before): 0.03 (±0.05) - GSM8K-SC (After): 0.14 (±0.07) - PRM800K-SC (Before): -0.02 (±0.04) - PRM800K-SC (After): -0.01 (±0.03) 8. **Gemma-3-12b-it**: - SCL15: 0.15 (±0.07) - GSM8K-SC (Before): 0.16 (±0.08) - GSM8K-SC (After): 0.22 (±0.09) - PRM800K-SC (Before): 0.07 (±0.06) - PRM800K-SC (After): 0.12 (±0.07) 9. **Phi-4-reasoning-plus**: - SCL15: 0.04 (±0.06) - GSM8K-SC (Before): -0.08 (±0.07) - GSM8K-SC (After): 0.15 (±0.08) - PRM800K-SC (Before): -0.15 (±0.09) - PRM800K-SC (After): -0.20 (±0.10) ### Key Observations - **Positive Trends**: - GSM8K-SC (After commit answer) consistently shows improved performance (higher values) across most models compared to "Before commit answer" versions. - SCL15 demonstrates the largest positive deviations (0.19 for Qwen-3-328), suggesting significant overestimation of blind spots. - **Negative Trends**: - PRM800K-SC (Before commit answer) frequently shows negative values (e.g., -0.20 for Phi-4-reasoning-plus), indicating underestimation of blind spots. - Qwen-3-148 (thinking) has the smallest confidence intervals (±0.04–0.08), suggesting higher measurement precision. - **Anomalies**: - Gemma-3-27b-it exhibits the largest improvement in GSM8K-SC (After commit answer: +0.06). - Phi-4-reasoning-plus shows the most drastic negative shift in PRM800K-SC (After commit answer: -0.20). ### Interpretation The data suggests that model updates ("After commit answer") generally reduce blind spots for GSM8K-SC and PRM800K-SC datasets, with SCL15 showing the most persistent overestimation. The negative values for PRM800K-SC (Before commit answer) across multiple models imply systematic underestimation of blind spots in this configuration. Confidence interval lengths vary significantly, with larger intervals (e.g., ±0.12 for QwQ-32B PRM800K-SC Before) indicating lower reliability in those estimates. The correlation between model size (e.g., Qwen-3-30B-A3B vs. Qwen-3-148) and performance trends warrants further investigation.