## Bar Chart: Blind Spot summary across datasets - 95% Confidence Intervals ### Overview The image is a bar chart comparing the self-correction blind spot across different models. The chart displays the mean self-correction blind spot for each model, with error bars representing 95% confidence intervals. The models are evaluated using different self-correction methods: SCL15, GSM8K-SC (Before commit answer), GSM8K-SC (After commit answer), PRM800K-SC (Before commit answer), and PRM800K-SC (After commit answer). ### Components/Axes * **Title:** Blind Spot summary across datasets - 95% Confidence Intervals * **X-axis:** Models (DeepSeek-R1-0528, QwQ-32B, Qwen3-235B-A22B (thinking), Qwen3-30B-A3B (thinking), Qwen3-14B (thinking), gemma-3-27b-it, Qwen3-32B (thinking), gemma-3-12b-it, Phi-4-reasoning-plus) * **Y-axis:** Self-Correction Blind Spot (ranging from -0.4 to 0.4, with increments of 0.1) * **Legend:** Located in the top-right corner. * Blue: SCL15 * Orange: GSM8K-SC (Before commit answer) * Light Orange: GSM8K-SC (After commit answer) * Green: PRM800K-SC (Before commit answer) * Light Green: PRM800K-SC (After commit answer) ### Detailed Analysis The chart presents data for nine different models, each evaluated using five different self-correction methods. The height of each bar represents the mean self-correction blind spot, and the error bars indicate the 95% confidence interval. * **DeepSeek-R1-0528:** * SCL15 (Blue): Approximately -0.12, confidence interval extends from approximately -0.18 to -0.06. * GSM8K-SC (Before commit answer) (Orange): Approximately -0.01, confidence interval extends from approximately -0.07 to 0.05. * GSM8K-SC (After commit answer) (Light Orange): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * PRM800K-SC (Before commit answer) (Green): Approximately 0.07, confidence interval extends from approximately 0.01 to 0.13. * PRM800K-SC (After commit answer) (Light Green): Approximately -0.05, confidence interval extends from approximately -0.11 to 0.01. * **QwQ-32B:** * SCL15 (Blue): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * GSM8K-SC (Before commit answer) (Orange): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * GSM8K-SC (After commit answer) (Light Orange): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * PRM800K-SC (Before commit answer) (Green): Approximately -0.22, confidence interval extends from approximately -0.28 to -0.16. * PRM800K-SC (After commit answer) (Light Green): Approximately -0.15, confidence interval extends from approximately -0.21 to -0.09. * **Qwen3-235B-A22B (thinking):** * SCL15 (Blue): Approximately 0.03, confidence interval extends from approximately -0.03 to 0.09. * GSM8K-SC (Before commit answer) (Orange): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * GSM8K-SC (After commit answer) (Light Orange): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * PRM800K-SC (Before commit answer) (Green): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * PRM800K-SC (After commit answer) (Light Green): Approximately -0.19, confidence interval extends from approximately -0.3 to -0.08. * **Qwen3-30B-A3B (thinking):** * SCL15 (Blue): Approximately 0.14, confidence interval extends from approximately 0.08 to 0.2. * GSM8K-SC (Before commit answer) (Orange): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * GSM8K-SC (After commit answer) (Light Orange): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * PRM800K-SC (Before commit answer) (Green): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * PRM800K-SC (After commit answer) (Light Green): Approximately -0.17, confidence interval extends from approximately -0.28 to -0.06. * **Qwen3-14B (thinking):** * SCL15 (Blue): Approximately 0.13, confidence interval extends from approximately 0.07 to 0.19. * GSM8K-SC (Before commit answer) (Orange): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * GSM8K-SC (After commit answer) (Light Orange): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * PRM800K-SC (Before commit answer) (Green): Approximately -0.04, confidence interval extends from approximately -0.1 to 0.02. * PRM800K-SC (After commit answer) (Light Green): Approximately -0.05, confidence interval extends from approximately -0.16 to 0.06. * **gemma-3-27b-it:** * SCL15 (Blue): Approximately -0.08, confidence interval extends from approximately -0.14 to -0.02. * GSM8K-SC (Before commit answer) (Orange): Approximately 0.19, confidence interval extends from approximately 0.13 to 0.25. * GSM8K-SC (After commit answer) (Light Orange): Approximately 0.23, confidence interval extends from approximately 0.17 to 0.29. * PRM800K-SC (Before commit answer) (Green): Approximately 0.07, confidence interval extends from approximately 0.01 to 0.13. * PRM800K-SC (After commit answer) (Light Green): Approximately 0.03, confidence interval extends from approximately -0.03 to 0.09. * **Qwen3-32B (thinking):** * SCL15 (Blue): Approximately 0.15, confidence interval extends from approximately 0.09 to 0.21. * GSM8K-SC (Before commit answer) (Orange): Approximately 0.01, confidence interval extends from approximately -0.05 to 0.07. * GSM8K-SC (After commit answer) (Light Orange): Approximately 0.2, confidence interval extends from approximately 0.14 to 0.26. * PRM800K-SC (Before commit answer) (Green): Approximately 0.15, confidence interval extends from approximately 0.09 to 0.21. * PRM800K-SC (After commit answer) (Light Green): Approximately -0.04, confidence interval extends from approximately -0.15 to 0.07. * **gemma-3-12b-it:** * SCL15 (Blue): Approximately 0.15, confidence interval extends from approximately 0.09 to 0.21. * GSM8K-SC (Before commit answer) (Orange): Approximately 0.16, confidence interval extends from approximately 0.1 to 0.22. * GSM8K-SC (After commit answer) (Light Orange): Approximately 0.22, confidence interval extends from approximately 0.16 to 0.28. * PRM800K-SC (Before commit answer) (Green): Approximately 0.07, confidence interval extends from approximately 0.01 to 0.13. * PRM800K-SC (After commit answer) (Light Green): Approximately 0.03, confidence interval extends from approximately -0.03 to 0.09. * **Phi-4-reasoning-plus:** * SCL15 (Blue): Approximately 0.05, confidence interval extends from approximately -0.01 to 0.11. * GSM8K-SC (Before commit answer) (Orange): Approximately -0.14, confidence interval extends from approximately -0.2 to -0.08. * GSM8K-SC (After commit answer) (Light Orange): Approximately 0.15, confidence interval extends from approximately 0.09 to 0.21. * PRM800K-SC (Before commit answer) (Green): Approximately -0.14, confidence interval extends from approximately -0.2 to -0.08. * PRM800K-SC (After commit answer) (Light Green): Approximately -0.23, confidence interval extends from approximately -0.34 to -0.12. ### Key Observations * The GSM8K-SC (After commit answer) method generally shows a higher self-correction blind spot compared to the GSM8K-SC (Before commit answer) method. * The PRM800K-SC (After commit answer) method often results in a negative self-correction blind spot, indicating a potential over-correction or bias. * The confidence intervals vary across models and methods, suggesting different levels of uncertainty in the estimated blind spots. * For some models, the SCL15 method shows a relatively high self-correction blind spot compared to other methods. ### Interpretation The bar chart provides a comparative analysis of the self-correction blind spot across different models and self-correction methods. The data suggests that the choice of self-correction method can significantly impact the model's performance and bias. The positive and negative values of the self-correction blind spot indicate the degree to which the model is either under-correcting or over-correcting its errors. The confidence intervals provide a measure of the reliability of these estimates. The chart highlights the importance of carefully selecting and tuning self-correction methods to optimize model performance and minimize bias. The observed differences across models suggest that the effectiveness of a particular self-correction method may depend on the specific architecture and training data of the model.