## Box Plot: ECE Diff and AUROC Diff Comparison ### Overview The image presents two box plots side-by-side, comparing the distributions of "ECE Diff" (Expected Calibration Error Difference) and "AUROC Diff" (Area Under the Receiver Operating Characteristic Curve Difference) across three different methods: "CoT" (Chain-of-Thought), "Multi-Step", and "Top-K". The box plots visually summarize the central tendency, spread, and skewness of the data for each method and metric. ### Components/Axes **Left Plot (ECE):** * **Title:** ECE * **Y-axis:** ECE Diff, with scale markers at 0, -10, -20, -30, -40, and -50. * **X-axis:** Categorical, representing the three methods: CoT, Multi-Step, and Top-K. **Right Plot (AUROC):** * **Title:** AUROC * **Y-axis:** AUROC Diff, with scale markers at -10, -5, 0, 5, 10, and 15. * **X-axis:** Categorical, representing the three methods: CoT, Multi-Step, and Top-K. **Legend (Implicit):** * CoT: Represented by blue boxes. * Multi-Step: Represented by orange boxes. * Top-K: Represented by green boxes. ### Detailed Analysis **Left Plot (ECE Diff):** * **CoT (Blue):** The box extends from approximately -24 to -8. The median is around -14. The upper whisker extends to approximately 2, and the lower whisker extends to approximately -56. * Trend: The ECE Diff for CoT is centered around -14, with a wide spread indicating high variability. * **Multi-Step (Orange):** The box extends from approximately -21 to -12. The median is around -16. The upper whisker extends to approximately 4, and the lower whisker extends to approximately -44. * Trend: The ECE Diff for Multi-Step is centered around -16, with a moderate spread. * **Top-K (Green):** The box extends from approximately -21 to -11. The median is around -16. The upper whisker extends to approximately 2, and the lower whisker extends to approximately -46. * Trend: The ECE Diff for Top-K is centered around -16, with a moderate spread. **Right Plot (AUROC Diff):** * **CoT (Blue):** The box extends from approximately -2 to 0. The median is around -1. The upper whisker extends to approximately 8, and the lower whisker extends to approximately -10. * Trend: The AUROC Diff for CoT is centered around -1, with a moderate spread. * **Multi-Step (Orange):** The box extends from approximately -1 to 4. The median is around 2. The upper whisker extends to approximately 15, and the lower whisker extends to approximately -5. * Trend: The AUROC Diff for Multi-Step is centered around 2, with a wide spread. * **Top-K (Green):** The box extends from approximately 4 to 10. The median is around 8. The upper whisker extends to approximately 17, and the lower whisker extends to approximately -3. * Trend: The AUROC Diff for Top-K is centered around 8, with a wide spread. ### Key Observations * For ECE Diff, all three methods have negative median values, indicating a general tendency to underestimate the calibration error. CoT has the widest spread. * For AUROC Diff, Top-K shows a significantly higher median value compared to CoT and Multi-Step, suggesting better performance in terms of discrimination. ### Interpretation The box plots provide a comparative view of the performance of three different methods (CoT, Multi-Step, and Top-K) based on two metrics: ECE Diff and AUROC Diff. * **ECE Diff:** The negative values across all methods suggest that, on average, these methods tend to underestimate the true calibration error. The wider spread for CoT indicates that its performance is more variable compared to Multi-Step and Top-K. * **AUROC Diff:** Top-K stands out with a higher median AUROC Diff, implying that it generally provides better discrimination compared to the other two methods. The spread of the data suggests that the performance of Multi-Step and Top-K can vary significantly. In summary, while all methods show a tendency to underestimate calibration error, Top-K appears to offer better discrimination performance based on the AUROC metric. The variability in performance, as indicated by the spread of the box plots, should also be considered when selecting a method for a specific application.

\n ## Box Plots: ECE and AUROC Difference Comparison ### Overview The image presents two box plots side-by-side, comparing the differences in Expected Calibration Error (ECE) and Area Under the Receiver Operating Characteristic curve (AUROC) across three different methods: Chain-of-Thought (CoT), Multi-Step, and Top-K. Each box plot visualizes the distribution of the difference between a baseline and each method. ### Components/Axes * **X-axis (Both Plots):** Method - CoT, Multi-Step, Top-K. * **Y-axis (Left Plot):** ECE Diff (approximately ranging from -55 to 5). * **Y-axis (Right Plot):** AUROC Diff (approximately ranging from -10 to 15). * **Box Plot Components:** Each box plot displays the median, quartiles (25th and 75th percentiles), and whiskers representing the range of the data, with potential outliers shown as individual points. * **Colors:** * CoT: Light Blue * Multi-Step: Orange/Red * Top-K: Green ### Detailed Analysis or Content Details **Left Plot: ECE Difference** * **CoT (Light Blue):** The median ECE difference is approximately -10. The box extends from roughly -20 to 0. Whiskers extend to approximately -45 and 5. * **Multi-Step (Orange/Red):** The median ECE difference is approximately -15. The box extends from roughly -25 to -10. Whiskers extend to approximately -50 and 0. * **Top-K (Green):** The median ECE difference is approximately -10. The box extends from roughly -20 to 0. Whiskers extend to approximately -40 and 5. **Right Plot: AUROC Difference** * **CoT (Light Blue):** The median AUROC difference is approximately 0. The box extends from roughly -5 to 5. Whiskers extend to approximately -10 and 10. * **Multi-Step (Orange/Red):** The median AUROC difference is approximately 2. The box extends from roughly -2 to 7. Whiskers extend to approximately -7 and 12. * **Top-K (Green):** The median AUROC difference is approximately 12. The box extends from roughly 7 to 15. Whiskers extend to approximately 2 and 18. ### Key Observations * **ECE:** Both Multi-Step and Top-K show a negative median ECE difference, indicating an improvement over the baseline. CoT also shows a negative median, but less pronounced than Multi-Step. * **AUROC:** Top-K demonstrates a significantly positive median AUROC difference, suggesting a substantial improvement. Multi-Step shows a modest positive difference, while CoT is close to zero. * **Variance:** The AUROC differences exhibit more variance than the ECE differences, as indicated by the longer whiskers and larger box sizes. * **Outliers:** There are some outliers visible in both plots, particularly for the Multi-Step method in the AUROC plot. ### Interpretation The data suggests that the Multi-Step and Top-K methods generally improve ECE, while Top-K significantly enhances AUROC performance compared to the baseline. CoT shows a slight improvement in ECE but has minimal impact on AUROC. The larger variance in AUROC differences indicates that the performance gains from Top-K and Multi-Step are more sensitive to the specific dataset or conditions. The presence of outliers suggests that there are instances where these methods perform substantially better or worse than the typical trend. The differences in ECE and AUROC suggest that these methods address different aspects of model calibration and performance. While Multi-Step and Top-K improve the model's confidence in its predictions (ECE), Top-K particularly excels at distinguishing between positive and negative cases (AUROC). This could be due to the Top-K method focusing on the most probable outputs, leading to better discrimination.

## Box Plot Comparison: ECE Diff and AUROC Diff ### Overview The image displays two side-by-side box plots comparing the distribution of difference scores for three methods (CoT, Multi-Step, Top-K) across two metrics: ECE (Expected Calibration Error) and AUROC (Area Under the Receiver Operating Characteristic curve). The plots are presented on a light gray background. ### Components/Axes **Left Plot:** * **Title:** "ECE" (centered at the top). * **Y-axis Label:** "ECE Diff" (rotated vertically on the left). * **Y-axis Scale:** Linear scale ranging from -50 to 0, with major tick marks at intervals of 10 (-50, -40, -30, -20, -10, 0). * **X-axis Categories:** Three categories labeled "CoT", "Multi-Step", and "Top-K" from left to right. * **Data Series (Box Plots):** * **CoT:** Blue box. * **Multi-Step:** Orange box. * **Top-K:** Green box. **Right Plot:** * **Title:** "AUROC" (centered at the top). * **Y-axis Label:** "AUROC Diff" (rotated vertically on the left). * **Y-axis Scale:** Linear scale ranging from -10 to 15, with major tick marks at intervals of 5 (-10, -5, 0, 5, 10, 15). * **X-axis Categories:** Same three categories as the left plot: "CoT", "Multi-Step", "Top-K". * **Data Series (Box Plots):** * **CoT:** Blue box. * **Multi-Step:** Orange box. * **Top-K:** Green box. **Legend/Spatial Grounding:** The legend is not a separate element. The method names ("CoT", "Multi-Step", "Top-K") are placed directly below their respective box plots on the x-axis. The color coding is consistent across both plots: Blue = CoT, Orange = Multi-Step, Green = Top-K. ### Detailed Analysis **Left Plot: ECE Diff** * **Trend Verification:** All three distributions are centered below zero, indicating a negative difference. The CoT distribution appears to have the widest overall range, while Top-K has the most compact interquartile range (IQR). * **CoT (Blue):** * Median (line inside box): Approximately -8. * Interquartile Range (IQR, box height): Spans from approximately -24 (25th percentile) to -4 (75th percentile). * Whiskers: Extend from a maximum of approximately +5 to a minimum of approximately -55. * **Multi-Step (Orange):** * Median: Approximately -18. * IQR: Spans from approximately -21 to -9. * Whiskers: Extend from approximately +3 to -44. * **Top-K (Green):** * Median: Approximately -21. * IQR: Spans from approximately -25 to -12. * Whiskers: Extend from approximately -11 to -46. **Right Plot: AUROC Diff** * **Trend Verification:** The distributions show a clear upward trend from CoT to Top-K. CoT is centered near zero, Multi-Step is positive, and Top-K is strongly positive. * **CoT (Blue):** * Median: Approximately -1. * IQR: Very narrow, spanning from approximately -2 to 0. * Whiskers: Extend from approximately +7 to -10. * **Multi-Step (Orange):** * Median: Approximately +3. * IQR: Spans from approximately -2 to +6. * Whiskers: Extend from approximately +15 to -5. * **Top-K (Green):** * Median: Approximately +8. * IQR: Spans from approximately +5 to +16. * Whiskers: Extend from approximately +17 to -6. ### Key Observations 1. **Metric Directionality:** The "ECE Diff" values are predominantly negative for all methods, while the "AUROC Diff" values are predominantly positive for Multi-Step and Top-K. 2. **Performance Ranking:** For AUROC Diff, there is a clear performance hierarchy: Top-K > Multi-Step > CoT. For ECE Diff, the medians are all negative and relatively close, with CoT showing the least negative median (closest to zero) but the highest variance. 3. **Variance:** CoT exhibits the largest variance (longest whiskers and tallest box) in the ECE Diff plot. In the AUROC Diff plot, Top-K shows the largest positive variance. 4. **Outliers:** No individual data points (outliers) are plotted beyond the whiskers in either chart. ### Interpretation This visualization compares the impact of three reasoning or sampling methods (Chain-of-Thought, Multi-Step, Top-K) on two key model performance metrics. The "Diff" likely represents the change in ECE and AUROC relative to a baseline (e.g., standard prompting). * **ECE Interpretation:** A negative ECE Diff suggests an improvement in calibration (lower ECE is better). All methods improve calibration, with Multi-Step and Top-K showing a slightly larger median improvement than CoT. However, CoT's performance is highly variable, sometimes leading to much larger improvements (whisker to -55) and sometimes to slight degradation (whisker to +5). * **AUROC Interpretation:** A positive AUROC Diff indicates improved discriminative performance (higher AUROC is better). Here, the methods show a clear stratified benefit: Top-K provides the largest and most consistent boost, followed by Multi-Step. CoT, on average, provides negligible change to AUROC but has high variance, with outcomes ranging from significant degradation (-10) to moderate improvement (+7). * **Overall Relationship:** The data suggests a potential trade-off or divergence in metric outcomes. While all methods tend to improve calibration (ECE), their effect on discriminative power (AUROC) varies dramatically. Top-K appears to be the most effective for boosting AUROC while also solidly improving calibration. CoT is the least reliable for improving AUROC and shows the most unpredictable results for calibration. This could imply that the mechanisms by which these methods alter model confidence (affecting ECE) and decision boundaries (affecting AUROC) are distinct.

## Box Plots: ECE and AUROC Performance Comparison ### Overview The image contains two side-by-side box plots comparing performance metrics (ECE and AUROC) across three methods: Chain-of-Thought (CoT), Multi-Step, and Top-K. Each plot uses distinct colors (blue, orange, green) to represent the methods, with legends positioned to the right of each plot. ### Components/Axes - **Left Plot (ECE)**: - **Y-Axis**: "ECE Diff" (Error Calibration Error Difference), ranging from -50 to 0. - **X-Axis**: Methods labeled "CoT", "Multi-Step", "Top-K". - **Legend**: Blue = CoT, Orange = Multi-Step, Green = Top-K. - **Right Plot (AUROC)**: - **Y-Axis**: "AUROC Diff" (Area Under the Receiver Operating Characteristic Curve Difference), ranging from -10 to 15. - **X-Axis**: Same methods as the left plot. - **Legend**: Same color coding as the left plot. ### Detailed Analysis #### ECE Plot - **CoT (Blue)**: - Median: ~-15. - Range: -50 (minimum) to 0 (maximum). - Interquartile Range (IQR): ~-20 to -10. - **Multi-Step (Orange)**: - Median: ~-20. - Range: -40 to 0. - IQR: ~-25 to -15. - **Top-K (Green)**: - Median: ~-25. - Range: -30 to -10. - IQR: ~-28 to -22. #### AUROC Plot - **CoT (Blue)**: - Median: ~-2. - Range: -10 to 5. - IQR: ~-4 to -1. - **Multi-Step (Orange)**: - Median: ~2. - Range: -5 to 15. - IQR: ~-2 to 5. - **Top-K (Green)**: - Median: ~8. - Range: 0 to 15. - IQR: ~5 to 12. ### Key Observations 1. **ECE Performance**: - Top-K consistently shows the lowest (most negative) ECE values, indicating better calibration. - CoT has the widest spread, suggesting higher variability in error calibration. 2. **AUROC Performance**: - Top-K achieves the highest median AUROC (~8) and the largest range (0–15), indicating superior discriminative ability. - CoT underperforms with a median of ~-2 and a narrower range (-10 to 5). 3. **Method Trends**: - Both metrics show Top-K outperforming other methods. - ECE values are uniformly negative, while AUROC values span both negative and positive ranges. ### Interpretation The data suggests that the **Top-K** method is the most effective across both evaluation metrics. In ECE, Top-K’s lower (more negative) values imply better calibration of predicted probabilities, while in AUROC, its higher values reflect stronger model discrimination between classes. The wider spread in Top-K’s AUROC indicates greater variability in performance, possibly due to sampling strategies or dataset characteristics. CoT and Multi-Step methods exhibit suboptimal performance, with CoT showing the least consistency in ECE and AUROC. These results may guide method selection in applications requiring reliable probabilistic predictions and robust classification.