## Scatter Plot Comparison: Model Performance ### Overview The image presents two scatter plots comparing the performance of three models: a Zero-Shot Classifier (red), a Verbal model (blue), and a Fine-tuned model (black dashed line). The left plot shows the relationship between Accuracy (x-axis) and ECE (Expected Calibration Error, y-axis), while the right plot shows the relationship between Accuracy (x-axis) and AUROC (Area Under the Receiver Operating Characteristic curve, y-axis). Each plot includes a regression line with a shaded confidence interval for the Zero-Shot Classifier and Verbal models. ### Components/Axes * **Legend:** Located at the top of the image. * Zero-Shot Classifier: Represented by red circles and a red regression line with a pink shaded confidence interval. * Verbal: Represented by blue circles and a blue regression line with a light blue shaded confidence interval. * Fine-tune: Represented by a black dashed horizontal line. * **Left Plot (ECE vs. Accuracy):** * Y-axis (ECE): Labeled "ECE" with a range from 0% to 60%, with tick marks at 0%, 20%, 40%, and 60%. * X-axis (Accuracy): Labeled "Accuracy" with a range from 35% to 50%, with tick marks at 35%, 40%, 45%, and 50%. * **Right Plot (AUROC vs. Accuracy):** * Y-axis (AUROC): Labeled "AUROC" with a range from 50% to 70%, with tick marks at 50%, 60%, and 70%. * X-axis (Accuracy): Labeled "Accuracy" with a range from 35% to 50%, with tick marks at 35%, 40%, 45%, and 50%. * Fine-tune: Represented by a black dashed horizontal line at approximately 72% AUROC. ### Detailed Analysis **Left Plot (ECE vs. Accuracy):** * **Zero-Shot Classifier (Red):** * Trend: Slightly positive, but relatively flat. * Data Points: Scattered across the plot. Approximate data points: (35%, 20%), (35%, 60%), (37%, 20%), (40%, 25%), (40%, 60%), (45%, 40%), (50%, 25%), (50%, 50%), (52%, 50%). * **Verbal (Blue):** * Trend: Slightly positive. * Data Points: Clustered around 40% ECE. Approximate data points: (35%, 45%), (37%, 40%), (42%, 40%), (45%, 42%), (52%, 40%). * **Fine-tune (Black Dashed Line):** * Constant ECE at approximately 5%. **Right Plot (AUROC vs. Accuracy):** * **Zero-Shot Classifier (Red):** * Trend: Positive. * Data Points: Approximate data points: (35%, 52%), (37%, 55%), (40%, 55%), (42%, 54%), (45%, 58%), (50%, 60%), (52%, 62%). * **Verbal (Blue):** * Trend: Positive. * Data Points: Approximate data points: (35%, 55%), (37%, 53%), (42%, 58%), (45%, 60%), (50%, 62%). * **Fine-tune (Black Dashed Line):** * Constant AUROC at approximately 72%. ### Key Observations * In the ECE vs. Accuracy plot, the Fine-tuned model has a significantly lower ECE than both the Zero-Shot Classifier and Verbal models, indicating better calibration. * In the AUROC vs. Accuracy plot, the Fine-tuned model has a higher AUROC than both the Zero-Shot Classifier and Verbal models, indicating better discrimination. * The Verbal model generally has a lower ECE and a higher AUROC than the Zero-Shot Classifier, suggesting better overall performance. * The accuracy range is relatively narrow, between 35% and 50%. ### Interpretation The plots suggest that fine-tuning leads to a model with superior calibration (lower ECE) and discrimination (higher AUROC) compared to the Zero-Shot Classifier and Verbal models. The Verbal model appears to offer a performance improvement over the Zero-Shot Classifier, but neither approaches the performance of the Fine-tuned model. The relatively flat trends for the Zero-Shot Classifier and Verbal models in the ECE plot suggest that increasing accuracy does not necessarily improve calibration for these models. The positive trends in the AUROC plot indicate that increasing accuracy does improve discrimination for all models. The Fine-tune model's horizontal line indicates that its performance is independent of the "Accuracy" metric shown on the x-axis.

\n ## Scatter Plots: Performance Comparison of Classifiers ### Overview The image presents two scatter plots comparing the performance of a "Zero-Shot Classifier" and a "Verbal" model, against a "Fine-tune" baseline. The plots visualize the relationship between Accuracy and two different metrics: Expected Calibration Error (ECE) in the left plot, and Area Under the Receiver Operating Characteristic curve (AUROC) in the right plot. Each plot includes a regression line with a shaded confidence interval for each model type. ### Components/Axes * **X-axis (Both Plots):** Accuracy, ranging from 35% to 50%, with markers at 35%, 40%, 45%, and 50%. * **Y-axis (Left Plot):** Expected Calibration Error (ECE), ranging from 0% to 60%, with markers at 0%, 20%, 40%, and 60%. * **Y-axis (Right Plot):** Area Under the ROC Curve (AUROC), ranging from 50% to 70%, with markers at 50%, 55%, 60%, 65%, and 70%. * **Legend (Top-Center):** * Pink circles: Zero-Shot Classifier * Blue circles: Verbal * Black dashed line: Fine-tune * **Horizontal Dashed Line (Both Plots):** Represents the Fine-tune baseline. The line is at 0% ECE for the left plot and 60% AUROC for the right plot. ### Detailed Analysis or Content Details **Left Plot (ECE vs. Accuracy):** * **Fine-tune Baseline:** A horizontal dashed black line at approximately 0% ECE. * **Zero-Shot Classifier (Pink):** The regression line slopes slightly upwards. * Approximate data points (visually estimated): * Accuracy 35%: ECE ~ 55% * Accuracy 40%: ECE ~ 45% * Accuracy 45%: ECE ~ 35% * Accuracy 50%: ECE ~ 25% * **Verbal (Blue):** The regression line is relatively flat. * Approximate data points (visually estimated): * Accuracy 35%: ECE ~ 42% * Accuracy 40%: ECE ~ 40% * Accuracy 45%: ECE ~ 38% * Accuracy 50%: ECE ~ 36% **Right Plot (AUROC vs. Accuracy):** * **Fine-tune Baseline:** A horizontal dashed black line at approximately 60% AUROC. * **Zero-Shot Classifier (Pink):** The regression line slopes upwards. * Approximate data points (visually estimated): * Accuracy 35%: AUROC ~ 55% * Accuracy 40%: AUROC ~ 58% * Accuracy 45%: AUROC ~ 62% * Accuracy 50%: AUROC ~ 65% * **Verbal (Blue):** The regression line slopes slightly upwards. * Approximate data points (visually estimated): * Accuracy 35%: AUROC ~ 55% * Accuracy 40%: AUROC ~ 57% * Accuracy 45%: AUROC ~ 60% * Accuracy 50%: AUROC ~ 62% ### Key Observations * In both plots, the Zero-Shot Classifier exhibits a positive correlation between Accuracy and the performance metric (ECE and AUROC). As Accuracy increases, ECE decreases and AUROC increases. * The Verbal model shows a weaker correlation. Its performance is relatively stable across the range of Accuracy values. * The Zero-Shot Classifier consistently performs worse than the Fine-tune baseline in terms of ECE (left plot), but performs similarly to the Fine-tune baseline in terms of AUROC (right plot). * The confidence intervals (shaded areas) around the regression lines indicate the variability in the data. ### Interpretation The data suggests that while the Zero-Shot Classifier's performance improves with increasing Accuracy, it suffers from calibration issues (high ECE). This means that its predicted probabilities are not well-aligned with the actual observed frequencies. However, its ability to discriminate between classes (AUROC) is comparable to a Fine-tuned model. The Verbal model appears to be more stable and well-calibrated, but its overall performance is not as sensitive to changes in Accuracy. The Fine-tune baseline provides a benchmark for expected performance. The Zero-Shot Classifier's ECE is significantly higher than the baseline, indicating a potential drawback. The AUROC values are close to the baseline, suggesting that the Zero-Shot Classifier can achieve similar discriminatory power with appropriate calibration adjustments. The plots highlight a trade-off between calibration and discrimination. The Zero-Shot Classifier excels in discrimination but requires calibration, while the Verbal model is well-calibrated but less discriminative. The choice of model depends on the specific application and the relative importance of these two factors.

## Scatter Plots: Zero-Shot Classifier vs. Verbal vs. Fine-tune Performance ### Overview The image displays two side-by-side scatter plots comparing the performance of three classification methods: "Zero-Shot Classifier," "Verbal," and "Fine-tune." The left plot evaluates Expected Calibration Error (ECE) against Accuracy, while the right plot evaluates Area Under the ROC Curve (AUROC) against Accuracy. Each plot includes individual data points, a linear regression trend line with a shaded confidence interval for the first two methods, and a horizontal dashed reference line for the "Fine-tune" method. ### Components/Axes * **Legend:** Positioned at the top center of the entire figure. * Pink circle: `Zero-Shot Classifier` * Blue circle: `Verbal` * Black dashed line: `Fine-tune` * **Left Plot (ECE vs. Accuracy):** * **Y-axis:** Label is `ECE`. Scale ranges from 0% to 60%, with major ticks at 0%, 20%, 40%, 60%. * **X-axis:** Label is `Accuracy`. Scale ranges from 35% to 50%, with major ticks at 35%, 40%, 45%, 50%. * **Right Plot (AUROC vs. Accuracy):** * **Y-axis:** Label is `AUROC`. Scale ranges from 50% to 70%, with major ticks at 50%, 60%, 70%. * **X-axis:** Label is `Accuracy`. Scale ranges from 35% to 50%, with major ticks at 35%, 40%, 45%, 50%. * **Data Series & Visual Elements:** * **Zero-Shot Classifier (Pink):** Individual pink dots scattered across the plot area. A solid pink regression line with a light pink shaded confidence interval is drawn through the data. * **Verbal (Blue):** Individual blue dots scattered across the plot area. A solid blue regression line with a light blue shaded confidence interval is drawn through the data. * **Fine-tune (Black Dashed):** A horizontal dashed black line, indicating a constant performance level for this method across the accuracy range shown. ### Detailed Analysis **Left Plot: ECE (Lower is Better)** * **Trend Verification:** Both the pink (Zero-Shot) and blue (Verbal) regression lines show a slight upward slope, suggesting a weak positive correlation between Accuracy and ECE for these methods. * **Data Points (Approximate):** * **Zero-Shot Classifier (Pink):** Points are widely scattered. Values range from approximately 10% to 60% ECE. Notable points include a cluster near 40% Accuracy/20% ECE and another near 50% Accuracy/50% ECE. * **Verbal (Blue):** Points are more tightly clustered than Zero-Shot. Values range from approximately 30% to 50% ECE. * **Fine-tune (Black Dashed Line):** Constant at approximately **5% ECE**, significantly lower than the other two methods across the entire accuracy range. **Right Plot: AUROC (Higher is Better)** * **Trend Verification:** Both the pink (Zero-Shot) and blue (Verbal) regression lines show a clear upward slope, indicating a positive correlation between Accuracy and AUROC. * **Data Points (Approximate):** * **Zero-Shot Classifier (Pink):** Points range from approximately 50% to 65% AUROC. There is a visible upward trend. * **Verbal (Blue):** Points range from approximately 55% to 62% AUROC, also showing an upward trend. * **Fine-tune (Black Dashed Line):** Constant at approximately **72% AUROC**, which is higher than all data points for the other two methods. ### Key Observations 1. **Superior Performance of Fine-tuning:** The "Fine-tune" method (dashed line) demonstrates both the best calibration (lowest ECE ~5%) and the best discriminative performance (highest AUROC ~72%) consistently, independent of the accuracy range plotted. 2. **Calibration vs. Discrimination Trade-off:** For the Zero-Shot and Verbal methods, higher Accuracy is associated with *worse* calibration (higher ECE) but *better* discrimination (higher AUROC). 3. **Variability:** The Zero-Shot Classifier shows significantly higher variance in ECE compared to the Verbal method, suggesting less consistent calibration. 4. **Performance Clustering:** The Verbal method's data points are more tightly clustered than the Zero-Shot method's, indicating more predictable performance. ### Interpretation This data suggests a fundamental trade-off between model calibration and raw discriminative power when using prompt-based (Zero-Shot, Verbal) methods versus a fully fine-tuned model. The fine-tuned model achieves a superior balance, excelling in both metrics. The positive correlation between Accuracy and AUROC is expected, as both measure aspects of correct classification. However, the simultaneous positive correlation between Accuracy and ECE for the prompt-based methods is a critical finding. It indicates that as these models become more accurate on this test set, they also become more *overconfident* in their predictions (higher ECE). This is a known issue with large language models used as zero-shot classifiers. The "Fine-tune" line acts as a gold-standard benchmark. The fact that it is horizontal implies its performance is stable and serves as a target. The gap between the dashed line and the scatter points quantifies the performance cost of using prompt-based methods instead of task-specific fine-tuning for this particular evaluation. The wider scatter of the Zero-Shot method highlights the instability and sensitivity of pure prompting compared to the more structured "Verbal" method (which may involve more engineered prompts or a specific verbalization format).

## Scatter Plots: ECE vs Accuracy and AUROC vs Accuracy ### Overview The image contains two side-by-side scatter plots comparing model performance metrics (ECE and AUROC) against accuracy. Both plots show data points for two classifier types (Zero-Shot and Verbal) with a reference line labeled "Fine-tune." The plots demonstrate relationships between accuracy and calibration (ECE) and discriminative power (AUROC). ### Components/Axes **Left Plot (ECE vs Accuracy):** - **X-axis**: Accuracy (35% to 50%, labeled in 5% increments) - **Y-axis**: Expected Calibration Error (ECE, 0% to 60%) - **Legend**: - Pink circles: Zero-Shot Classifier - Blue circles: Verbal - Dashed black line: Fine-tune - **Visual Elements**: - Shaded pink region around Zero-Shot line (confidence interval) - Shaded blue region around Verbal line **Right Plot (AUROC vs Accuracy):** - **X-axis**: Accuracy (35% to 50%, same scale as left plot) - **Y-axis**: Area Under Receiver Operating Characteristic Curve (AUROC, 50% to 70%) - **Legend**: Same as left plot - **Visual Elements**: - Shaded pink region around Zero-Shot line - Shaded blue region around Verbal line ### Detailed Analysis **Left Plot (ECE):** - **Zero-Shot Classifier (pink)**: - Data points cluster between 20-40% ECE - Line shows slight upward trend (ECE increases with accuracy) - Shaded region spans ~10% of ECE values - **Verbal (blue)**: - Data points cluster between 30-45% ECE - Line shows stronger upward trend than Zero-Shot - Shaded region spans ~15% of ECE values - **Fine-tune line**: Horizontal dashed line at ~35% ECE **Right Plot (AUROC):** - **Zero-Shot Classifier (pink)**: - Data points cluster between 50-60% AUROC - Line shows moderate upward trend - Shaded region spans ~5% of AUROC values - **Verbal (blue)**: - Data points cluster between 55-65% AUROC - Line shows stronger upward trend than Zero-Shot - Shaded region spans ~10% of AUROC values - **Fine-tune line**: Horizontal dashed line at ~70% AUROC ### Key Observations 1. **Positive Correlation**: Both ECE and AUROC increase with accuracy in all models 2. **Verbal Advantage**: Verbal models consistently outperform Zero-Shot in both metrics at similar accuracy levels 3. **Calibration vs Discrimination**: - Verbal models show better calibration (lower ECE) and higher AUROC - Zero-Shot models demonstrate more variability in performance 4. **Fine-tune Thresholds**: - ECE target: ~35% (dashed line) - AUROC target: ~70% (dashed line) 5. **Shaded Regions**: Indicate measurement uncertainty, with Verbal showing greater variability ### Interpretation The data suggests that Verbal models achieve better performance across both calibration and discrimination metrics compared to Zero-Shot models at equivalent accuracy levels. The upward trends indicate that higher accuracy generally improves both calibration and discriminative power. However, the shaded regions reveal that individual model performance varies significantly, particularly for Verbal models which show wider confidence intervals. The Fine-tune reference lines establish clear performance benchmarks, with Verbal models approaching or exceeding these targets in both metrics. The consistent outperformance of Verbal models suggests architectural or training advantages over Zero-Shot approaches in this evaluation framework.