## Chart: Model Performance Comparison ### Overview The image presents two line charts comparing the performance of different language models (LLaMA-2 7B Chat, LLaMA-2 13B Chat, and Mistral 7B Instruct) across varying sample sizes. The left chart displays the Expected Calibration Error (ECE), while the right chart shows the Area Under the Receiver Operating Characteristic Curve (AUROC). Baseline performance is indicated by horizontal dashed lines for "Zero-Shot Classifier" and "Sampling". ### Components/Axes * **X-axis (both charts):** Samples (logarithmic scale), ranging from 10² to 10⁴. * **Y-axis (left chart):** ECE, ranging from 0.1 to 0.2. * **Y-axis (right chart):** AUROC, ranging from 0.6 to 0.7. * **Legend (top):** * Dark Blue: LLaMA-2 7B Chat * Light Blue: LLaMA-2 13B Chat * Teal: Mistral 7B Instruct * Red Dashed: Zero-Shot Classifier * Purple Dashed: Sampling ### Detailed Analysis **Left Chart: ECE** * **LLaMA-2 7B Chat (Dark Blue):** The ECE starts around 0.14 at 10² samples and decreases to approximately 0.07 at 10³ samples. It then plateaus and slightly increases to around 0.08 at 10⁴ samples. * **LLaMA-2 13B Chat (Light Blue):** The ECE starts around 0.17 at 10² samples and decreases to approximately 0.08 at 10³ samples. It then plateaus and slightly increases to around 0.09 at 10⁴ samples. * **Mistral 7B Instruct (Teal):** The ECE starts around 0.23 at 10² samples and decreases to approximately 0.10 at 10³ samples. It then plateaus and slightly increases to around 0.11 at 10⁴ samples. * **Zero-Shot Classifier (Red Dashed):** The ECE is constant at approximately 0.14. * **Sampling (Purple Dashed):** The ECE is constant at approximately 0.13. **Right Chart: AUROC** * **LLaMA-2 7B Chat (Dark Blue):** The AUROC starts around 0.55 at 10² samples and increases to approximately 0.72 at 10⁴ samples. * **LLaMA-2 13B Chat (Light Blue):** The AUROC starts around 0.58 at 10² samples and increases to approximately 0.73 at 10⁴ samples. * **Mistral 7B Instruct (Teal):** The AUROC starts around 0.60 at 10² samples and increases to approximately 0.74 at 10⁴ samples. * **Zero-Shot Classifier (Red Dashed):** The AUROC is constant at approximately 0.59. * **Sampling (Purple Dashed):** The AUROC is constant at approximately 0.53. ### Key Observations * As the number of samples increases, the ECE generally decreases for all three language models, indicating better calibration. * As the number of samples increases, the AUROC generally increases for all three language models, indicating better classification performance. * Mistral 7B Instruct generally outperforms the LLaMA-2 models in both ECE and AUROC, especially at lower sample sizes. * The Zero-Shot Classifier and Sampling baselines remain constant across all sample sizes. ### Interpretation The charts demonstrate the impact of increasing the number of samples on the performance of different language models. The decreasing ECE and increasing AUROC values suggest that with more data, the models become better calibrated and more accurate in their classifications. The Mistral 7B Instruct model appears to be the most effective among the three, showing superior performance compared to the LLaMA-2 models. The horizontal baselines provide a reference point, highlighting the improvement achieved by the language models compared to simpler classification methods. The logarithmic scale on the x-axis suggests that the initial increase in samples has a more significant impact on performance than later increases.

## Line Chart: Model Calibration and Performance vs. Sample Size ### Overview The image presents two line charts comparing the calibration and performance of different Large Language Models (LLMs) – LLaMA-2 (7B Chat and 13B Chat) and Mistral (7B Instruct) – as a function of the number of samples used. The left chart displays Expected Calibration Error (ECE), while the right chart shows Area Under the Receiver Operating Characteristic curve (AUROC). Both charts include baseline performance metrics for Zero-Shot Classifier and Sampling methods. ### Components/Axes * **X-axis (Both Charts):** "Samples" - Logarithmic scale, ranging from 10² to 10⁴. * **Left Chart Y-axis:** "ECE" - Ranging from 0.0 to 0.25. * **Right Chart Y-axis:** "AUROC" - Ranging from 0.5 to 0.8. * **Legend (Top-Center):** * LLaMA-2 7B Chat (Dark Blue Solid Line) * LLaMA-2 13B Chat (Light Blue Solid Line) * Mistral 7B Instruct (Teal Solid Line) * Zero-Shot Classifier (Red Dashed Line) * Sampling (Red Dashed Line) ### Detailed Analysis or Content Details **Left Chart (ECE):** * **LLaMA-2 7B Chat (Dark Blue):** Starts at approximately ECE = 0.22 at 10² samples, decreases to approximately ECE = 0.09 at 10³ samples, and stabilizes around ECE = 0.08 at 10⁴ samples. * **LLaMA-2 13B Chat (Light Blue):** Starts at approximately ECE = 0.21 at 10² samples, decreases to approximately ECE = 0.09 at 10³ samples, and stabilizes around ECE = 0.08 at 10⁴ samples. * **Mistral 7B Instruct (Teal):** Starts at approximately ECE = 0.23 at 10² samples, decreases to approximately ECE = 0.11 at 10³ samples, and stabilizes around ECE = 0.09 at 10⁴ samples. * **Zero-Shot Classifier (Red Dashed):** Horizontal line at approximately ECE = 0.16. * **Sampling (Red Dashed):** Horizontal line at approximately ECE = 0.16. **Right Chart (AUROC):** * **LLaMA-2 7B Chat (Dark Blue):** Starts at approximately AUROC = 0.62 at 10² samples, increases to approximately AUROC = 0.72 at 10³ samples, and stabilizes around AUROC = 0.74 at 10⁴ samples. * **LLaMA-2 13B Chat (Light Blue):** Starts at approximately AUROC = 0.64 at 10² samples, increases to approximately AUROC = 0.74 at 10³ samples, and stabilizes around AUROC = 0.76 at 10⁴ samples. * **Mistral 7B Instruct (Teal):** Starts at approximately AUROC = 0.66 at 10² samples, increases to approximately AUROC = 0.75 at 10³ samples, and stabilizes around AUROC = 0.77 at 10⁴ samples. * **Zero-Shot Classifier (Red Dashed):** Horizontal line at approximately AUROC = 0.61. * **Sampling (Red Dashed):** Horizontal line at approximately AUROC = 0.61. ### Key Observations * All models show a decreasing ECE with increasing sample size, indicating improved calibration. * All models show an increasing AUROC with increasing sample size, indicating improved performance. * Mistral 7B Instruct generally exhibits slightly higher AUROC values than the LLaMA-2 models. * The LLaMA-2 13B Chat model performs slightly better than the 7B Chat model in both ECE and AUROC. * The Zero-Shot Classifier and Sampling baselines perform consistently worse than all the LLMs across both metrics. ### Interpretation The data suggests that increasing the number of samples used for evaluation improves both the calibration (ECE) and performance (AUROC) of all three LLMs. This is expected, as more samples provide a more robust estimate of the model's true capabilities. The Mistral 7B Instruct model appears to be slightly better calibrated and more performant than the LLaMA-2 models, particularly at larger sample sizes. The consistent underperformance of the Zero-Shot Classifier and Sampling baselines highlights the benefits of using fine-tuned LLMs for this task. The convergence of the lines at 10⁴ samples suggests that the models are approaching a point of diminishing returns in terms of calibration and performance gains with further increases in sample size. The difference between ECE and AUROC provides a nuanced view of model quality: a low ECE indicates that the model's predicted probabilities are well-aligned with its actual accuracy, while a high AUROC indicates that the model is generally good at distinguishing between positive and negative examples.

\n ## Line Charts: Model Calibration (ECE) and Classification Performance (AUROC) vs. Training Samples ### Overview The image displays two side-by-side line charts comparing the performance of three large language models (LLMs) as a function of the number of training samples used. The left chart measures Expected Calibration Error (ECE), and the right chart measures Area Under the Receiver Operating Characteristic curve (AUROC). Both charts include baseline performance lines for a "Zero-Shot Classifier" and a "Sampling" method. ### Components/Axes * **Legend (Top Center):** A shared legend identifies three model series: * **LLaMA-2 7B Chat:** Dark purple solid line. * **LLaMA-2 13B Chat:** Blue solid line. * **Mistral 7B Instruct:** Teal solid line. * **Baseline Legend (Below Main Legend):** Identifies two horizontal dashed lines: * **Zero-Shot Classifier:** Red dashed line. * **Sampling:** Purple dashed line. * **Left Chart (ECE):** * **Y-axis:** Label "ECE". Scale ranges from approximately 0.05 to 0.25. Major ticks at 0.1 and 0.2. * **X-axis:** Label "Samples". Logarithmic scale with major ticks at 10², 10³, and 10⁴. * **Right Chart (AUROC):** * **Y-axis:** Label "AUROC". Scale ranges from approximately 0.55 to 0.75. Major ticks at 0.6 and 0.7. * **X-axis:** Label "Samples". Identical logarithmic scale to the left chart (10², 10³, 10⁴). * **Data Series:** Each model series is plotted with a shaded region around the central line, indicating confidence intervals or variance. ### Detailed Analysis **Left Chart - ECE (Lower is Better):** * **Trend Verification:** All three model lines show a clear downward trend as the number of samples increases, indicating improved calibration (lower error). * **Data Points (Approximate):** * **LLaMA-2 7B Chat (Dark Purple):** Starts at ~0.15 (10² samples), decreases to ~0.10 (10³ samples), and ends at ~0.08 (10⁴ samples). * **LLaMA-2 13B Chat (Blue):** Starts at ~0.14 (10² samples), decreases to ~0.09 (10³ samples), and ends at ~0.07 (10⁴ samples). * **Mistral 7B Instruct (Teal):** Starts highest at ~0.22 (10² samples), decreases sharply to ~0.12 (10³ samples), and ends at ~0.09 (10⁴ samples). It shows a slight upward bump between 10³ and 10⁴ samples. * **Baselines (Horizontal Dashed Lines):** * **Zero-Shot Classifier (Red):** Constant at ~0.15. * **Sampling (Purple):** Constant at ~0.14. * **Spatial Grounding:** The baselines are positioned in the upper half of the chart. All model lines start near or above these baselines at 10² samples and fall significantly below them by 10⁴ samples. **Right Chart - AUROC (Higher is Better):** * **Trend Verification:** All three model lines show a clear upward trend as the number of samples increases, indicating improved classification performance. * **Data Points (Approximate):** * **LLaMA-2 7B Chat (Dark Purple):** Starts at ~0.60 (10² samples), increases to ~0.68 (10³ samples), and ends at ~0.72 (10⁴ samples). * **LLaMA-2 13B Chat (Blue):** Starts at ~0.58 (10² samples), increases to ~0.66 (10³ samples), and ends at ~0.70 (10⁴ samples). * **Mistral 7B Instruct (Teal):** Starts at ~0.64 (10² samples), increases to ~0.70 (10³ samples), and ends highest at ~0.74 (10⁴ samples). * **Baselines (Horizontal Dashed Lines):** * **Zero-Shot Classifier (Red):** Constant at ~0.60. * **Sampling (Purple):** Constant at ~0.56. * **Spatial Grounding:** The baselines are positioned in the lower half of the chart. All model lines start at or above the Zero-Shot baseline and end well above both baselines. ### Key Observations 1. **Inverse Relationship:** There is a clear inverse relationship between ECE and AUROC for all models; as performance (AUROC) improves with more data, calibration error (ECE) decreases. 2. **Model Comparison:** Mistral 7B Instruct starts with the worst calibration (highest ECE) but best initial performance (highest AUROC) at low samples (10²). By 10⁴ samples, LLaMA-2 13B Chat achieves the best calibration (lowest ECE), while Mistral achieves the best performance (highest AUROC). 3. **Data Efficiency:** All models surpass the "Zero-Shot Classifier" baseline in both metrics with as few as 10² samples. They surpass the "Sampling" baseline shortly thereafter. 4. **Convergence:** The performance gap between models narrows as the number of samples increases, particularly for ECE. ### Interpretation This data demonstrates the critical impact of fine-tuning sample size on both the reliability (calibration) and effectiveness (discriminative power) of LLMs for classification tasks. * **Calibration vs. Performance:** The charts show that calibration (ECE) and raw performance (AUROC) are related but distinct axes of model quality. A model can be well-performing but poorly calibrated, or vice-versa, especially in low-data regimes. * **Value of Data:** The consistent trends indicate that increasing the fine-tuning dataset size from 100 to 10,000 samples yields significant, monotonic benefits for both metrics across all tested models. This suggests the models are not yet saturated at 10⁴ samples. * **Model Selection Implications:** The choice between LLaMA-2 and Mistral may depend on the application's priority. If calibration is paramount (e.g., for risk assessment), LLaMA-2 13B Chat appears superior with sufficient data. If maximizing discriminative power is the sole goal, Mistral 7B Instruct shows a slight edge at high sample counts. * **Baseline Context:** The "Zero-Shot" and "Sampling" baselines provide a crucial reference point, showing that even minimal fine-tuning (100 samples) provides a substantial boost over these methods. The flat baselines highlight that these methods do not benefit from the additional training data being supplied to the other models.

## Line Chart: Model Performance vs. Sample Size ### Overview The image contains two line charts comparing the performance of different language models and methods across increasing sample sizes (10² to 10⁴). The left subplot measures Expected Calibration Error (ECE), while the right subplot measures Area Under the Receiver Operating Characteristic curve (AUROC). Performance is visualized with confidence intervals (shaded regions) and benchmark baselines (horizontal dashed lines). ### Components/Axes - **X-axis**: Samples (logarithmic scale: 10², 10³, 10⁴) - **Y-axis (Left)**: ECE (0.0 to 0.2) - **Y-axis (Right)**: AUROC (0.5 to 0.7) - **Legends**: - **Top-left**: Model variants (LLama-2 7B Chat, LLama-2 13B Chat, Mistral 7B Instruct) - **Top-right**: Method types (Zero-Shot Classifier, Sampling) - **Line styles**: - Solid lines: Model variants - Dashed lines: Benchmark baselines - Shaded regions: 95% confidence intervals ### Detailed Analysis #### ECE Subplot (Left) - **Zero-Shot Classifier (red dashed)**: Horizontal line at ~0.15 across all sample sizes. - **Sampling (purple dashed)**: Horizontal line at ~0.1 across all sample sizes. - **LLama-2 7B Chat (dark blue solid)**: - Starts at ~0.18 (10² samples), dips to ~0.12 (10³), then rises to ~0.14 (10⁴). - **LLama-2 13B Chat (blue solid)**: - Starts at ~0.16 (10²), dips to ~0.11 (10³), then rises to ~0.13 (10⁴). - **Mistral 7B Instruct (teal solid)**: - Starts at ~0.17 (10²), dips to ~0.10 (10³), then rises to ~0.12 (10⁴). #### AUROC Subplot (Right) - **Zero-Shot Classifier (red dashed)**: Horizontal line at ~0.6 across all sample sizes. - **Sampling (purple dashed)**: Horizontal line at ~0.55 across all sample sizes. - **LLama-2 7B Chat (dark blue solid)**: - Starts at ~0.58 (10²), rises to ~0.68 (10³), then plateaus at ~0.67 (10⁴). - **LLama-2 13B Chat (blue solid)**: - Starts at ~0.59 (10²), rises to ~0.72 (10³), then plateaus at ~0.71 (10⁴). - **Mistral 7B Instruct (teal solid)**: - Starts at ~0.61 (10²), rises to ~0.74 (10³), then plateaus at ~0.73 (10⁴). ### Key Observations 1. **Performance Trends**: - All models improve performance as sample size increases, approaching the Zero-Shot baseline. - Mistral 7B Instruct and LLama-2 13B Chat outperform the 7B variants in both metrics. - Sampling method underperforms compared to model-based approaches. 2. **Confidence Intervals**: - Shaded regions indicate variability, with wider intervals at lower sample sizes (10²) and narrowing as samples increase. 3. **Baseline Comparison**: - Both ECE and AUROC trends show models converging toward the Zero-Shot baseline as sample size grows, suggesting diminishing returns beyond ~10³ samples. ### Interpretation The data demonstrates that: - **Model scale matters**: The 13B variant of LLama-2 and Mistral 7B Instruct achieve higher AUROC and lower ECE than their 7B counterparts, indicating better generalization. - **Sample efficiency**: Performance gains are most pronounced between 10² and 10³ samples, with diminishing returns at 10⁴. - **Method limitations**: The Sampling approach lags behind model-based methods, suggesting it may not leverage model capacity effectively. - **Calibration vs. Accuracy**: While AUROC improves with scale, ECE trends show models becoming more calibrated (lower error) as they approach the Zero-Shot baseline. This suggests that larger models and instruction-tuned variants (e.g., Mistral) are more sample-efficient, but performance plateaus near the Zero-Shot baseline, highlighting the need for better alignment or training strategies to surpass this ceiling.