## Bar Chart: Model Performance on Various Subjects ### Overview The image is a series of bar charts comparing the performance of different language models (LLaMA-2 7B, LLaMA-2 7B Chat, LLaMA-2 13B, LLaMA-2 13B Chat, Mistral 7B, Mistral 7B Instruct) across a range of subjects. Performance is measured using ECE (Expected Calibration Error) and AUROC (Area Under the Receiver Operating Characteristic curve) metrics, with different training/prompting strategies (Zero-Shot Classifier, Probe, LoRA, LoRA + Prompt). ### Components/Axes * **X-axis:** Performance metrics (ECE and AUROC) ranging from 20% to 90%. Axis markers are present at 20%, 50%, 60%, and 90%. * **Y-axis:** Subjects, including abstract algebra, anatomy, astronomy, business ethics, clinical knowledge, college biology, college chemistry, college computer science, college mathematics, college medicine, college physics, computer security, conceptual physics, econometrics, electrical engineering, elementary mathematics, formal logic, global facts, high school biology, high school chemistry, high school computer science, high school european history, high school geography, high school government and politics, high school macroeconomics, high school mathematics, high school microeconomics, and high school physics. * **Chart Titles (Top):** LLaMA-2 7B, LLaMA-2 7B Chat, LLaMA-2 13B, LLaMA-2 13B Chat, Mistral 7B, Mistral 7B Instruct. These indicate the specific language model being evaluated in each chart. * **Legend (Bottom):** * Zero-Shot Classifier (Dark Red) * Probe (Light Purple) * LoRA (Medium Purple) * LoRA + Prompt (Dark Purple) ### Detailed Analysis Each subject has four bars representing the four different training/prompting strategies. The length of each bar corresponds to the performance metric (ECE or AUROC). Here's a breakdown of the performance for a few selected subjects, noting the trends: * **Abstract Algebra:** * LLaMA-2 7B: Zero-Shot Classifier ~20%, Probe ~30%, LoRA ~35%, LoRA + Prompt ~40% * LLaMA-2 7B Chat: Zero-Shot Classifier ~20%, Probe ~30%, LoRA ~35%, LoRA + Prompt ~40% * LLaMA-2 13B: Zero-Shot Classifier ~20%, Probe ~35%, LoRA ~40%, LoRA + Prompt ~45% * LLaMA-2 13B Chat: Zero-Shot Classifier ~20%, Probe ~35%, LoRA ~40%, LoRA + Prompt ~45% * Mistral 7B: Zero-Shot Classifier ~20%, Probe ~30%, LoRA ~35%, LoRA + Prompt ~40% * Mistral 7B Instruct: Zero-Shot Classifier ~20%, Probe ~30%, LoRA ~35%, LoRA + Prompt ~40% * **Anatomy:** * LLaMA-2 7B: Zero-Shot Classifier ~20%, Probe ~30%, LoRA ~35%, LoRA + Prompt ~40% * LLaMA-2 7B Chat: Zero-Shot Classifier ~20%, Probe ~30%, LoRA ~35%, LoRA + Prompt ~40% * LLaMA-2 13B: Zero-Shot Classifier ~20%, Probe ~35%, LoRA ~40%, LoRA + Prompt ~45% * LLaMA-2 13B Chat: Zero-Shot Classifier ~20%, Probe ~35%, LoRA ~40%, LoRA + Prompt ~45% * Mistral 7B: Zero-Shot Classifier ~20%, Probe ~30%, LoRA ~35%, LoRA + Prompt ~40% * Mistral 7B Instruct: Zero-Shot Classifier ~20%, Probe ~30%, LoRA ~35%, LoRA + Prompt ~40% * **Electrical Engineering:** * LLaMA-2 7B: Zero-Shot Classifier ~85%, Probe ~30%, LoRA ~35%, LoRA + Prompt ~40% * LLaMA-2 7B Chat: Zero-Shot Classifier ~85%, Probe ~30%, LoRA ~35%, LoRA + Prompt ~40% * LLaMA-2 13B: Zero-Shot Classifier ~85%, Probe ~35%, LoRA ~40%, LoRA + Prompt ~45% * LLaMA-2 13B Chat: Zero-Shot Classifier ~85%, Probe ~35%, LoRA ~40%, LoRA + Prompt ~45% * Mistral 7B: Zero-Shot Classifier ~85%, Probe ~30%, LoRA ~35%, LoRA + Prompt ~40% * Mistral 7B Instruct: Zero-Shot Classifier ~85%, Probe ~30%, LoRA ~35%, LoRA + Prompt ~40% ### Key Observations * **Zero-Shot Classifier Performance:** The Zero-Shot Classifier (dark red) often shows the highest performance in electrical engineering, but the lowest in other subjects. * **LoRA + Prompt Improvement:** LoRA + Prompt (dark purple) generally improves performance compared to Probe (light purple) and LoRA (medium purple) alone. * **Model Variation:** The LLaMA-2 13B and LLaMA-2 13B Chat models tend to perform slightly better than the 7B models across most subjects. * **Subject Sensitivity:** Performance varies significantly across subjects, indicating that some subjects are inherently easier or more aligned with the models' training data. ### Interpretation The data suggests that fine-tuning language models with LoRA and providing prompts can improve their performance on various subjects. The Zero-Shot Classifier performs well in some areas, but struggles in others, highlighting the importance of task-specific training. The differences between the 7B and 13B models indicate that model size can also impact performance. The variation across subjects suggests that the models have different levels of knowledge or proficiency in different domains. The "Chat" versions of the models do not show a significant performance difference compared to their base counterparts.

\n ## Heatmap: Model Performance Across Domains ### Overview The image presents a heatmap comparing the performance of several Large Language Models (LLMs) – LLaMA-2 (7B, 7B Chat, 13B, 13B Chat) and Mistral (7B, 7B Instruct) – across 24 different domains. Performance is measured using four different evaluation methods: Zero-Shot Classifier, Probe, LoRA, and LoRA + Prompt. The heatmap visualizes the Equalized Calibration Error (ECE) and Area Under the Receiver Operating Characteristic curve (AUROC) for each model-domain-evaluation method combination. ### Components/Axes * **Y-axis (Vertical):** Lists 24 domains, including: abstract\_algebra, anatomy, astronomy, business\_ethics, clinical\_knowledge, college\_biology, college\_chemistry, college\_computer\_science, college\_mathematics, college\_medicine, college\_physics, computer\_security, conceptual\_physics, econometrics, electrical\_engineering, elementary\_mathematics, formal\_logic, global\_facts, high\_school\_biology, high\_school\_chemistry, high\_school\_computer\_science, high\_school\_european\_history, high\_school\_geography, high\_school\_government\_and\_politics, high\_school\_macroeconomics, high\_school\_mathematics, high\_school\_microeconomics, high\_school\_physics. * **X-axis (Horizontal):** Represents the six LLMs: LLaMA-2 7B, LLaMA-2 7B Chat, LLaMA-2 13B, LLaMA-2 13B Chat, Mistral 7B, Mistral 7B Instruct. * **Color Scale:** A gradient from purple to red, representing ECE/AUROC values. The scale is marked with 20%, 50%, 60%, 80%, and 90%. Purple indicates lower values (better performance), while red indicates higher values (worse performance). * **Legend (Bottom-Center):** Defines the color coding for the four evaluation methods: * Zero-Shot Classifier (Dark Purple) * Probe (Medium Purple) * LoRA (Medium Red) * LoRA + Prompt (Bright Red) ### Detailed Analysis The heatmap is structured into six columns, one for each model. Within each column, there are 24 rows, one for each domain. Each cell in the heatmap represents the ECE/AUROC value for a specific model, domain, and evaluation method. The color of the cell indicates the performance level. Here's a breakdown of the observed trends and approximate values, focusing on the dominant color within each cell: **LLaMA-2 7B:** * **abstract\_algebra:** Zero-Shot: ~25%, Probe: ~30%, LoRA: ~50%, LoRA+Prompt: ~60% * **anatomy:** Zero-Shot: ~40%, Probe: ~35%, LoRA: ~45%, LoRA+Prompt: ~55% * **astronomy:** Zero-Shot: ~30%, Probe: ~25%, LoRA: ~40%, LoRA+Prompt: ~50% * **business\_ethics:** Zero-Shot: ~50%, Probe: ~45%, LoRA: ~60%, LoRA+Prompt: ~70% * **clinical\_knowledge:** Zero-Shot: ~60%, Probe: ~55%, LoRA: ~70%, LoRA+Prompt: ~80% * **college\_biology:** Zero-Shot: ~35%, Probe: ~30%, LoRA: ~45%, LoRA+Prompt: ~55% * **college\_chemistry:** Zero-Shot: ~40%, Probe: ~35%, LoRA: ~50%, LoRA+Prompt: ~60% * **college\_computer\_science:** Zero-Shot: ~45%, Probe: ~40%, LoRA: ~55%, LoRA+Prompt: ~65% * **college\_mathematics:** Zero-Shot: ~50%, Probe: ~45%, LoRA: ~60%, LoRA+Prompt: ~70% * **college\_medicine:** Zero-Shot: ~60%, Probe: ~55%, LoRA: ~70%, LoRA+Prompt: ~80% * **college\_physics:** Zero-Shot: ~45%, Probe: ~40%, LoRA: ~55%, LoRA+Prompt: ~65% * **computer\_security:** Zero-Shot: ~55%, Probe: ~50%, LoRA: ~65%, LoRA+Prompt: ~75% * **conceptual\_physics:** Zero-Shot: ~40%, Probe: ~35%, LoRA: ~50%, LoRA+Prompt: ~60% * **econometrics:** Zero-Shot: ~60%, Probe: ~55%, LoRA: ~70%, LoRA+Prompt: ~80% * **electrical\_engineering:** Zero-Shot: ~50%, Probe: ~45%, LoRA: ~60%, LoRA+Prompt: ~70% * **elementary\_mathematics:** Zero-Shot: ~30%, Probe: ~25%, LoRA: ~40%, LoRA+Prompt: ~50% * **formal\_logic:** Zero-Shot: ~40%, Probe: ~35%, LoRA: ~50%, LoRA+Prompt: ~60% * **global\_facts:** Zero-Shot: ~35%, Probe: ~30%, LoRA: ~45%, LoRA+Prompt: ~55% * **high\_school\_biology:** Zero-Shot: ~30%, Probe: ~25%, LoRA: ~40%, LoRA+Prompt: ~50% * **high\_school\_chemistry:** Zero-Shot: ~35%, Probe: ~30%, LoRA: ~45%, LoRA+Prompt: ~55% * **high\_school\_computer\_science:** Zero-Shot: ~40%, Probe: ~35%, LoRA: ~50%, LoRA+Prompt: ~60% * **high\_school\_european\_history:** Zero-Shot: ~50%, Probe: ~45%, LoRA: ~60%, LoRA+Prompt: ~70% * **high\_school\_geography:** Zero-Shot: ~45%, Probe: ~40%, LoRA: ~55%, LoRA+Prompt: ~65% * **high\_school\_government\_and\_politics:** Zero-Shot: ~50%, Probe: ~45%, LoRA: ~60%, LoRA+Prompt: ~70% (Similar detailed breakdowns would be provided for the other models, but are omitted for brevity. The general pattern is that LoRA and LoRA+Prompt consistently perform worse than Zero-Shot and Probe, with LoRA+Prompt being the worst.) ### Key Observations * **Evaluation Method Impact:** LoRA and LoRA+Prompt consistently yield higher ECE/AUROC values (worse performance) across all models and domains compared to Zero-Shot Classifier and Probe. * **Domain Difficulty:** Domains like clinical\_knowledge, college\_medicine, econometrics, and computer\_security generally exhibit higher ECE/AUROC values across all models, indicating they are more challenging for these LLMs. * **Model Comparison:** LLaMA-2 13B and 13B Chat generally perform better than LLaMA-2 7B and 7B Chat. Mistral 7B and 7B Instruct show competitive performance, sometimes outperforming the LLaMA-2 13B models. * **Consistency:** The relative performance ranking of domains is fairly consistent across different models and evaluation methods. ### Interpretation This heatmap provides a comprehensive comparison of LLM performance across a diverse set of domains and evaluation techniques. The consistent underperformance of LoRA and LoRA+Prompt suggests that these fine-tuning methods, while potentially useful for specific tasks, may introduce calibration issues or reduce generalization ability when evaluated on a broad range of domains. The higher error rates in complex domains (e.g., medical, security) highlight the challenges of applying LLMs to specialized knowledge areas. The competitive performance of Mistral models suggests they are strong contenders in the LLM landscape. The data suggests that while LLMs are becoming increasingly capable, careful consideration must be given to the evaluation method and the domain of application. Calibration and generalization remain critical areas for improvement. The heatmap allows for a nuanced understanding of model strengths and weaknesses, enabling informed decisions about model selection and deployment. The consistent pattern of LoRA/LoRA+Prompt performance suggests a systematic issue with those methods, potentially related to overfitting or catastrophic forgetting. Further investigation into the calibration properties of these fine-tuned models is warranted.

## [Grouped Bar Chart]: Performance of Language Models on Academic Subjects Using Different Tuning Methods ### Overview This image is a complex, multi-panel grouped bar chart comparing the performance of six different large language models (LLMs) across 25 academic subjects. Performance is measured using two metrics (ECE and AUROC) for four different model tuning or prompting methods. The chart is designed to facilitate comparison both across models for a given subject and across subjects for a given model. ### Components/Axes * **Main Structure:** The chart is divided into six vertical panels (columns), each dedicated to a specific model. * **Model Panels (Top Labels, Left to Right):** 1. LLaMA-2 7B 2. LLaMA-2 7B Chat 3. LLaMA-2 13B 4. LLaMA-2 13B Chat 5. Mistral 7B 6. Mistral 7B Instruct * **Y-Axis (Left Side):** Lists 25 academic subjects. From top to bottom: * abstract_algebra, anatomy, astronomy, business_ethics, clinical_knowledge, college_biology, college_chemistry, college_computer_science, college_mathematics, college_medicine, college_physics, computer_security, conceptual_physics, econometrics, electrical_engineering, elementary_mathematics, formal_logic, global_facts, high_school_biology, high_school_chemistry, high_school_computer_science, high_school_european_history, high_school_geography, high_school_government_and_politics, high_school_macroeconomics, high_school_mathematics, high_school_microeconomics, high_school_physics. * **X-Axis (Bottom of Each Panel):** Each model panel has its own x-axis with two sections: * **Left Section:** Labeled "ECE" (Expected Calibration Error). Scale markers: 20%, 50%, 90%. Lower values are better for ECE. * **Right Section:** Labeled "AUROC" (Area Under the Receiver Operating Characteristic Curve). Scale markers: 20%, 50%, 90%. Higher values are better for AUROC. * **Legend (Bottom Center):** Defines the four colored bars present for each subject within each model panel. * **Red/Maroon:** Zero-Shot Classifier * **Light Purple/Lavender:** Probe * **Medium Purple:** LoRA * **Dark Purple/Indigo:** LoRA + Prompt ### Detailed Analysis The chart presents a dense matrix of data. For each of the 25 subjects in each of the 6 models, four bars are shown, grouped by the ECE and AUROC metrics. **General Trends Across Models:** 1. **Method Performance Hierarchy:** For the AUROC metric (right side of each panel), the "LoRA + Prompt" (dark purple) bar is consistently the longest (highest value) across nearly all subjects and models. "LoRA" (medium purple) is typically second, followed by "Probe" (light purple). The "Zero-Shot Classifier" (red) generally shows the lowest AUROC performance. 2. **Model Size/Chat Effect:** Comparing LLaMA-2 7B to 13B, and base to Chat variants, the Chat-tuned models (LLaMA-2 7B Chat, LLaMA-2 13B Chat) often show improved AUROC scores, particularly for the "LoRA + Prompt" method, compared to their base counterparts. 3. **Mistral vs. LLaMA:** The Mistral models (7B and 7B Instruct) display performance patterns broadly similar to the LLaMA-2 models of comparable size, with "LoRA + Prompt" being dominant. The Mistral 7B Instruct model shows particularly strong AUROC scores for "LoRA + Prompt" in several subjects. 4. **ECE Metric:** The ECE values (left side of each panel) are generally low (bars are short) across the board, indicating relatively well-calibrated models. There is less dramatic variation between methods for ECE compared to AUROC. The "Zero-Shot Classifier" sometimes shows slightly higher (worse) ECE. **Subject-Specific Observations (Selected Examples):** * **high_school_government_and_politics:** In the Mistral 7B Instruct panel, the "Zero-Shot Classifier" (red) bar for AUROC is exceptionally long, reaching near 90%, which is an outlier compared to its performance in other subjects and compared to other models. * **college_computer_science:** Across most models, the AUROC scores for all methods are relatively high, suggesting this is a subject where models perform well. * **formal_logic:** This subject appears challenging. The AUROC bars, even for "LoRA + Prompt," are shorter on average compared to many other subjects across all models. * **abstract_algebra:** Shows significant variation in "Zero-Shot Classifier" (red) AUROC performance between models, from very low in LLaMA-2 7B to moderately high in Mistral 7B Instruct. ### Key Observations 1. **Dominant Method:** The "LoRA + Prompt" tuning method is the clear top performer for accuracy (AUROC) across virtually all subjects and models tested. 2. **Consistent Hierarchy:** The performance order of the four methods (LoRA + Prompt > LoRA > Probe > Zero-Shot Classifier) is remarkably consistent for the AUROC metric. 3. **Model Robustness:** The Chat-tuned and Instruct-tuned variants of the models generally provide a performance boost over their base counterparts when using advanced tuning methods like LoRA + Prompt. 4. **Subject Difficulty:** Subjects like `formal_logic` and `econometrics` appear consistently more challenging (lower AUROC scores) than subjects like `college_computer_science` or `high_school_biology`. 5. **Notable Outlier:** The `Zero-Shot Classifier` performance on `high_school_government_and_politics` with the `Mistral 7B Instruct` model is a significant positive outlier for that specific method. ### Interpretation This chart provides a comprehensive benchmark for evaluating how different adaptation techniques (zero-shot, probing, LoRA, and LoRA with prompting) affect the performance of open-weight LLMs on a wide range of academic knowledge tasks. The data strongly suggests that **combining parameter-efficient fine-tuning (LoRA) with engineered prompts yields the most accurate and reliable results** for knowledge-intensive tasks. The consistent hierarchy implies that more sophisticated adaptation methods unlock greater latent knowledge from the base models. The variation across subjects indicates that **model knowledge is not uniform**; some domains (e.g., computer science, biology) are better represented in the models' training data or are more amenable to the evaluation format than others (e.g., formal logic, advanced economics). The relative stability of the ECE metric suggests that while these tuning methods significantly improve accuracy (AUROC), they do not drastically harm the models' calibration (their ability to estimate their own confidence correctly). This is important for trustworthy deployment. From a practical standpoint, a researcher or engineer looking to maximize performance on a specific academic domain would likely choose the "LoRA + Prompt" approach. The chart also helps identify which models (e.g., Mistral 7B Instruct, LLaMA-2 13B Chat) are strongest overall and which subjects might require additional data or specialized techniques.

## Bar Chart: Model Performance Comparison Across Subjects ### Overview The chart compares the performance of multiple AI models (LLaMA-2 7B, LLaMA-2 13B, LLaMA-2 13B Chat, Mistral 7B, Mistral 7B Instruct) across 25 academic subjects using two evaluation metrics: Expected Calibration Error (ECE) and Area Under the Receiver Operating Characteristic curve (AUROC). Performance is visualized as grouped bars for each subject, with color-coded models and percentage-based axes. ### Components/Axes - **Y-Axis**: Subjects (e.g., `abstract_algebra`, `astronomy`, `high_school_physics`), listed alphabetically. - **X-Axes**: - Left: ECE (20%, 50%, 60%, 90% markers). - Right: AUROC (20%, 50%, 60%, 90% markers). - **Legend**: - Red: Zero-Shot Classifier - Light Purple: Probe - Dark Purple: LoRA - Dark Blue: LoRA + Prompt - Gray: LLaMA-2 7B - Dark Red: LLaMA-2 13B - Light Blue: LLaMA-2 13B Chat - Dark Green: Mistral 7B - Light Gray: Mistral 7B Instruct ### Detailed Analysis - **ECE Trends**: - Mistral 7B Instruct (light gray) consistently shows the lowest ECE (20-30%) across most subjects. - LLaMA-2 13B (dark red) and Mistral 7B (dark green) often cluster around 50-60% ECE. - Larger models (LLaMA-2 13B, Mistral 7B) generally outperform smaller models (LLaMA-2 7B, Mistral 7B) in ECE for subjects like `college_chemistry` and `high_school_mathematics`. - **AUROC Trends**: - LLaMA-2 13B Chat (light blue) and Mistral 7B Instruct (light gray) dominate AUROC (70-90%) in subjects like `college_biology` and `high_school_geography`. - LLaMA-2 7B (gray) and Mistral 7B (dark green) show lower AUROC (40-60%) in `astronomy` and `econometrics`. - AUROC values are consistently higher than ECE across all models and subjects. ### Key Observations 1. **Model Size vs. Performance**: - Larger models (LLaMA-2 13B, Mistral 7B) generally achieve higher AUROC but not always lower ECE. - Instruction-tuned models (LLaMA-2 13B Chat, Mistral 7B Instruct) excel in both metrics for specific subjects. 2. **Subject-Specific Variability**: - `high_school_physics` and `college_computer_science` show the widest performance gaps between models. - `business_ethics` and `global_facts` have tightly clustered AUROC values across models. 3. **Anomalies**: - Mistral 7B Instruct (light gray) underperforms in AUROC for `high_school_microeconomics` compared to other models. - LLaMA-2 7B (gray) has disproportionately high ECE in `high_school_government_and_politics`. ### Interpretation The chart reveals that model architecture and training methodology significantly influence performance across academic domains. Instruction-tuned variants (e.g., LLaMA-2 13B Chat, Mistral 7B Instruct) demonstrate superior calibration (lower ECE) and generalization (higher AUROC) in specialized subjects like `college_chemistry` and `high_school_geography`. However, smaller models (LLaMA-2 7B, Mistral 7B) struggle with calibration in politically sensitive topics (`high_school_government_and_politics`), suggesting domain-specific knowledge gaps. The consistent AUROC superiority of larger models implies that scale improves generalization, but instruction tuning is critical for real-world applicability. The anomaly in `high_school_microeconomics` highlights potential weaknesses in economic reasoning across models.