## Bar Chart: Model Performance Comparison on Various Topics ### Overview The image presents a series of bar charts comparing the performance of different language models (LLaMA-2 7B, LLaMA-2 7B Chat, LLaMA-2 13B, LLaMA-2 13B Chat, Mistral 7B, and Mistral 7B Instruct) across a range of topics. Performance is measured using ECE (Expected Calibration Error) and AUROC (Area Under the Receiver Operating Characteristic curve) metrics, with different training methods (Zero-Shot Classifier, Probe, LoRA, and LoRA + Prompt). ### Components/Axes * **X-axis:** Performance metrics (ECE and AUROC) ranging from 20% to 90%. * **Y-axis:** List of topics, including: * high\_school\_psychology * high\_school\_statistics * high\_school\_us\_history * high\_school\_world\_history * human\_aging * human\_sexuality * international\_law * jurisprudence * logical\_fallacies * machine\_learning * management * marketing * medical\_genetics * miscellaneous * moral\_disputes * moral\_scenarios * nutrition * philosophy * prehistory * professional\_accounting * professional\_law * professional\_medicine * professional\_psychology * public\_relations * security\_studies * sociology * us\_foreign\_policy * virology * world\_religions * **Legend:** Located at the bottom of the chart. * Zero-Shot Classifier (Dark Red) * Probe (Light Purple) * LoRA (Medium Purple) * LoRA + Prompt (Dark Purple) ### Detailed Analysis Each model has two bars for each topic, one for ECE and one for AUROC. Each bar is further divided into four segments, representing the performance of each training method. **Model-Specific Observations:** * **LLaMA-2 7B:** Performance varies significantly across topics and training methods. Zero-Shot Classifier often shows lower performance compared to other methods. * **LLaMA-2 7B Chat:** Similar trends to LLaMA-2 7B, but with some variations in performance across topics. * **LLaMA-2 13B:** Generally shows improved performance compared to the 7B models, particularly in AUROC. * **LLaMA-2 13B Chat:** Similar to LLaMA-2 13B, with slight variations. * **Mistral 7B:** Performance is generally competitive with LLaMA-2 13B models. * **Mistral 7B Instruct:** Shows a more consistent performance across topics, often outperforming other models, especially in AUROC. **Training Method Observations:** * **Zero-Shot Classifier (Dark Red):** Generally the lowest performing method, especially in ECE. * **Probe (Light Purple):** Performance varies, sometimes better than Zero-Shot but often lower than LoRA methods. * **LoRA (Medium Purple):** Generally improves performance compared to Zero-Shot and Probe. * **LoRA + Prompt (Dark Purple):** Often the best performing method, especially in AUROC. **Topic-Specific Observations:** * Some topics, like "logical\_fallacies" and "machine\_learning," show relatively high performance across all models and methods. * Other topics, like "high\_school\_psychology" and "world\_religions," tend to have lower performance. **Example Data Points (Approximate):** * **LLaMA-2 7B, high\_school\_psychology:** * Zero-Shot Classifier (ECE): ~20% * Probe (ECE): ~30% * LoRA (ECE): ~35% * LoRA + Prompt (ECE): ~40% * Zero-Shot Classifier (AUROC): ~30% * Probe (AUROC): ~40% * LoRA (AUROC): ~45% * LoRA + Prompt (AUROC): ~50% * **Mistral 7B Instruct, machine\_learning:** * Zero-Shot Classifier (ECE): ~50% * Probe (ECE): ~60% * LoRA (ECE): ~65% * LoRA + Prompt (ECE): ~70% * Zero-Shot Classifier (AUROC): ~70% * Probe (AUROC): ~80% * LoRA (AUROC): ~85% * LoRA + Prompt (AUROC): ~90% ### Key Observations * LoRA and LoRA + Prompt generally outperform Zero-Shot and Probe methods. * 13B models and Mistral models tend to perform better than the 7B models. * Mistral 7B Instruct shows consistent high performance. * Performance varies significantly across different topics. ### Interpretation The data suggests that model size and training method significantly impact performance on various topics. LoRA and LoRA + Prompt are effective techniques for improving model accuracy and calibration. The Mistral 7B Instruct model appears to be particularly well-suited for these tasks, possibly due to its architecture or training data. The varying performance across topics highlights the importance of considering domain-specific knowledge when evaluating language models. The ECE and AUROC metrics provide complementary insights into model performance, with ECE measuring calibration and AUROC measuring discrimination ability.

## Comparative Performance Chart: LLM Evaluation Across Domains ### Overview This chart compares the performance of several Large Language Models (LLMs) – LLaMA-2 (7B, 7B Chat, 13B, 13B Chat) and Mistral (7B, 7B Instruct) – across 26 different domains. Performance is evaluated using four metrics: Zero-Shot Classifier, Probe, LoRA, and LoRA + Prompt. The chart uses a heatmap-style visualization, where the length of each horizontal bar represents the performance score (AUROC) for a given LLM and domain. ### Components/Axes * **Y-Axis (Vertical):** Lists 26 domains/categories: high_school_psychology, high_school_statistics, high_school_us_history, high_school_world_history, human_aging, human_sexuality, international_law, jurisprudence, logical_fallacies, machine_learning, management, marketing, medical_genetics, miscellaneous, moral_disputes, moral_scenarios, nutrition, philosophy, prehistory, professional_accounting, professional_law, professional_medicine, professional_psychology, public_relations, safety_studies, sociology, us_foreign_policy, virology, world_religions. * **X-Axis (Horizontal):** Represents the six LLMs being compared: LLaMA-2 7B, LLaMA-2 7B Chat, LLaMA-2 13B, LLaMA-2 13B Chat, Mistral 7B, Mistral 7B Instruct. * **Color Legend (Bottom):** * Purple: Zero-Shot Classifier * Blue: Probe * Orange: LoRA * Pink: LoRA + Prompt * **X-Axis Scale (Bottom):** AUROC values ranging from 20% to 90%, with markers at 20%, 50%, 60%, 80%, and 90%. ### Detailed Analysis The chart consists of 26 rows (domains) and 6 columns (LLMs), with each cell containing four horizontal bars representing the four evaluation metrics. The length of each bar corresponds to the AUROC score. I will analyze each LLM and metric, noting trends and approximate values. Due to the visual nature of the chart, values are approximate. **LLaMA-2 7B:** * **Zero-Shot Classifier (Purple):** Generally performs poorly, with most bars around 20-40%. Notable exceptions include machine_learning (~50%), moral_disputes (~45%), and professional_law (~40%). * **Probe (Blue):** Shows moderate improvement over Zero-Shot, with most bars between 40-60%. Strongest performance in machine_learning (~70%), professional_law (~65%), and moral_disputes (~60%). * **LoRA (Orange):** Significant improvement, with most bars between 60-80%. Highest scores in machine_learning (~85%), professional_law (~80%), and moral_disputes (~75%). * **LoRA + Prompt (Pink):** Further improvement, with most bars between 70-90%. Peak performance in machine_learning (~90%), professional_law (~85%), and moral_disputes (~80%). **LLaMA-2 7B Chat:** * **Zero-Shot Classifier (Purple):** Similar to LLaMA-2 7B, generally 20-40%, with some domains reaching ~50%. * **Probe (Blue):** Moderate improvement, generally 40-60%. * **LoRA (Orange):** Significant improvement, generally 60-80%. * **LoRA + Prompt (Pink):** Further improvement, generally 70-90%. Performance is generally slightly higher than LLaMA-2 7B across all metrics. **LLaMA-2 13B:** * **Zero-Shot Classifier (Purple):** Generally better than the 7B models, with most bars between 30-50%. * **Probe (Blue):** Improved performance, generally 50-70%. * **LoRA (Orange):** Strong performance, generally 70-90%. * **LoRA + Prompt (Pink):** Excellent performance, with many bars reaching 80-90%. **LLaMA-2 13B Chat:** * **Zero-Shot Classifier (Purple):** Similar to LLaMA-2 13B, generally 30-50%. * **Probe (Blue):** Improved performance, generally 50-70%. * **LoRA (Orange):** Strong performance, generally 70-90%. * **LoRA + Prompt (Pink):** Excellent performance, with many bars reaching 80-90%. Performance is generally slightly higher than LLaMA-2 13B across all metrics. **Mistral 7B:** * **Zero-Shot Classifier (Purple):** Generally performs well, often exceeding 50%. * **Probe (Blue):** Strong performance, frequently above 60%. * **LoRA (Orange):** Excellent performance, often reaching 80-90%. * **LoRA + Prompt (Pink):** Very strong performance, with many bars at or near 90%. **Mistral 7B Instruct:** * **Zero-Shot Classifier (Purple):** Similar to Mistral 7B, generally above 50%. * **Probe (Blue):** Strong performance, frequently above 60%. * **LoRA (Orange):** Excellent performance, often reaching 80-90%. * **LoRA + Prompt (Pink):** Very strong performance, with many bars at or near 90%. Performance is generally slightly higher than Mistral 7B across all metrics. ### Key Observations * **LoRA + Prompt consistently yields the highest performance** across all LLMs and domains. * **Larger models (13B) generally outperform smaller models (7B)**, especially with LoRA and LoRA + Prompt. * **Mistral models consistently outperform LLaMA-2 models**, particularly in Zero-Shot and Probe evaluations. * **Machine learning, professional law, and moral disputes consistently show higher AUROC scores** across all models and metrics, suggesting these domains are easier for the LLMs to evaluate. * **Human sexuality and prehistory consistently show lower AUROC scores**, indicating these domains are more challenging. ### Interpretation This chart demonstrates the effectiveness of fine-tuning techniques (LoRA and LoRA + Prompt) in improving the performance of LLMs across a diverse set of domains. The consistent superiority of LoRA + Prompt suggests that providing targeted prompts alongside LoRA adaptation significantly enhances the models' ability to generalize and perform well. The better performance of Mistral models suggests architectural or training data differences contribute to their superior capabilities. The varying performance across domains highlights the challenges of creating general-purpose LLMs and the need for domain-specific adaptation. The lower scores in areas like human sexuality and prehistory could be due to data scarcity, inherent complexity, or biases in the training data. The chart provides a valuable comparative analysis for selecting and fine-tuning LLMs for specific applications.

## Grouped Bar Chart: Language Model Performance Across Academic Subjects (ECE & AUROC) ### Overview This image is a complex, multi-panel grouped bar chart comparing the performance of six different Large Language Models (LLMs) across 30 academic subjects. Performance is measured using two metrics: Expected Calibration Error (ECE) and Area Under the Receiver Operating Characteristic Curve (AUROC). For each model and subject, four different methods are evaluated: Zero-Shot Classifier, Probe, LoRA, and LoRA + Prompt. ### Components/Axes * **Chart Type:** Multi-panel grouped horizontal bar chart. * **Panels (Columns):** Six distinct panels, each representing a different LLM. From left to right: 1. LLaMA-2 7B 2. LLaMA-2 7B Chat 3. LLaMA-2 13B 4. LLaMA-2 13B Chat 5. Mistral 7B 6. Mistral 7B Instruct * **Y-Axis (Rows):** Lists 30 academic subjects. From top to bottom: `high_school_psychology`, `high_school_statistics`, `high_school_us_history`, `high_school_world_history`, `human_aging`, `human_sexuality`, `international_law`, `jurisprudence`, `logical_fallacies`, `machine_learning`, `management`, `marketing`, `medical_genetics`, `miscellaneous`, `moral_disputes`, `moral_scenarios`, `nutrition`, `philosophy`, `prehistory`, `professional_accounting`, `professional_law`, `professional_medicine`, `professional_psychology`, `public_relations`, `security_studies`, `sociology`, `us_foreign_policy`, `virology`, `world_religions`. * **X-Axis (Metrics):** Each panel has two sub-columns at the bottom, labeled: * **ECE** (Expected Calibration Error): Scale from 0% to approximately 20%. Lower values are better. * **AUROC** (Area Under the ROC Curve): Scale from 50% to 90%. Higher values are better. * **Legend (Bottom Center):** Four colored bars define the methods: * **Maroon/Dark Red:** Zero-Shot Classifier * **Light Purple/Lavender:** Probe * **Medium Purple:** LoRA * **Dark Purple/Indigo:** LoRA + Prompt * **Spatial Layout:** The legend is positioned at the bottom center of the entire figure. Each of the six model panels is arranged vertically, with subjects listed on the far left. Within each panel, for every subject, four horizontal bars are grouped together, corresponding to the four methods in the legend. ### Detailed Analysis **General Trends Across All Models:** 1. **Metric Comparison:** For nearly all subjects and models, the AUROC bars (right sub-column) are significantly longer than the ECE bars (left sub-column), indicating that models generally achieve moderate to good discriminative ability (AUROC > 50%) while maintaining relatively low calibration error (ECE < ~15%). 2. **Method Performance Hierarchy:** A consistent pattern is visible across most subjects and models: * **Zero-Shot Classifier (Maroon):** Typically shows the shortest bars for AUROC and often the longest bars for ECE, indicating the poorest performance. * **Probe (Light Purple):** Shows a clear improvement over Zero-Shot, with longer AUROC bars and shorter ECE bars. * **LoRA (Medium Purple) & LoRA + Prompt (Dark Purple):** These two methods consistently perform the best. Their bars are often very close in length, with LoRA + Prompt showing a slight, consistent edge in AUROC (longer bar) and a slight edge in ECE (shorter bar) over LoRA alone in many cases. **Model-Specific Observations:** * **LLaMA-2 7B vs. 7B Chat:** The Chat variant generally shows improved AUROC scores (longer bars) across many subjects compared to the base 7B model, particularly for the fine-tuned methods (Probe, LoRA). * **LLaMA-2 13B vs. 13B Chat:** Similar to the 7B pair, the 13B Chat model often outperforms the base 13B model in AUROC, though the difference appears less dramatic than in the 7B case. * **Mistral 7B vs. 7B Instruct:** The Instruct variant shows a very strong performance boost over the base Mistral 7B. The AUROC bars for LoRA/LoRA+Prompt in the Instruct model are frequently among the longest in the entire chart, often exceeding 80%. * **Scale (7B vs. 13B):** Comparing LLaMA-2 7B to 13B, the larger 13B model shows a modest but visible improvement in AUROC for most methods and subjects. **Subject-Specific Highlights (Approximate Values):** * **High ECE (Poor Calibration):** The `machine_learning` subject often shows relatively high ECE values (bars extending further left) across models, especially for the Zero-Shot method. `moral_scenarios` also shows notable ECE. * **High AUROC (Strong Discrimination):** Subjects like `high_school_psychology`, `professional_psychology`, and `sociology` frequently show very high AUROC scores (>80%) for the best-performing methods (LoRA + Prompt) across multiple models. * **Challenging Subjects:** Subjects like `professional_law`, `international_law`, and `jurisprudence` tend to have shorter AUROC bars overall, suggesting they are more difficult for the models to classify correctly. ### Key Observations 1. **Fine-Tuning is Crucial:** The most striking pattern is the substantial performance gap between the Zero-Shot Classifier and all other methods (Probe, LoRA, LoRA+Prompt). This demonstrates that some form of adaptation or fine-tuning is essential for strong performance on these academic tasks. 2. **Instruction Tuning Matters:** The "Chat" and "Instruct" variants of models consistently outperform their base counterparts, highlighting the value of instruction-following training for these types of knowledge-based QA tasks. 3. **LoRA + Prompt is the Top Performer:** The dark purple bars (LoRA + Prompt) are almost universally the longest for AUROC and the shortest for ECE, indicating this combined approach yields the best calibrated and most accurate models. 4. **Performance Variability:** There is significant variability in model performance across different academic subjects, indicating that model knowledge and reasoning ability are not uniform across domains. ### Interpretation This chart provides a comprehensive benchmark of LLM capabilities across a wide spectrum of academic knowledge. The data suggests several key insights: 1. **The Necessity of Adaptation:** The poor performance of the Zero-Shot Classifier underscores that simply using a pre-trained LLM's internal knowledge is insufficient for high-accuracy, well-calibrated classification on specialized academic topics. Parameter-efficient fine-tuning methods like LoRA are highly effective. 2. **Synergy of Methods:** The consistent, albeit sometimes small, superiority of "LoRA + Prompt" over "LoRA" alone suggests that combining parameter-efficient fine-tuning with optimized prompting creates a synergistic effect, leading to better model performance than either technique in isolation. 3. **Model Architecture and Training Impact:** The comparison between base and chat/instruct models, and between 7B and 13B parameter models, illustrates that both the training objective (instruction tuning) and model scale contribute positively to performance. The Mistral 7B Instruct model's strong showing suggests its training regimen is particularly effective for this evaluation setup. 4. **Domain-Specific Challenges:** The variation in performance across subjects (e.g., law vs. psychology) implies that the underlying training data of these models may be imbalanced, or that some domains require more complex reasoning that is harder for the models to capture. This has implications for using LLMs as general-purpose knowledge engines. **In summary, the chart is a detailed map of LLM performance, revealing that while base models have foundational knowledge, achieving high accuracy and reliability on academic tasks requires targeted fine-tuning and prompting strategies, with the combination of LoRA and prompt engineering emerging as the most robust approach among those tested.**

## Bar Chart: Model Performance Across Subjects ### Overview The image is a grouped bar chart comparing the performance of four AI models (LLaMA-2 7B, LLaMA-2 13B, Mistral 7B, Mistral 7B Instruct) across 25 subjects (e.g., high school psychology, human sexuality, international law) using two metrics: Expected Calibration Error (ECE) and Area Under the Receiver Operating Characteristic curve (AUROC). Performance is evaluated using four classifier types: Zero-Shot, Probe, LoRA, and LoRA + Prompt. ### Components/Axes - **Y-Axis**: Subjects (25 categories, e.g., "high_school_psychology", "human_aging", "world_religions"). - **X-Axis**: Metrics (ECE and AUROC, each with 20%, 50%, 60%, 90% thresholds). - **Legend**: - Red: Zero-Shot Classifier - Light Purple: Probe - Dark Purple: LoRA - Black: LoRA + Prompt - **Models**: - LLaMA-2 7B (leftmost group) - LLaMA-2 13B (second group) - Mistral 7B (third group) - Mistral 7B Instruct (rightmost group) ### Detailed Analysis - **ECE Trends**: - All models show ECE values clustered around 20-50% for most subjects. - LoRA + Prompt (black bars) generally has the lowest ECE across subjects. - Zero-Shot (red bars) often has the highest ECE, especially in "high_school_psychology" and "human_sexuality". - **AUROC Trends**: - AUROC values range from 50-90%. - LoRA + Prompt (black bars) consistently achieves the highest AUROC, particularly in "jurisprudence" and "professional_medicine". - Probe (light purple bars) performs well in "management" and "marketing". - Zero-Shot (red bars) has the lowest AUROC in "moral_disputes" and "philosophy". ### Key Observations 1. **LoRA + Prompt Dominance**: Outperforms other classifiers in AUROC for 18/25 subjects (e.g., "international_law", "virology"). 2. **Probe Strength**: Excels in applied domains like "marketing" and "public_relations". 3. **Zero-Shot Weaknesses**: Struggles in abstract or nuanced subjects (e.g., "moral_scenarios", "us_foreign_policy"). 4. **Model Size Impact**: LLaMA-2 13B generally outperforms LLaMA-2 7B in AUROC, but Mistral 7B Instruct matches or exceeds LLaMA-2 13B in 12 subjects. ### Interpretation The data demonstrates that **fine-tuning (LoRA) combined with prompting** significantly improves model reliability (lower ECE) and accuracy (higher AUROC) across diverse domains. The Probe classifier shows domain-specific strengths, suggesting it may be optimized for certain applications. Zero-Shot performance highlights the limitations of general-purpose models without adaptation. Notably, Mistral 7B Instruct achieves competitive results despite smaller parameter count, indicating architectural efficiency. The stark contrast between LoRA + Prompt and Zero-Shot in subjects like "moral_disputes" underscores the importance of contextual adaptation for ethical reasoning tasks.