## Heatmap: Per-Subject Scores: Agreement Rate ### Overview The heatmap visualizes agreement rates between different AI models (rows) and subjects (columns). Scores range from 0.0 (no agreement) to 0.6 (high agreement), with darker red indicating higher agreement. The data suggests model performance varies significantly by subject domain. ### Components/Axes - **Rows (Subjects)**: 50+ academic/professional disciplines (e.g., "astronomy," "professional_law," "machine_learning") - **Columns (Models)**: 6 AI models: - Llama-3-2-3B - Llama-3-1-8B - Qwen2.5-3B - Qwen2.5-7B - Qwen2.5-14B - Qwen2.5-32B - **Color Scale**: - 0.0 (light yellow) → 0.6 (dark red) - Legend positioned vertically on the right ### Detailed Analysis 1. **Highest Agreement Scores**: - **Qwen2.5-7B**: - Professional_law (0.60) - International_law (0.30) - Management (0.30) - **Llama-3-1-8B**: - Astronomy (0.40) - Econometrics (0.20) - Electrical_engineering (0.20) - **Qwen2.5-32B**: - Business_ethics (0.20) - High_school_government_and_politics (0.10) 2. **Notable Patterns**: - **Specialized vs. General Subjects**: - Qwen models excel in specialized fields (e.g., "professional_law," "machine_learning") - Llama models perform better in general science/math subjects (e.g., "astronomy," "econometrics") - **Consistency**: - Qwen2.5-7B shows the highest overall scores (max 0.60) - Llama-3-2-3B has the fewest non-zero scores (only 3 subjects with ≥0.10) 3. **Zero Agreement**: - Many subjects (e.g., "abstract_algebra," "anatomy") show 0.00 across all models, suggesting either: - No training data for these topics - No testing performed on these subjects ### Key Observations - **Model Strengths**: - Qwen2.5-7B dominates in legal/ethics subjects - Llama-3-1-8B leads in STEM fields - Qwen2.5-32B shows moderate performance in social sciences - **Outliers**: - "Professional_law" (0.60) is the only score reaching the maximum agreement threshold - "High_school_government_and_politics" (0.10) is the lowest non-zero score ### Interpretation The data reveals significant model specialization: 1. **Qwen2.5-7B** appears optimized for legal/ethical reasoning, with the highest agreement in "professional_law" (0.60) and "international_law" (0.30). 2. **Llama-3-1-8B** demonstrates strength in quantitative subjects, with notable scores in "astronomy" (0.40) and "econometrics" (0.20). 3. The absence of scores for many subjects (e.g., "abstract_algebra") suggests either: - Deliberate exclusion from testing - Insufficient training data for these domains 4. The Qwen models generally outperform Llama models in specialized professional fields, while Llama models show better generalization in STEM subjects. The heatmap highlights the importance of model architecture and training data in determining subject-specific performance, with no single model excelling across all disciplines.