## Heatmap: Per-Subject Scores: Sycophant with Knowledge
### Overview
This heatmap visualizes the performance scores of various AI models across 50+ academic and professional subjects. Scores range from 0.0 (light yellow) to 0.5 (dark red), with darker colors indicating higher scores. The data compares six models: Llama-3-2-3B, Llama-3-1-8B, Qwen2.5-3B, Qwen2.5-7B, Qwen2.5-14B, and Qwen2.5-32B.
### Components/Axes
- **Rows (Subjects)**: 50+ academic/professional disciplines (e.g., "abstract_algebra," "business_ethics," "professional_psychology").
- **Columns (Models)**: Six AI models with varying parameter sizes.
- **Color Scale**: 0.0 (light yellow) to 0.5 (dark red), representing normalized scores.
- **Legend**: Positioned on the right, with a gradient from yellow (low) to red (high).
### Detailed Analysis
#### Subject-Specific Scores
1. **Highest Scores**:
- **Llama-3-2-3B**:
- `abstract_algebra` (0.50), `college_chemistry` (0.50), `professional_psychology` (0.40).
- **Qwen2.5-32B**:
- `business_ethics` (0.33), `professional_law` (0.25), `security_studies` (0.33).
2. **Notable Mid-Range Scores**:
- **Llama-3-1-8B**:
- `college_chemistry` (0.50), `high_school_physics` (0.25), `high_school_psychology` (0.30).
- **Qwen2.5-14B**:
- `college_chemistry` (0.12), `high_school_chemistry` (0.17), `high_school_mathematics` (0.17).
3. **Low/No Scores**:
- Many subjects (e.g., `astronomy`, `clinical_knowledge`, `virology`) show 0.00 scores across all models, suggesting either no data or poor performance.
#### Model Performance Trends
- **Llama-3-2-3B** dominates in **abstract reasoning** (`abstract_algebra`, `college_chemistry`) and **professional domains** (`professional_psychology`).
- **Qwen2.5-32B** excels in **applied ethics** (`business_ethics`, `professional_law`) and **security studies**.
- **Qwen2.5-7B** shows moderate performance in **high school-level subjects** (e.g., `high_school_physics` at 0.17).
- **Llama-3-1-8B** has strong scores in **chemistry** and **psychology** but weaker in **mathematics** (0.00 for `college_mathematics`).
### Key Observations
1. **Model Specialization**:
- Larger models (e.g., Qwen2.5-32B) perform better in **applied disciplines** (law, ethics), while smaller models (Llama-3-2-3B) excel in **theoretical subjects** (algebra, chemistry).
- No model achieves high scores in **medical genetics** (max 0.17) or **virology** (0.00).
2. **Gaps in Coverage**:
- Many subjects (e.g., `astronomy`, `virology`) have no scores, indicating potential knowledge gaps or evaluation limitations.
3. **Consistency**:
- `college_chemistry` is a consistent high-performer across models (0.50 for Llama-3-2-3B, 0.12 for Qwen2.5-14B).
### Interpretation
The heatmap reveals that model performance is **highly domain-specific**, with no single model dominating across all subjects. Larger models (Qwen2.5-32B) show broader applicability in **professional and ethical domains**, while smaller models (Llama-3-2-3B) excel in **theoretical and scientific fields**. The absence of scores in certain subjects suggests either:
- **Data limitations** (e.g., lack of training data for niche topics like `virology`),
- **Evaluation challenges** (e.g., subjective scoring in `moral_scenarios`),
- Or **model biases** (e.g., poor performance in `human_sexuality` across all models).
This analysis underscores the importance of **model selection based on use case**—e.g., choosing Llama-3-2-3B for chemistry problems or Qwen2.5-32B for legal ethics. Further investigation into the training data and evaluation methodologies for these models would clarify the root causes of these patterns.
