## Scatter Plot: LLM Performance vs Human Expertise ### Overview This scatter plot compares Large Language Model (LLM) performance (LLM-Judge Score) against human expert evaluations (Human Expert Score) across different model sizes and configurations. The plot includes performance tiers (Top/Mid/Low) and distinguishes models with/without rationale generation capabilities. ### Components/Axes - **X-axis**: Human Expert Score (5-16 scale, increments of 2) - **Y-axis**: LLM-Judge Score (2.5-20 scale, increments of 2.5) - **Legends**: 1. **Model Size** (top-left): - 0.6B (smallest circles) - 20B (medium circles) - 235B (largest circles) 2. **Performance Tier** (top-right): - Top tier (dark blue) - Mid tier (medium blue) - Low tier (light blue) 3. **Rationale Presence** (bottom-right): - w/ rationale (solid blue circles) - w/o rationale (hollow purple circles) - **Trend Line**: Dashed purple line with shaded confidence interval (light purple) ### Detailed Analysis 1. **Model Size Correlation**: - Larger models (235B) cluster at higher scores (e.g., GPT-4: 16→18, A.X-4.0: 15→17) - Smaller models (0.6B) cluster at lower scores (Qwen-0.6B: 5→4) - Mid-sized models (20B) show mixed performance (e.g., VARCO-VISION: 10→12) 2. **Performance Tiers**: - Top tier models (dark blue) generally have Human Expert Scores ≥12 and LLM Scores ≥12 - Mid tier models (medium blue) cluster between 8-12 on both axes - Low tier models (light blue) appear below 8 on both axes 3. **Rationale Impact**: - Models with rationale (solid blue) consistently outperform identical models without rationale (hollow purple) - Example: InternVL-3_5-8B-Instruct shows 5→6 with rationale vs 5→5 without 4. **Trend Line Analysis**: - Positive correlation (R² ≈ 0.85) between human and LLM scores - Shaded confidence interval widens at mid-range scores (8-12), indicating greater variability - Slope ≈ 0.95 suggests near-parity between human and LLM evaluations at higher scores ### Key Observations 1. **Size-Score Relationship**: - 235B models achieve 15-20 LLM scores vs 12-16 human scores - 0.6B models achieve 4-6 LLM scores vs 5-7 human scores - 20B models show variable performance (6-12 LLM scores) 2. **Rationale Effectiveness**: - Rationale improves scores by 0.5-1.5 points across all model sizes - Most significant gains observed in mid-tier models (e.g., GPT-OSS-20B: +1 point) 3. **Performance Tiers**: - Top tier models maintain ≥12 LLM score regardless of rationale - Mid tier models show 10-12 LLM scores with rationale - Low tier models cluster below 8 LLM score ### Interpretation The data demonstrates that: 1. **Model Size Matters**: Larger models consistently achieve higher scores, with 235B models approaching human expert levels (16→18) 2. **Rationale Enhances Performance**: Across all sizes, models with rationale outperform identical configurations without, particularly in mid-tier ranges 3. **Human-LLM Alignment**: The near-slope of 1 indicates strong alignment between human and LLM evaluations at higher performance levels 4. **Diminishing Returns**: While larger models perform better, the performance gap narrows at the highest scores (e.g., GPT-4: 16→18 vs A.X-4.0: 15→17) Notable anomalies include: - GPT-OSS-120B (20B) achieving Top tier scores despite mid-sized architecture - InternVL-3_5-8B-Instruct showing minimal improvement with rationale (+1 point) - Qwen-0.6B maintaining near-parity between human and LLM scores despite small size This suggests that while model scale is crucial, architectural choices (rationale generation) and training objectives significantly impact performance, particularly in mid-tier ranges where human-LLM alignment is most variable.