## Scatter Plot: Generalization Gap vs. Factor Diversity
### Overview
The chart visualizes the relationship between **Factor Diversity** (number of active factors) and **Generalization Gap** (absolute difference between validation accuracy and test accuracy) across three task categories: Math, Science, and Logic. A red dashed line at 0.1 marks the "Good Generalization Threshold."
### Components/Axes
- **X-axis**: Factor Diversity (# of Active Factors) – ranges from 0 to 5.
- **Y-axis**: Generalization Gap |(Val Acc - Test Acc)| – ranges from -0.10 to 0.30.
- **Legend**:
- Blue: Math Tasks
- Orange: Science Tasks
- Green: Logic Tasks
- **Threshold Line**: Red dashed line at y = 0.1.
### Detailed Analysis
1. **Math Tasks (Blue)**:
- **MULTIARITH**: (1, -0.05) – below threshold.
- **GSM-Hard**: (2, 0.10) – on threshold.
- **AQUA-RAT**: (3, 0.04) – below threshold.
- **GSM8K**: (3, 0.03) – below threshold.
2. **Science Tasks (Orange)**:
- **GPQA Chemistry**: (3, 0.10) – on threshold.
- **GPQA Physics**: (3.5, 0.24) – well above threshold.
3. **Logic Tasks (Green)**:
- **BBH Logic-5**: (2, 0.00) – on threshold.
- **BBH Date**: (1, 0.16) – above threshold.
### Key Observations
- **Highest Generalization Gap**: GPQA Physics (0.24) – Science Task with 3.5 active factors.
- **Lowest Generalization Gap**: MULTIARITH (-0.05) – Math Task with 1 active factor.
- **Threshold Crossings**:
- 3 tasks (GSM-Hard, GPQA Chemistry, BBH Logic-5) meet/exceed the 0.1 threshold.
- 5 tasks fall below the threshold, including GPQA Physics despite high factor diversity.
### Interpretation
The data suggests that **Factor Diversity does not consistently correlate with Generalization Gap**. While Science Tasks (e.g., GPQA Physics) often exhibit larger gaps, some Math and Logic Tasks (e.g., GSM8K, BBH Logic-5) achieve near-zero gaps even with fewer factors. The threshold line highlights that only a minority of tasks meet the "Good Generalization" criterion, emphasizing the need for task-specific optimization strategies. Notably, GPQA Physics’s high gap despite 3.5 factors challenges the assumption that increased complexity inherently improves generalization.
