## Heatmap: Correlation Coefficients Between K-MetBench and KMMMLU Redux Benchmarks
### Overview
This heatmap visualizes Kendall's correlation coefficients between K-MetBench benchmarks (rows) and KMMMLU Redux comparison benchmarks (columns). Values range from -1 (blue) to 1 (red), with darker red indicating stronger positive correlation. The data suggests strong alignment between most K-MetBench benchmarks and KMMMLU Redux, except for the Multimodal benchmark.
### Components/Axes
- **X-axis (Columns)**: Labeled "KMMMLU Redux Comparison Benchmarks" with categories:
`KMMMLU` | `All` | `39` | `All-39`
- **Y-axis (Rows)**: Labeled "K-MetBench" with categories:
`Total` | `Reasoning` | `Part 1` | `Part 2` | `Part 3` | `Part 4` | `Part 5` | `Text-Only` | `Multimodal` | `Korean`
- **Legend**: Right-aligned colorbar titled "Kendall's Correlation Coefficient" with gradient from blue (-1) to red (1).
### Detailed Analysis
- **Total Row**:
`KMMMLU`: 0.56 (light red) | `All`: 0.78 (dark red) | `39`: 0.70 (dark red) | `All-39`: 0.78 (dark red)
- **Reasoning Row**:
`KMMMLU`: 0.61 | `All`: 0.66 | `39`: 0.65 | `All-39`: 0.66
- **Part 1-5 Rows**:
All parts show high correlations (0.51–0.79), with `Part 4` and `Part 5` peaking at 0.79 in `All` and `All-39`.
- **Text-Only Row**:
`KMMMLU`: 0.56 | `All`: 0.78 | `39`: 0.71 | `All-39`: 0.78
- **Multimodal Row**:
Outlier with weak correlations: `KMMMLU`: 0.20 (blue) | `All`: 0.29 | `39`: 0.34 | `All-39`: 0.29
- **Korean Row**:
`KMMMLU`: 0.57 | `All`: 0.70 | `39`: 0.61 | `All-39`: 0.70
### Key Observations
1. **Strong Overall Correlation**: Most benchmarks (e.g., `All`, `All-39`) show coefficients ≥0.65, indicating robust alignment.
2. **Multimodal Anomaly**: The `Multimodal` benchmark deviates significantly (0.20–0.34), suggesting poor alignment or distinct characteristics.
3. **Consistency Across Sub-Benchmarks**: `Part 1–5` and `Text-Only` maintain high correlations (>0.69) across all KMMMLU Redux categories.
4. **Korean Benchmark**: Performs similarly to others but shows slightly lower values in `39` (0.61 vs. 0.70 in `All-39`).
### Interpretation
- **Alignment Insights**: High correlations (0.56–0.79) imply K-MetBench benchmarks are reliable proxies for KMMMLU Redux performance, except for `Multimodal`.
- **Multimodal Discrepancy**: The low scores for `Multimodal` may reflect architectural differences (e.g., multimodal inputs vs. text-only tasks) or evaluation methodology mismatches.
- **Korean Benchmark**: Suggests cultural or linguistic specificity in task design, as its `39` score lags behind others.
- **Part Consistency**: Uniform high scores across `Part 1–5` indicate these sub-benchmarks share similar evaluation principles.
The data underscores the generalizability of K-MetBench benchmarks to KMMMLU Redux, with the exception of multimodal tasks, which may require separate evaluation frameworks.
