## Heatmap: Tokenizer Rank vs. p-value Distribution
### Overview
The image displays a heatmap visualizing the relationship between tokenizer rank and p-values across 18 ranked tokenizers. The color gradient transitions from blue (low p-values) to red (high p-values), with a diagonal pattern of intermediate values separating the two regions.
### Components/Axes
- **X-axis (Horizontal)**: "Tokenizer Rank" (1 to 18)
- **Y-axis (Vertical)**: "p-value vs. Lower Ranked Tokenizers" (1 to 18)
- **Color Bar (Right)**: Labeled "p-value" with a scale from 0.00 (blue) to 1.00 (red)
- **Grid**: Black gridlines separating cells
- **Annotations**: No embedded text in cells
### Detailed Analysis
1. **Top-Left Region (High p-values)**:
- Ranks 1–5 (y-axis) vs. 1–5 (x-axis) show dominant red shades.
- Example: Cell (1,1) = ~0.90, (2,2) = ~0.85, (3,3) = ~0.75.
- Gradual transition to orange in cells like (4,4) (~0.60) and (5,5) (~0.55).
2. **Diagonal Band (Intermediate p-values)**:
- Cells along the diagonal (e.g., 6–12 vs. 6–12) show mixed gray/blue shades.
- Example: (10,10) = ~0.15, (12,12) = ~0.10.
3. **Bottom-Right Region (Low p-values)**:
- Ranks 13–18 (y-axis) vs. 13–18 (x-axis) are predominantly blue.
- Example: (18,18) = ~0.01, (16,16) = ~0.02.
4. **Edge Cases**:
- Cell (17,17) = ~0.03 (light blue).
- Cell (15,15) = ~0.04 (light blue).
### Key Observations
- **Dominant Pattern**: A clear diagonal division separates high p-values (top-left) from low p-values (bottom-right).
- **Statistical Significance**: Higher-ranked tokenizers (1–5) exhibit weaker statistical significance (higher p-values) when compared to lower-ranked ones.
- **Threshold Effect**: The diagonal band suggests a potential cutoff where p-values drop below ~0.10 for ranks ≥10.
### Interpretation
The heatmap implies that tokenizer rankings correlate with statistical significance in their performance. Higher-ranked tokenizers (1–5) show less significant p-values when compared to themselves, while lower-ranked tokenizers (13–18) demonstrate stronger significance. The diagonal band may represent a critical threshold where p-values transition from non-significant (≥0.10) to significant (<0.10). This could reflect diminishing returns in tokenizer utility as rank increases, or a methodological artifact in the ranking process. The absence of extreme outliers suggests a consistent trend across the dataset.
