## Heatmap: P-value Comparison of Tokenizers ### Overview The image is a heatmap displaying the p-values resulting from a comparison of different tokenizers. The heatmap is triangular, showing pairwise comparisons between tokenizers ranked from 1 to 18. The color intensity represents the p-value, ranging from blue (low p-value) to red (high p-value). The heatmap is only filled in the lower triangle. ### Components/Axes * **X-axis:** "Tokenizer Rank" ranging from 1 to 18. * **Y-axis:** "p-value vs. Lower Ranked Tokenizers" ranging from 1 to 18. * **Colorbar (right side):** "p-value" ranging from 0.00 (blue) to 1.00 (red). The colorbar has the following markers: 0.00, 0.01, 0.02, 0.03, 0.04, 0.05, 0.10, 0.20, 0.30, 0.40, 0.50, 0.60, 0.70, 0.80, 0.90, 1.00. ### Detailed Analysis The heatmap is a lower triangular matrix, meaning only the comparisons where the y-axis tokenizer rank is greater than or equal to the x-axis tokenizer rank are shown. Each cell represents the p-value of comparing the tokenizer on the y-axis to the tokenizer on the x-axis. Here's a breakdown of the p-values for some specific tokenizer comparisons: * **Tokenizer 1 vs. Tokenizer 2:** p-value is approximately 0.80 (red). * **Tokenizer 1 vs. Tokenizer 3:** p-value is approximately 0.70 (red). * **Tokenizer 1 vs. Tokenizer 4:** p-value is approximately 0.90 (red). * **Tokenizer 1 vs. Tokenizer 5:** p-value is approximately 0.10 (light gray). * **Tokenizer 1 vs. Tokenizer 6:** p-value is approximately 0.10 (light gray). * **Tokenizer 1 vs. Tokenizer 7:** p-value is approximately 0.03 (light blue). * **Tokenizer 1 vs. Tokenizer 8:** p-value is approximately 0.02 (blue). * **Tokenizer 1 vs. Tokenizer 9:** p-value is approximately 0.10 (light gray). * **Tokenizer 2 vs. Tokenizer 3:** p-value is approximately 0.70 (red). * **Tokenizer 2 vs. Tokenizer 4:** p-value is approximately 0.80 (red). * **Tokenizer 2 vs. Tokenizer 5:** p-value is approximately 0.60 (red). * **Tokenizer 2 vs. Tokenizer 6:** p-value is approximately 0.10 (light gray). * **Tokenizer 2 vs. Tokenizer 7:** p-value is approximately 0.04 (light blue). * **Tokenizer 2 vs. Tokenizer 8:** p-value is approximately 0.03 (light blue). * **Tokenizer 2 vs. Tokenizer 9:** p-value is approximately 0.02 (blue). * **Tokenizer 3 vs. Tokenizer 4:** p-value is approximately 0.90 (red). * **Tokenizer 3 vs. Tokenizer 5:** p-value is approximately 0.10 (light gray). * **Tokenizer 3 vs. Tokenizer 6:** p-value is approximately 0.20 (light orange). * **Tokenizer 3 vs. Tokenizer 7:** p-value is approximately 0.04 (light blue). * **Tokenizer 3 vs. Tokenizer 3:** p-value is approximately 0.70 (red). * **Tokenizer 4 vs. Tokenizer 4:** p-value is approximately 0.90 (red). * **Tokenizer 4 vs. Tokenizer 5:** p-value is approximately 0.70 (red). * **Tokenizer 4 vs. Tokenizer 6:** p-value is approximately 0.20 (light orange). * **Tokenizer 4 vs. Tokenizer 7:** p-value is approximately 0.03 (light blue). * **Tokenizer 5 vs. Tokenizer 5:** p-value is approximately 0.10 (light gray). * **Tokenizer 5 vs. Tokenizer 6:** p-value is approximately 0.10 (light gray). * **Tokenizer 5 vs. Tokenizer 7:** p-value is approximately 0.02 (blue). * **Tokenizer 6 vs. Tokenizer 6:** p-value is approximately 0.10 (light gray). * **Tokenizer 6 vs. Tokenizer 7:** p-value is approximately 0.02 (blue). * **Tokenizer 7 vs. Tokenizer 7:** p-value is approximately 0.70 (red). * **Tokenizer 8 vs. Tokenizer 8:** p-value is approximately 0.03 (light blue). * **Tokenizer 9 vs. Tokenizer 9:** p-value is approximately 0.70 (red). * **Tokenizer 10 vs. Tokenizer 10:** p-value is approximately 0.10 (light gray). * **Tokenizer 11 vs. Tokenizer 11:** p-value is approximately 0.10 (light gray). * **Tokenizer 12 vs. Tokenizer 12:** p-value is approximately 0.70 (red). * **Tokenizer 13 vs. Tokenizer 13:** p-value is approximately 0.05 (light gray). * **Tokenizer 14 vs. Tokenizer 14:** p-value is approximately 0.10 (light gray). * **Tokenizer 15 vs. Tokenizer 15:** p-value is approximately 0.70 (red). * **Tokenizer 16 vs. Tokenizer 16:** p-value is approximately 0.05 (light gray). * **Tokenizer 17 vs. Tokenizer 17:** p-value is approximately 0.70 (red). * **Tokenizer 18 vs. Tokenizer 18:** p-value is approximately 0.00 (blue). ### Key Observations * The top-left portion of the heatmap (comparing lower-ranked tokenizers) generally shows higher p-values (red/orange), indicating less significant differences between those tokenizers. * The bottom-left portion of the heatmap (comparing higher-ranked tokenizers to lower-ranked ones) generally shows lower p-values (blue), indicating more significant differences. * There are some exceptions to the general trend, with some cells showing unexpected p-values. ### Interpretation The heatmap visualizes the statistical significance of differences between various tokenizers. Lower p-values suggest that the tokenizers being compared produce significantly different results. The general trend suggests that higher-ranked tokenizers tend to perform differently from lower-ranked ones, while lower-ranked tokenizers are more similar to each other. The specific p-values can be used to identify which tokenizers are statistically different and to guide the selection of appropriate tokenizers for specific tasks. The high p-values along the diagonal are expected, as they represent the comparison of a tokenizer with itself.

\n ## Heatmap: p-value vs. Tokenizer Rank ### Overview This image presents a heatmap visualizing the relationship between the p-value and the rank of different tokenizers. The heatmap displays p-values for comparisons between a tokenizer and all lower-ranked tokenizers. The color intensity represents the p-value, with warmer colors (red) indicating higher p-values and cooler colors (blue) indicating lower p-values. The heatmap is structured as an 18x18 grid, representing the comparison of each tokenizer (ranked 1 to 18) against all tokenizers with lower ranks. Black outlines separate the cells. ### Components/Axes * **X-axis:** "Tokenizer Rank" - Ranges from 1 to 18, representing the rank of the tokenizer. * **Y-axis:** "p-value vs. Lower Ranked Tokenizers" - Ranges from 1 to 18, representing the rank of the tokenizers being compared against. * **Color Scale (Legend):** Located on the right side of the image. It maps color intensity to p-value. * 0.00 (Light Blue) * 0.01 * 0.02 * 0.03 * 0.04 * 0.05 (Medium Blue) * 0.10 * 0.20 * 0.30 * 0.40 * 0.50 * 0.60 * 0.70 * 0.80 * 0.90 * 1.00 (Dark Red) ### Detailed Analysis The heatmap shows a clear diagonal pattern. The cells along the main diagonal (where the tokenizer rank on the x-axis equals the tokenizer rank on the y-axis) are generally lighter in color, indicating higher p-values. As you move away from the diagonal, the colors become progressively darker blue, indicating lower p-values. Here's a breakdown of approximate p-value ranges based on color and position: * **Rank 1:** * Rank 1 vs. Rank 1: ~0.95 (Dark Red) * Rank 1 vs. Rank 2: ~0.85 (Orange-Red) * Rank 1 vs. Rank 3: ~0.75 (Orange) * Rank 1 vs. Rank 4: ~0.60 (Orange) * Rank 1 vs. Rank 5: ~0.50 (Orange) * Rank 1 vs. Rank 6: ~0.40 (Light Orange) * Rank 1 vs. Rank 7: ~0.30 (Light Blue) * Rank 1 vs. Rank 8: ~0.20 (Light Blue) * Rank 1 vs. Rank 9: ~0.10 (Light Blue) * Rank 1 vs. Rank 10: ~0.05 (Medium Blue) * Rank 1 vs. Rank 11: ~0.03 (Dark Blue) * Rank 1 vs. Rank 12: ~0.02 (Dark Blue) * Rank 1 vs. Rank 13: ~0.01 (Dark Blue) * Rank 1 vs. Rank 14: ~0.01 (Dark Blue) * Rank 1 vs. Rank 15: ~0.01 (Dark Blue) * Rank 1 vs. Rank 16: ~0.01 (Dark Blue) * Rank 1 vs. Rank 17: ~0.00 (Dark Blue) * Rank 1 vs. Rank 18: ~0.00 (Dark Blue) * **Rank 2:** * Rank 2 vs. Rank 1: ~0.85 (Orange-Red) * Rank 2 vs. Rank 2: ~0.90 (Dark Red) * Rank 2 vs. Rank 3: ~0.70 (Orange) * Rank 2 vs. Rank 4: ~0.55 (Orange) * Rank 2 vs. Rank 5: ~0.45 (Light Orange) * Rank 2 vs. Rank 6: ~0.35 (Light Blue) * Rank 2 vs. Rank 7: ~0.25 (Light Blue) * Rank 2 vs. Rank 8: ~0.15 (Light Blue) * Rank 2 vs. Rank 9: ~0.05 (Medium Blue) * Rank 2 vs. Rank 10: ~0.03 (Dark Blue) * Rank 2 vs. Rank 11: ~0.02 (Dark Blue) * Rank 2 vs. Rank 12: ~0.01 (Dark Blue) * Rank 2 vs. Rank 13: ~0.01 (Dark Blue) * Rank 2 vs. Rank 14: ~0.01 (Dark Blue) * Rank 2 vs. Rank 15: ~0.01 (Dark Blue) * Rank 2 vs. Rank 16: ~0.01 (Dark Blue) * Rank 2 vs. Rank 17: ~0.00 (Dark Blue) * Rank 2 vs. Rank 18: ~0.00 (Dark Blue) * **Rank 18:** * Rank 18 vs. Rank 1: ~0.00 (Dark Blue) * Rank 18 vs. Rank 2: ~0.00 (Dark Blue) * Rank 18 vs. Rank 3: ~0.00 (Dark Blue) * Rank 18 vs. Rank 4: ~0.00 (Dark Blue) * Rank 18 vs. Rank 5: ~0.00 (Dark Blue) * Rank 18 vs. Rank 6: ~0.00 (Dark Blue) * Rank 18 vs. Rank 7: ~0.00 (Dark Blue) * Rank 18 vs. Rank 8: ~0.00 (Dark Blue) * Rank 18 vs. Rank 9: ~0.00 (Dark Blue) * Rank 18 vs. Rank 10: ~0.00 (Dark Blue) * Rank 18 vs. Rank 11: ~0.00 (Dark Blue) * Rank 18 vs. Rank 12: ~0.00 (Dark Blue) * Rank 18 vs. Rank 13: ~0.00 (Dark Blue) * Rank 18 vs. Rank 14: ~0.00 (Dark Blue) * Rank 18 vs. Rank 15: ~0.00 (Dark Blue) * Rank 18 vs. Rank 16: ~0.00 (Dark Blue) * Rank 18 vs. Rank 17: ~0.01 (Dark Blue) * Rank 18 vs. Rank 18: ~0.95 (Dark Red) ### Key Observations * The p-values generally decrease as the rank difference between the two tokenizers increases. * The highest p-values are observed when comparing a tokenizer to itself (diagonal). * There is a noticeable gradient from red (high p-value) to blue (low p-value) as you move away from the diagonal. * The lower-right corner of the heatmap (comparing lower-ranked tokenizers) consistently shows very low p-values. ### Interpretation This heatmap suggests that higher-ranked tokenizers are statistically significantly different from lower-ranked tokenizers. The high p-values along the diagonal indicate that each tokenizer is very similar to itself (as expected). As we compare a tokenizer to those with lower ranks, the p-values decrease, indicating a growing statistical difference. This implies that the ranking system effectively differentiates between tokenizers based on some underlying performance metric. The consistently low p-values in the lower-right corner suggest that the lowest-ranked tokenizers are significantly different from all higher-ranked tokenizers. This could indicate that these tokenizers are substantially less effective or perform differently than the others. The heatmap provides a visual representation of the statistical significance of differences between tokenizers, allowing for a quick assessment of their relative performance.

## Heatmap: Statistical Significance (p-values) of Tokenizer Rank Comparisons ### Overview The image is a triangular heatmap visualizing p-values from statistical comparisons between tokenizers of different ranks. The chart displays a matrix where each cell represents the p-value resulting from a comparison between a tokenizer at a specific rank (x-axis) and a tokenizer at a lower rank (y-axis). The color intensity indicates the magnitude of the p-value, with a clear threshold at 0.05 for statistical significance. ### Components/Axes * **Chart Type:** Lower-triangular heatmap (the upper triangle is empty). * **X-Axis:** Labeled **"Tokenizer Rank"**. It has numerical markers from **1 to 18**, increasing from left to right. * **Y-Axis:** Labeled **"p-value vs. Lower Ranked Tokenizers"**. It has numerical markers from **1 to 18**, increasing from top to bottom. * **Color Scale/Legend:** Located on the right side. It is a vertical gradient bar labeled **"p-value"**. * The scale ranges from **0.00 (dark blue)** to **1.00 (dark red)**. * A critical threshold is marked at **0.05**, where the color transitions from shades of blue (p < 0.05) to shades of orange/red (p > 0.05). * Specific labeled ticks on the scale are: 0.00, 0.01, 0.02, 0.03, 0.04, 0.05, 0.10, 0.20, 0.30, 0.40, 0.50, 0.60, 0.70, 0.80, 0.90, 1.00. * **Visual Encoding:** * **Color:** Represents the p-value. Blue hues indicate low p-values (statistically significant difference), while orange/red hues indicate high p-values (no significant difference). * **Black Borders:** Certain cells are outlined with a thick black border. These borders are used to highlight specific cells, likely those with p-values below a certain threshold (e.g., p < 0.05) or of particular interest. ### Detailed Analysis The heatmap is a lower-triangular matrix, meaning it only shows comparisons where the rank on the y-axis is greater than or equal to the rank on the x-axis (i.e., comparing a higher-numbered rank to a lower-numbered rank). **Spatial and Color Pattern Analysis:** 1. **Top-Left Region (Ranks 1-6):** This area contains a mix of colors. Cells comparing very low ranks (e.g., Rank 1 vs. 2, Rank 2 vs. 3) show orange to red colors, indicating high p-values (p > 0.10, often > 0.30). This suggests no statistically significant difference between the performance of the very top-ranked tokenizers. Several of these cells have black borders. 2. **Diagonal and Near-Diagonal:** Cells comparing ranks that are close together (e.g., Rank 5 vs. 6, Rank 9 vs. 10) often show light orange or beige colors, with p-values frequently in the 0.10 to 0.40 range. Many of these cells are bordered in black. 3. **Bottom-Left Region (High y-rank vs. Low x-rank):** This large region is dominated by deep blue colors. For example, comparisons like Rank 18 vs. 1, Rank 15 vs. 2, or Rank 12 vs. 3 all show very dark blue, corresponding to p-values near **0.00 to 0.02**. This indicates a highly statistically significant difference when comparing a low-ranked tokenizer to a much higher-ranked one. 4. **Trend:** There is a clear gradient from the top-right (high p-values, red/orange) to the bottom-left (low p-values, blue). As the difference in rank between the two tokenizers being compared increases (moving down and to the left on the matrix), the p-value decreases dramatically. **Key Data Points (Approximate p-values from color):** * **Rank 1 vs. Rank 2:** p ≈ 0.60 - 0.70 (orange-red, bordered) * **Rank 2 vs. Rank 3:** p ≈ 0.50 - 0.60 (orange, bordered) * **Rank 5 vs. Rank 6:** p ≈ 0.20 - 0.30 (light orange, bordered) * **Rank 9 vs. Rank 10:** p ≈ 0.10 - 0.20 (beige, bordered) * **Rank 10 vs. Rank 11:** p ≈ 0.04 - 0.05 (light blue/grey, bordered) * **Rank 14 vs. Rank 15:** p ≈ 0.03 - 0.04 (light blue, bordered) * **Rank 17 vs. Rank 18:** p ≈ 0.10 - 0.20 (light orange, bordered) * **Rank 18 vs. Rank 1:** p ≈ 0.00 - 0.01 (dark blue) * **Rank 15 vs. Rank 3:** p ≈ 0.01 - 0.02 (dark blue) * **Rank 12 vs. Rank 5:** p ≈ 0.02 - 0.03 (medium blue) ### Key Observations 1. **Significant Hierarchy:** The data strongly suggests a performance hierarchy among the tokenizers. Tokenizers with lower rank numbers (1, 2, 3...) are not significantly different from each other (high p-values), but they are significantly different from tokenizers with much higher rank numbers (low p-values). 2. **Clustering at the Top:** The top 5-6 ranked tokenizers form a cluster where intra-group comparisons yield non-significant p-values. 3. **Clear Significance Threshold:** The color break at p=0.05 visually separates statistically significant comparisons (blue) from non-significant ones (orange/red). The black borders appear to primarily, but not exclusively, highlight cells with p-values near or above this threshold. 4. **Asymmetry:** The comparison is directional ("vs. Lower Ranked Tokenizers"). The heatmap only shows one direction of the pairwise comparison (e.g., Rank 5 vs. Rank 10 is shown, but Rank 10 vs. Rank 5 is not, as it would be in the empty upper triangle). ### Interpretation This heatmap is a statistical visualization tool likely used in machine learning or natural language processing research to evaluate tokenizer performance. The "Tokenizer Rank" probably corresponds to an ordering based on a performance metric (e.g., compression efficiency, downstream task accuracy). The data demonstrates that **performance differences are only statistically meaningful between tokenizers that are far apart in the ranking**. The top-performing tokenizers (ranks 1-6) are statistically indistinguishable from one another, forming a "top tier." However, any tokenizer in this top tier is significantly better than a tokenizer from the lower ranks (e.g., ranks 12-18). This suggests a plateau of performance at the top, with a clear drop-off to lower-performing models. The black borders likely serve to draw the viewer's attention to specific comparisons of interest, perhaps those that are "borderline" significant (p ≈ 0.05) or comparisons between adjacent ranks that the researchers wanted to highlight. The overall pattern validates the ranking system by showing that large rank differences correspond to large, statistically verifiable performance gaps.

## Heatmap: Tokenizer Rank vs. p-value Distribution ### Overview The image displays a heatmap visualizing the relationship between tokenizer rank and p-values across 18 ranked tokenizers. The color gradient transitions from blue (low p-values) to red (high p-values), with a diagonal pattern of intermediate values separating the two regions. ### Components/Axes - **X-axis (Horizontal)**: "Tokenizer Rank" (1 to 18) - **Y-axis (Vertical)**: "p-value vs. Lower Ranked Tokenizers" (1 to 18) - **Color Bar (Right)**: Labeled "p-value" with a scale from 0.00 (blue) to 1.00 (red) - **Grid**: Black gridlines separating cells - **Annotations**: No embedded text in cells ### Detailed Analysis 1. **Top-Left Region (High p-values)**: - Ranks 1–5 (y-axis) vs. 1–5 (x-axis) show dominant red shades. - Example: Cell (1,1) = ~0.90, (2,2) = ~0.85, (3,3) = ~0.75. - Gradual transition to orange in cells like (4,4) (~0.60) and (5,5) (~0.55). 2. **Diagonal Band (Intermediate p-values)**: - Cells along the diagonal (e.g., 6–12 vs. 6–12) show mixed gray/blue shades. - Example: (10,10) = ~0.15, (12,12) = ~0.10. 3. **Bottom-Right Region (Low p-values)**: - Ranks 13–18 (y-axis) vs. 13–18 (x-axis) are predominantly blue. - Example: (18,18) = ~0.01, (16,16) = ~0.02. 4. **Edge Cases**: - Cell (17,17) = ~0.03 (light blue). - Cell (15,15) = ~0.04 (light blue). ### Key Observations - **Dominant Pattern**: A clear diagonal division separates high p-values (top-left) from low p-values (bottom-right). - **Statistical Significance**: Higher-ranked tokenizers (1–5) exhibit weaker statistical significance (higher p-values) when compared to lower-ranked ones. - **Threshold Effect**: The diagonal band suggests a potential cutoff where p-values drop below ~0.10 for ranks ≥10. ### Interpretation The heatmap implies that tokenizer rankings correlate with statistical significance in their performance. Higher-ranked tokenizers (1–5) show less significant p-values when compared to themselves, while lower-ranked tokenizers (13–18) demonstrate stronger significance. The diagonal band may represent a critical threshold where p-values transition from non-significant (≥0.10) to significant (<0.10). This could reflect diminishing returns in tokenizer utility as rank increases, or a methodological artifact in the ranking process. The absence of extreme outliers suggests a consistent trend across the dataset.