## Heatmap: AUROC Performance Comparison ### Overview The image is a heatmap displaying AUROC (Area Under the Receiver Operating Characteristic curve) values for different categories across three different models or conditions, labeled as *t_g*, *t_p*, and *d_LR*. The heatmap uses a color gradient from red (low AUROC) to yellow (high AUROC) to represent the performance of each category. ### Components/Axes * **Title:** AUROC * **Columns (Models/Conditions):** * *t_g* (top) * *t_p* (top) * *d_LR* (top) * **Rows (Categories):** * cities * neg\_cities * sp\_en\_trans * neg\_sp\_en\_trans * inventors * neg\_inventors * animal\_class * neg\_animal\_class * element\_symb * neg\_element\_symb * facts * neg\_facts * **Colorbar (AUROC Scale):** Ranges from 0.0 (red) to 1.0 (yellow). ### Detailed Analysis or Content Details Here's a breakdown of the AUROC values for each category and model: * **cities:** * *t_g*: 1.00 (yellow) * *t_p*: 1.00 (yellow) * *d_LR*: 1.00 (yellow) * **neg\_cities:** * *t_g*: 1.00 (yellow) * *t_p*: 0.00 (red) * *d_LR*: 1.00 (yellow) * **sp\_en\_trans:** * *t_g*: 1.00 (yellow) * *t_p*: 1.00 (yellow) * *d_LR*: 1.00 (yellow) * **neg\_sp\_en\_trans:** * *t_g*: 1.00 (yellow) * *t_p*: 0.00 (red) * *d_LR*: 1.00 (yellow) * **inventors:** * *t_g*: 0.97 (yellow) * *t_p*: 0.97 (yellow) * *d_LR*: 0.95 (yellow) * **neg\_inventors:** * *t_g*: 0.98 (yellow) * *t_p*: 0.04 (red) * *d_LR*: 0.98 (yellow) * **animal\_class:** * *t_g*: 1.00 (yellow) * *t_p*: 1.00 (yellow) * *d_LR*: 1.00 (yellow) * **neg\_animal\_class:** * *t_g*: 1.00 (yellow) * *t_p*: 0.01 (red) * *d_LR*: 1.00 (yellow) * **element\_symb:** * *t_g*: 1.00 (yellow) * *t_p*: 1.00 (yellow) * *d_LR*: 1.00 (yellow) * **neg\_element\_symb:** * *t_g*: 1.00 (yellow) * *t_p*: 0.00 (red) * *d_LR*: 1.00 (yellow) * **facts:** * *t_g*: 0.95 (yellow) * *t_p*: 0.88 (yellow) * *d_LR*: 0.95 (yellow) * **neg\_facts:** * *t_g*: 0.89 (yellow) * *t_p*: 0.10 (red) * *d_LR*: 0.91 (yellow) ### Key Observations * *t_g* and *d_LR* consistently show high AUROC values (close to 1.00) across all categories. * *t_p* shows significantly lower AUROC values (close to 0.00) for the "neg\_" prefixed categories (neg\_cities, neg\_sp\_en\_trans, neg\_inventors, neg\_animal\_class, neg\_element\_symb, neg\_facts). * The "neg\_" prefixed categories generally represent negative examples or counterfactuals of the corresponding positive categories. ### Interpretation The heatmap suggests that models *t_g* and *d_LR* perform well in distinguishing between positive and negative examples across all categories. However, model *t_p* struggles significantly with the "neg\_" prefixed categories, indicating a potential issue in handling negative examples or counterfactuals. This could be due to the model being biased towards positive examples or having difficulty in understanding the relationships between positive and negative counterparts. The high AUROC values for *t_g* and *d_LR* indicate strong performance in these tasks, while the near-zero values for *t_p* on negative examples suggest a failure to correctly classify these instances. ```

\n ## Heatmap: Performance Metrics for Different Categories ### Overview This image presents a heatmap displaying performance metrics for various categories. The heatmap has three columns representing different metrics: *t_G*, *AUROC_tp*, and *d_LR*. The rows represent different categories and their negative counterparts. The color intensity indicates the value of the metric, with yellow representing higher values and red representing lower values. ### Components/Axes * **Rows (Categories):** * cities * neg\_cities * sp\_en\_trans * neg\_sp\_en\_trans * inventors * neg\_inventors * animal\_class * neg\_animal\_class * element\_symb * neg\_element\_symb * facts * neg\_facts * **Columns (Metrics):** * t_G (Top-left) * AUROC_tp (Center) * d_LR (Top-right) * **Color Scale (Bottom-right):** Ranges from 0.0 (red) to 1.0 (yellow). * **Title:** "AUROC" is present above the columns. ### Detailed Analysis The heatmap displays numerical values at the intersection of each row and column. The values are as follows: | Category | t_G | AUROC_tp | d_LR | | ------------------ | -------- | -------- | -------- | | cities | 1.00 | 1.00 | 1.00 | | neg\_cities | 1.00 | 0.00 | 1.00 | | sp\_en\_trans | 1.00 | 1.00 | 1.00 | | neg\_sp\_en\_trans | 1.00 | 0.00 | 1.00 | | inventors | 0.97 | 0.97 | 0.95 | | neg\_inventors | 0.98 | 0.04 | 0.98 | | animal\_class | 1.00 | 1.00 | 1.00 | | neg\_animal\_class | 1.00 | 0.01 | 1.00 | | element\_symb | 1.00 | 1.00 | 1.00 | | neg\_element\_symb | 1.00 | 0.00 | 1.00 | | facts | 0.95 | 0.88 | 0.95 | | neg\_facts | 0.89 | 0.10 | 0.91 | **Trends:** * **t_G:** Most categories have a value of 1.00. *neg\_facts* has the lowest value at 0.89. * **AUROC_tp:** A clear pattern emerges: positive categories (e.g., cities, sp\_en\_trans) consistently score 1.00, while their negative counterparts (e.g., neg\_cities, neg\_sp\_en\_trans) score very low, close to 0.00. *neg\_inventors* and *neg\_animal\_class* also have very low scores. *facts* has a moderate score of 0.88. * **d_LR:** Values are generally high, mostly 1.00, across all categories. *neg\_facts* has the lowest value at 0.91. ### Key Observations * The negative categories consistently perform poorly on the *AUROC_tp* metric, indicating a difficulty in distinguishing between positive and negative instances for these categories. * The *t_G* and *d_LR* metrics are relatively stable across all categories, suggesting consistent performance in these aspects. * *neg\_facts* consistently shows the lowest performance across multiple metrics. ### Interpretation This heatmap likely represents the performance of a model or system in classifying or identifying different types of entities or concepts. The categories represent different types of data (cities, inventors, animal classes, etc.), and the metrics evaluate different aspects of performance. * **t_G** might represent a threshold or a measure of confidence. * **AUROC_tp** (Area Under the Receiver Operating Characteristic curve for true positives) indicates the model's ability to correctly identify positive instances. The low scores for negative categories suggest the model struggles to differentiate between true positives and false positives for those categories. * **d_LR** (Likelihood Ratio) measures the evidence provided by the model in favor of a positive instance. The consistent high performance on *t_G* and *d_LR* suggests the model is generally confident in its predictions, but the low *AUROC_tp* scores for negative categories indicate a potential bias or difficulty in handling negative instances. The *neg\_facts* category appears to be particularly problematic, requiring further investigation. The data suggests that the model is better at identifying the presence of a concept than its absence.

## Heatmap: AUROC Performance Across Categories and Metrics ### Overview The image is a heatmap visualizing the Area Under the Receiver Operating Characteristic curve (AUROC) scores for three different metrics (`t_g`, `t_p`, `d_LR`) across twelve distinct categories. The categories appear to be datasets or tasks, with some having a "neg_" prefix, likely indicating negative or adversarial versions. The heatmap uses a color scale from red (0.0) to yellow (1.0) to represent the AUROC score, with exact numerical values overlaid on each cell. ### Components/Axes * **Title:** "AUROC" (centered at the top). * **Column Headers (Metrics):** Three columns labeled `t_g`, `t_p`, and `d_LR` (from left to right). * **Row Labels (Categories):** Twelve categories listed vertically on the left side: 1. `cities` 2. `neg_cities` 3. `sp_en_trans` 4. `neg_sp_en_trans` 5. `inventors` 6. `neg_inventors` 7. `animal_class` 8. `neg_animal_class` 9. `element_symb` 10. `neg_element_symb` 11. `facts` 12. `neg_facts` * **Color Scale/Legend:** A vertical color bar located on the far right of the chart. It maps colors to AUROC values, ranging from **0.0 (red)** at the bottom to **1.0 (yellow)** at the top. Intermediate markers are at 0.2, 0.4, 0.6, and 0.8. * **Data Cells:** A 12x3 grid where each cell contains a numerical AUROC value and is colored according to the scale. ### Detailed Analysis The following table reconstructs the data from the heatmap. The color description is based on the visual mapping from the legend. | Category | `t_g` (AUROC) | `t_p` (AUROC) | `d_LR` (AUROC) | | :--- | :--- | :--- | :--- | | **cities** | 1.00 (Yellow) | 1.00 (Yellow) | 1.00 (Yellow) | | **neg_cities** | 1.00 (Yellow) | 0.00 (Red) | 1.00 (Yellow) | | **sp_en_trans** | 1.00 (Yellow) | 1.00 (Yellow) | 1.00 (Yellow) | | **neg_sp_en_trans** | 1.00 (Yellow) | 0.00 (Red) | 1.00 (Yellow) | | **inventors** | 0.97 (Yellow) | 0.97 (Yellow) | 0.95 (Yellow) | | **neg_inventors** | 0.98 (Yellow) | 0.04 (Red) | 0.98 (Yellow) | | **animal_class** | 1.00 (Yellow) | 1.00 (Yellow) | 1.00 (Yellow) | | **neg_animal_class** | 1.00 (Yellow) | 0.01 (Red) | 1.00 (Yellow) | | **element_symb** | 1.00 (Yellow) | 1.00 (Yellow) | 1.00 (Yellow) | | **neg_element_symb** | 1.00 (Yellow) | 0.00 (Red) | 1.00 (Yellow) | | **facts** | 0.95 (Yellow) | 0.88 (Yellow) | 0.95 (Yellow) | | **neg_facts** | 0.89 (Yellow) | 0.10 (Red) | 0.91 (Yellow) | **Trend Verification per Metric:** * **`t_g` (Left Column):** The line of values shows consistently high performance (AUROC ≥ 0.89). The trend is nearly perfect (1.00) for most categories, with slight dips for `inventors` (0.97), `facts` (0.95), and `neg_facts` (0.89). This metric appears robust across both standard and "neg_" categories. * **`t_p` (Middle Column):** This column shows a stark, binary trend. For standard categories (`cities`, `sp_en_trans`, `inventors`, `animal_class`, `element_symb`, `facts`), the AUROC is high (0.88 to 1.00). For their corresponding "neg_" prefixed categories, the AUROC drops dramatically to near zero (0.00 to 0.10), with the exception of `neg_inventors` at 0.04. This indicates the `t_p` metric is highly sensitive to the distinction between standard and "neg_" versions of the tasks. * **`d_LR` (Right Column):** This metric shows uniformly high performance (AUROC ≥ 0.91) across all categories, mirroring the robustness of `t_g`. The lowest score is for `neg_facts` (0.91). ### Key Observations 1. **Perfect Scores:** The metrics `t_g` and `d_LR` achieve a perfect AUROC of 1.00 on 7 out of 12 categories each. 2. **Catastrophic Failure of `t_p`:** The `t_p` metric fails completely (AUROC ≤ 0.10) on all categories prefixed with "neg_", except for a very low score of 0.04 on `neg_inventors`. 3. **Resilience of `t_g` and `d_LR`:** Both `t_g` and `d_LR` maintain high performance on the "neg_" categories, showing no significant drop compared to their standard counterparts. 4. **Hardest Category:** The `facts` and `neg_facts` categories yield the lowest scores across all three metrics, suggesting these tasks are more challenging for the models being evaluated. 5. **Spatial Layout:** The legend is positioned to the right of the main data grid. The column headers are centered above their respective data columns. Row labels are left-aligned. ### Interpretation This heatmap likely compares the performance of three different detection or classification methods (`t_g`, `t_p`, `d_LR`) on a set of benchmark tasks, some of which are adversarial or negative examples (the "neg_" categories). The data suggests a fundamental difference in how these metrics operate: * **`t_p` is a brittle metric.** It performs perfectly on standard tasks but fails catastrophically on their negative counterparts. This implies it may be overfit to specific features present in the standard data that are absent or inverted in the negative sets. It is not a reliable measure for adversarial robustness. * **`t_g` and `d_LR` are robust metrics.** They maintain high performance regardless of whether the category is standard or negative. This indicates they capture more generalizable and reliable signals for the underlying task, making them suitable for evaluating model performance in adversarial settings. The near-perfect scores for `t_g` and `d_LR` on most tasks could imply that the underlying models have mastered these benchmarks, or that the benchmarks themselves may not be sufficiently challenging to differentiate model capabilities beyond a certain point. The relative difficulty of the `facts` category provides a better point of comparison. The investigation would benefit from examining why `t_p` is so uniquely sensitive to the "neg_" transformation.

## Heatmap: Performance Metrics Across Categories ### Overview The image is a heatmap comparing three performance metrics (t_g, t_p, d_LR) across 12 categories. Values range from 0.00 (red) to 1.00 (yellow), with a color gradient indicating performance strength. The heatmap reveals systematic differences in metric performance between original and negated categories. ### Components/Axes - **X-axis (Columns)**: - t_g (green threshold) - t_p (purple threshold) - d_LR (distance to latent representation) - **Y-axis (Rows)**: - Categories: cities, neg_cities, sp_en_trans, neg_sp_en_trans, inventors, neg_inventors, animal_class, neg_animal_class, element_symb, neg_element_symb, facts, neg_facts - **Legend**: - Vertical color bar on the right (0.00 = red, 1.00 = yellow) - Spatial grounding: Legend occupies the rightmost 20% of the image, aligned vertically ### Detailed Analysis 1. **t_g Column**: - All values = 1.00 (yellow) - Spatial grounding: Uniform yellow across all rows - Trend: Perfect performance across all categories 2. **t_p Column**: - Original categories: - cities (1.00), sp_en_trans (1.00), inventors (0.97), animal_class (1.00), element_symb (1.00), facts (0.88) - Negated categories: - neg_cities (0.00), neg_sp_en_trans (0.00), neg_inventors (0.04), neg_animal_class (0.01), neg_element_symb (0.00), neg_facts (0.10) - Spatial grounding: Red dominates negated categories; yellow in original categories 3. **d_LR Column**: - All values ≥ 0.91 (yellow to light orange) - Spatial grounding: Consistent high performance across all rows - Notable: neg_facts (0.91) shows slight deviation from perfect score ### Key Observations 1. **t_p Sensitivity**: - Negated categories show dramatic drops in t_p (0.00-0.10 vs 0.88-1.00 in original) - Example: neg_inventors (t_p=0.04) vs inventors (t_p=0.97) 2. **d_LR Robustness**: - Maintains high scores (0.91-1.00) across all categories - Contrasts with t_p's category-specific performance 3. **t_g Consistency**: - Perfect scores (1.00) across all categories - Suggests uniform threshold effectiveness ### Interpretation The data demonstrates that: 1. **t_p metric** is highly sensitive to category negation, showing near-zero performance in negated categories (e.g., neg_cities, neg_sp_en_trans) 2. **d_LR metric** maintains high performance regardless of negation, suggesting it measures a more fundamental property 3. **t_g metric** shows perfect consistency across all categories, indicating uniform threshold effectiveness 4. The neg_inventors category is an outlier with exceptionally low t_p (0.04) despite high d_LR (0.98), suggesting unique challenges in this category's representation This pattern implies that t_p may be measuring category-specific features that are lost in negation, while d_LR captures more generalizable representations. The perfect t_g scores suggest the threshold itself is optimally calibrated across all categories.