## Heatmap: AUROC Scores for Different Categories ### Overview The image is a heatmap displaying AUROC (Area Under the Receiver Operating Characteristic curve) scores for different categories across three different methods or models, labeled as *t_g*, *t_p*, and *d_LR*. The heatmap uses a color gradient from red (low AUROC) to yellow (high AUROC) to represent the scores. The categories are listed on the left side of the heatmap. ### Components/Axes * **Title:** AUROC * **Columns:** * *t_g* (left column) * *t_p* (middle column) * *d_LR* (right column) * **Rows (Categories):** * cities * neg\_cities * sp\_en\_trans * neg\_sp\_en\_trans * inventors * neg\_inventors * animal\_class * neg\_animal\_class * element\_symb * neg\_element\_symb * facts * neg\_facts * **Color Scale (Legend):** Located on the right side of the heatmap. The scale ranges from 0.0 (red) to 1.0 (yellow). ### Detailed Analysis or Content Details Here's a breakdown of the AUROC scores for each category and method: * **cities:** * *t_g*: 1.00 (yellow) * *t_p*: 1.00 (yellow) * *d_LR*: 1.00 (yellow) * **neg\_cities:** * *t_g*: 1.00 (yellow) * *t_p*: 0.00 (red) * *d_LR*: 1.00 (yellow) * **sp\_en\_trans:** * *t_g*: 1.00 (yellow) * *t_p*: 1.00 (yellow) * *d_LR*: 1.00 (yellow) * **neg\_sp\_en\_trans:** * *t_g*: 1.00 (yellow) * *t_p*: 0.00 (red) * *d_LR*: 1.00 (yellow) * **inventors:** * *t_g*: 0.93 (yellow) * *t_p*: 0.94 (yellow) * *d_LR*: 0.93 (yellow) * **neg\_inventors:** * *t_g*: 0.97 (yellow) * *t_p*: 0.07 (red) * *d_LR*: 0.98 (yellow) * **animal\_class:** * *t_g*: 1.00 (yellow) * *t_p*: 0.99 (yellow) * *d_LR*: 1.00 (yellow) * **neg\_animal\_class:** * *t_g*: 1.00 (yellow) * *t_p*: 0.03 (red) * *d_LR*: 1.00 (yellow) * **element\_symb:** * *t_g*: 1.00 (yellow) * *t_p*: 1.00 (yellow) * *d_LR*: 1.00 (yellow) * **neg\_element\_symb:** * *t_g*: 1.00 (yellow) * *t_p*: 0.00 (red) * *d_LR*: 1.00 (yellow) * **facts:** * *t_g*: 0.95 (yellow) * *t_p*: 0.92 (yellow) * *d_LR*: 0.94 (yellow) * **neg\_facts:** * *t_g*: 0.92 (yellow) * *t_p*: 0.13 (red) * *d_LR*: 0.88 (yellow) ### Key Observations * The *t_g* and *d_LR* columns generally show high AUROC scores (mostly yellow), indicating good performance for these methods across most categories. * The *t_p* column shows significantly lower AUROC scores (red) for the "neg\_" categories (neg\_cities, neg\_sp\_en\_trans, neg\_inventors, neg\_animal\_class, neg\_element\_symb, neg\_facts), indicating poor performance for these categories with this method. * For positive categories (cities, sp\_en\_trans, inventors, animal\_class, element\_symb, facts), all three methods (*t_g*, *t_p*, and *d_LR*) show high AUROC scores. ### Interpretation The heatmap suggests that the *t_g* and *d_LR* methods perform well across all categories, while the *t_p* method struggles with the "neg\_" categories. This could indicate that the *t_p* method is not effective at distinguishing between negative examples in these categories. The high AUROC scores for the positive categories across all methods suggest that all three methods are good at identifying positive examples. The "neg\_" prefix likely indicates negative sampling or a similar technique, and the poor performance of *t_p* on these categories warrants further investigation. It is possible that *t_p* is overfitting to the positive examples or is not properly handling the negative examples in these specific categories.

\n ## Heatmap: Performance Metrics for Different Categories ### Overview This image presents a heatmap displaying performance metrics for ten categories and their corresponding negative counterparts. The metrics are represented by color intensity, with a scale ranging from 0.0 to 1.0. The heatmap is divided into three columns, each representing a different metric: *t_G*, *AUROC_tp*, and *d_LR*. The rows represent different categories, including both positive and negative examples. ### Components/Axes * **Rows (Categories):** cities, neg\_cities, sp\_en\_trans, neg\_sp\_en\_trans, inventors, neg\_inventors, animal\_class, neg\_animal\_class, element\_symb, neg\_element\_symb, facts, neg\_facts. * **Columns (Metrics):** * *t_G* (Top-left column) * *AUROC_tp* (Center column) - Area Under the Receiver Operating Characteristic curve for the positive class. * *d_LR* (Bottom-right column) - Log-likelihood ratio. * **Color Scale:** A vertical color bar on the right side of the heatmap indicates the mapping between color intensity and metric values. The scale ranges from approximately 0.0 (dark red) to 1.0 (yellow). * **Legend:** The color scale acts as the legend. ### Detailed Analysis The heatmap displays numerical values at the intersection of each category and metric. Here's a breakdown of the values, row by row: * **cities:** *t_G* = 1.00, *AUROC_tp* = 1.00, *d_LR* = 1.00 * **neg\_cities:** *t_G* = 1.00, *AUROC_tp* = 0.00, *d_LR* = 1.00 * **sp\_en\_trans:** *t_G* = 1.00, *AUROC_tp* = 1.00, *d_LR* = 1.00 * **neg\_sp\_en\_trans:** *t_G* = 1.00, *AUROC_tp* = 0.00, *d_LR* = 1.00 * **inventors:** *t_G* = 0.93, *AUROC_tp* = 0.94, *d_LR* = 0.93 * **neg\_inventors:** *t_G* = 0.97, *AUROC_tp* = 0.07, *d_LR* = 0.98 * **animal\_class:** *t_G* = 1.00, *AUROC_tp* = 0.99, *d_LR* = 1.00 * **neg\_animal\_class:** *t_G* = 1.00, *AUROC_tp* = 0.03, *d_LR* = 1.00 * **element\_symb:** *t_G* = 1.00, *AUROC_tp* = 1.00, *d_LR* = 1.00 * **neg\_element\_symb:** *t_G* = 1.00, *AUROC_tp* = 0.00, *d_LR* = 1.00 * **facts:** *t_G* = 0.95, *AUROC_tp* = 0.92, *d_LR* = 0.94 * **neg\_facts:** *t_G* = 0.92, *AUROC_tp* = 0.13, *d_LR* = 0.88 **Trends:** * For the *t_G* metric, most categories achieve a score of 1.00, except for "inventors" (0.93) and "facts" (0.95), and "neg_facts" (0.92). * The *AUROC_tp* metric shows a clear pattern: positive categories (cities, sp\_en\_trans, animal\_class, element\_symb) generally have values close to 1.00, while their negative counterparts (neg\_cities, neg\_sp\_en\_trans, neg\_animal\_class, neg\_element\_symb) have values close to 0.00. "inventors" and "facts" show intermediate values, while their negative counterparts show very low values. * The *d_LR* metric is consistently high (close to 1.00) for most categories, with "neg\_facts" being the lowest at 0.88. ### Key Observations * The negative examples consistently exhibit low *AUROC_tp* values, indicating poor performance in distinguishing positive from negative instances for those categories. * The *t_G* metric is generally high across all categories, suggesting good performance in a different aspect of the evaluation. * The *d_LR* metric is relatively stable across all categories, indicating a consistent ability to discriminate between classes. * The heatmap clearly differentiates between positive and negative examples based on the *AUROC_tp* metric. ### Interpretation This heatmap likely represents the performance of a classification model on different categories of data. The categories appear to be related to knowledge or information retrieval (cities, inventors, facts, etc.). The "neg\_" prefix indicates negative examples, likely created through some form of adversarial or contrastive learning. The high *t_G* values suggest the model is generally good at identifying relevant information. However, the low *AUROC_tp* values for the negative examples indicate that the model struggles to correctly identify *non*-examples of these categories. This could be due to several factors, such as: * **Data Imbalance:** The negative examples might be underrepresented in the training data. * **Feature Overlap:** The features used to represent the categories might not be sufficiently discriminative between positive and negative examples. * **Adversarial Examples:** The negative examples might be specifically designed to fool the model. The *d_LR* metric provides a measure of the model's confidence in its predictions. The relatively high values across all categories suggest that the model is generally confident in its classifications, even when it is incorrect (as evidenced by the low *AUROC_tp* for negative examples). This could indicate that the model is overconfident or that the features are not providing sufficient information to make accurate predictions. The heatmap provides valuable insights into the strengths and weaknesses of the model, highlighting areas where further improvement is needed. Specifically, addressing the poor performance on negative examples should be a priority.

## Heatmap Chart: AUROC Scores Across Categories and Methods ### Overview The image displays a heatmap chart titled "AUROC" (Area Under the Receiver Operating Characteristic Curve), which is a performance metric for classification models. The chart compares the AUROC scores of three different methods or models (labeled as columns) across twelve different categories or datasets (labeled as rows). The values range from 0.00 to 1.00, with a color scale indicating performance: bright yellow represents a perfect score of 1.0, transitioning through orange to red for scores approaching 0.0. ### Components/Axes * **Chart Title:** "AUROC" (centered at the top). * **Column Headers (Methods/Models):** * `t_g` (left column) * `t_p` (middle column) * `d_LR` (right column) * **Row Labels (Categories/Datasets):** Listed vertically on the left side. From top to bottom: 1. `cities` 2. `neg_cities` 3. `sp_en_trans` 4. `neg_sp_en_trans` 5. `inventors` 6. `neg_inventors` 7. `animal_class` 8. `neg_animal_class` 9. `element_symb` 10. `neg_element_symb` 11. `facts` 12. `neg_facts` * **Color Scale/Legend:** Positioned vertically on the far right. It is a gradient bar labeled from `0.0` (bottom, red) to `1.0` (top, yellow), with intermediate markers at `0.2`, `0.4`, `0.6`, and `0.8`. This scale maps the numerical AUROC values to colors in the heatmap cells. ### Detailed Analysis The heatmap contains a grid of 12 rows by 3 columns, with each cell displaying a numerical AUROC value and colored according to the scale. **Column `t_g` (Left):** * **Visual Trend:** This column shows consistently high performance. Almost all cells are bright yellow, indicating near-perfect scores. * **Data Points (Top to Bottom):** * `cities`: 1.00 * `neg_cities`: 1.00 * `sp_en_trans`: 1.00 * `neg_sp_en_trans`: 1.00 * `inventors`: 0.93 (slightly less yellow) * `neg_inventors`: 0.97 * `animal_class`: 1.00 * `neg_animal_class`: 1.00 * `element_symb`: 1.00 * `neg_element_symb`: 1.00 * `facts`: 0.95 * `neg_facts`: 0.92 **Column `t_p` (Middle):** * **Visual Trend:** This column exhibits extreme variability. It contains both perfect scores (bright yellow) and very low scores (deep red), creating a stark, alternating pattern. * **Data Points (Top to Bottom):** * `cities`: 1.00 * `neg_cities`: 0.00 (deep red) * `sp_en_trans`: 1.00 * `neg_sp_en_trans`: 0.00 (deep red) * `inventors`: 0.94 * `neg_inventors`: 0.07 (red) * `animal_class`: 0.99 * `neg_animal_class`: 0.03 (deep red) * `element_symb`: 1.00 * `neg_element_symb`: 0.00 (deep red) * `facts`: 0.92 * `neg_facts`: 0.13 (red) **Column `d_LR` (Right):** * **Visual Trend:** Similar to `t_g`, this column shows very high and stable performance across all categories, with all cells appearing bright yellow. * **Data Points (Top to Bottom):** * `cities`: 1.00 * `neg_cities`: 1.00 * `sp_en_trans`: 1.00 * `neg_sp_en_trans`: 1.00 * `inventors`: 0.93 * `neg_inventors`: 0.98 * `animal_class`: 1.00 * `neg_animal_class`: 1.00 * `element_symb`: 1.00 * `neg_element_symb`: 1.00 * `facts`: 0.94 * `neg_facts`: 0.88 (slightly less yellow than others in this column) ### Key Observations 1. **Method Performance Disparity:** Methods `t_g` and `d_LR` demonstrate robust, high performance (AUROC ≥ 0.88) across all twelve categories. In contrast, method `t_p` is highly unstable. 2. **Pattern in `t_p` Failures:** The `t_p` method fails catastrophically (AUROC ≤ 0.13) on every category prefixed with "neg_" (`neg_cities`, `neg_sp_en_trans`, `neg_inventors`, `neg_animal_class`, `neg_element_symb`). It performs perfectly or near-perfectly on their positive counterparts. 3. **Category Difficulty:** The `inventors` and `facts` categories (and their negations) appear slightly more challenging for all methods, as they are the only rows where scores dip below 0.95 for the high-performing models. 4. **Spatial Layout:** The legend is positioned to the right of the data grid. The row labels are left-aligned, and column headers are centered above their respective data columns. The numerical values are centered within each colored cell. ### Interpretation This heatmap likely evaluates different techniques (`t_g`, `t_p`, `d_LR`) for a binary classification task across various datasets. The "neg_" prefix suggests these are negated or adversarial versions of the base tasks (e.g., distinguishing non-cities from something else). The data suggests that `t_g` and `d_LR` are reliable, generalizable methods. The `t_p` method, however, reveals a critical flaw: it appears to rely on a superficial feature or bias present in the positive examples of the base tasks but completely absent or inverted in the negated tasks. This causes its performance to collapse to near-random (or worse) on the "neg_" datasets. This pattern is a classic sign of a model that has not learned the true underlying concept but has instead "cheated" by exploiting dataset-specific artifacts. The near-perfect scores for `t_g` and `d_LR` on most tasks could indicate either very effective models or potentially overly simplistic evaluation datasets. The slight performance dip on `inventors` and `facts` might point to these being more complex or noisy categories. The chart effectively communicates not just raw performance, but the *robustness* and *failure modes* of the compared methods.

## Heatmap: AUROC Metrics Across Categories ### Overview The image is a heatmap comparing three performance metrics (t_g, t_p, d_LR) across 12 categories. Values range from 0.00 to 1.00, with a color gradient from red (low) to yellow (high). The legend on the right maps colors to numerical values. ### Components/Axes - **Columns**: - `t_g` (leftmost column) - `t_p` (middle column) - `d_LR` (rightmost column) - **Rows**: Categories (e.g., cities, neg_cities, sp_en_trans, etc.) - **Legend**: Vertical color bar labeled "AUROC" with values from 0.0 (red) to 1.0 (yellow). ### Detailed Analysis | Category | t_g | t_p | d_LR | Color Notes | |------------------------|-------|-------|-------|--------------------------------------| | cities | 1.00 | 1.00 | 1.00 | Yellow (highest value) | | neg_cities | 1.00 | 0.00 | 1.00 | Red (lowest value) | | sp_en_trans | 1.00 | 1.00 | 1.00 | Yellow | | neg_sp_en_trans | 1.00 | 0.00 | 1.00 | Red | | inventors | 0.93 | 0.94 | 0.93 | Light yellow | | neg_inventors | 0.97 | 0.07 | 0.98 | Red (t_p) | | animal_class | 1.00 | 0.99 | 1.00 | Yellow | | neg_animal_class | 1.00 | 0.03 | 1.00 | Red (t_p) | | element_symb | 1.00 | 1.00 | 1.00 | Yellow | | neg_element_symb | 1.00 | 0.00 | 1.00 | Red | | facts | 0.95 | 0.92 | 0.94 | Light yellow | | neg_facts | 0.92 | 0.13 | 0.88 | Red (t_p) | ### Key Observations 1. **High Performance**: Most categories achieve near-perfect scores (1.00) in `t_g` and `d_LR`, with `t_p` also high except for negated categories. 2. **Negated Categories**: All "neg_" prefixed rows show drastically lower `t_p` values (e.g., neg_cities: 0.00, neg_inventors: 0.07), suggesting poor performance in this metric. 3. **Consistency in d_LR**: The `d_LR` metric remains consistently high (>0.88) across all categories, indicating robustness in this measure. 4. **Outliers**: - `neg_facts` has the lowest `d_LR` (0.88), slightly below others. - `neg_animal_class` has the lowest `t_p` (0.03). ### Interpretation - **t_p Discrepancy**: The `t_p` metric shows significant drops for negated categories, implying potential issues in handling negated terms or false positives in these cases. - **Robustness of d_LR**: The high `d_LR` values suggest the model maintains strong discriminative ability across all categories, even when `t_p` falters. - **Negation Impact**: The consistent underperformance of negated categories in `t_p` highlights a possible weakness in the model's ability to handle negated semantics, warranting further investigation into feature engineering or model architecture for such cases.