## Heatmap: AUROC for Projections a^Tt ### Overview The image presents two side-by-side heatmaps comparing Area Under the Receiver Operating Characteristic (AUROC) values for different combinations of training and test sets under two projection scenarios: (1) no projections applied, and (2) projections of `t_G` and `t_P` applied. The heatmaps use a color gradient from red (low AUROC) to yellow (high AUROC), with numerical values explicitly labeled in each cell. --- ### Components/Axes - **X-axis (Train Set "cities")**: Subcategories: - `cities` - `+ neg_cities` - `+ cities_conj` - `+ cities_disj` - **Y-axis (Test Set)**: Subcategories: - `cities` - `neg_cities` - `facts` - `neg_facts` - `facts_conj` - `facts_disj` - **Legend**: A color bar on the right maps AUROC values from 0.0 (red) to 1.0 (yellow). - **Key Titles**: - Main title: "AUROC for Projections a^Tt" - Subtitles: - Left heatmap: "Projected out: None" - Right heatmap: "Projected out: t_G and t_P" --- ### Detailed Analysis #### Left Heatmap (Projected out: None) - **Trend**: AUROC values are generally high (≥0.76) across most combinations, with the highest values (1.00–0.99) along the diagonal where train and test sets match. - **Notable Values**: - `cities` vs. `cities`: 1.00 - `neg_cities` vs. `cities`: 0.79 - `facts` vs. `cities`: 0.92 - `facts_disj` vs. `cities`: 0.56 #### Right Heatmap (Projected out: t_G and t_P) - **Trend**: AUROC values drop significantly compared to the left heatmap, with many cells in red/orange (≤0.63). The diagonal still shows high values (0.98–1.00), but cross-category performance degrades sharply. - **Notable Values**: - `cities` vs. `neg_cities`: 0.02 (lowest value) - `facts` vs. `cities`: 0.23 - `facts_disj` vs. `cities`: 0.33 - `cities_disj` vs. `cities_disj`: 0.63 --- ### Key Observations 1. **Diagonal Dominance**: Both heatmaps show high AUROC values (0.98–1.00) along the diagonal, indicating strong performance when train and test sets match. 2. **Projection Impact**: Projecting out `t_G` and `t_P` reduces AUROC values by 30–50% for cross-category comparisons (e.g., `neg_cities` vs. `cities` drops from 0.79 to 0.02). 3. **Category Sensitivity**: - `facts` and `neg_facts` show moderate performance in the left heatmap but collapse under projection (e.g., `facts` vs. `cities` drops from 0.92 to 0.23). - `facts_conj` and `facts_disj` exhibit the most severe degradation when projections are applied. --- ### Interpretation The data demonstrates that projecting out `t_G` and `t_P` severely limits the model's ability to generalize across different categories. While the unprojected scenario maintains high AUROC values even for dissimilar categories (e.g., `facts` vs. `cities` at 0.92), the projected scenario collapses performance for these cases (0.23). This suggests that `t_G` and `t_P` contain critical information for distinguishing between categories, and their removal disrupts the model's discriminative power. The diagonal dominance in both heatmaps highlights the importance of matching train-test distributions for optimal performance. **Notable Anomaly**: The extreme drop in AUROC for `neg_cities` vs. `cities` under projection (0.02) indicates a near-total failure to distinguish these categories when `t_G` and `t_P` are removed, possibly due to overlapping features in the projected space.