## Heatmap: AUROC for Projections a^Tt
### Overview
The image presents two heatmaps comparing the Area Under the Receiver Operating Characteristic Curve (AUROC) for different projections. The left heatmap shows results when no projections are used ("Projected out: None"), while the right heatmap shows results when projections tG and tP are used ("Projected out: tG and tP"). The heatmaps display AUROC values for various test sets (cities, neg_cities, facts, neg_facts, facts_conj, facts_disj) against different training sets derived from "cities". The color intensity represents the AUROC value, with yellow indicating higher values and red indicating lower values.
### Components/Axes
* **Title:** AUROC for Projections a^Tt
* **X-axis (Train Set "cities"):** cities, + neg\_cities, + cities\_conj, + cities\_disj
* **Y-axis (Test Set):** cities, neg\_cities, facts, neg\_facts, facts\_conj, facts\_disj
* **Heatmap 1 Title:** Projected out: None
* **Heatmap 2 Title:** Projected out: tG and tP
* **Colorbar:** Ranges from 0.0 (red) to 1.0 (yellow). Increments of 0.2.
### Detailed Analysis
**Heatmap 1: Projected out: None**
| Test Set | cities | + neg\_cities | + cities\_conj | + cities\_disj |
| :---------- | :----- | :------------ | :------------- | :------------- |
| cities | 1.00 | 1.00 | 1.00 | 0.99 |
| neg\_cities | 0.46 | 1.00 | 1.00 | 0.99 |
| facts | 0.92 | 0.95 | 0.96 | 0.96 |
| neg\_facts | 0.47 | 0.91 | 0.89 | 0.89 |
| facts\_conj | 0.72 | 0.74 | 0.80 | 0.80 |
| facts\_disj | 0.64 | 0.70 | 0.76 | 0.78 |
* **cities:** The AUROC values are consistently high (0.99-1.00) across all training sets.
* **neg\_cities:** The AUROC value is low (0.46) when trained on "cities" alone, but high (0.99-1.00) when trained on the other sets.
* **facts:** The AUROC values are consistently high (0.92-0.96) across all training sets.
* **neg\_facts:** The AUROC value is relatively low (0.47) when trained on "cities" alone, but higher (0.89-0.91) when trained on the other sets.
* **facts\_conj:** The AUROC values range from 0.72 to 0.80.
* **facts\_disj:** The AUROC values range from 0.64 to 0.78.
**Heatmap 2: Projected out: tG and tP**
| Test Set | cities | + neg\_cities | + cities\_conj | + cities\_disj |
| :---------- | :----- | :------------ | :------------- | :------------- |
| cities | 1.00 | 1.00 | 1.00 | 0.99 |
| neg\_cities | 0.11 | 1.00 | 1.00 | 0.99 |
| facts | 0.22 | 0.21 | 0.36 | 0.37 |
| neg\_facts | 0.48 | 0.25 | 0.23 | 0.25 |
| facts\_conj | 0.41 | 0.47 | 0.80 | 0.80 |
| facts\_disj | 0.39 | 0.46 | 0.76 | 0.79 |
* **cities:** The AUROC values are consistently high (0.99-1.00) across all training sets.
* **neg\_cities:** The AUROC value is very low (0.11) when trained on "cities" alone, but high (0.99-1.00) when trained on the other sets.
* **facts:** The AUROC values are low (0.21-0.37) across all training sets.
* **neg\_facts:** The AUROC values are low (0.23-0.48) across all training sets.
* **facts\_conj:** The AUROC values are lower (0.41-0.47) when trained on "cities" and "+ neg_cities", but higher (0.80) when trained on "+ cities_conj" and "+ cities_disj".
* **facts\_disj:** The AUROC values are lower (0.39-0.46) when trained on "cities" and "+ neg_cities", but higher (0.76-0.79) when trained on "+ cities_conj" and "+ cities_disj".
### Key Observations
* Projecting out tG and tP significantly reduces the AUROC for "facts" and "neg_facts" test sets, regardless of the training set.
* For "neg_cities", projecting out tG and tP drastically reduces the AUROC when trained on "cities" alone.
* Training on "+ cities_conj" and "+ cities_disj" generally improves the AUROC compared to training on "cities" or "+ neg_cities", especially for "facts_conj" and "facts_disj" when tG and tP are projected out.
* The "cities" test set maintains high AUROC values regardless of the projection or training set.
### Interpretation
The heatmaps illustrate the impact of projecting out tG and tP on the performance of different models. Projecting out tG and tP seems to negatively affect the model's ability to generalize to "facts" and "neg_facts", suggesting that these projections contain information relevant to those test sets. The "cities" test set appears to be less sensitive to these projections. The improvement in AUROC when training on "+ cities_conj" and "+ cities_disj" suggests that these training sets provide more robust features for certain test sets, especially when tG and tP are projected out. The low AUROC for "neg_cities" when trained only on "cities" and projecting out tG and tP indicates a significant difference in the feature space between positive and negative city examples, and that the projections are important for distinguishing them.
