## Heatmap: AUROC for Projections a^Tt ### Overview The image presents two heatmaps comparing the Area Under the Receiver Operating Characteristic curve (AUROC) for different projections. The left heatmap shows results when no projections are used ("Projected out: None"), while the right heatmap shows results when projections t_G and t_P are used ("Projected out: t_G and t_P"). The heatmaps compare performance across different test sets (rows) and train sets (columns), with the color intensity indicating the AUROC score. ### Components/Axes * **Title:** AUROC for Projections a^Tt * **X-axis (Train Set):** "cities", "+ neg\_cities", "+ cities\_conj", "+ cities\_disj" * **Y-axis (Test Set):** "cities", "neg\_cities", "facts", "neg\_facts", "facts\_conj", "facts\_disj" * **Colorbar:** Ranges from 0.0 to 1.0, with colors transitioning from red (low AUROC) to yellow (high AUROC). * 0.0: Red * 0.2: Orange-Red * 0.4: Orange * 0.6: Yellow-Orange * 0.8: Yellow * 1.0: Bright Yellow * **Heatmap 1 Title:** Projected out: None * **Heatmap 2 Title:** Projected out: t_G and t_P ### Detailed Analysis **Heatmap 1: Projected out: None** | Test Set | cities | + neg\_cities | + cities\_conj | + cities\_disj | | :---------- | :----- | :------------- | :------------- | :------------- | | cities | 1.00 | 1.00 | 1.00 | 1.00 | | neg\_cities | 0.80 | 1.00 | 1.00 | 1.00 | | facts | 0.93 | 0.95 | 0.96 | 0.96 | | neg\_facts | 0.53 | 0.92 | 0.90 | 0.90 | | facts\_conj | 0.77 | 0.83 | 0.85 | 0.85 | | facts\_disj | 0.65 | 0.73 | 0.76 | 0.77 | * **cities:** All values are 1.00, indicating perfect performance. * **neg\_cities:** Starts at 0.80 with "cities" training set, then increases to 1.00 for all other training sets. * **facts:** Values range from 0.93 to 0.96, showing consistently high performance. * **neg\_facts:** Starts at 0.53 with "cities" training set, then increases to around 0.90 for other training sets. * **facts\_conj:** Values range from 0.77 to 0.85. * **facts\_disj:** Values range from 0.65 to 0.77. **Heatmap 2: Projected out: t_G and t_P** | Test Set | cities | + neg\_cities | + cities\_conj | + cities\_disj | | :---------- | :----- | :------------- | :------------- | :------------- | | cities | 1.00 | 1.00 | 1.00 | 0.99 | | neg\_cities | 0.14 | 1.00 | 1.00 | 0.99 | | facts | 0.22 | 0.20 | 0.42 | 0.44 | | neg\_facts | 0.39 | 0.19 | 0.27 | 0.29 | | facts\_conj | 0.26 | 0.36 | 0.82 | 0.83 | | facts\_disj | 0.33 | 0.47 | 0.75 | 0.77 | * **cities:** Values are close to 1.00, except for the last value which is 0.99. * **neg\_cities:** Starts at 0.14 with "cities" training set, then increases to around 1.00 for other training sets. * **facts:** Values range from 0.20 to 0.44, showing lower performance compared to the "None" projection. * **neg\_facts:** Values range from 0.19 to 0.39, showing lower performance compared to the "None" projection. * **facts\_conj:** Values range from 0.26 to 0.83. * **facts\_disj:** Values range from 0.33 to 0.77. ### Key Observations * When no projections are used, the model performs very well on the "cities" and "neg\_cities" test sets, achieving near-perfect AUROC scores. * Projecting out t_G and t_P significantly reduces performance on the "cities" and "neg\_cities" test sets when trained on "cities" alone. * Training on "+ neg\_cities", "+ cities\_conj", and "+ cities\_disj" generally improves performance compared to training on "cities" alone, especially when projections are used. * The "facts", "neg\_facts", "facts\_conj", and "facts\_disj" test sets show lower AUROC scores compared to "cities" and "neg\_cities", particularly when projections are used. ### Interpretation The heatmaps illustrate the impact of projecting out t_G and t_P on the AUROC scores for different test and train set combinations. The results suggest that projecting out these features can significantly degrade performance, especially when the model is trained on a limited dataset like "cities" alone. This indicates that t_G and t_P contain important information for distinguishing between positive and negative examples in the "cities" and "neg\_cities" test sets. The improved performance when training on combined datasets ("+ neg\_cities", "+ cities\_conj", "+ cities\_disj") suggests that these datasets provide a more diverse and representative training signal, mitigating the negative impact of projecting out t_G and t_P. The lower AUROC scores for the "facts", "neg\_facts", "facts\_conj", and "facts\_disj" test sets may indicate that these datasets are more challenging or require different features for optimal performance.

## Heatmap: AUROC for Projections Aᵀt ### Overview The image presents two heatmaps displaying Area Under the Receiver Operating Characteristic curve (AUROC) values for different combinations of train and test sets. The heatmaps compare performance when projecting out different variables (None vs. τ_G and τ_P). The color scale ranges from red (low AUROC, ~0.0) to yellow (high AUROC, ~1.0). ### Components/Axes * **Title:** "AUROC for Projections Aᵀt" * **Subtitles:** * Left: "Projected out: None" * Right: "Projected out: τ_G and τ_P" * **X-axis (Train Set "cities"):** Categories: "cities", "+ neg\_cities", "+ cities\_conj", "+ cities\_disj" * **Y-axis (Test Set):** Categories: "cities", "neg\_cities", "facts", "neg\_facts", "facts\_conj", "facts\_disj" * **Color Scale:** Ranges from approximately 0.0 (red) to 1.0 (yellow). The scale is positioned on the right side of the image. * **Legend:** Located on the right side of the image, showing the mapping between color and AUROC value. ### Detailed Analysis or Content Details **Left Heatmap (Projected out: None)** The left heatmap shows AUROC values when no variables are projected out. The values generally decrease as you move down the Y-axis (from "cities" to "facts\_disj"). * **cities vs. cities:** 1.00 * **cities vs. neg\_cities:** 0.80 * **cities vs. facts:** 0.93 * **cities vs. neg\_facts:** 0.53 * **cities vs. facts\_conj:** 0.77 * **cities vs. facts\_disj:** 0.65 * **+ neg\_cities vs. cities:** 1.00 * **+ neg\_cities vs. neg\_cities:** 1.00 * **+ neg\_cities vs. facts:** 0.95 * **+ neg\_cities vs. neg\_facts:** 0.92 * **+ neg\_cities vs. facts\_conj:** 0.83 * **+ neg\_cities vs. facts\_disj:** 0.73 * **+ cities\_conj vs. cities:** 1.00 * **+ cities\_conj vs. neg\_cities:** 1.00 * **+ cities\_conj vs. facts:** 0.96 * **+ cities\_conj vs. neg\_facts:** 0.90 * **+ cities\_conj vs. facts\_conj:** 0.85 * **+ cities\_conj vs. facts\_disj:** 0.76 * **+ cities\_disj vs. cities:** 1.00 * **+ cities\_disj vs. neg\_cities:** 1.00 * **+ cities\_disj vs. facts:** 0.96 * **+ cities\_disj vs. neg\_facts:** 0.90 * **+ cities\_disj vs. facts\_conj:** 0.85 * **+ cities\_disj vs. facts\_disj:** 0.77 **Right Heatmap (Projected out: τ_G and τ_P)** The right heatmap shows AUROC values when τ_G and τ_P are projected out. The values are generally lower than in the left heatmap, especially for combinations involving "facts" and "neg\_facts". * **cities vs. cities:** 1.00 * **cities vs. neg\_cities:** 0.14 * **cities vs. facts:** 0.22 * **cities vs. neg\_facts:** 0.39 * **cities vs. facts\_conj:** 0.26 * **cities vs. facts\_disj:** 0.33 * **+ neg\_cities vs. cities:** 1.00 * **+ neg\_cities vs. neg\_cities:** 1.00 * **+ neg\_cities vs. facts:** 0.20 * **+ neg\_cities vs. neg\_facts:** 0.19 * **+ neg\_cities vs. facts\_conj:** 0.36 * **+ neg\_cities vs. facts\_disj:** 0.47 * **+ cities\_conj vs. cities:** 1.00 * **+ cities\_conj vs. neg\_cities:** 1.00 * **+ cities\_conj vs. facts:** 0.42 * **+ cities\_conj vs. neg\_facts:** 0.27 * **+ cities\_conj vs. facts\_conj:** 0.82 * **+ cities\_conj vs. facts\_disj:** 0.75 * **+ cities\_disj vs. cities:** 1.00 * **+ cities\_disj vs. neg\_cities:** 1.00 * **+ cities\_disj vs. facts:** 0.44 * **+ cities\_disj vs. neg\_facts:** 0.29 * **+ cities\_disj vs. facts\_conj:** 0.83 * **+ cities\_disj vs. facts\_disj:** 0.77 ### Key Observations * The AUROC values are generally higher when no variables are projected out (left heatmap). * Projecting out τ_G and τ_P significantly reduces the AUROC values, particularly when the test set includes "facts" or "neg\_facts". * The highest AUROC values are consistently observed when the train and test sets are both "cities". * The lowest AUROC values in the right heatmap are observed when the test set is "neg\_facts" and the train set is "+ neg\_cities" (0.19). ### Interpretation The data suggests that the projections τ_G and τ_P are important for distinguishing between "cities" and "facts" (or their variations). When these projections are removed, the ability to discriminate between these categories is substantially reduced, as evidenced by the lower AUROC values in the right heatmap. This implies that τ_G and τ_P capture information relevant to differentiating between city-related data and factual data. The consistently high AUROC values when both train and test sets are "cities" indicate that the model performs very well at identifying "cities" within "cities". However, performance degrades when the test set includes "facts" or "neg\_facts", suggesting that the model struggles to generalize to these different data types, especially when τ_G and τ_P are removed. The significant drop in AUROC when projecting out τ_G and τ_P for "facts" and "neg\_facts" suggests these projections are crucial for representing the characteristics that distinguish factual information from city-related information. The model relies heavily on these projections to perform well on these types of data.

## Comparative Heatmap Chart: AUROC for Projections a^T t ### Overview The image displays two side-by-side heatmaps comparing the Area Under the Receiver Operating Characteristic curve (AUROC) performance of a model under two different projection conditions. The overall title is "AUROC for Projections a^T t". The left heatmap shows results when "Projected out: None", and the right heatmap shows results when "Projected out: t_G and t_P". Performance is measured across various test sets when the model is trained on different combinations of data, all based on a core "cities" dataset. ### Components/Axes * **Main Title:** "AUROC for Projections a^T t" (Top center, spanning both charts). * **Left Heatmap Subtitle:** "Projected out: None" (Top left). * **Right Heatmap Subtitle:** "Projected out: t_G and t_P" (Top right). * **Y-Axis (Both Heatmaps):** Labeled "Test Set". Categories from top to bottom: * `cities` * `neg_cities` * `facts` * `neg_facts` * `facts_conj` * `facts_disj` * **X-Axis (Both Heatmaps):** Labeled "Train Set 'cities'". Categories from left to right: * `cities` * `+ neg_cities` * `+ cities_conj` * `+ cities_disj` * **Color Bar (Right side):** A vertical scale indicating AUROC values. The scale runs from 0.0 (dark red) to 1.0 (bright yellow), with intermediate markers at 0.2, 0.4, 0.6, and 0.8. ### Detailed Analysis The heatmaps contain numerical AUROC values in each cell. Values are transcribed below with the format: `[Test Set] | [Train Set Condition]: [AUROC Value]`. **Left Heatmap (Projected out: None):** * **cities:** `cities`: 1.00 | `+ neg_cities`: 1.00 | `+ cities_conj`: 1.00 | `+ cities_disj`: 1.00 * **neg_cities:** `cities`: 0.80 | `+ neg_cities`: 1.00 | `+ cities_conj`: 1.00 | `+ cities_disj`: 1.00 * **facts:** `cities`: 0.93 | `+ neg_cities`: 0.95 | `+ cities_conj`: 0.96 | `+ cities_disj`: 0.96 * **neg_facts:** `cities`: 0.53 | `+ neg_cities`: 0.92 | `+ cities_conj`: 0.90 | `+ cities_disj`: 0.90 * **facts_conj:** `cities`: 0.77 | `+ neg_cities`: 0.83 | `+ cities_conj`: 0.85 | `+ cities_disj`: 0.85 * **facts_disj:** `cities`: 0.65 | `+ neg_cities`: 0.73 | `+ cities_conj`: 0.76 | `+ cities_disj`: 0.77 **Right Heatmap (Projected out: t_G and t_P):** * **cities:** `cities`: 1.00 | `+ neg_cities`: 1.00 | `+ cities_conj`: 1.00 | `+ cities_disj`: 0.99 * **neg_cities:** `cities`: 0.14 | `+ neg_cities`: 1.00 | `+ cities_conj`: 1.00 | `+ cities_disj`: 0.99 * **facts:** `cities`: 0.22 | `+ neg_cities`: 0.20 | `+ cities_conj`: 0.42 | `+ cities_disj`: 0.44 * **neg_facts:** `cities`: 0.39 | `+ neg_cities`: 0.19 | `+ cities_conj`: 0.27 | `+ cities_disj`: 0.29 * **facts_conj:** `cities`: 0.26 | `+ neg_cities`: 0.36 | `+ cities_conj`: 0.82 | `+ cities_disj`: 0.83 * **facts_disj:** `cities`: 0.33 | `+ neg_cities`: 0.47 | `+ cities_conj`: 0.75 | `+ cities_disj`: 0.77 ### Key Observations 1. **Performance Collapse with Projection:** The most striking pattern is the dramatic drop in AUROC for most test sets when moving from the left heatmap (no projection) to the right heatmap (projecting out t_G and t_P). This is visually represented by the shift from predominantly yellow cells to predominantly red/orange cells. 2. **Robustness of the `cities` Test Set:** The `cities` test set maintains near-perfect performance (AUROC ≈ 1.00) across all training conditions in both projection settings. It is the only test set unaffected by the projection. 3. **Impact on Negated Data:** The `neg_cities` test set shows extreme sensitivity. Without projection, training on `cities` alone yields a moderate 0.80, which improves to 1.00 with additional data. With projection, training on `cities` alone collapses to 0.14 (worse than random), but recovers to 1.00 when `neg_cities` is included in the training set. 4. **Generalization to "facts":** Performance on the `facts` and `neg_facts` test sets is generally high without projection but suffers severely with projection, especially when the training set is limited to `cities` or `+ neg_cities`. Including conjunctive/disjunctive data (`+ cities_conj`, `+ cities_disj`) provides partial recovery. 5. **Conjunctive/Disjunctive Test Sets:** The `facts_conj` and `facts_disj` test sets show a similar pattern: poor performance with projection when trained on basic sets, but significant recovery (AUROC > 0.75) when the training set includes the corresponding conjunctive or disjunctive data (`+ cities_conj` or `+ cities_disj`). ### Interpretation This chart investigates the role of specific model components or directions, denoted as `t_G` and `t_P`, in generalization. The "projection out" operation likely removes the influence of these components from the model's representations. * **Core Finding:** The components `t_G` and `t_P` appear to be **critical for generalization** beyond the specific `cities` task. Their removal (right heatmap) causes performance to plummet on all test sets except the in-distribution `cities` set. This suggests these components encode broad, transferable knowledge. * **Task-Specific vs. General Knowledge:** The model's perfect performance on `cities` even after projection indicates that knowledge specific to that task is stored in other components. The catastrophic failure on `neg_cities` (when trained only on `cities`) after projection implies that understanding negation relies heavily on these general components (`t_G`, `t_P`). * **Data Efficiency and Compositionality:** Including negated or compositional data (`+ neg_cities`, `+ cities_conj`, etc.) in training can compensate for the loss of `t_G` and `t_P` to a significant degree. This demonstrates that the model can learn these reasoning skills directly from data, but under normal conditions (left heatmap), it preferentially uses the more efficient, general-purpose `t_G` and `t_P` components. * **Peircean Investigation:** The chart acts as a diagnostic tool. By systematically removing components (`t_G`, `t_P`) and testing on varied logical forms (negation, conjunction, disjunction), the researchers can ablate and identify which parts of the model are responsible for which reasoning capabilities. The stark contrast between the two heatmaps provides strong evidence that `t_G` and `t_P` are not merely task-specific features but are fundamental to the model's ability to generalize its understanding.

## Heatmap: AUROC for Projections a^Tt ### Overview The image presents two side-by-side heatmaps comparing the Area Under the Receiver Operating Characteristic (AUROC) values for different test and train set combinations under two projection scenarios: "Projected out: None" (left) and "Projected out: t_G and t_P" (right). The heatmaps use a color gradient from red (low AUROC) to yellow (high AUROC) to represent performance metrics. --- ### Components/Axes - **X-axis (Train Set "cities")**: - Categories: `cities`, `+ neg_cities`, `+ cities_conj`, `+ cities_disj` - **Y-axis (Test Set)**: - Categories: `cities`, `neg_cities`, `facts`, `neg_facts`, `facts_conj`, `facts_disj` - **Legend**: - Vertical color bar on the right with values from 0.0 (red) to 1.0 (yellow). - **Main Title**: "AUROC for Projections a^Tt" - **Subtitles**: - Left: "Projected out: None" - Right: "Projected out: t_G and t_P" --- ### Detailed Analysis #### Left Section ("Projected out: None") | Test Set \ Train Set | cities | + neg_cities | + cities_conj | + cities_disj | |----------------------|--------|--------------|---------------|---------------| | **cities** | 1.00 | 1.00 | 1.00 | 1.00 | | **neg_cities** | 0.80 | 1.00 | 1.00 | 1.00 | | **facts** | 0.93 | 0.95 | 0.96 | 0.96 | | **neg_facts** | 0.53 | 0.92 | 0.90 | 0.90 | | **facts_conj** | 0.77 | 0.83 | 0.85 | 0.85 | | **facts_disj** | 0.65 | 0.73 | 0.76 | 0.77 | #### Right Section ("Projected out: t_G and t_P") | Test Set \ Train Set | cities | + neg_cities | + cities_conj | + cities_disj | |----------------------|--------|--------------|---------------|---------------| | **cities** | 1.00 | 1.00 | 1.00 | 0.99 | | **neg_cities** | 0.14 | 1.00 | 1.00 | 0.99 | | **facts** | 0.22 | 0.20 | 0.42 | 0.44 | | **neg_facts** | 0.39 | 0.19 | 0.27 | 0.29 | | **facts_conj** | 0.26 | 0.36 | 0.82 | 0.83 | | **facts_disj** | 0.33 | 0.47 | 0.75 | 0.77 | --- ### Key Observations 1. **High AUROC in "Projected out: None"**: - All test sets achieve near-perfect AUROC (1.00) when trained on `cities` and `+ neg_cities` in the left section. - Negative categories (`neg_cities`, `neg_facts`) show moderate to high performance (0.53–0.92). 2. **Significant Drop in "Projected out: t_G and t_P"**: - **Negative categories** (e.g., `neg_cities`, `neg_facts`) experience drastic declines: - `neg_cities` drops from 0.80 (left) to 0.14 (right). - `neg_facts` drops from 0.53 (left) to 0.39 (right). - **Positive categories** (e.g., `cities`, `facts`) also decline but less severely: - `facts` drops from 0.93 (left) to 0.22 (right). - **Conjunction/disjunction categories** (`facts_conj`, `facts_disj`) show mixed results, with `facts_conj` improving slightly in the right section (0.82 vs. 0.77). 3. **Color Gradient Consistency**: - Red cells (low AUROC) dominate the right section for negative categories, while yellow cells (high AUROC) dominate the left section. --- ### Interpretation - **Impact of Projection**: Projecting out `t_G` and `t_P` severely degrades the model's ability to distinguish negative instances (`neg_cities`, `neg_facts`), suggesting these features are critical for performance. - **Stability of Positive Categories**: `cities` and `facts` retain higher AUROC values even after projection, indicating robustness in positive class discrimination. - **Conjunction/Disjunction Behavior**: The improvement in `facts_conj` and `facts_disj` under projection might reflect reduced noise or overfitting, but this is offset by the loss in negative class performance. - **Practical Implications**: The model's reliance on `t_G` and `t_P` for negative class discrimination highlights a potential vulnerability in scenarios where these features are removed. This analysis underscores the trade-off between feature projection and model performance, emphasizing the importance of retaining key features for negative class tasks.