## Heatmap: Dataset Performance Comparison ### Overview The image is a heatmap visualizing the performance of models trained on different datasets and tested on different datasets. The color intensity represents the performance score, ranging from blue (low) to red (high). The x-axis represents the "Test dataset" and the y-axis represents the "Train dataset". ### Components/Axes * **X-axis (Test dataset):** TriviaQA, HotpotQA, Movies, Winobias, Winogrande, NLI, IMDB, Math, HotpotQA\_WC, NQ\_WC * **Y-axis (Train dataset):** TriviaQA, HotpotQA, Movies, Winobias, Winogrande, NLI, IMDB, Math, HotpotQA\_WC, NQ\_WC * **Color Scale:** Ranges from 0.0 (blue) to 1.0 (red), with intermediate shades representing values in between. ### Detailed Analysis The heatmap displays performance scores for each combination of training and testing datasets. The values are explicitly written in each cell. Here's a breakdown of the values: * **TriviaQA:** * Trained on TriviaQA: 0.82 * Trained on HotpotQA: 0.69 * Trained on Movies: 0.69 * Trained on Winobias: 0.53 * Trained on Winogrande: 0.52 * Trained on NLI: 0.52 * Trained on IMDB: 0.59 * Trained on Math: 0.82 * Trained on HotpotQA\_WC: 0.50 * Trained on NQ\_WC: 0.55 * **HotpotQA:** * Trained on TriviaQA: 0.76 * Trained on HotpotQA: 0.82 * Trained on Movies: 0.70 * Trained on Winobias: 0.54 * Trained on Winogrande: 0.53 * Trained on NLI: 0.51 * Trained on IMDB: 0.59 * Trained on Math: 0.79 * Trained on HotpotQA\_WC: 0.63 * Trained on NQ\_WC: 0.55 * **Movies:** * Trained on TriviaQA: 0.70 * Trained on HotpotQA: 0.58 * Trained on Movies: 0.82 * Trained on Winobias: 0.60 * Trained on Winogrande: 0.51 * Trained on NLI: 0.56 * Trained on IMDB: 0.54 * Trained on Math: 0.54 * Trained on HotpotQA\_WC: 0.52 * Trained on NQ\_WC: 0.56 * **Winobias:** * Trained on TriviaQA: 0.63 * Trained on HotpotQA: 0.60 * Trained on Movies: 0.62 * Trained on Winobias: 0.91 * Trained on Winogrande: 0.53 * Trained on NLI: 0.52 * Trained on IMDB: 0.77 * Trained on Math: 0.74 * Trained on HotpotQA\_WC: 0.56 * Trained on NQ\_WC: 0.51 * **Winogrande:** * Trained on TriviaQA: 0.61 * Trained on HotpotQA: 0.55 * Trained on Movies: 0.60 * Trained on Winobias: 0.65 * Trained on Winogrande: 0.65 * Trained on NLI: 0.62 * Trained on IMDB: 0.86 * Trained on Math: 0.54 * Trained on HotpotQA\_WC: 0.50 * Trained on NQ\_WC: 0.53 * **NLI:** * Trained on TriviaQA: 0.57 * Trained on HotpotQA: 0.53 * Trained on Movies: 0.59 * Trained on Winobias: 0.57 * Trained on Winogrande: 0.52 * Trained on NLI: 0.94 * Trained on IMDB: 0.70 * Trained on Math: 0.56 * Trained on HotpotQA\_WC: 0.51 * Trained on NQ\_WC: 0.53 * **IMDB:** * Trained on TriviaQA: 0.60 * Trained on HotpotQA: 0.53 * Trained on Movies: 0.62 * Trained on Winobias: 0.66 * Trained on Winogrande: 0.52 * Trained on NLI: 0.67 * Trained on IMDB: 0.97 * Trained on Math: 0.57 * Trained on HotpotQA\_WC: 0.58 * Trained on NQ\_WC: 0.52 * **Math:** * Trained on TriviaQA: 0.62 * Trained on HotpotQA: 0.53 * Trained on Movies: 0.57 * Trained on Winobias: 0.51 * Trained on Winogrande: 0.51 * Trained on NLI: 0.51 * Trained on IMDB: 0.74 * Trained on Math: 0.96 * Trained on HotpotQA\_WC: 0.54 * Trained on NQ\_WC: 0.56 * **HotpotQA\_WC:** * Trained on TriviaQA: 0.67 * Trained on HotpotQA: 0.68 * Trained on Movies: 0.55 * Trained on Winobias: 0.51 * Trained on Winogrande: 0.53 * Trained on NLI: 0.58 * Trained on IMDB: 0.78 * Trained on Math: 0.75 * Trained on HotpotQA\_WC: 0.77 * Trained on NQ\_WC: 0.50 * **NQ\_WC:** * Trained on TriviaQA: 0.66 * Trained on HotpotQA: 0.56 * Trained on Movies: 0.68 * Trained on Winobias: 0.58 * Trained on Winogrande: 0.55 * Trained on NLI: 0.53 * Trained on IMDB: 0.53 * Trained on Math: 0.56 * Trained on HotpotQA\_WC: 0.54 * Trained on NQ\_WC: 0.75 ### Key Observations * The diagonal elements (training and testing on the same dataset) generally have higher scores, indicating that models perform best on data similar to what they were trained on. * IMDB and Math datasets show high performance when trained and tested on themselves (0.97 and 0.96 respectively). * Winobias and NLI also show high performance when trained and tested on themselves (0.91 and 0.94 respectively). * Cross-dataset performance varies, with some datasets showing better generalization than others. ### Interpretation The heatmap illustrates the transferability of models across different datasets. Training a model on a specific dataset and testing it on the same dataset generally yields the best performance. However, the performance drops when the model is tested on a different dataset, indicating that the models are not perfectly generalizable. The extent of the performance drop varies depending on the similarity between the training and testing datasets. For example, models trained on TriviaQA perform relatively well on HotpotQA, suggesting that these two datasets share some common characteristics. Conversely, models trained on NQ\_WC tend to perform worse on other datasets. The high diagonal values suggest that the models are learning dataset-specific features, rather than generalizable knowledge.

## Heatmap: Train-Test Dataset Performance Correlation ### Overview This image presents a heatmap visualizing the correlation between different training datasets and test datasets. The color intensity represents the correlation coefficient, ranging from 0.0 to 1.0. The heatmap displays the performance of various models trained on one dataset and evaluated on another. ### Components/Axes * **X-axis:** Test dataset. Categories are: TriviaQA, HotpotQA, Movies, Winobias, Winogrande, NLI, IMDB, Math, HotpotQA\_WC, NQ\_WC. * **Y-axis:** Train dataset. Categories are: TriviaQA, HotpotQA, Movies, Winobias, Winogrande, NLI, IMDB, Math, HotpotQA\_WC, NQ\_WC. * **Color Scale (Legend):** Located on the right side of the heatmap. Ranges from blue (0.0) to red (1.0), representing low to high correlation. * 0.0 is represented by a light blue color. * 1.0 is represented by a dark red color. * **Labels:** Each cell in the heatmap contains a numerical value representing the correlation coefficient. ### Detailed Analysis The heatmap shows the correlation coefficients between each pair of train and test datasets. Here's a breakdown of the values, reading row by row (Train dataset vs. Test datasets): * **TriviaQA (Train):** * TriviaQA (Test): 0.82 * HotpotQA (Test): 0.69 * Movies (Test): 0.69 * Winobias (Test): 0.53 * Winogrande (Test): 0.52 * NLI (Test): 0.52 * IMDB (Test): 0.59 * Math (Test): 0.82 * HotpotQA\_WC (Test): 0.50 * NQ\_WC (Test): 0.55 * **HotpotQA (Train):** * TriviaQA (Test): 0.76 * HotpotQA (Test): 0.82 * Movies (Test): 0.70 * Winobias (Test): 0.54 * Winogrande (Test): 0.53 * NLI (Test): 0.51 * IMDB (Test): 0.59 * Math (Test): 0.79 * HotpotQA\_WC (Test): 0.63 * NQ\_WC (Test): 0.55 * **Movies (Train):** * TriviaQA (Test): 0.70 * HotpotQA (Test): 0.58 * Movies (Test): 0.82 * Winobias (Test): 0.60 * Winogrande (Test): 0.51 * NLI (Test): 0.56 * IMDB (Test): 0.54 * Math (Test): 0.54 * HotpotQA\_WC (Test): 0.52 * NQ\_WC (Test): 0.56 * **Winobias (Train):** * TriviaQA (Test): 0.63 * HotpotQA (Test): 0.60 * Movies (Test): 0.60 * Winobias (Test): 0.91 * Winogrande (Test): 0.53 * NLI (Test): 0.52 * IMDB (Test): 0.77 * Math (Test): 0.74 * HotpotQA\_WC (Test): 0.56 * NQ\_WC (Test): 0.51 * **Winogrande (Train):** * TriviaQA (Test): 0.61 * HotpotQA (Test): 0.55 * Movies (Test): 0.60 * Winobias (Test): 0.65 * Winogrande (Test): 0.62 * NLI (Test): 0.86 * IMDB (Test): 0.54 * Math (Test): 0.50 * HotpotQA\_WC (Test): 0.53 * NQ\_WC (Test): 0.53 * **NLI (Train):** * TriviaQA (Test): 0.57 * HotpotQA (Test): 0.53 * Movies (Test): 0.59 * Winobias (Test): 0.57 * Winogrande (Test): 0.52 * NLI (Test): 0.94 * IMDB (Test): 0.70 * Math (Test): 0.56 * HotpotQA\_WC (Test): 0.51 * NQ\_WC (Test): 0.53 * **IMDB (Train):** * TriviaQA (Test): 0.60 * HotpotQA (Test): 0.53 * Movies (Test): 0.62 * Winobias (Test): 0.66 * Winogrande (Test): 0.52 * NLI (Test): 0.67 * IMDB (Test): 0.97 * Math (Test): 0.57 * HotpotQA\_WC (Test): 0.58 * NQ\_WC (Test): 0.52 * **Math (Train):** * TriviaQA (Test): 0.62 * HotpotQA (Test): 0.53 * Movies (Test): 0.57 * Winobias (Test): 0.51 * Winogrande (Test): 0.51 * NLI (Test): 0.51 * IMDB (Test): 0.74 * Math (Test): 0.96 * HotpotQA\_WC (Test): 0.54 * NQ\_WC (Test): 0.56 * **HotpotQA\_WC (Train):** * TriviaQA (Test): 0.67 * HotpotQA (Test): 0.68 * Movies (Test): 0.55 * Winobias (Test): 0.51 * Winogrande (Test): 0.53 * NLI (Test): 0.58 * IMDB (Test): 0.78 * Math (Test): 0.75 * HotpotQA\_WC (Test): 0.77 * NQ\_WC (Test): 0.50 * **NQ\_WC (Train):** * TriviaQA (Test): 0.66 * HotpotQA (Test): 0.56 * Movies (Test): 0.68 * Winobias (Test): 0.58 * Winogrande (Test): 0.55 * NLI (Test): 0.53 * IMDB (Test): 0.53 * Math (Test): 0.56 * HotpotQA\_WC (Test): 0.54 * NQ\_WC (Test): 0.75 ### Key Observations * The highest correlations are observed when a dataset is used for both training and testing (diagonal elements), with values close to 1.0 (e.g., Winobias-Winobias: 0.91, NLI-NLI: 0.94, IMDB-IMDB: 0.97, Math-Math: 0.96). * The correlation between TriviaQA and Math is relatively high (0.82), suggesting some shared characteristics or transferability between these datasets. * The correlations between datasets are generally lower for datasets like Winogrande, NLI, and Math when tested against datasets like HotpotQA or Movies. * The "WC" datasets (HotpotQA\_WC and NQ\_WC) show moderate correlations with other datasets, but generally lower than the original datasets. ### Interpretation This heatmap demonstrates the degree to which models trained on one dataset generalize to other datasets. High correlation coefficients indicate that a model trained on one dataset is likely to perform well on another. The diagonal dominance suggests that models perform best when tested on data similar to what they were trained on. The lower off-diagonal values highlight the challenges of transfer learning and the importance of dataset selection. The "WC" datasets, potentially representing a different data collection or processing method, exhibit lower correlations, suggesting they may have different characteristics than the original datasets. This information is valuable for selecting appropriate training data for specific tasks and understanding the limitations of models trained on particular datasets. The heatmap provides a quantitative assessment of dataset similarity and transferability, which can guide model development and evaluation.

\n ## Heatmap: Cross-Dataset Performance ### Overview The image is a heatmap visualizing a matrix of numerical performance scores (likely accuracy, F1, or a similar metric) between different "Train datasets" (rows) and "Test datasets" (columns). The values range from 0.0 to 1.0, with a color gradient from blue (low) to red (high) indicating the score. The chart is designed to show how well a model trained on one dataset generalizes to another. ### Components/Axes * **Y-Axis (Vertical):** Labeled **"Train dataset"**. It lists 10 datasets used for training: 1. TriviaQA 2. HotpotQA 3. Movies 4. Winobias 5. Winogrande 6. NLI 7. IMDB 8. Math 9. HotpotQA_WC 10. NQ_WC * **X-Axis (Horizontal):** Labeled **"Test dataset"**. It lists the same 10 datasets used for testing, in the same order as the Y-axis. * **Color Scale/Legend:** Located on the right side of the chart. It is a vertical bar showing the mapping of color to numerical value. * **Range:** 0.0 (bottom, blue) to 1.0 (top, red). * **Key Markers:** 0.0, 0.2, 0.4, 0.6, 0.8, 1.0. * **Data Grid:** A 10x10 grid of colored cells. Each cell contains a numerical value (to two decimal places) representing the performance score for the corresponding Train-Test dataset pair. ### Detailed Analysis The following table reconstructs the entire data matrix. Values are transcribed directly from the image. Rows represent the "Train dataset" and columns represent the "Test dataset". | Train \ Test | TriviaQA | HotpotQA | Movies | Winobias | Winogrande | NLI | IMDB | Math | HotpotQA_WC | NQ_WC | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | **TriviaQA** | **0.82** | 0.69 | 0.69 | 0.53 | 0.52 | 0.52 | 0.59 | 0.82 | 0.50 | 0.55 | | **HotpotQA** | 0.76 | **0.82** | 0.70 | 0.54 | 0.53 | 0.51 | 0.59 | 0.79 | 0.63 | 0.55 | | **Movies** | 0.70 | 0.58 | **0.82** | 0.60 | 0.51 | 0.56 | 0.54 | 0.54 | 0.52 | 0.56 | | **Winobias** | 0.63 | 0.60 | 0.62 | **0.91** | 0.53 | 0.52 | 0.77 | 0.74 | 0.56 | 0.51 | | **Winogrande** | 0.61 | 0.55 | 0.60 | 0.65 | **0.65** | 0.62 | 0.86 | 0.54 | 0.50 | 0.53 | | **NLI** | 0.57 | 0.53 | 0.59 | 0.57 | 0.52 | **0.94** | 0.70 | 0.56 | 0.51 | 0.53 | | **IMDB** | 0.60 | 0.53 | 0.62 | 0.66 | 0.52 | 0.67 | **0.97** | 0.57 | 0.58 | 0.52 | | **Math** | 0.62 | 0.53 | 0.57 | 0.51 | 0.51 | 0.51 | 0.74 | **0.96** | 0.54 | 0.56 | | **HotpotQA_WC** | 0.67 | 0.68 | 0.55 | 0.51 | 0.53 | 0.58 | 0.78 | 0.75 | **0.77** | 0.50 | | **NQ_WC** | 0.66 | 0.56 | 0.68 | 0.58 | 0.55 | 0.53 | 0.53 | 0.56 | 0.54 | **0.75** | **Trend Verification & Spatial Grounding:** * **Diagonal Trend:** The cells where the Train and Test dataset are the same (the main diagonal from top-left to bottom-right) are consistently the highest values in their respective rows and are colored dark red. This indicates strong within-dataset performance. * **High Off-Diagonal Values:** Notable high scores exist between related datasets. For example: * Train: **Winobias** (0.91) -> Test: **Winobias** (dark red). * Train: **IMDB** (0.97) -> Test: **IMDB** (darkest red on the chart). * Train: **NLI** (0.94) -> Test: **NLI** (dark red). * Train: **Winogrande** (0.86) -> Test: **IMDB** (medium red). * Train: **TriviaQA** (0.82) -> Test: **Math** (medium red). * **Low Values:** The lowest scores (lightest colors, near 0.5) are often found in the lower-right quadrant of the matrix, particularly when training on QA datasets (TriviaQA, HotpotQA) and testing on others, or vice-versa. ### Key Observations 1. **Strongest Performance:** The single highest score is **0.97** for the **IMDB** train/test pair. 2. **Weakest Performance:** The lowest scores appear to be around **0.50-0.51**. Examples include: * Train: **TriviaQA** -> Test: **HotpotQA_WC** (0.50) * Train: **HotpotQA_WC** -> Test: **NQ_WC** (0.50) * Train: **Winobias** -> Test: **NQ_WC** (0.51) * Train: **NLI** -> Test: **HotpotQA_WC** (0.51) 3. **Dataset Clusters:** Some datasets show stronger cross-performance: * **IMDB** and **Winogrande** have a high mutual score (Train Winogrande -> Test IMDB = 0.86). * **TriviaQA** and **Math** show a surprisingly high transfer (Train TriviaQA -> Test Math = 0.82). * **HotpotQA** and **HotpotQA_WC** show moderate transfer (0.63 and 0.68 in respective directions). 4. **Asymmetry:** Performance is not always symmetric. For example: * Train on **Winobias**, Test on **IMDB**: **0.77** * Train on **IMDB**, Test on **Winobias**: **0.66** ### Interpretation This heatmap provides a diagnostic view of model generalization across diverse NLP tasks (Question Answering, Commonsense Reasoning, Sentiment Analysis, Natural Language Inference, etc.). * **What it demonstrates:** The high diagonal values confirm that models perform best when tested on the same distribution they were trained on. The off-diagonal values reveal the **transfer learning potential** between datasets. High off-diagonal scores suggest the datasets share underlying features or task structures that a model can leverage. * **Relationships between elements:** The matrix acts as a similarity map. Datasets that are "close" in this map (high mutual scores, like IMDB and Winogrande) are likely more similar in the skills they require or the data patterns they contain. Datasets with low mutual scores are more distinct. * **Notable anomalies/patterns:** * The very high transfer from **TriviaQA to Math** (0.82) is intriguing and suggests the reasoning or retrieval skills from TriviaQA might be highly applicable to the Math dataset used here. * The **IMDB** dataset appears to be both very easy to master (0.97 self-score) and a good source for training models that perform well on other tasks (e.g., 0.86 on Winogrande, 0.78 on HotpotQA_WC). This could indicate it's a strong, general-purpose sentiment or text feature dataset. * The **QA datasets (TriviaQA, HotpotQA, NQ_WC)** generally show lower cross-performance with other dataset types, suggesting their specific QA format or knowledge requirements are less transferable to tasks like sentiment analysis (IMDB) or natural language inference (NLI). * The **"WC" variants** (HotpotQA_WC, NQ_WC) likely stand for "Without Context" or a similar modification. Their generally lower scores compared to their parent datasets suggest the context is a crucial component for performance on those tasks. In essence, this chart is a tool for understanding task relatedness and predicting how a model trained for one purpose might fare in another, guiding decisions about multi-task learning, data selection, and model robustness.

## Heatmap: Cross-Dataset Performance Correlation ### Overview This heatmap visualizes the correlation or similarity scores between different question-answering datasets when used as training and test sets. Values range from 0.0 (no correlation) to 1.0 (perfect correlation), with darker red indicating higher similarity. ### Components/Axes - **X-axis (Test dataset)**: TriviaQA, HotpotQA, Movies, Winobias, Winogrande, NLI, IMDB, Math, HotpotQA_WC, NQ_WC - **Y-axis (Train dataset)**: Same as X-axis, listed vertically - **Color legend**: Blue (0.0) to Red (1.0), with intermediate shades representing incremental values - **Cell values**: Numerical scores embedded in each cell (e.g., 0.82, 0.69) ### Detailed Analysis #### Train dataset rows: 1. **TriviaQA**: - Highest self-correlation: 0.82 (self-test) - Strongest cross-correlation: 0.82 with HotpotQA (test) - Weakest: 0.50 with HotpotQA_WC (test) 2. **HotpotQA**: - Self-correlation: 0.76 - Strongest cross: 0.82 with Movies (test) - Weakest: 0.51 with Winogrande (test) 3. **Movies**: - Self-correlation: 0.82 - Strongest cross: 0.82 with TriviaQA (test) - Weakest: 0.52 with IMDb (test) 4. **Winobias**: - Self-correlation: 0.91 (highest in dataset) - Strongest cross: 0.77 with IMDb (test) - Weakest: 0.51 with NQ_WC (test) 5. **Winogrande**: - Self-correlation: 0.65 - Strongest cross: 0.86 with IMDb (test) - Weakest: 0.50 with TriviaQA (test) 6. **NLI**: - Self-correlation: 0.94 (highest in dataset) - Strongest cross: 0.97 with IMDb (test) - Weakest: 0.51 with Math (test) 7. **IMDB**: - Self-correlation: 0.97 (highest in dataset) - Strongest cross: 0.96 with Math (test) - Weakest: 0.52 with HotpotQA_WC (test) 8. **Math**: - Self-correlation: 0.96 - Strongest cross: 0.97 with IMDb (test) - Weakest: 0.51 with Winobias (test) 9. **HotpotQA_WC**: - Self-correlation: 0.67 - Strongest cross: 0.78 with IMDb (test) - Weakest: 0.50 with NQ_WC (test) 10. **NQ_WC**: - Self-correlation: 0.75 - Strongest cross: 0.75 with HotpotQA_WC (test) - Weakest: 0.50 with TriviaQA (test) ### Key Observations 1. **Diagonal dominance**: All datasets show highest scores when trained and tested on the same dataset (e.g., IMDb-IMDb: 0.97, Math-Math: 0.96) 2. **Generalization gaps**: Cross-dataset performance varies significantly: - Strongest cross-generalization: IMDb-NLI (0.97) - Weakest cross-generalization: Winobias-NQ_WC (0.51) 3. **WC datasets**: HotpotQA_WC and NQ_WC show moderate self-correlation (0.67-0.75) but poor performance on other datasets (<0.55 in most cases) 4. **Language understanding clusters**: TriviaQA, HotpotQA, and Movies form a cluster with moderate cross-correlation (0.58-0.70) 5. **Knowledge-intensive datasets**: IMDb and Math show high mutual correlation (0.96-0.97) ### Interpretation This heatmap reveals critical insights about dataset relationships and model generalization: 1. **Overfitting risk**: High diagonal values (0.96-0.97) suggest models trained on specific datasets may overfit, performing poorly on dissimilar test sets 2. **Knowledge transfer limitations**: The weakest scores (0.50-0.53) between QA datasets (e.g., TriviaQA-HotpotQA_WC) indicate limited transferability between different question types 3. **WC dataset challenges**: The WC (with context) variants show significantly lower performance across all tests, suggesting contextual augmentation may reduce model adaptability 4. **Knowledge domain clustering**: IMDb and Math demonstrate near-perfect mutual correlation (0.96-0.97), implying shared underlying knowledge structures 5. **Practical implications**: For real-world deployment, models trained on diverse datasets (e.g., IMDb+NLI) may outperform single-dataset trained models when facing mixed queries The data suggests that while specialized training yields high performance on specific tasks, cross-dataset generalization remains a significant challenge, particularly for WC variants and knowledge-intensive domains.