## Heatmaps: Model Performance Comparison ### Overview The image presents two heatmaps comparing the performance of two language models, Mistral and LLaMA-2, under different training and evaluation conditions. The left heatmap, titled "Probe," shows performance when using a probe. The right heatmap, titled "LoRA + Prompt," shows performance when using LoRA and Prompt. The heatmaps visualize the performance of each model (Mistral and LLaMA-2) when trained on either Mistral or LLaMA-2 data. The color intensity represents the performance score, with higher scores indicated by lighter colors and lower scores by darker colors. ### Components/Axes * **Titles:** "Probe" (left heatmap), "LoRA + Prompt" (right heatmap) * **Y-axis Label:** "Model" * **Y-axis Categories:** Mistral, LLaMA-2 * **X-axis Label:** "Trained On" * **X-axis Categories:** Mistral, LLaMA-2 * **Color Scale (Right Side of Each Heatmap):** * 0.8 (Top, Lightest Color) * 0.7 * 0.6 * 0.5 (Bottom, Darkest Color) * Right Heatmap: * 0.80 (Top, Lightest Color) * 0.75 * 0.70 * 0.65 (Bottom, Darkest Color) ### Detailed Analysis **Left Heatmap: Probe** * **Mistral (Model) Trained On Mistral:** Dark purple, indicating a low performance score of approximately 0.55. * **Mistral (Model) Trained On LLaMA-2:** Light orange, indicating a high performance score of approximately 0.78. * **LLaMA-2 (Model) Trained On Mistral:** Red, indicating a medium-high performance score of approximately 0.68. * **LLaMA-2 (Model) Trained On LLaMA-2:** Dark purple, indicating a low performance score of approximately 0.55. **Right Heatmap: LoRA + Prompt** * **Mistral (Model) Trained On Mistral:** Dark purple, indicating a low performance score of approximately 0.66. * **Mistral (Model) Trained On LLaMA-2:** Red-orange, indicating a high performance score of approximately 0.77. * **LLaMA-2 (Model) Trained On Mistral:** Dark purple, indicating a low performance score of approximately 0.66. * **LLaMA-2 (Model) Trained On LLaMA-2:** Red, indicating a medium-high performance score of approximately 0.73. ### Key Observations * In the "Probe" configuration, both models perform significantly better when trained on the *other* model's data. Mistral performs best when trained on LLaMA-2, and LLaMA-2 performs better when trained on Mistral. * In the "LoRA + Prompt" configuration, Mistral still performs better when trained on LLaMA-2, but the difference is less pronounced. LLaMA-2 performs better when trained on LLaMA-2. * The "LoRA + Prompt" configuration generally results in higher performance scores compared to the "Probe" configuration, especially for LLaMA-2. ### Interpretation The heatmaps suggest that the models exhibit a degree of specialization or overfitting to their own training data when using a probe. When using LoRA and Prompt, the models are more robust and generalize better. The fact that Mistral performs well when trained on LLaMA-2 data, regardless of the evaluation method, suggests that LLaMA-2 data might contain information that is beneficial for Mistral. The "LoRA + Prompt" method appears to improve the performance of both models, particularly LLaMA-2, indicating that it is a more effective training strategy. The lower performance when trained on their own data suggests a lack of diversity or potential biases in the original training datasets.

\n ## Heatmap: Model Performance Comparison ### Overview The image presents two heatmaps side-by-side, comparing the performance of two language models – Mistral and LLaMA-2 – under two different training conditions: "Probe" and "LoRA + Prompt". The heatmaps visualize a metric (likely a correlation or similarity score) based on which model was trained on which dataset. The color intensity represents the value of the metric, with darker colors indicating lower values and lighter colors indicating higher values. ### Components/Axes * **Y-axis (Vertical):** "Model" with categories: Mistral, LLaMA-2. * **X-axis (Horizontal):** "Trained On" with categories: Mistral, LLaMA-2. * **Color Scale (Right):** Ranges from approximately 0.65 (dark purple) to 0.80 (light orange). * **Titles:** "Probe" (left heatmap), "LoRA + Prompt" (right heatmap). * **Legend:** A color gradient is provided on the right side of both heatmaps, indicating the mapping between color and metric value. ### Detailed Analysis or Content Details **Heatmap 1: Probe** * **Mistral / Mistral:** Approximately 0.78 (orange). * **Mistral / LLaMA-2:** Approximately 0.68 (red). * **LLaMA-2 / Mistral:** Approximately 0.67 (red). * **LLaMA-2 / LLaMA-2:** Approximately 0.55 (dark purple). **Heatmap 2: LoRA + Prompt** * **Mistral / Mistral:** Approximately 0.79 (orange). * **Mistral / LLaMA-2:** Approximately 0.72 (red). * **LLaMA-2 / Mistral:** Approximately 0.73 (red). * **LLaMA-2 / LLaMA-2:** Approximately 0.68 (red). ### Key Observations * In both heatmaps, training a model on its own dataset (Mistral on Mistral, LLaMA-2 on LLaMA-2) yields the highest metric values. * The "Probe" heatmap shows a more pronounced difference between training on the same dataset versus a different dataset. The LLaMA-2 model trained on LLaMA-2 has a significantly lower value (0.55) compared to the other values. * The "LoRA + Prompt" heatmap shows less variation. The values are generally higher, and the difference between training on the same vs. different datasets is less dramatic. * Mistral consistently performs better than LLaMA-2 when trained on LLaMA-2 data, in both training conditions. ### Interpretation The data suggests that both models perform best when trained on data from the same distribution as their pre-training data. The "Probe" heatmap indicates a stronger dependency on this alignment for LLaMA-2, as its performance drops significantly when trained on Mistral data. The "LoRA + Prompt" method appears to mitigate this dependency to some extent, as the performance difference between training on the same vs. different datasets is smaller. The higher values in the "LoRA + Prompt" heatmap overall suggest that this training method is more effective at generalizing across datasets or adapting to different data distributions. The LoRA (Low-Rank Adaptation) technique, combined with prompt engineering, likely allows the models to better leverage information from datasets different from their original training data. The difference in performance between the two models when trained on the other model's data could indicate differences in their architectures or pre-training objectives. Mistral's ability to maintain relatively higher performance when trained on LLaMA-2 data suggests it may be more robust or adaptable.

\n ## Heatmap Pair: Model Performance Comparison (Probe vs. LoRA + Prompt) ### Overview The image displays two side-by-side heatmaps comparing the performance of two machine learning models (Mistral and LLaMA-2) under two different adaptation methods: "Probe" (left) and "LoRA + Prompt" (right). The heatmaps visualize a performance metric (likely accuracy or a similar score) based on which model was used for training and which model is being evaluated. ### Components/Axes * **Chart Type:** Two 2x2 heatmaps. * **Y-Axis (Vertical):** Labeled **"Model"**. The two categories are **"Mistral"** (top row) and **"LLaMA-2"** (bottom row). This axis represents the model being evaluated or probed. * **X-Axis (Horizontal):** Labeled **"Trained On"**. The two categories are **"Mistral"** (left column) and **"LLaMA-2"** (right column). This axis represents the model on which the training or adaptation was performed. * **Color Scale/Legend:** * **Left Heatmap (Probe):** A vertical color bar on the right side of the heatmap. The scale ranges from **0.5** (dark purple/black) to **0.8** (bright orange/red). Intermediate markers are at **0.6** and **0.7**. * **Right Heatmap (LoRA + Prompt):** A vertical color bar on the right side. The scale ranges from **0.65** (dark purple) to **0.80** (bright orange/red). Intermediate markers are at **0.70** and **0.75**. * **Titles:** The left heatmap is titled **"Probe"**. The right heatmap is titled **"LoRA + Prompt"**. ### Detailed Analysis **Left Heatmap: "Probe"** * **Cell (Mistral Model, Trained On Mistral):** Color is a medium-dark purple. Estimated value: **~0.65**. * **Cell (Mistral Model, Trained On LLaMA-2):** Color is bright orange-red. Estimated value: **~0.78**. * **Cell (LLaMA-2 Model, Trained On Mistral):** Color is a bright red-pink. Estimated value: **~0.75**. * **Cell (LLaMA-2 Model, Trained On LLaMA-2):** Color is very dark purple/black. Estimated value: **~0.52**. **Right Heatmap: "LoRA + Prompt"** * **Cell (Mistral Model, Trained On Mistral):** Color is dark purple. Estimated value: **~0.68**. * **Cell (Mistral Model, Trained On LLaMA-2):** Color is bright orange-red. Estimated value: **~0.79**. * **Cell (LLaMA-2 Model, Trained On Mistral):** Color is dark purple. Estimated value: **~0.67**. * **Cell (LLaMA-2 Model, Trained On LLaMA-2):** Color is bright red-pink. Estimated value: **~0.76**. ### Key Observations 1. **Cross-Model Training Advantage:** In both adaptation methods, training on a *different* model than the one being evaluated yields significantly higher performance. The brightest cells (highest values) are always in the off-diagonal positions (Mistral model trained on LLaMA-2, and LLaMA-2 model trained on Mistral). 2. **Method Comparison - LLaMA-2 on LLaMA-2:** The most dramatic difference is for the LLaMA-2 model when trained on itself. With the "Probe" method, this is the worst-performing combination (~0.52). With "LoRA + Prompt," it becomes one of the best-performing combinations (~0.76). 3. **Method Comparison - Mistral on Mistral:** The performance for Mistral trained on itself improves slightly from "Probe" (~0.65) to "LoRA + Prompt" (~0.68). 4. **Overall Performance Range:** The "LoRA + Prompt" method appears to have a higher performance floor (minimum ~0.67) compared to the "Probe" method (minimum ~0.52), suggesting it may be a more robust adaptation technique. ### Interpretation The data suggests a strong **negative transfer or interference** when a model is probed or adapted using only its own pre-trained weights (the diagonal cells in the "Probe" heatmap). This could indicate that the probing method alone is insufficient to elicit good performance from the base model on the target task. Conversely, the **LoRA + Prompt** method appears to successfully mitigate this issue, especially for LLaMA-2. The technique seems to enable effective **knowledge transfer or adaptation** when applied across different model architectures (the off-diagonal cells), which consistently show high performance. The fact that LLaMA-2's performance on itself jumps so dramatically with LoRA + Prompt implies that this method is particularly effective at unlocking or reorganizing the model's internal knowledge for the given task, whereas simple probing fails to do so. The consistent high performance of cross-model training (e.g., Mistral model trained on LLaMA-2 data/weights) under both methods is notable. It may suggest that the task benefits from the features or representations learned by a different but related model architecture, or that the training process effectively distills knowledge from one model into another. The "LoRA + Prompt" method seems to refine and stabilize this cross-model transfer, raising the lower bound of performance.

## Heatmap: Model Performance Comparison (Probe vs LoRA + Prompt) ### Overview The image contains two side-by-side heatmaps comparing model performance metrics. The left heatmap is labeled "Probe," and the right is labeled "LoRA + Prompt." Both heatmaps evaluate two models ("Mistral" and "LLaMA-2") trained on two datasets ("Mistral" and "LLaMA-2"). Performance is visualized using a color gradient from dark purple (low) to light orange (high), with numerical scales provided. --- ### Components/Axes - **X-axis (Trained On)**: - Categories: "Mistral," "LLaMA-2" - **Y-axis (Model)**: - Categories: "Mistral," "LLaMA-2" - **Legend**: - Color gradient: Dark purple (0.5) → Light orange (0.8 for Probe, 0.75 for LoRA + Prompt) - Positioned on the right side of each heatmap. - **Titles**: - Top heatmap: "Probe" - Bottom heatmap: "LoRA + Prompt" --- ### Detailed Analysis #### Probe Heatmap (Left) - **Mistral (Model) trained on Mistral (Dataset)**: 0.8 (light orange) - **Mistral (Model) trained on LLaMA-2 (Dataset)**: 0.7 (orange) - **LLaMA-2 (Model) trained on Mistral (Dataset)**: 0.65 (red) - **LLaMA-2 (Model) trained on LLaMA-2 (Dataset)**: 0.6 (dark purple) #### LoRA + Prompt Heatmap (Right) - **Mistral (Model) trained on Mistral (Dataset)**: 0.75 (orange) - **Mistral (Model) trained on LLaMA-2 (Dataset)**: 0.7 (red-orange) - **LLaMA-2 (Model) trained on Mistral (Dataset)**: 0.7 (red-orange) - **LLaMA-2 (Model) trained on LLaMA-2 (Dataset)**: 0.65 (red) --- ### Key Observations 1. **Probe vs LoRA + Prompt**: - Probe consistently shows higher performance values across all model/dataset combinations. - LoRA + Prompt reduces performance slightly (e.g., Mistral on Mistral drops from 0.8 to 0.75). 2. **Model Consistency**: - Models trained on their native dataset (e.g., Mistral on Mistral) outperform cross-dataset training. - LLaMA-2 shows the largest performance drop when trained on Mistral (0.65 in Probe, 0.7 in LoRA + Prompt). 3. **Color Correlation**: - Darker purple (lower values) corresponds to LLaMA-2 trained on Mistral in both heatmaps. - Light orange (highest values) corresponds to Mistral trained on Mistral in Probe. --- ### Interpretation The data suggests that model performance is strongly tied to the alignment between training dataset and model architecture. The Probe setup achieves higher scores, indicating that additional LoRA + Prompt techniques may introduce trade-offs in performance. Notably, LLaMA-2 exhibits greater sensitivity to cross-dataset training, with a 0.05 drop in Probe and 0.05 drop in LoRA + Prompt when trained on Mistral. This implies architectural mismatches between models and datasets have a more pronounced impact on LLaMA-2. The consistent color coding across heatmaps reinforces the reliability of these trends.