## Heatmap: Performance Metrics by Scenario and Transformation
### Overview
This image displays two stacked heatmaps evaluating model performance across three distinct scenarios (ID, CMP, OOD) and six different transformations (f1, f2, f1 ∘ f1, f1 ∘ f2, f2 ∘ f1, f2 ∘ f2). The top heatmap measures performance using the "BLEU Score," while the bottom heatmap measures performance using "Exact Match (%)."
### Components/Axes
* **Shared X-Axis (Bottom):** Labeled "Transformation." The categories are: `f1`, `f2`, `f1 ∘ f1`, `f1 ∘ f2`, `f2 ∘ f1`, `f2 ∘ f2`.
* **Shared Y-Axis (Left):** Labeled "Scenario." The categories are: `ID`, `CMP`, `OOD`.
* **Top Heatmap Legend:** "BLEU Score," ranging from 0.0 (blue) to 1.0 (red).
* **Bottom Heatmap Legend:** "Exact Match (%)," ranging from 0 (blue) to 100 (red).
### Detailed Analysis
#### Top Heatmap: BLEU Score
* **ID Row:** The values are consistently **1.00** across all six transformations. The color is solid red.
* **CMP Row:**
* `f1`: 0.71
* `f2`: 0.62
* `f1 ∘ f1`: 0.65
* `f1 ∘ f2`: 0.68
* `f2 ∘ f1`: 0.32
* `f2 ∘ f2`: 0.16
* *Trend:* Performance is highest at `f1` (0.71) and degrades significantly as the transformations involve `f2` in the first position (0.32 and 0.16).
* **OOD Row:**
* `f1`: 0.00
* `f2`: 0.00
* `f1 ∘ f1`: 0.46
* `f1 ∘ f2`: 0.35
* `f2 ∘ f1`: 0.40
* `f2 ∘ f2`: 0.35
* *Trend:* Performance is non-existent (0.00) for single transformations (`f1`, `f2`) but shows moderate performance (0.35–0.46) for all composite transformations.
#### Bottom Heatmap: Exact Match (%)
* **ID Row:** The values are consistently **100** across all six transformations. The color is solid red.
* **CMP Row:** The values are consistently **0** across all six transformations. The color is solid blue.
* **OOD Row:** The values are consistently **0** across all six transformations. The color is solid blue.
### Key Observations
* **Perfect ID Performance:** The model achieves perfect scores (1.00 BLEU, 100 Exact Match) in the ID (In-Distribution) scenario regardless of the transformation applied.
* **Exact Match Failure:** The model fails to achieve any exact matches in the CMP (Comparison) and OOD (Out-of-Distribution) scenarios, resulting in a score of 0 across all transformations.
* **BLEU vs. Exact Match Discrepancy:** Despite having 0 Exact Match in CMP and OOD, the BLEU scores are non-zero (ranging from 0.16 to 0.71). This indicates that while the model is not producing identical outputs to the ground truth, it is generating text that shares significant n-gram overlap or semantic similarity.
* **Transformation Impact:** In the CMP scenario, transformations starting with `f1` yield higher BLEU scores (0.62–0.71) compared to those starting with `f2` (0.16–0.32), suggesting `f2` is a more challenging or destructive transformation for the model in this context.
### Interpretation
The data suggests a model that is highly specialized or overfitted to the "ID" (In-Distribution) data, as it handles all transformations perfectly. The complete failure to achieve "Exact Match" in CMP and OOD scenarios, contrasted with the non-zero BLEU scores, implies that the model is capable of generating "near-miss" or semantically relevant text, but lacks the precision required for exact reproduction in these more difficult scenarios. The performance drop in the CMP row when `f2` is applied suggests that `f2` introduces complexity or noise that the model is less equipped to handle compared to `f1`. The OOD scenario's inability to handle single transformations (`f1`, `f2`) but relative success with composite transformations is an anomaly that might suggest the model requires a specific "compositional" structure to trigger its generalization capabilities.
