## Heatmap Analysis ### Overview The image displays two heatmaps comparing the performance of Recurrent Neural Networks (RNNs) with different configurations (L=4, R=2) and Vanilla Neural Networks (L=8, R=1) on a dataset. The heatmaps are divided into three stages: Training, Test ID, and Test OOD, with each stage further divided into two categories: r1 and r2. ### Components/Axes - **Training**: The first row of the heatmaps shows the performance of the RNN with L=4 and R=2 during training. - **Test ID**: The second row shows the performance during test ID, where the model is evaluated on unseen data. - **Test OOD**: The third row shows the performance during test OOD, where the model is evaluated on out-of-distribution data. - **Stage 1, Stage 2, Stage 3**: The heatmaps are segmented into three stages, each representing a different phase of the training process. - **r1, r2**: The heatmaps are further divided into two categories, r1 and r2, which could represent different versions or configurations of the model. ### Detailed Analysis - **Training**: The RNN with L=4 and R=2 shows a higher performance in the training phase compared to the Vanilla Neural Network with L=8 and R=1. - **Test ID**: During test ID, the RNN with L=4 and R=2 maintains a higher performance than the Vanilla Neural Network with L=8 and R=1. - **Test OOD**: In the test OOD phase, the RNN with L=4 and R=2 again shows a higher performance than the Vanilla Neural Network with L=8 and R=1. - **Effective depth**: The bottom right of the image shows a heatmap of effective depth, which is a measure of the model's ability to generalize to unseen data. The RNN with L=4 and R=2 has a higher effective depth compared to the Vanilla Neural Network with L=8 and R=1. ### Key Observations - The RNN with L=4 and R=2 consistently outperforms the Vanilla Neural Network with L=8 and R=1 across all three stages and categories. - The effective depth of the RNN with L=4 and R=2 is higher, indicating better generalization capabilities. ### Interpretation The data suggests that the RNN with L=4 and R=2 is more effective at handling out-of-distribution data compared to the Vanilla Neural Network with L=8 and R=1. This could be due to the RNN's ability to capture long-term dependencies in the data, which is crucial for out-of-distribution scenarios. The higher effective depth of the RNN with L=4 and R=2 also indicates that it is better at generalizing to unseen data, which is a desirable property in machine learning models.

## Heatmap: Model Performance Across Stages and Effective Depth ### Overview The image presents a comparative heatmap analysis of two neural network architectures (Recurrent and Vanilla) across three stages (Stage 1-3) and eight effective depth levels (1-8). Performance metrics are visualized through color gradients representing "Bridge (%)" and "Target (%)" values, with separate panels for Training, Test ID, and Test OOD conditions. ### Components/Axes - **X-axis**: Effective depth (1-8) - **Y-axis**: - Recurrent model: Stage 1 (r1/r2), Stage 2 (r1/r2), Stage 3 (r1/r2) - Vanilla model: Stage 1, Stage 2, Stage 3 - **Legend**: - Bridge (%) (green gradient: 0-100%) - Target (%) (blue gradient: 0-100%) - **Panels**: - Left: Recurrent (L=4, R=2) - Right: Vanilla (L=8, R=1) - **Conditions**: Training, Test ID, Test OOD (horizontal grouping) ### Detailed Analysis #### Recurrent Model (L=4, R=2) - **Training**: - Stage 1: r1 shows 80-90% Bridge (dark green) at depths 1-3, dropping to 40-50% (light green) at depths 4-8. r2 shows 60-70% Bridge (medium green) at depths 1-4, declining to 20-30% (light green) at depths 5-8. - Stage 2: r1 maintains 70-80% Bridge (dark green) at depths 1-4, dropping to 30-40% (light green) at depths 5-8. r2 shows 50-60% Bridge (medium green) at depths 1-5, declining to 10-20% (light green) at depths 6-8. - Stage 3: r1 exhibits 60-70% Bridge (medium green) at depths 1-5, dropping to 10-20% (light green) at depths 6-8. r2 shows 40-50% Bridge (medium green) at depths 1-6, declining to 5-10% (light green) at depths 7-8. - **Test ID**: - Similar patterns to Training, with slightly higher Bridge percentages in early depths (e.g., Stage 1 r1: 85-95% Bridge at depths 1-3). - **Test OOD**: - Bridge percentages drop significantly: Stage 1 r1 shows 50-60% (medium green) at depths 1-3, declining to 10-20% (light green) at depths 4-8. r2 follows a similar trend but with shallower declines. #### Vanilla Model (L=8, R=1) - **Training**: - Stage 1: Bridge starts at 90% (dark green) at depth 1, dropping to 50% (medium green) at depth 4, then to 20% (light green) at depths 5-8. - Stage 2: Bridge declines from 80% (dark green) at depth 1 to 30% (light green) at depths 5-8. - Stage 3: Bridge drops from 70% (medium green) at depth 1 to 10% (light green) at depths 4-8. - **Test ID**: - Bridge percentages mirror Training but with slightly higher retention in early depths (e.g., Stage 1: 95% at depth 1, 60% at depth 4). - **Test OOD**: - Bridge percentages decline sharply: Stage 1 drops from 70% (medium green) at depth 1 to 10% (light green) at depths 3-8. Stage 3 shows near-zero Bridge (white) at depths 2-8. ### Key Observations 1. **Recurrent Advantage**: The Recurrent model consistently maintains higher Bridge percentages across all stages and depths compared to the Vanilla model, particularly in Test OOD conditions. 2. **Depth Sensitivity**: Both models show performance degradation with increasing effective depth, but the Recurrent model's decline is more gradual. 3. **Target vs. Bridge**: Target percentages (blue) generally follow inverse patterns to Bridge, with higher values in later depths for the Recurrent model. 4. **Stage Progression**: Performance deteriorates across stages for both models, with Stage 3 showing the most significant drops. ### Interpretation The data suggests that the Recurrent architecture (L=4, R=2) demonstrates superior robustness to varying effective depths and out-of-distribution (OOD) conditions compared to the Vanilla model (L=8, R=1). The Recurrent model's multi-stage design with recurrent connections (R=2) appears to better preserve learned representations across depths, maintaining higher Bridge percentages (shared knowledge transfer) even in challenging OOD scenarios. The Vanilla model's single-stage structure (R=1) with greater depth (L=8) suffers from more severe performance degradation, particularly in later stages and deeper layers. This aligns with the hypothesis that recurrent architectures with shallower depth but recurrent connections may be more effective for tasks requiring cross-depth knowledge integration.