## Heatmap: Model Performance Across Stages and Effective Depth ### Overview The image presents a comparative heatmap analysis of two neural network architectures (Recurrent and Vanilla) across three stages (Stage 1-3) and eight effective depth levels (1-8). Performance metrics are visualized through color gradients representing "Bridge (%)" and "Target (%)" values, with separate panels for Training, Test ID, and Test OOD conditions. ### Components/Axes - **X-axis**: Effective depth (1-8) - **Y-axis**: - Recurrent model: Stage 1 (r1/r2), Stage 2 (r1/r2), Stage 3 (r1/r2) - Vanilla model: Stage 1, Stage 2, Stage 3 - **Legend**: - Bridge (%) (green gradient: 0-100%) - Target (%) (blue gradient: 0-100%) - **Panels**: - Left: Recurrent (L=4, R=2) - Right: Vanilla (L=8, R=1) - **Conditions**: Training, Test ID, Test OOD (horizontal grouping) ### Detailed Analysis #### Recurrent Model (L=4, R=2) - **Training**: - Stage 1: r1 shows 80-90% Bridge (dark green) at depths 1-3, dropping to 40-50% (light green) at depths 4-8. r2 shows 60-70% Bridge (medium green) at depths 1-4, declining to 20-30% (light green) at depths 5-8. - Stage 2: r1 maintains 70-80% Bridge (dark green) at depths 1-4, dropping to 30-40% (light green) at depths 5-8. r2 shows 50-60% Bridge (medium green) at depths 1-5, declining to 10-20% (light green) at depths 6-8. - Stage 3: r1 exhibits 60-70% Bridge (medium green) at depths 1-5, dropping to 10-20% (light green) at depths 6-8. r2 shows 40-50% Bridge (medium green) at depths 1-6, declining to 5-10% (light green) at depths 7-8. - **Test ID**: - Similar patterns to Training, with slightly higher Bridge percentages in early depths (e.g., Stage 1 r1: 85-95% Bridge at depths 1-3). - **Test OOD**: - Bridge percentages drop significantly: Stage 1 r1 shows 50-60% (medium green) at depths 1-3, declining to 10-20% (light green) at depths 4-8. r2 follows a similar trend but with shallower declines. #### Vanilla Model (L=8, R=1) - **Training**: - Stage 1: Bridge starts at 90% (dark green) at depth 1, dropping to 50% (medium green) at depth 4, then to 20% (light green) at depths 5-8. - Stage 2: Bridge declines from 80% (dark green) at depth 1 to 30% (light green) at depths 5-8. - Stage 3: Bridge drops from 70% (medium green) at depth 1 to 10% (light green) at depths 4-8. - **Test ID**: - Bridge percentages mirror Training but with slightly higher retention in early depths (e.g., Stage 1: 95% at depth 1, 60% at depth 4). - **Test OOD**: - Bridge percentages decline sharply: Stage 1 drops from 70% (medium green) at depth 1 to 10% (light green) at depths 3-8. Stage 3 shows near-zero Bridge (white) at depths 2-8. ### Key Observations 1. **Recurrent Advantage**: The Recurrent model consistently maintains higher Bridge percentages across all stages and depths compared to the Vanilla model, particularly in Test OOD conditions. 2. **Depth Sensitivity**: Both models show performance degradation with increasing effective depth, but the Recurrent model's decline is more gradual. 3. **Target vs. Bridge**: Target percentages (blue) generally follow inverse patterns to Bridge, with higher values in later depths for the Recurrent model. 4. **Stage Progression**: Performance deteriorates across stages for both models, with Stage 3 showing the most significant drops. ### Interpretation The data suggests that the Recurrent architecture (L=4, R=2) demonstrates superior robustness to varying effective depths and out-of-distribution (OOD) conditions compared to the Vanilla model (L=8, R=1). The Recurrent model's multi-stage design with recurrent connections (R=2) appears to better preserve learned representations across depths, maintaining higher Bridge percentages (shared knowledge transfer) even in challenging OOD scenarios. The Vanilla model's single-stage structure (R=1) with greater depth (L=8) suffers from more severe performance degradation, particularly in later stages and deeper layers. This aligns with the hypothesis that recurrent architectures with shallower depth but recurrent connections may be more effective for tasks requiring cross-depth knowledge integration.