## Heatmap: Latent State Convergence ||x - x*|| ### Overview This heatmap visualizes the convergence of latent states (represented by phrases) over iterations during test time. The color intensity reflects the logarithmic distance between the current latent state and a target state x*, with darker colors indicating smaller distances (closer convergence) and brighter colors indicating larger distances. ### Components/Axes - **X-axis**: "Iterations at Test Time" (0 to 60, linear scale). - **Y-axis**: Phrases representing latent states (e.g., "Go the 's", "Faust is a complex...", "One of the most significant..."). - **Color Legend**: Logarithmic scale from 10⁰ (purple, low distance) to 10² (yellow, high distance). Positioned on the right, vertically aligned. ### Detailed Analysis 1. **Initial State (Iteration 0)**: - Most phrases start with high log distances (yellow/green), indicating significant divergence from x*. - Exceptions: Phrases like "Go the 's" and "Faust is a complex..." begin with moderate distances (green). 2. **Convergence Trends**: - **Rapid Convergence**: Phrases like "One of the most significant..." drop sharply to low distances (purple) within ~10 iterations. - **Slow Convergence**: Phrases like "Go the 's" and "Faust is a complex..." maintain higher distances (green/yellow) across all iterations, showing slower alignment with x*. - **Stable Convergence**: Phrases like "the nature of knowledge" and "the limits of human understanding" exhibit gradual transitions from green to purple, stabilizing around iteration 30–40. 3. **Logarithmic Scale Impact**: - The log scale amplifies differences in early iterations (e.g., 10⁰ to 10¹ represents a 10x distance change), while later iterations show finer-grained changes (e.g., 10⁰ to 10⁻¹). ### Key Observations - **Phrase-Specific Variability**: Convergence rates differ significantly between phrases, suggesting latent states encode distinct semantic or syntactic properties. - **Persistent Divergence**: Certain phrases (e.g., "Go the 's") retain higher distances even at iteration 60, indicating potential ambiguity or complexity in their representation. - **Logarithmic Nonlinearity**: The log scale reveals exponential decay in distance for some phrases, while others show linear-like convergence. ### Interpretation The heatmap demonstrates that latent states converge at heterogeneous rates, likely reflecting the complexity of the phrases they represent. Phrases with higher initial distances (e.g., "Go the 's") may involve more nuanced or context-dependent meanings, requiring more iterations for the model to align with x*. The log scale highlights early-stage divergence, emphasizing the importance of initial iterations in state alignment. This pattern could inform model optimization strategies, such as prioritizing training on phrases with slower convergence to improve overall latent state fidelity.