## Heatmap Visualization: Token-Level vs Step-Level Analysis of Chemical Problem Solving ### Overview The image presents a comparative analysis of two solution paths (False and True) to a chemistry problem about carbon atom counting in a chemical product. It includes four heatmaps visualizing the relationship between "Thinking" and "Summary" components at both token-level and step-level analysis, with color-coded correctness indicators. ### Components/Axes **Heatmap Structure:** - **X-axis (Horizontal):** "Thinking" component (0-150 scale) - **Y-axis (Vertical):** "Summary" component (0-150 scale) - **Legend:** - Green = Correct analysis - Red = Incorrect analysis - **Annotations:** - "High Noise" labels - Arrows indicating information flow/analysis lock-in - Text boxes highlighting key steps **Spatial Positioning:** - Top-left: Question box with chemistry problem - Middle: Two solution paths (False/True) with step-by-step reasoning - Bottom: Four heatmaps arranged in 2x2 grid: - Top-left: Token Level (Correct) - Top-right: Token Level (Wrong) - Bottom-left: Step Level (Correct) - Bottom-right: Step Level (Wrong) ### Detailed Analysis **Token Level Heatmaps:** 1. **Correct (Green):** - Diagonal green band from (0,0) to (150,150) - Indicates strong correlation between thinking and summary components - "High Noise" annotation in bottom-right quadrant 2. **Incorrect (Red):** - Red band concentrated in bottom-left quadrant - "High Noise" annotation in top-right quadrant - Weak diagonal correlation compared to correct version **Step Level Heatmaps:** 1. **Correct (Green):** - Smooth green gradient from top-left to bottom-right - Clear information flow from thinking to summary - No noise annotations 2. **Incorrect (Red):** - Red band with "Analysis Lock-in" annotation - Steeper diagonal than correct version - Weak final flow indicated by red arrow ### Key Observations 1. Token-level analysis shows higher noise in incorrect solutions 2. Step-level correct solutions demonstrate smoother information flow 3. Incorrect step-level analysis exhibits "lock-in" effects 4. Correct solutions maintain stronger thinking-summary correlation 5. Noise patterns differ between token and step-level evaluations ### Interpretation The visualization demonstrates that step-level analysis provides more reliable chemical problem-solving than token-level analysis. The "Analysis Lock-in" phenomenon in incorrect step-level solutions suggests cognitive biases persisting through reasoning steps. The "High Noise" annotations in token-level incorrect solutions indicate greater variability in surface-level processing errors. The smooth information flow in correct step-level solutions aligns with the True solution path's systematic approach to molecular counting, while the False path's abrupt carbon count change (10→11) correlates with the analysis lock-in pattern. This supports the conclusion that structured reasoning steps outperform superficial token analysis in complex chemical calculations.