## Heatmap Grid: Loss Landscapes for Fully Connected Neural Networks (FCNNs) ### Overview The image displays a 4x2 grid of eight heatmap plots, labeled (a) through (h). Each plot visualizes the loss landscape of a Fully Connected Neural Network (FCNN) in a 2D parameter subspace defined by axes `α` and `β`. The plots compare two different network architectures (FCNN 1 and FCNN 2), two parameter initialization methods ("random" and "Hessian"), and two dataset splits ("train" and "test"). The color intensity represents the loss value at each coordinate pair (α, β). ### Components/Axes * **Grid Structure:** 4 rows x 2 columns. * **Subplot Labels:** (a), (b), (c), (d), (e), (f), (g), (h) in the top-left corner of each plot. * **Subplot Titles:** Each title follows the format: `[Model] ([Initialization], [Dataset])`. * (a) FCNN 1 (random,train) * (b) FCNN 1 (random,test) * (c) FCNN 1 (Hessian,train) * (d) FCNN 1 (Hessian,test) * (e) FCNN 2 (random,train) * (f) FCNN 2 (random,test) * (g) FCNN 2 (Hessian,train) * (h) FCNN 2 (Hessian,test) * **Axes:** * **X-axis (Horizontal):** Labeled `α` at the bottom of the grid (plots g, h). Range: -0.1 to 0.1. Major ticks at -0.1, 0, 0.1. * **Y-axis (Vertical):** Labeled `β` on the left side of the grid (plots a, c, e, g). Range: -0.1 to 0.1. Major ticks at -0.1, 0, 0.1. * **Color Bar (Legend):** Located to the right of each heatmap. It maps color to loss value. * **Color Scale:** A gradient from dark green (low loss) through yellow to dark red (high loss). * **Labels & Scale:** * Plots (a), (b), (e), (f) [Random Init]: Labeled `loss [×10⁻¹]`. Scale ranges from 0.0 to 10.0. This indicates the displayed values should be multiplied by 0.1 (e.g., 10.0 represents a loss of 1.0). * Plots (c), (d) [FCNN 1, Hessian Init]: Labeled `loss [×10⁻²]`. Scale ranges from 0.0 to 10.0. This indicates the displayed values should be multiplied by 0.01 (e.g., 10.0 represents a loss of 0.1). * Plots (g), (h) [FCNN 2, Hessian Init]: Labeled `loss`. Scale ranges from 0.0 to 10.0. No multiplier is indicated, suggesting the values are as displayed. * **Reference Lines:** White dashed lines cross at (α=0, β=0) in each plot. * **Marked Point:** A black 'x' marks a specific point in each landscape, likely the converged solution or minimum found during training. ### Detailed Analysis **1. Random Initialization Landscapes (Plots a, b, e, f):** * **Trend/Shape:** The low-loss region (green) forms a broad, roughly circular or elliptical basin centered near (0,0). Loss increases radially outward, transitioning to yellow and then red at the edges of the [-0.1, 0.1] domain. * **Marked Point ('x'):** Located precisely at the intersection of the white dashed lines, (α=0, β=0), in all four random initialization plots. * **Loss Scale:** The color bar multiplier of `×10⁻¹` indicates the loss values in these landscapes are an order of magnitude larger than those in the Hessian-initialized FCNN 1 plots. * **Train vs. Test:** The landscapes for train (a, e) and test (b, f) are visually very similar for each respective model, suggesting the loss surface geometry generalizes well. **2. Hessian Initialization Landscapes (Plots c, d, g, h):** * **Trend/Shape:** The landscapes are dominated by vertical bands. The lowest loss (green) forms a vertical stripe centered around α=0. Loss increases as |α| increases, moving into yellow and red regions on the left and right sides. The variation along the β-axis is much less pronounced compared to the α-axis. * **Marked Point ('x') Position:** * **FCNN 1 (c, d):** The 'x' is offset from the center. In (c) train, it is at approximately (α=0, β≈0.03). In (d) test, it is at approximately (α=0, β≈0.09). * **FCNN 2 (g, h):** The 'x' is offset in the negative β direction. In both (g) train and (h) test, it is at approximately (α=0, β≈-0.03). * **Loss Scale:** * FCNN 1 (c, d): The `×10⁻²` multiplier indicates these loss values are 100 times smaller than those in the random initialization plots for FCNN 1. * FCNN 2 (g, h): The scale has no multiplier, suggesting loss values are directly comparable to the 0-10 range, but the landscape shape is fundamentally different from its random initialization counterpart. ### Key Observations 1. **Dramatic Landscape Change with Initialization:** The most striking feature is the complete transformation of the loss landscape geometry when switching from random to Hessian-based initialization. Random init yields isotropic (circular) basins, while Hessian init creates anisotropic (vertically elongated) valleys. 2. **Consistency Across Train/Test:** For a given model and initialization, the train and test loss landscapes are remarkably similar in shape and scale. This implies the local geometry of the loss surface around the solution is determined more by the model and initialization than by the specific data split. 3. **Divergent Convergence Points:** The black 'x' marks show that Hessian initialization leads the optimization to converge to different points in the (α, β) subspace compared to random initialization (which always finds (0,0)). Furthermore, the converged point differs between FCNN 1 and FCNN 2 under Hessian init. 4. **Scale of Loss:** The loss values at the minima (green regions) are significantly lower for Hessian-initialized FCNN 1 (scale ~0.01) compared to random initialization (scale ~0.1). FCNN 2 with Hessian init shows loss values on a 0-10 scale, but the landscape shape is the key difference. ### Interpretation This visualization provides a Peircean investigation into the **effect of parameter initialization on the geometry of neural network loss landscapes and optimization outcomes**. * **What the Data Suggests:** The "random" initialization places the starting point in a region where the loss surface is relatively symmetric and bowl-shaped around the minimum. The "Hessian" initialization, which uses curvature information, places the starting point in a region where the loss surface is highly elongated—a narrow valley. This suggests the Hessian-based method finds a path to a solution that lies in a flatter, more stable region of the parameter space (as indicated by the vertical banding, showing low curvature in the β direction). * **Relationship Between Elements:** The side-by-side comparison of train and test plots for each condition acts as a control, confirming that the observed landscape properties are intrinsic to the model and initialization, not an artifact of overfitting. The difference between FCNN 1 and FCNN 2 under the same initialization shows that network architecture also influences the resulting landscape. * **Notable Anomalies/Insights:** The fact that the converged point ('x') for Hessian init is not at (0,0) is critical. It shows that the "optimal" solution found depends on the initialization trajectory. The vertical banding in Hessian plots indicates that the loss is much more sensitive to changes in `α` than in `β`. This could imply that the `β` parameter corresponds to a direction in weight space that is less important for the model's function, or that the Hessian initialization has already optimized along that direction. The stark contrast between the landscapes underscores that **optimization does not occur in a vacuum; the starting point fundamentally shapes the path taken and the final solution discovered**, with potential implications for generalization and robustness.