2404.19756v5

Model: nemotron-free

# KAN: Kolmogorov–Arnold Networks > zmliu@mit.edu Abstract Inspired by the Kolmogorov-Arnold representation theorem, we propose Kolmogorov-Arnold Networks (KANs) as promising alternatives to Multi-Layer Perceptrons (MLPs). While MLPs have fixed activation functions on nodes (“neurons”), KANs have learnable activation functions on edges (“weights”). KANs have no linear weights at all – every weight parameter is replaced by a univariate function parametrized as a spline. We show that this seemingly simple change makes KANs outperform MLPs in terms of accuracy and interpretability, on small-scale AI + Science tasks. For accuracy, smaller KANs can achieve comparable or better accuracy than larger MLPs in function fitting tasks. Theoretically and empirically, KANs possess faster neural scaling laws than MLPs. For interpretability, KANs can be intuitively visualized and can easily interact with human users. Through two examples in mathematics and physics, KANs are shown to be useful “collaborators” helping scientists (re)discover mathematical and physical laws. In summary, KANs are promising alternatives for MLPs, opening opportunities for further improving today’s deep learning models which rely heavily on MLPs. <details> <summary>2404.19756v5/x1.png Details</summary> ![ba006cd5490adb526c6315af16e8ab81a5462c699548b2f6017a686d190eb51d](http://localhost:8000/v1/image/ba006cd5490adb526c6315af16e8ab81a5462c699548b2f6017a686d190eb51d) ### Visual Description # Technical Document Extraction: Multi-Layer Perceptron (MLP) vs. Kolmogorov-Arnold Network (KAN) ## Table Structure | Model | Multi-Layer Perceptron (MLP) | Kolmogorov-Arnold Network (KAN) | |-------|------------------------------|----------------------------------| | Theorem | Universal Approximation Theorem | Kolmogorov-Arnold Representation Theorem | | Formula (Shallow) | $ f(\mathbf{x}) \approx \sum_{i=1}^{N(\epsilon)} a_i\sigma(\mathbf{w}_i \cdot \mathbf{x} + b_i) $ | $ f(\mathbf{x}) = \sum_{q=1}^{2n+1} \Phi_q\left(\sum_{p=1}^n \phi_{q,p}(x_p)\right) $ | | Model (Shallow) | Diagram (a) | Diagram (b) | | Formula (Deep) | $ \text{MLP}(\mathbf{x}) = (\mathbf{W}_3 \circ \sigma_2 \circ \mathbf{W}_2 \circ \sigma_1 \circ \mathbf{W}_1)(\mathbf{x}) $ | $ \text{KAN}(\mathbf{x}) = (\Phi_3 \circ \Phi_2 \circ \Phi_1)(\mathbf{x}) $ | | Model (Deep) | Diagram (c) | Diagram (d) | --- ## Key Components and Flow ### 1. **Model (Shallow)** #### (a) Multi-Layer Perceptron (MLP) - **Fixed Activation Functions**: Applied to nodes (σ). - **Learnable Weights**: Applied to edges (w, b). - **Diagram (a)**: - Nodes represented as squares with σ symbols. - Edges with red (weights) and blue (bias) lines. - Output node aggregates inputs via summation. #### (b) Kolmogorov-Arnold Network (KAN) - **Learnable Activation Functions**: Applied to edges (Φ). - **Sum Operation**: Applied to nodes. - **Diagram (b)**: - Nodes represented as circles with Φ symbols. - Edges with black lines. - Output node aggregates inputs via summation. --- ### 2. **Model (Deep)** #### (c) Multi-Layer Perceptron (MLP) - **Layers**: - **Input Layer**: Linear, learnable weights (W₁). - **Hidden Layers**: Nonlinear, fixed activation (σ₁, σ₂). - **Output Layer**: Nonlinear, fixed activation (σ). - **Diagram (c)**: - Three layers (W₁, W₂, W₃) with σ₁, σ₂, σ. - Red lines for weights, blue for bias, black for activation functions. #### (d) Kolmogorov-Arnold Network (KAN) - **Layers**: - **Input Layer**: Linear, learnable weights (Φ₁). - **Hidden Layers**: Nonlinear, learnable activation (Φ₂, Φ₃). - **Diagram (d)**: - Three layers (Φ₁, Φ₂, Φ₃) with Φ symbols. - Black lines for edges, Φ symbols for activation functions. --- ## Mathematical Formulas ### MLP (Shallow) \[ f(\mathbf{x}) \approx \sum_{i=1}^{N(\epsilon)} a_i\sigma(\mathbf{w}_i \cdot \mathbf{x} + b_i) \] - **σ**: Fixed sigmoid activation. - **w, b**: Learnable weights and biases. ### KAN (Shallow) \[ f(\mathbf{x}) = \sum_{q=1}^{2n+1} \Phi_q\left(\sum_{p=1}^n \phi_{q,p}(x_p)\right) \] - **Φ**: Learnable activation functions. - **φ**: Input-dependent functions. ### MLP (Deep) \[ \text{MLP}(\mathbf{x}) = (\mathbf{W}_3 \circ \sigma_2 \circ \mathbf{W}_2 \circ \sigma_1 \circ \mathbf{W}_1)(\mathbf{x}) \] - **σ₁, σ₂, σ**: Fixed nonlinear activations. - **W₁, W₂, W₃**: Learnable weight matrices. ### KAN (Deep) \[ \text{KAN}(\mathbf{x}) = (\Phi_3 \circ \Phi_2 \circ \Phi_1)(\mathbf{x}) \] - **Φ₁, Φ₂, Φ₃**: Learnable nonlinear activations. --- ## Diagram Analysis ### Spatial Grounding - **Legend**: Not explicitly present; inferred from diagram annotations. - **Color Coding**: - **Red**: Learnable weights (MLP) / Non-learnable weights (KAN). - **Blue**: Bias terms (MLP). - **Black**: Activation functions (MLP) / Learnable activations (KAN). ### Trend Verification - **MLP (Shallow)**: Linear combination of fixed sigmoid activations. - **KAN (Shallow)**: Sum of learnable activation functions. - **MLP (Deep)**: Sequential composition of linear and fixed nonlinear layers. - **KAN (Deep)**: Sequential composition of learnable nonlinear functions. --- ## Critical Observations 1. **Activation Flexibility**: - MLP: Fixed activations (σ) on nodes. - KAN: Learnable activations (Φ) on edges. 2. **Operations**: - MLP: Summation at nodes. - KAN: Summation at nodes with learnable Φ functions. 3. **Scalability**: - MLP: Depth increases via stacked linear+nonlinear layers. - KAN: Depth increases via chained learnable Φ functions. --- ## Language Notes - **Primary Language**: English. - **No Non-English Text Detected**. </details> Figure 0.1: Multi-Layer Perceptrons (MLPs) vs. Kolmogorov-Arnold Networks (KANs) 1 Introduction Multi-layer perceptrons (MLPs) [1, 2, 3], also known as fully-connected feedforward neural networks, are foundational building blocks of today’s deep learning models. The importance of MLPs can never be overstated, since they are the default models in machine learning for approximating nonlinear functions, due to their expressive power guaranteed by the universal approximation theorem [3]. However, are MLPs the best nonlinear regressors we can build? Despite the prevalent use of MLPs, they have significant drawbacks. In transformers [4] for example, MLPs consume almost all non-embedding parameters and are typically less interpretable (relative to attention layers) without post-analysis tools [5]. We propose a promising alternative to MLPs, called Kolmogorov-Arnold Networks (KANs). Whereas MLPs are inspired by the universal approximation theorem, KANs are inspired by the Kolmogorov-Arnold representation theorem [6, 7, 8]. Like MLPs, KANs have fully-connected structures. However, while MLPs place fixed activation functions on nodes (“neurons”), KANs place learnable activation functions on edges (“weights”), as illustrated in Figure 0.1. As a result, KANs have no linear weight matrices at all: instead, each weight parameter is replaced by a learnable 1D function parametrized as a spline. KANs’ nodes simply sum incoming signals without applying any non-linearities. One might worry that KANs are hopelessly expensive, since each MLP’s weight parameter becomes KAN’s spline function. Fortunately, KANs usually allow much smaller computation graphs than MLPs. Unsurprisingly, the possibility of using Kolmogorov-Arnold representation theorem to build neural networks has been studied [9, 10, 11, 12, 13, 14, 15, 16]. However, most work has stuck with the original depth-2 width-( $2n+1$ ) representation, and many did not have the chance to leverage more modern techniques (e.g., back propagation) to train the networks. In [12], a depth-2 width-( $2n+1$ ) representation was investigated, with breaking of the curse of dimensionality observed both empirically and with an approximation theory given compositional structures of the function. Our contribution lies in generalizing the original Kolmogorov-Arnold representation to arbitrary widths and depths, revitalizing and contextualizing it in today’s deep learning world, as well as using extensive empirical experiments to highlight its potential for AI + Science due to its accuracy and interpretability. Despite their elegant mathematical interpretation, KANs are nothing more than combinations of splines and MLPs, leveraging their respective strengths and avoiding their respective weaknesses. Splines are accurate for low-dimensional functions, easy to adjust locally, and able to switch between different resolutions. However, splines have a serious curse of dimensionality (COD) problem, because of their inability to exploit compositional structures. MLPs, on the other hand, suffer less from COD thanks to their feature learning, but are less accurate than splines in low dimensions, because of their inability to optimize univariate functions. The link between MLPs using ReLU-k as activation functions and splines have been established in [17, 18]. To learn a function accurately, a model should not only learn the compositional structure (external degrees of freedom), but should also approximate well the univariate functions (internal degrees of freedom). KANs are such models since they have MLPs on the outside and splines on the inside. As a result, KANs can not only learn features (thanks to their external similarity to MLPs), but can also optimize these learned features to great accuracy (thanks to their internal similarity to splines). For example, given a high dimensional function $$ \displaystyle f(x_{1},\cdots,x_{N})=\exp\left(\frac{1}{N}\sum_{i=1}^{N}{\rm sin% }^{2}(x_{i})\right), $$ splines would fail for large $N$ due to COD; MLPs can potentially learn the the generalized additive structure, but they are very inefficient for approximating the exponential and sine functions with say, ReLU activations. In contrast, KANs can learn both the compositional structure and the univariate functions quite well, hence outperforming MLPs by a large margin (see Figure 3.1). Throughout this paper, we will use extensive numerical experiments to show that KANs can lead to accuracy and interpretability improvement over MLPs, at least on small-scale AI + Science tasks. The organization of the paper is illustrated in Figure 2.1. In Section 2, we introduce the KAN architecture and its mathematical foundation, introduce network simplification techniques to make KANs interpretable, and introduce a grid extension technique to make KANs more accurate. In Section 3, we show that KANs are more accurate than MLPs for data fitting: KANs can beat the curse of dimensionality when there is a compositional structure in data, achieving better scaling laws than MLPs. We also demonstrate the potential of KANs in PDE solving via a simple example of the Poisson equation. In Section 4, we show that KANs are interpretable and can be used for scientific discoveries. We use two examples from mathematics (knot theory) and physics (Anderson localization) to demonstrate that KANs can be helpful “collaborators” for scientists to (re)discover math and physical laws. Section 5 summarizes related works. In Section 6, we conclude by discussing broad impacts and future directions. Codes are available at https://github.com/KindXiaoming/pykan and can also be installed via pip install pykan. 2 Kolmogorov–Arnold Networks (KAN) <details> <summary>2404.19756v5/extracted/6190555/figs/flowchart.png Details</summary> ![be3cfdc22298b428273cb2fc6afeaae4ea881ea1a2a84c0249603e04d8835b65](http://localhost:8000/v1/image/be3cfdc22298b428273cb2fc6afeaae4ea881ea1a2a84c0249603e04d8835b65) ### Visual Description # Technical Document Extraction: Kolmogorov-Arnold Networks (KANs) ## Key Components and Flow ### 1. **Historical Figures and Conceptual Foundation** - **Kolmogorov** (left portrait): Mathematical logician, foundational work in function approximation. - **Arnold** (right portrait): Mathematician, contributed to the development of the Kolmogorov-Arnold representation theorem. - **Network Diagram**: Grid of interconnected nodes labeled "Network" (symbolizing traditional neural networks). - **KAN Diagram**: Complex interconnected nodes labeled "KAN" (Kolmogorov-Arnold Network), representing a hybrid of mathematical and neural architectures. ### 2. **Core Equation and Claims** - **Boxed Text**: ``` KANs are both accurate & interpretable! ``` - **Mathematical Expressions**: - `f(x) = J₀(20x)` (Bessel function of the first kind, order 0, scaled by 20). - `exp(J₀(20x) + y²)` (Exponential function combining Bessel and quadratic terms). ### 3. **Performance Graph: Test RMSE vs. Parameters** - **Axes**: - **Y-axis**: `test RMSE` (log scale, `10⁻¹` to `10⁻⁷`). - **X-axis**: `Number of parameters` (log scale, `10¹` to `10⁵`). - **Legend**: - **Blue**: KAN (lowest RMSE, steepest decline). - **Orange**: MLP (depth 2). - **Green**: MLP (depth 3). - **Red**: MLP (depth 4). - **Purple**: MLP (depth 5). - **Dashed Red**: Theory (KAN). - **Dashed Black**: Theory (ID). - **Trend**: KANs outperform MLPs across all parameter scales, with RMSE decreasing exponentially as parameters increase. ### 4. **Flowchart: KAN Methodology and Applications** - **Nodes and Arrows**: - **Mathematical** (Blue): - Kolmogorov-Arnold Theorem (Section 2.1). - KAN scaling laws (Section 2.3). - **Accurate** (Green): - Methodology: Grid extension (Section 2.4). - Application: Data fitting, PDE (Section 3). - **Interpretable** (Red): - Methodology: Simplification (Section 2.5). - Application: AI for math & physics (Section 4). ### 5. **Diagrammatic Representations** - **KAN Architecture**: - Nodes with embedded functions (e.g., `x`, `y`, `J₀(20x)`). - Connections represent function composition and parameter scaling. - **Flowchart Structure**: - Hierarchical organization of KAN properties (Mathematical → Accurate → Interpretable). - Cross-references to technical sections (e.g., "Section 2.1" for Kolmogorov-Arnold Theorem). ### 6. **Critical Observations** - **Accuracy**: KANs achieve lower test RMSE than MLPs of comparable depth, validating their theoretical efficiency. - **Interpretability**: KANs retain mathematical transparency (e.g., explicit use of Bessel functions), unlike black-box MLPs. - **Scalability**: KAN scaling laws (Section 2.3) suggest polynomial growth in performance with parameter count, contrasting with exponential MLP growth. ### 7. **Section References** - **Section 2.1**: Kolmogorov-Arnold Theorem. - **Section 2.3**: KAN scaling laws. - **Section 2.4**: Grid extension methodology. - **Section 2.5**: Simplification methodology. - **Section 3**: Applications in data fitting and PDEs. - **Section 4**: Applications in AI for mathematics and physics. ## Conclusion The image illustrates the theoretical and practical advantages of KANs over traditional MLPs, emphasizing their mathematical rigor, accuracy, and interpretability. The performance graph and flowchart provide a roadmap for implementing KANs in scientific and engineering domains. </details> Figure 2.1: Our proposed Kolmogorov-Arnold networks are in honor of two great late mathematicians, Andrey Kolmogorov and Vladimir Arnold. KANs are mathematically sound, accurate and interpretable. Multi-Layer Perceptrons (MLPs) are inspired by the universal approximation theorem. We instead focus on the Kolmogorov-Arnold representation theorem, which can be realized by a new type of neural network called Kolmogorov-Arnold networks (KAN). We review the Kolmogorov-Arnold theorem in Section 2.1, to inspire the design of Kolmogorov-Arnold Networks in Section 2.2. In Section 2.3, we provide theoretical guarantees for the expressive power of KANs and their neural scaling laws, relating them to existing approximation and generalization theories in the literature. In Section 2.4, we propose a grid extension technique to make KANs increasingly more accurate. In Section 2.5, we propose simplification techniques to make KANs interpretable. 2.1 Kolmogorov-Arnold Representation theorem Vladimir Arnold and Andrey Kolmogorov established that if $f$ is a multivariate continuous function on a bounded domain, then $f$ can be written as a finite composition of continuous functions of a single variable and the binary operation of addition. More specifically, for a smooth $f:[0,1]^{n}→\mathbb{R}$ , $$ f(\mathbf{x})=f(x_{1},\cdots,x_{n})=\sum_{q=1}^{2n+1}\Phi_{q}\left(\sum_{p=1}^% {n}\phi_{q,p}(x_{p})\right), $$ where $\phi_{q,p}:[0,1]→\mathbb{R}$ and $\Phi_{q}:\mathbb{R}→\mathbb{R}$ . In a sense, they showed that the only true multivariate function is addition, since every other function can be written using univariate functions and sum. One might naively consider this great news for machine learning: learning a high-dimensional function boils down to learning a polynomial number of 1D functions. However, these 1D functions can be non-smooth and even fractal, so they may not be learnable in practice [19, 20]. Because of this pathological behavior, the Kolmogorov-Arnold representation theorem was basically sentenced to death in machine learning, regarded as theoretically sound but practically useless [19, 20]. However, we are more optimistic about the usefulness of the Kolmogorov-Arnold theorem for machine learning. First of all, we need not stick to the original Eq. (2.1) which has only two-layer non-linearities and a small number of terms ( $2n+1$ ) in the hidden layer: we will generalize the network to arbitrary widths and depths. Secondly, most functions in science and daily life are often smooth and have sparse compositional structures, potentially facilitating smooth Kolmogorov-Arnold representations. The philosophy here is close to the mindset of physicists, who often care more about typical cases rather than worst cases. After all, our physical world and machine learning tasks must have structures to make physics and machine learning useful or generalizable at all [21]. 2.2 KAN architecture <details> <summary>2404.19756v5/extracted/6190555/figs/spline_notation.png Details</summary> ![6866b55201bec04880edd15b9e5b8f3d10174b47547108bd4faaed2fecef37fa](http://localhost:8000/v1/image/6866b55201bec04880edd15b9e5b8f3d10174b47547108bd4faaed2fecef37fa) ### Visual Description # Technical Document Extraction ## Network Diagram (Left Side) ### Nodes - **Top Layer Nodes**: - `X_{2,1}` (central node) - **Intermediate Layer Nodes**: - `X_{1,1}`, `X_{1,2}`, `X_{1,3}`, `X_{1,4}`, `X_{1,5}` - **Bottom Layer Nodes**: - `X_{0,1}`, `X_{0,2}` ### Connections - **Top-to-Intermediate Connections**: - `x̃_{1,1,1}`, `x̃_{1,1,2}`, `x̃_{1,1,3}`, `x̃_{1,1,4}`, `x̃_{1,1,5}` - `x̃_{1,2,1}`, `x̃_{1,2,2}`, `x̃_{1,2,3}`, `x̃_{1,2,4}`, `x̃_{1,2,5}` - `x̃_{1,3,1}`, `x̃_{1,3,2}`, `x̃_{1,3,3}`, `x̃_{1,3,4}`, `x̃_{1,3,5}` - `x̃_{1,4,1}`, `x̃_{1,4,2}`, `x̃_{1,4,3}`, `x̃_{1,4,4}`, `x̃_{1,4,5}` - `x̃_{1,5,1}`, `x̃_{1,5,2}` - **Intermediate-to-Bottom Connections**: - `x̃_{0,1,1}`, `x̃_{0,1,2}`, `x̃_{0,1,3}`, `x̃_{0,1,4}`, `x̃_{0,1,5}` - `x̃_{0,2,1}`, `x̃_{0,2,2}`, `x̃_{0,2,3}`, `x̃_{0,2,4}`, `x̃_{0,2,5}` ### Function Blocks - **Top Layer Functions**: - `φ_{1,1,1}`, `φ_{1,1,2}`, `φ_{1,1,3}`, `φ_{1,1,4}`, `φ_{1,1,5}` - `φ_{1,2,1}`, `φ_{1,2,2}`, `φ_{1,2,3}`, `φ_{1,2,4}`, `φ_{1,2,5}` - `φ_{1,3,1}`, `φ_{1,3,2}`, `φ_{1,3,3}`, `φ_{1,3,4}`, `φ_{1,3,5}` - `φ_{1,4,1}`, `φ_{1,4,2}`, `φ_{1,4,3}`, `φ_{1,4,4}`, `φ_{1,4,5}` - `φ_{1,5,1}`, `φ_{1,5,2}` - **Bottom Layer Functions**: - `φ_{0,1,1}`, `φ_{0,1,2}`, `φ_{0,1,3}`, `φ_{0,1,4}`, `φ_{0,1,5}` - `φ_{0,2,1}`, `φ_{0,2,2}`, `φ_{0,2,3}`, `φ_{0,2,4}`, `φ_{0,2,5}` ## Graph and Equations (Right Side) ### Main Graph - **Function**: `φ(x)` - **Equation**: - `φ(x) = ∑_{i=0}^7 c_i B_i(x)` - **Time Axis**: - Range: `t_{-3}` to `t_8` - Key Points: `t_{-3}`, `t_{-2}`, `t_{-1}`, `t_0`, `t_1`, `t_2`, `t_3`, `t_4`, `t_5`, `t_6`, `t_7`, `t_8` - **Grid Extension**: - **Extended Function**: `φ(x) = ∑_{i=0}^{12} c'_i B'_i(x)` - **Extended Time Axis**: - Range: `t'_{-3}` to `t'_{13}` - Key Points: `t'_{-3}`, `t'_{-2}`, `t'_{-1}`, `t'_0`, `t'_1`, `t'_2`, `t'_3`, `t'_4`, `t'_5`, `t'_6`, `t'_7`, `t'_8`, `t'_9`, `t'_{10}`, `t'_{11}`, `t'_{12}`, `t'_{13}` - **Basis Functions**: - `B_0(x)`, `B_1(x)`, `B_2(x)`, `B_3(x)`, `B_4(x)`, `B_5(x)`, `B_6(x)`, `B_7(x)` - Extended Basis Functions: `B'_0(x)`, `B'_1(x)`, ..., `B'_{12}(x)` - **Coefficients**: - Original: `c_0`, `c_1`, ..., `c_7` - Extended: `c'_0`, `c'_1`, ..., `c'_{12}` ### Legend - **Colors**: - `B_0(x)`: Red - `B_1(x)`: Green - `B_2(x)`: Blue - `B_3(x)`: Purple - `B_4(x)`: Cyan - `B_5(x)`: Orange - `B_6(x)`: Pink - `B_7(x)`: Dark Blue - Extended Basis Functions: Not explicitly labeled but implied by continuation. ### Additional Notes - **Grid Extension Indicator**: Dashed arrow pointing from original to extended time axis. - **G Values**: - `G_1 = 5` (original grid size) - `G_2 = 10` (extended grid size) - **Key**: `k = 3` (possibly a parameter for the network or grid configuration). </details> Figure 2.2: Left: Notations of activations that flow through the network. Right: an activation function is parameterized as a B-spline, which allows switching between coarse-grained and fine-grained grids. Suppose we have a supervised learning task consisting of input-output pairs $\{\mathbf{x}_{i},y_{i}\}$ , where we want to find $f$ such that $y_{i}≈ f(\mathbf{x}_{i})$ for all data points. Eq. (2.1) implies that we are done if we can find appropriate univariate functions $\phi_{q,p}$ and $\Phi_{q}$ . This inspires us to design a neural network which explicitly parametrizes Eq. (2.1). Since all functions to be learned are univariate functions, we can parametrize each 1D function as a B-spline curve, with learnable coefficients of local B-spline basis functions (see Figure 2.2 right). Now we have a prototype of KAN, whose computation graph is exactly specified by Eq. (2.1) and illustrated in Figure 0.1 (b) (with the input dimension $n=2$ ), appearing as a two-layer neural network with activation functions placed on edges instead of nodes (simple summation is performed on nodes), and with width $2n+1$ in the middle layer. As mentioned, such a network is known to be too simple to approximate any function arbitrarily well in practice with smooth splines! We therefore generalize our KAN to be wider and deeper. It is not immediately clear how to make KANs deeper, since Kolmogorov-Arnold representations correspond to two-layer KANs. To the best of our knowledge, there is not yet a “generalized” version of the theorem that corresponds to deeper KANs. The breakthrough occurs when we notice the analogy between MLPs and KANs. In MLPs, once we define a layer (which is composed of a linear transformation and nonlinearties), we can stack more layers to make the network deeper. To build deep KANs, we should first answer: “what is a KAN layer?” It turns out that a KAN layer with $n_{\rm in}$ -dimensional inputs and $n_{\rm out}$ -dimensional outputs can be defined as a matrix of 1D functions $$ \displaystyle{\mathbf{\Phi}}=\{\phi_{q,p}\},\qquad p=1,2,\cdots,n_{\rm in},% \qquad q=1,2\cdots,n_{\rm out}, $$ where the functions $\phi_{q,p}$ have trainable parameters, as detaild below. In the Kolmogov-Arnold theorem, the inner functions form a KAN layer with $n_{\rm in}=n$ and $n_{\rm out}=2n+1$ , and the outer functions form a KAN layer with $n_{\rm in}=2n+1$ and $n_{\rm out}=1$ . So the Kolmogorov-Arnold representations in Eq. (2.1) are simply compositions of two KAN layers. Now it becomes clear what it means to have deeper Kolmogorov-Arnold representations: simply stack more KAN layers! Let us introduce some notation. This paragraph will be a bit technical, but readers can refer to Figure 2.2 (left) for a concrete example and intuitive understanding. The shape of a KAN is represented by an integer array $$ \displaystyle[n_{0},n_{1},\cdots,n_{L}], $$ where $n_{i}$ is the number of nodes in the $i^{\rm th}$ layer of the computational graph. We denote the $i^{\rm th}$ neuron in the $l^{\rm th}$ layer by $(l,i)$ , and the activation value of the $(l,i)$ -neuron by $x_{l,i}$ . Between layer $l$ and layer $l+1$ , there are $n_{l}n_{l+1}$ activation functions: the activation function that connects $(l,i)$ and $(l+1,j)$ is denoted by $$ \displaystyle\phi_{l,j,i},\quad l=0,\cdots,L-1,\quad i=1,\cdots,n_{l},\quad j=% 1,\cdots,n_{l+1}. $$ The pre-activation of $\phi_{l,j,i}$ is simply $x_{l,i}$ ; the post-activation of $\phi_{l,j,i}$ is denoted by $\tilde{x}_{l,j,i}\equiv\phi_{l,j,i}(x_{l,i})$ . The activation value of the $(l+1,j)$ neuron is simply the sum of all incoming post-activations: $$ x_{l+1,j}=\sum_{i=1}^{n_{l}}\tilde{x}_{l,j,i}=\sum_{i=1}^{n_{l}}\phi_{l,j,i}(x% _{l,i}),\qquad j=1,\cdots,n_{l+1}. $$ In matrix form, this reads $$ \mathbf{x}_{l+1}=\underbrace{\begin{pmatrix}\phi_{l,1,1}(\cdot)&\phi_{l,1,2}(% \cdot)&\cdots&\phi_{l,1,n_{l}}(\cdot)\\ \phi_{l,2,1}(\cdot)&\phi_{l,2,2}(\cdot)&\cdots&\phi_{l,2,n_{l}}(\cdot)\\ \vdots&\vdots&&\vdots\\ \phi_{l,n_{l+1},1}(\cdot)&\phi_{l,n_{l+1},2}(\cdot)&\cdots&\phi_{l,n_{l+1},n_{% l}}(\cdot)\\ \end{pmatrix}}_{\mathbf{\Phi}_{l}}\mathbf{x}_{l}, $$ where ${\mathbf{\Phi}}_{l}$ is the function matrix corresponding to the $l^{\rm th}$ KAN layer. A general KAN network is a composition of $L$ layers: given an input vector $\mathbf{x}_{0}∈\mathbb{R}^{n_{0}}$ , the output of KAN is $$ {\rm KAN}(\mathbf{x})=(\mathbf{\Phi}_{L-1}\circ\mathbf{\Phi}_{L-2}\circ\cdots% \circ\mathbf{\Phi}_{1}\circ\mathbf{\Phi}_{0})\mathbf{x}. $$ We can also rewrite the above equation to make it more analogous to Eq. (2.1), assuming output dimension $n_{L}=1$ , and define $f(\mathbf{x})\equiv{\rm KAN}(\mathbf{x})$ : $$ f(\mathbf{x})=\sum_{i_{L-1}=1}^{n_{L-1}}\phi_{L-1,i_{L},i_{L-1}}\left(\sum_{i_% {L-2}=1}^{n_{L-2}}\cdots\left(\sum_{i_{2}=1}^{n_{2}}\phi_{2,i_{3},i_{2}}\left(% \sum_{i_{1}=1}^{n_{1}}\phi_{1,i_{2},i_{1}}\left(\sum_{i_{0}=1}^{n_{0}}\phi_{0,% i_{1},i_{0}}(x_{i_{0}})\right)\right)\right)\cdots\right), $$ which is quite cumbersome. In contrast, our abstraction of KAN layers and their visualizations are cleaner and intuitive. The original Kolmogorov-Arnold representation Eq. (2.1) corresponds to a 2-Layer KAN with shape $[n,2n+1,1]$ . Notice that all the operations are differentiable, so we can train KANs with back propagation. For comparison, an MLP can be written as interleaving of affine transformations $\mathbf{W}$ and non-linearities $\sigma$ : $$ {\rm MLP}(\mathbf{x})=(\mathbf{W}_{L-1}\circ\sigma\circ\mathbf{W}_{L-2}\circ% \sigma\circ\cdots\circ\mathbf{W}_{1}\circ\sigma\circ\mathbf{W}_{0})\mathbf{x}. $$ It is clear that MLPs treat linear transformations and nonlinearities separately as $\mathbf{W}$ and $\sigma$ , while KANs treat them all together in $\mathbf{\Phi}$ . In Figure 0.1 (c) and (d), we visualize a three-layer MLP and a three-layer KAN, to clarify their differences. Implementation details. Although a KAN layer Eq. (2.5) looks extremely simple, it is non-trivial to make it well optimizable. The key tricks are: 1. Residual activation functions. We include a basis function $b(x)$ (similar to residual connections) such that the activation function $\phi(x)$ is the sum of the basis function $b(x)$ and the spline function: $$ \displaystyle\phi(x)=w_{b}b(x)+w_{s}{\rm spline}(x). $$ We set $$ \displaystyle b(x)={\rm silu}(x)=x/(1+e^{-x}) $$ in most cases. ${\rm spline}(x)$ is parametrized as a linear combination of B-splines such that $$ \displaystyle{\rm spline}(x)=\sum_{i}c_{i}B_{i}(x) $$ where $c_{i}$ s are trainable (see Figure 2.2 for an illustration). In principle $w_{b}$ and $w_{s}$ are redundant since it can be absorbed into $b(x)$ and ${\rm spline}(x)$ . However, we still include these factors (which are by default trainable) to better control the overall magnitude of the activation function. 1. Initialization scales. Each activation function is initialized to have $w_{s}=1$ and ${\rm spline}(x)≈ 0$ This is done by drawing B-spline coefficients $c_{i}\sim\mathcal{N}(0,\sigma^{2})$ with a small $\sigma$ , typically we set $\sigma=0.1$ .. $w_{b}$ is initialized according to the Xavier initialization, which has been used to initialize linear layers in MLPs. 1. Update of spline grids. We update each grid on the fly according to its input activations, to address the issue that splines are defined on bounded regions but activation values can evolve out of the fixed region during training Other possibilities are: (a) the grid is learnable with gradient descent, e.g., [22]; (b) use normalization such that the input range is fixed. We tried (b) at first but its performance is inferior to our current approach.. Parameter count. For simplicity, let us assume a network 1. of depth $L$ , 1. with layers of equal width $n_{0}=n_{1}=·s=n_{L}=N$ , 1. with each spline of order $k$ (usually $k=3$ ) on $G$ intervals (for $G+1$ grid points). Then there are in total $O(N^{2}L(G+k))\sim O(N^{2}LG)$ parameters. In contrast, an MLP with depth $L$ and width $N$ only needs $O(N^{2}L)$ parameters, which appears to be more efficient than KAN. Fortunately, KANs usually require much smaller $N$ than MLPs, which not only saves parameters, but also achieves better generalization (see e.g., Figure 3.1 and 3.3) and facilitates interpretability. We remark that for 1D problems, we can take $N=L=1$ and the KAN network in our implementation is nothing but a spline approximation. For higher dimensions, we characterize the generalization behavior of KANs with a theorem below. 2.3 KAN’s Approximation Abilities and Scaling Laws Recall that in Eq. (2.1), the 2-Layer width- $(2n+1)$ representation may be non-smooth. However, deeper representations may bring the advantages of smoother activations. For example, the 4-variable function $$ \displaystyle f(x_{1},x_{2},x_{3},x_{4})=\exp\left({\sin}(x_{1}^{2}+x_{2}^{2})% +{\sin}(x_{3}^{2}+x_{4}^{2})\right) $$ can be smoothly represented by a $[4,2,1,1]$ KAN which is 3-Layer, but may not admit a 2-Layer KAN with smooth activations. To facilitate an approximation analysis, we still assume smoothness of activations, but allow the representations to be arbitrarily wide and deep, as in Eq. (2.7). To emphasize the dependence of our KAN on the finite set of grid points, we use $\mathbf{\Phi}_{l}^{G}$ and $\Phi_{l,i,j}^{G}$ below to replace the notation $\mathbf{\Phi}_{l}$ and $\Phi_{l,i,j}$ used in Eq. (2.5) and (2.6). **Theorem 2.1 (Approximation theory, KAT)** *Let $\mathbf{x}=(x_{1},x_{2},·s,x_{n})$ . Suppose that a function $f(\mathbf{x})$ admits a representation $$ f=(\mathbf{\Phi}_{L-1}\circ\mathbf{\Phi}_{L-2}\circ\cdots\circ\mathbf{\Phi}_{1% }\circ\mathbf{\Phi}_{0})\mathbf{x}\,, $$ as in Eq. (2.7), where each one of the $\Phi_{l,i,j}$ are $(k+1)$ -times continuously differentiable. Then there exists a constant $C$ depending on $f$ and its representation, such that we have the following approximation bound in terms of the grid size $G$ : there exist $k$ -th order B-spline functions $\Phi_{l,i,j}^{G}$ such that for any $0≤ m≤ k$ , we have the bound $$ \|f-(\mathbf{\Phi}^{G}_{L-1}\circ\mathbf{\Phi}^{G}_{L-2}\circ\cdots\circ% \mathbf{\Phi}^{G}_{1}\circ\mathbf{\Phi}^{G}_{0})\mathbf{x}\|_{C^{m}}\leq CG^{-% k-1+m}\,. $$ Here we adopt the notation of $C^{m}$ -norm measuring the magnitude of derivatives up to order $m$ : $$ \|g\|_{C^{m}}=\max_{|\beta|\leq m}\sup_{x\in[0,1]^{n}}\left|D^{\beta}g(x)% \right|. $$* * Proof* By the classical 1D B-spline theory [23] and the fact that $\Phi_{l,i,j}$ as continuous functions can be uniformly bounded on a bounded domain, we know that there exist finite-grid B-spline functions $\Phi_{l,i,j}^{G}$ such that for any $0≤ m≤ k$ , $$ \|(\Phi_{l,i,j}\circ\mathbf{\Phi}_{l-1}\circ\mathbf{\Phi}_{l-2}\circ\cdots% \circ\mathbf{\Phi}_{1}\circ\mathbf{\Phi}_{0})\mathbf{x}-(\Phi_{l,i,j}^{G}\circ% \mathbf{\Phi}_{l-1}\circ\mathbf{\Phi}_{l-2}\circ\cdots\circ\mathbf{\Phi}_{1}% \circ\mathbf{\Phi}_{0})\mathbf{x}\|_{C^{m}}\leq CG^{-k-1+m}\,, $$ with a constant $C$ independent of $G$ . We fix those B-spline approximations. Therefore we have that the residue $R_{l}$ defined via $$ R_{l}\coloneqq(\mathbf{\Phi}^{G}_{L-1}\circ\cdots\circ\mathbf{\Phi}^{G}_{l+1}% \circ\mathbf{\Phi}_{l}\circ\mathbf{\Phi}_{l-1}\circ\cdots\circ\mathbf{\Phi}_{0% })\mathbf{x}-(\mathbf{\Phi}_{L-1}^{G}\circ\cdots\circ\mathbf{\Phi}_{l+1}^{G}% \circ\mathbf{\Phi}_{l}^{G}\circ\mathbf{\Phi}_{l-1}\circ\cdots\circ\mathbf{\Phi% }_{0})\mathbf{x} $$ satisfies $$ \|R_{l}\|_{C^{m}}\leq CG^{-k-1+m}\,, $$ with a constant independent of $G$ . Finally notice that $$ f-(\mathbf{\Phi}^{G}_{L-1}\circ\mathbf{\Phi}^{G}_{L-2}\circ\cdots\circ\mathbf{% \Phi}^{G}_{1}\circ\mathbf{\Phi}^{G}_{0})\mathbf{x}=R_{L-1}+R_{L-2}+\cdots+R_{1% }+R_{0}\,, $$ we know that (2.15) holds. ∎ We know that asymptotically, provided that the assumption in Theorem 2.1 holds, KANs with finite grid size can approximate the function well with a residue rate independent of the dimension, hence beating curse of dimensionality! This comes naturally since we only use splines to approximate 1D functions. In particular, for $m=0$ , we recover the accuracy in $L^{∞}$ norm, which in turn provides a bound of RMSE on the finite domain, which gives a scaling exponent $k+1$ . Of course, the constant $C$ is dependent on the representation; hence it will depend on the dimension. We will leave the discussion of the dependence of the constant on the dimension as a future work. We remark that although the Kolmogorov-Arnold theorem Eq. (2.1) corresponds to a KAN representation with shape $[d,2d+1,1]$ , its functions are not necessarily smooth. On the other hand, if we are able to identify a smooth representation (maybe at the cost of extra layers or making the KAN wider than the theory prescribes), then Theorem 2.1 indicates that we can beat the curse of dimensionality (COD). This should not come as a surprise since we can inherently learn the structure of the function and make our finite-sample KAN approximation interpretable. Neural scaling laws: comparison to other theories. Neural scaling laws are the phenomenon where test loss decreases with more model parameters, i.e., $\ell\propto N^{-\alpha}$ where $\ell$ is test RMSE, $N$ is the number of parameters, and $\alpha$ is the scaling exponent. A larger $\alpha$ promises more improvement by simply scaling up the model. Different theories have been proposed to predict $\alpha$ . Sharma & Kaplan [24] suggest that $\alpha$ comes from data fitting on an input manifold of intrinsic dimensionality $d$ . If the model function class is piecewise polynomials of order $k$ ( $k=1$ for ReLU), then the standard approximation theory implies $\alpha=(k+1)/d$ from the approximation theory. This bound suffers from the curse of dimensionality, so people have sought other bounds independent of $d$ by leveraging compositional structures. In particular, Michaud et al. [25] considered computational graphs that only involve unary (e.g., squared, sine, exp) and binary ( $+$ and $×$ ) operations, finding $\alpha=(k+1)/d^{*}=(k+1)/2$ , where $d^{*}=2$ is the maximum arity. Poggio et al. [19] leveraged the idea of compositional sparsity and proved that given function class $W_{m}$ (function whose derivatives are continuous up to $m$ -th order), one needs $N=O(\epsilon^{-\frac{2}{m}})$ number of parameters to achieve error $\epsilon$ , which is equivalent to $\alpha=\frac{m}{2}$ . Our approach, which assumes the existence of smooth Kolmogorov-Arnold representations, decomposes the high-dimensional function into several 1D functions, giving $\alpha=k+1$ (where $k$ is the piecewise polynomial order of the splines). We choose $k=3$ cubic splines so $\alpha=4$ which is the largest and best scaling exponent compared to other works. We will show in Section 3.1 that this bound $\alpha=4$ can in fact be achieved empirically with KANs, while previous work [25] reported that MLPs have problems even saturating slower bounds (e.g., $\alpha=1$ ) and plateau quickly. Of course, we can increase $k$ to match the smoothness of functions, but too high $k$ might be too oscillatory, leading to optimization issues. Comparison between KAT and UAT. The power of fully-connected neural networks is justified by the universal approximation theorem (UAT), which states that given a function and error tolerance $\epsilon>0$ , a two-layer network with $k>N(\epsilon)$ neurons can approximate the function within error $\epsilon$ . However, the UAT guarantees no bound for how $N(\epsilon)$ scales with $\epsilon$ . Indeed, it suffers from the COD, and $N$ has been shown to grow exponentially with $d$ in some cases [21]. The difference between KAT and UAT is a consequence that KANs take advantage of the intrinsically low-dimensional representation of the function while MLPs do not. In KAT, we highlight quantifying the approximation error in the compositional space. In the literature, generalization error bounds, taking into account finite samples of training data, for a similar space have been studied for regression problems; see [26, 27], and also specifically for MLPs with ReLU activations [28]. On the other hand, for general function spaces like Sobolev or Besov spaces, the nonlinear $n$ -widths theory [29, 30, 31] indicates that we can never beat the curse of dimensionality, while MLPs with ReLU activations can achieve the tight rate [32, 33, 34]. This fact again motivates us to consider functions of compositional structure, the much "nicer" functions that we encounter in practice and in science, to overcome the COD. Compared with MLPs, we may use a smaller architecture in practice, since we learn general nonlinear activation functions; see also [28] where the depth of the ReLU MLPs needs to reach at least $\log n$ to have the desired rate, where $n$ is the number of samples. Indeed, we will show that KANs are nicely aligned with symbolic functions while MLPs are not. 2.4 For accuracy: Grid Extension <details> <summary>2404.19756v5/x2.png Details</summary> ![99654236f6272b2729dc2f322c45839ac4cf05a44d381705a5f4f89e92a74949](http://localhost:8000/v1/image/99654236f6272b2729dc2f322c45839ac4cf05a44d381705a5f4f89e92a74949) ### Visual Description # Technical Document Extraction: KAN Model Performance Analysis ## Main Title Fitting f(x, y) = exp(sin(πx)) + y² --- ### Top Left Graph: KAN [2,5,1] #### Axes - **X-axis**: `step` (0 to 1800) - **Y-axis**: `RMSE` (log scale: 10⁻⁹ to 10¹) #### Legend - `train` (blue line) - `test` (orange line) #### Key Elements - Red dashed vertical line labeled `interpolation threshold` - Grid size annotations: `grid=3`, `grid=5`, `grid=10`, `grid=20`, `grid=50`, `grid=100`, `grid=200`, `grid=500`, `grid=1000` #### Trends - **Train RMSE**: Gradual decline with minor fluctuations. - **Test RMSE**: Sharp spikes at `grid=500` and `grid=1000`, followed by steep drops. - Interpolation threshold occurs near `step=1000`. --- ### Top Right Graph: KAN [2,1,1] #### Axes - **X-axis**: `step` (0 to 1800) - **Y-axis**: `RMSE` (log scale: 10⁻⁹ to 10¹) #### Legend - `train` (blue line) - `test` (orange line) #### Key Elements - Red dashed vertical line labeled `interpolation threshold` - Grid size annotations: `grid=3`, `grid=5`, `grid=10`, `grid=20`, `grid=50`, `grid=100`, `grid=200`, `grid=500`, `grid=1000` #### Trends - **Train RMSE**: Stable until `step=1000`, then sharp decline. - **Test RMSE**: Multiple plateaus and spikes, with a critical rise at `grid=1000`. --- ### Bottom Left Graph: Training Loss vs Grid Size G #### Axes - **X-axis**: `grid size G` (log scale: 10¹ to 10²) - **Y-axis**: `test loss` (log scale: 10⁻⁶ to 10⁻²) #### Legend - `KAN [2,5,1] sqrt(mean of squared)` (black line) - `KAN [2,1,1] sqrt(mean of squared)` (blue line) - `KAN [2,1,1] sqrt(median of squared)` (light blue line) #### Annotations - Dashed lines labeled `G⁻²`, `G⁻³`, `G⁻⁴` #### Trends - All models show decaying loss with increasing G. - `KAN [2,1,1] sqrt(median of squared)` decays fastest (aligns with `G⁻⁴`). --- ### Bottom Right Graph: Training Time vs Grid Size G #### Axes - **X-axis**: `grid size G` (log scale: 10¹ to 10³) - **Y-axis**: `training time (seconds/step)` (log scale: 10⁻¹ to 10¹) #### Legend - `KAN [2,5,1]` (black line) - `KAN [2,1,1]` (blue line) #### Trends - **KAN [2,5,1]**: Gradual increase, sharp spike at `G=1000`. - **KAN [2,1,1]**: Slower growth, with a plateau at `G=100`. --- ### Cross-Referenced Observations 1. **Interpolation Threshold**: - Both top graphs show a red dashed line at `step=1000`, indicating a critical point for model performance. 2. **Grid Size Impact**: - Larger grid sizes (e.g., `G=1000`) correlate with higher test RMSE spikes and longer training times. 3. **Model Complexity**: - `KAN [2,5,1]` exhibits higher training times and more volatile test RMSE compared to `KAN [2,1,1]`. 4. **Loss Metrics**: - `sqrt(median of squared)` loss metric for `KAN [2,1,1]` achieves the fastest decay (G⁻⁴). --- ### Summary The graphs illustrate the trade-offs between model complexity (grid size) and performance metrics (RMSE, training loss, training time). `KAN [2,1,1]` with `sqrt(median of squared)` loss demonstrates optimal efficiency, while `KAN [2,5,1]` requires careful grid size selection to avoid overfitting and excessive computational cost. </details> Figure 2.3: We can make KANs more accurate by grid extension (fine-graining spline grids). Top left (right): training dynamics of a $[2,5,1]$ ( $[2,1,1]$ ) KAN. Both models display staircases in their loss curves, i.e., loss suddently drops then plateaus after grid extension. Bottom left: test RMSE follows scaling laws against grid size $G$ . Bottom right: training time scales favorably with grid size $G$ . In principle, a spline can be made arbitrarily accurate to a target function as the grid can be made arbitrarily fine-grained. This good feature is inherited by KANs. By contrast, MLPs do not have the notion of “fine-graining”. Admittedly, increasing the width and depth of MLPs can lead to improvement in performance (“neural scaling laws”). However, these neural scaling laws are slow (discussed in the last section). They are also expensive to obtain, because models of varying sizes are trained independently. By contrast, for KANs, one can first train a KAN with fewer parameters and then extend it to a KAN with more parameters by simply making its spline grids finer, without the need to retraining the larger model from scratch. We next describe how to perform grid extension (illustrated in Figure 2.2 right), which is basically fitting a new fine-grained spline to an old coarse-grained spline. Suppose we want to approximate a 1D function $f$ in a bounded region $[a,b]$ with B-splines of order $k$ . A coarse-grained grid with $G_{1}$ intervals has grid points at $\{t_{0}=a,t_{1},t_{2},·s,t_{G_{1}}=b\}$ , which is augmented to $\{t_{-k},·s,t_{-1},t_{0},·s,t_{G_{1}},t_{G_{1}+1},·s,t_{G_{1}+k}\}$ . There are $G_{1}+k$ B-spline basis functions, with the $i^{\rm th}$ B-spline $B_{i}(x)$ being non-zero only on $[t_{-k+i},t_{i+1}]$ $(i=0,·s,G_{1}+k-1)$ . Then $f$ on the coarse grid is expressed in terms of linear combination of these B-splines basis functions $f_{\rm coarse}(x)=\sum_{i=0}^{G_{1}+k-1}c_{i}B_{i}(x)$ . Given a finer grid with $G_{2}$ intervals, $f$ on the fine grid is correspondingly $f_{\rm fine}(x)=\sum_{j=0}^{G_{2}+k-1}c_{j}^{\prime}B_{j}^{\prime}(x)$ . The parameters $c^{\prime}_{j}$ s can be initialized from the parameters $c_{i}$ by minimizing the distance between $f_{\rm fine}(x)$ to $f_{\rm coarse}(x)$ (over some distribution of $x$ ): $$ \{c_{j}^{\prime}\}=\underset{\{c_{j}^{\prime}\}}{\rm argmin}\ \mathop{\mathbb{% E}}_{x\sim p(x)}\left(\sum_{j=0}^{G_{2}+k-1}c_{j}^{\prime}B_{j}^{\prime}(x)-% \sum_{i=0}^{G_{1}+k-1}c_{i}B_{i}(x)\right)^{2}, $$ which can be implemented by the least squares algorithm. We perform grid extension for all splines in a KAN independently. Toy example: staricase-like loss curves. We use a toy example $f(x,y)={\rm exp}({\rm sin}(\pi x)+y^{2})$ to demonstrate the effect of grid extension. In Figure 2.3 (top left), we show the train and test RMSE for a $[2,5,1]$ KAN. The number of grid points starts as 3, increases to a higher value every 200 LBFGS steps, ending up with 1000 grid points. It is clear that every time fine graining happens, the training loss drops faster than before (except for the finest grid with 1000 points, where optimization ceases to work probably due to bad loss landscapes). However, the test losses first go down then go up, displaying a U-shape, due to the bias-variance tradeoff (underfitting vs. overfitting). We conjecture that the optimal test loss is achieved at the interpolation threshold when the number of parameters match the number of data points. Since our training samples are 1000 and the total parameters of a $[2,5,1]$ KAN is $15G$ ( $G$ is the number of grid intervals), we expect the interpolation threshold to be $G=1000/15≈ 67$ , which roughly agrees with our experimentally observed value $G\sim 50$ . Small KANs generalize better. Is this the best test performance we can achieve? Notice that the synthetic task can be represented exactly by a $[2,1,1]$ KAN, so we train a $[2,1,1]$ KAN and present the training dynamics in Figure 2.3 top right. Interestingly, it can achieve even lower test losses than the $[2,5,1]$ KAN, with clearer staircase structures and the interpolation threshold is delayed to a larger grid size as a result of fewer parameters. This highlights a subtlety of choosing KAN architectures. If we do not know the problem structure, how can we determine the minimal KAN shape? In Section 2.5, we will propose a method to auto-discover such minimal KAN architecture via regularization and pruning. Scaling laws: comparison with theory. We are also interested in how the test loss decreases as the number of grid parameters increases. In Figure 2.3 (bottom left), a [2,1,1] KAN scales roughly as ${\rm test\ RMSE}\propto G^{-3}$ . However, according to the Theorem 2.1, we would expect ${\rm test\ RMSE}\propto G^{-4}$ . We found that the errors across samples are not uniform. This is probably attributed to boundary effects [25]. In fact, there are a few samples that have significantly larger errors than others, making the overall scaling slow down. If we plot the square root of the median (not mean) of the squared losses, we get a scaling closer to $G^{-4}$ . Despite this suboptimality (probably due to optimization), KANs still have much better scaling laws than MLPs, for data fitting (Figure 3.1) and PDE solving (Figure 3.3). In addition, the training time scales favorably with the number of grid points $G$ , shown in Figure 2.3 bottom right When $G=1000$ , training becomes significantly slower, which is specific to the use of the LBFGS optimizer with line search. We conjecture that the loss landscape becomes bad for $G=1000$ , so line search with trying to find an optimal step size within maximal iterations without early stopping.. External vs Internal degrees of freedom. A new concept that KANs highlights is a distinction between external versus internal degrees of freedom (parameters). The computational graph of how nodes are connected represents external degrees of freedom (“dofs”), while the grid points inside an activation function are internal degrees of freedom. KANs benefit from the fact that they have both external dofs and internal dofs. External dofs (that MLPs also have but splines do not) are responsible for learning compositional structures of multiple variables. Internal dofs (that splines also have but MLPs do not) are responsible for learning univariate functions. 2.5 For Interpretability: Simplifying KANs and Making them interactive One loose end from the last subsection is that we do not know how to choose the KAN shape that best matches the structure of a dataset. For example, if we know that the dataset is generated via the symbolic formula $f(x,y)={\rm exp}({\rm sin}(\pi x)+y^{2})$ , then we know that a $[2,1,1]$ KAN is able to express this function. However, in practice we do not know the information a priori, so it would be nice to have approaches to determine this shape automatically. The idea is to start from a large enough KAN and train it with sparsity regularization followed by pruning. We will show that these pruned KANs are much more interpretable than non-pruned ones. To make KANs maximally interpretable, we propose a few simplification techniques in Section 2.5.1, and an example of how users can interact with KANs to make them more interpretable in Section 2.5.2. 2.5.1 Simplification techniques 1. Sparsification. For MLPs, L1 regularization of linear weights is used to favor sparsity. KANs can adapt this high-level idea, but need two modifications: 1. There is no linear “weight” in KANs. Linear weights are replaced by learnable activation functions, so we should define the L1 norm of these activation functions. 1. We find L1 to be insufficient for sparsification of KANs; instead an additional entropy regularization is necessary (see Appendix C for more details). We define the L1 norm of an activation function $\phi$ to be its average magnitude over its $N_{p}$ inputs, i.e., $$ \left|\phi\right|_{1}\equiv\frac{1}{N_{p}}\sum_{s=1}^{N_{p}}\left|\phi(x^{(s)}% )\right|. $$ Then for a KAN layer $\mathbf{\Phi}$ with $n_{\rm in}$ inputs and $n_{\rm out}$ outputs, we define the L1 norm of $\mathbf{\Phi}$ to be the sum of L1 norms of all activation functions, i.e., $$ \left|\mathbf{\Phi}\right|_{1}\equiv\sum_{i=1}^{n_{\rm in}}\sum_{j=1}^{n_{\rm out% }}\left|\phi_{i,j}\right|_{1}. $$ In addition, we define the entropy of $\mathbf{\Phi}$ to be $$ S(\mathbf{\Phi})\equiv-\sum_{i=1}^{n_{\rm in}}\sum_{j=1}^{n_{\rm out}}\frac{% \left|\phi_{i,j}\right|_{1}}{\left|\mathbf{\Phi}\right|_{1}}{\rm log}\left(% \frac{\left|\phi_{i,j}\right|_{1}}{\left|\mathbf{\Phi}\right|_{1}}\right). $$ The total training objective $\ell_{\rm total}$ is the prediction loss $\ell_{\rm pred}$ plus L1 and entropy regularization of all KAN layers: $$ \ell_{\rm total}=\ell_{\rm pred}+\lambda\left(\mu_{1}\sum_{l=0}^{L-1}\left|% \mathbf{\Phi}_{l}\right|_{1}+\mu_{2}\sum_{l=0}^{L-1}S(\mathbf{\Phi}_{l})\right), $$ where $\mu_{1},\mu_{2}$ are relative magnitudes usually set to $\mu_{1}=\mu_{2}=1$ , and $\lambda$ controls overall regularization magnitude. 2. Visualization. When we visualize a KAN, to get a sense of magnitudes, we set the transparency of an activation function $\phi_{l,i,j}$ proportional to ${\rm tanh}(\beta A_{l,i,j})$ where $\beta=3$ . Hence, functions with small magnitude appear faded out to allow us to focus on important ones. 3. Pruning. After training with sparsification penalty, we may also want to prune the network to a smaller subnetwork. We sparsify KANs on the node level (rather than on the edge level). For each node (say the $i^{\rm th}$ neuron in the $l^{\rm th}$ layer), we define its incoming and outgoing score as $$ I_{l,i}=\underset{k}{\rm max}(\left|\phi_{l-1,i,k}\right|_{1}),\qquad O_{l,i}=% \underset{j}{\rm max}(\left|\phi_{l+1,j,i}\right|_{1}), $$ and consider a node to be important if both incoming and outgoing scores are greater than a threshold hyperparameter $\theta=10^{-2}$ by default. All unimportant neurons are pruned. 4. Symbolification. In cases where we suspect that some activation functions are in fact symbolic (e.g., ${\rm cos}$ or ${\rm log}$ ), we provide an interface to set them to be a specified symbolic form, fix_symbolic(l,i,j,f) can set the $(l,i,j)$ activation to be $f$ . However, we cannot simply set the activation function to be the exact symbolic formula, since its inputs and outputs may have shifts and scalings. So, we obtain preactivations $x$ and postactivations $y$ from samples, and fit affine parameters $(a,b,c,d)$ such that $y≈ cf(ax+b)+d$ . The fitting is done by iterative grid search of $a,b$ and linear regression. Besides these techniques, we provide additional tools that allow users to apply more fine-grained control to KANs, listed in Appendix A. 2.5.2 A toy example: how humans can interact with KANs <details> <summary>2404.19756v5/extracted/6190555/figs/sr.png Details</summary> ![d8f656b82a04cf84d9950fbce215cf926d30e6ace1c2f7e87463375b0428057e](http://localhost:8000/v1/image/d8f656b82a04cf84d9950fbce215cf926d30e6ace1c2f7e87463375b0428057e) ### Visual Description # Technical Document Extraction: Neural Network Optimization Flowchart ## Diagram Overview The image depicts a six-step process for optimizing a neural network architecture, transforming it from a complex structure to a simplified symbolic formula. The workflow involves sparsification, pruning, activation function customization, parameter tuning, and final formula derivation. --- ### Step 1: Train with Sparsification - **Input Variables**: - `x` (horizontal axis) - `y` (vertical axis) - **Network Architecture**: - Multi-layer perceptron with dense connections - Hidden layers represented by interconnected nodes - Output layer labeled `exp(sin(πx) + y²)` - **Key Feature**: - Initial dense connectivity pattern (all-to-all connections between layers) --- ### Step 2: Prune - **Process**: - Connection reduction from dense to sparse architecture - Removal of redundant pathways - **Visualization**: - Grayed-out connections indicate pruned links - Simplified network topology with fewer inter-layer connections --- ### Step 3a: Set Sine - **Activation Function**: - Sine function applied to specific nodes - Visualized by sinusoidal curve in gray box - **Placement**: - Located in upper-right branch of pruned network --- ### Step 3b: Set Squared - **Activation Function**: - Squared function (y²) applied to nodes - Visualized by parabolic curve in gray box - **Placement**: - Located in lower-right branch of pruned network --- ### Step 3c: Set Exponential - **Activation Function**: - Exponential function applied to nodes - Visualized by exponential curve in gray box - **Placement**: - Located in central branch of pruned network --- ### Step 4: Train Affine Parameters - **Process**: - Fine-tuning of linear transformation parameters (weights/biases) - Red-highlighted boxes indicate trainable parameters - **Visualization**: - Curved lines show parameter optimization trajectories --- ### Step 5: Output Symbolic Formula - **Final Expression**: ``` 1.0e^(1.0y² + 1.0 sin(3.14x)) ``` - **Derivation**: - Combines exponential, sine, and polynomial components - Coefficients (1.0) indicate normalized parameters --- ### Step 6: Number Snap - **Simplified Formula**: ``` e^y² + sin(πx) ``` - **Process**: - Coefficient simplification (1.0 → omitted) - π approximation (3.14 → π) --- ## Key Technical Components 1. **Architecture Evolution**: - Dense → Sparse → Custom Activation → Optimized Parameters 2. **Mathematical Transformations**: - Exponential: `e^(...)` - Sine: `sin(πx)` - Polynomial: `y²` 3. **Optimization Strategy**: - Iterative refinement through sparsification and parameter tuning 4. **Symbolic Representation**: - Final formula captures essential network behavior with minimal complexity --- ## Critical Observations 1. **Pruning Impact**: - Reduces network complexity while preserving key functional relationships 2. **Activation Function Customization**: - Different functions applied to distinct network branches 3. **Parameter Optimization**: - Coefficients refined to 1.0 through training process 4. **Formula Simplification**: - "Number Snap" step removes redundant coefficients for cleaner representation This flowchart illustrates a systematic approach to neural network compression while maintaining mathematical interpretability of the final model. </details> Figure 2.4: An example of how to do symbolic regression with KAN. Above we have proposed a number of simplification techniques for KANs. We can view these simplification choices as buttons one can click on. A user interacting with these buttons can decide which button is most promising to click next to make KANs more interpretable. We use an example below to showcase how a user could interact with a KAN to obtain maximally interpretable results. Let us again consider the regression task $$ \displaystyle f(x,y)=\exp\left({\sin}(\pi x)+y^{2}\right). $$ Given data points $(x_{i},y_{i},f_{i})$ , $i=1,2,·s,N_{p}$ , a hypothetical user Alice is interested in figuring out the symbolic formula. The steps of Alice’s interaction with the KANs are described below (illustrated in Figure 2.4): Step 1: Training with sparsification. Starting from a fully-connected $[2,5,1]$ KAN, training with sparsification regularization can make it quite sparse. 4 out of 5 neurons in the hidden layer appear useless, hence we want to prune them away. Step 2: Pruning. Automatic pruning is seen to discard all hidden neurons except the last one, leaving a $[2,1,1]$ KAN. The activation functions appear to be known symbolic functions. Step 3: Setting symbolic functions. Assuming that the user can correctly guess these symbolic formulas from staring at the KAN plot, they can set $$ \displaystyle\texttt{fix\_symbolic(1,0,0,`exp')}. $$ In case the user has no domain knowledge or no idea which symbolic functions these activation functions might be, we provide a function suggest_symbolic to suggest symbolic candidates. Step 4: Further training. After symbolifying all the activation functions in the network, the only remaining parameters are the affine parameters. We continue training these affine parameters, and when we see the loss dropping to machine precision, we know that we have found the correct symbolic expression. Step 5: Output the symbolic formula. Sympy is used to compute the symbolic formula of the output node. The user obtains $1.0e^{1.0y^{2}+1.0{\rm sin}(3.14x)}$ , which is the true answer (we only displayed two decimals for $\pi$ ). Remark: Why not symbolic regression (SR)? It is reasonable to use symbolic regression for this example. However, symbolic regression methods are in general brittle and hard to debug. They either return a success or a failure in the end without outputting interpretable intermediate results. In contrast, KANs do continuous search (with gradient descent) in function space, so their results are more continuous and hence more robust. Moreover, users have more control over KANs as compared to SR due to KANs’ transparency. The way we visualize KANs is like displaying KANs’ “brain” to users, and users can perform “surgery” (debugging) on KANs. This level of control is typically unavailable for SR. We will show examples of this in Section 4.4. More generally, when the target function is not symbolic, symbolic regression will fail but KANs can still provide something meaningful. For example, a special function (e.g., a Bessel function) is impossible to SR to learn unless it is provided in advance, but KANs can use splines to approximate it numerically anyway (see Figure 4.1 (d)). 3 KANs are accurate In this section, we demonstrate that KANs are more effective at representing functions than MLPs in various tasks (regression and PDE solving). When comparing two families of models, it is fair to compare both their accuracy (loss) and their complexity (number of parameters). We will show that KANs display more favorable Pareto Frontiers than MLPs. Moreover, in Section 3.5, we show that KANs can naturally work in continual learning without catastrophic forgetting. 3.1 Toy datasets <details> <summary>2404.19756v5/x3.png Details</summary> ![aea30289d7797988ccd180b5e6f65ef6f2464f2af13df30bb813bf67f3e34a89](http://localhost:8000/v1/image/aea30289d7797988ccd180b5e6f65ef6f2464f2af13df30bb813bf67f3e34a89) ### Visual Description # Technical Document Extraction: Test RMSE vs. Number of Parameters ## Overview The image contains five subplots comparing the **test RMSE** (Root Mean Square Error) of different neural network architectures (KAN and MLP) across varying function complexities. Each subplot plots RMSE against the **number of parameters** (log scale) for different model depths. Theoretical convergence rates (e.g., $N^{-4}$, $N^{-2}$) are indicated by dashed lines. --- ### Subplot 1: $f(x) = J_0(20x)$ - **X-axis**: Number of parameters (log scale: $10^1$ to $10^5$) - **Y-axis**: Test RMSE (log scale: $10^{-1}$ to $10^{-7}$) - **Legend**: - **KAN (depth 2)**: Blue line (solid) - **MLP (depth 2)**: Orange line (solid) - **MLP (depth 3)**: Green line (solid) - **MLP (depth 4)**: Red line (solid) - **MLP (depth 5)**: Purple line (solid) - **Theory (KAN)**: Red dashed line ($N^{-4}$) - **Theory (ID)**: Black dashed line ($N^{-4}$) - **Key Trends**: - KAN (depth 2) closely follows the $N^{-4}$ theoretical trend. - MLP models (depths 2–5) show slower convergence, with RMSE plateauing at higher parameter counts. - Theory (ID) aligns with KAN's performance. --- ### Subplot 2: $f(x, y) = \exp(\sin(\pi x) + y^2)$ - **X-axis**: Number of parameters (log scale: $10^1$ to $10^5$) - **Y-axis**: Test RMSE (log scale: $10^{-1}$ to $10^{-7}$) - **Legend**: - **KAN (depth 2)**: Blue line (solid) - **MLP (depth 2)**: Orange line (solid) - **MLP (depth 3)**: Green line (solid) - **MLP (depth 4)**: Red line (solid) - **MLP (depth 5)**: Purple line (solid) - **Theory (KAN)**: Red dashed line ($N^{-4}$) - **Theory (ID)**: Black dashed line ($N^{-2}$) - **Key Trends**: - KAN (depth 2) follows $N^{-4}$ theory but deviates at higher parameters. - MLP (depth 5) outperforms shallower MLPs, approaching $N^{-2}$ theory. - Theory (ID) ($N^{-2}$) aligns with MLP (depth 5) at large parameter counts. --- ### Subplot 3: $f(x, y) = xy$ - **X-axis**: Number of parameters (log scale: $10^1$ to $10^5$) - **Y-axis**: Test RMSE (log scale: $10^{-2}$ to $10^{-8}$) - **Legend**: - **KAN (depth 2)**: Blue line (solid) - **MLP (depth 2)**: Orange line (solid) - **MLP (depth 3)**: Green line (solid) - **MLP (depth 4)**: Red line (solid) - **MLP (depth 5)**: Purple line (solid) - **Theory (KAN)**: Red dashed line ($N^{-4}$) - **Theory (ID)**: Black dashed line ($N^{-2}$) - **Key Trends**: - KAN (depth 2) closely matches $N^{-4}$ theory. - MLP (depth 5) converges to $N^{-2}$ theory at large parameter counts. - All models show improved RMSE with increasing parameters. --- ### Subplot 4: $f(x_1, \dots, x_{100}) = \exp\left(\frac{1}{100} \sum_{i=1}^{100} \sin^2\left(\frac{\pi x_i}{2}\right)\right)$ - **X-axis**: Number of parameters (log scale: $10^3$ to $10^5$) - **Y-axis**: Test RMSE (log scale: $10^{-1}$ to $10^{-5}$) - **Legend**: - **KAN (depth 2)**: Blue line (solid) - **MLP (depth 2)**: Orange line (solid) - **MLP (depth 3)**: Green line (solid) - **MLP (depth 4)**: Red line (solid) - **MLP (depth 5)**: Purple line (solid) - **Theory (KAN)**: Red dashed line ($N^{-4}$) - **Theory (ID)**: Black dashed line ($N^{-0.04}$) - **Key Trends**: - KAN (depth 2) follows $N^{-4}$ theory but underperforms at $10^5$ parameters. - MLP models (depths 2–5) show minimal improvement, with RMSE plateauing near $10^{-4}$. - Theory (ID) ($N^{-0.04}$) suggests near-linear convergence. --- ### Subplot 5: $f(x_1, x_2, x_3, x_4) = \exp(\sin(x_1^2 + x_2^2) + \sin(x_3^2 + x_4^2)$ - **X-axis**: Number of parameters (log scale: $10^1$ to $10^5$) - **Y-axis**: Test RMSE (log scale: $10^{-1}$ to $10^{-7}$) - **Legend**: - **KAN (depth 2)**: Blue line (solid) - **MLP (depth 2)**: Orange line (solid) - **MLP (depth 3)**: Green line (solid) - **MLP (depth 4)**: Red line (solid) - **MLP (depth 5)**: Purple line (solid) - **Theory (KAN)**: Red dashed line ($N^{-4}$) - **Theory (ID)**: Black dashed line ($N^{-2}$) - **Key Trends**: - KAN (depth 2) follows $N^{-4}$ theory but deviates at higher parameters. - MLP (depth 5) outperforms shallower MLPs, approaching $N^{-2}$ theory. - Theory (ID) ($N^{-2}$) aligns with MLP (depth 5) at large parameter counts. </details> Figure 3.1: Compare KANs to MLPs on five toy examples. KANs can almost saturate the fastest scaling law predicted by our theory $(\alpha=4)$ , while MLPs scales slowly and plateau quickly. In Section 2.3, our theory suggested that test RMSE loss $\ell$ scales as $\ell\propto N^{-4}$ with model parameters $N$ . However, this relies on the existence of a Kolmogorov-Arnold representation. As a sanity check, we construct five examples we know have smooth KA representations: 1. $f(x)=J_{0}(20x)$ , which is the Bessel function. Since it is a univariate function, it can be represented by a spline, which is a $[1,1]$ KAN. 1. $f(x,y)={\rm exp}({\rm sin}(\pi x)+y^{2})$ . We know that it can be exactly represented by a $[2,1,1]$ KAN. 1. $f(x,y)=xy$ . We know from Figure 4.1 that it can be exactly represented by a $[2,2,1]$ KAN. 1. A high-dimensional example $f(x_{1},·s,x_{100})={\rm exp}(\frac{1}{100}\sum_{i=1}^{100}{\rm sin}^{2}(% \frac{\pi x_{i}}{2}))$ which can be represented by a $[100,1,1]$ KAN. 1. A four-dimensional example $f(x_{1},x_{2},x_{3},x_{4})={\rm exp}(\frac{1}{2}({\rm sin}(\pi(x_{1}^{2}+x_{2}% ^{2}))+{\rm sin}(\pi(x_{3}^{2}+x_{4}^{2}))))$ which can be represented by a $[4,4,2,1]$ KAN. We train these KANs by increasing grid points every 200 steps, in total covering $G=\{3,5,10,20,50,100,200,500,1000\}$ . We train MLPs with different depths and widths as baselines. Both MLPs and KANs are trained with LBFGS for 1800 steps in total. We plot test RMSE as a function of the number of parameters for KANs and MLPs in Figure 3.1, showing that KANs have better scaling curves than MLPs, especially for the high-dimensional example. For comparison, we plot the lines predicted from our KAN theory as red dashed ( $\alpha=k+1=4$ ), and the lines predicted from Sharma & Kaplan [24] as black-dashed ( $\alpha=(k+1)/d=4/d$ ). KANs can almost saturate the steeper red lines, while MLPs struggle to converge even as fast as the slower black lines and plateau quickly. We also note that for the last example, the 2-Layer KAN $[4,9,1]$ behaves much worse than the 3-Layer KAN (shape $[4,2,2,1]$ ). This highlights the greater expressive power of deeper KANs, which is the same for MLPs: deeper MLPs have more expressive power than shallower ones. Note that we have adopted the vanilla setup where both KANs and MLPs are trained with LBFGS without advanced techniques, e.g., switching between Adam and LBFGS, or boosting [35]. We leave the comparison of KANs and MLPs in advanced setups for future work. 3.2 Special functions <details> <summary>2404.19756v5/x4.png Details</summary> ![d716b65f2eae90841a955d589f3bd15a049d0de9f88cce0403c790284316a5bd](http://localhost:8000/v1/image/d716b65f2eae90841a955d589f3bd15a049d0de9f88cce0403c790284316a5bd) ### Visual Description # RMSE vs. Number of Parameters Analysis ## General Structure - **Grid Layout**: 15 subplots arranged in 3 rows (5 charts per row). - **Axes**: - **X-axis**: `number of parameters` (log scale: 10¹ to 10⁴). - **Y-axis**: `RMSE` (log scale: 10⁻⁷ to 10⁻¹). - **Legends**: - **KAN train**: Solid blue line. - **KAN test**: Dashed blue line. - **MLP train**: Solid orange line. - **MLP test**: Dashed orange line. --- ## Chart-Specific Observations ### Top Row 1. **ellipj** - KAN train/test and MLP train/test lines show similar trends. - RMSE decreases sharply for KAN models as parameters increase. - MLP models plateau at higher RMSE values (~10⁻⁵) despite parameter growth. 2. **ellipkinc** - KAN models outperform MLPs significantly at higher parameter counts. - KAN test RMSE drops below 10⁻⁵, while MLP test remains ~10⁻⁴. 3. **ellipeinc** - KAN train/test lines converge tightly, indicating stable training. - MLP test RMSE stabilizes near 10⁻⁴ after ~10² parameters. 4. **jv** - KAN test RMSE declines steeply, reaching ~10⁻⁶ at 10³ parameters. - MLP test RMSE plateaus at ~10⁻⁴, showing limited scalability. 5. **yv** - KAN models achieve RMSE <10⁻⁵ at 10³ parameters. - MLP test RMSE remains ~10⁻⁴, with minimal improvement beyond 10² parameters. --- ### Middle Row 6. **kv** - KAN train/test lines overlap, suggesting minimal overfitting. - MLP test RMSE plateaus at ~10⁻⁴, while KAN test drops to ~10⁻⁶. 7. **iv** - KAN test RMSE decreases to ~10⁻⁶ at 10³ parameters. - MLP test RMSE remains ~10⁻⁴, with no significant improvement. 8. **lpmv_m_0** - KAN test RMSE drops sharply to ~10⁻⁶ at 10² parameters. - MLP test RMSE plateaus at ~10⁻⁴, showing limited scalability. 9. **lpmv_m_1** - KAN test RMSE reaches ~10⁻⁶ at 10² parameters. - MLP test RMSE stabilizes at ~10⁻⁴, with no further gains. 10. **lpmv_m_2** - KAN test RMSE declines to ~10⁻⁶ at 10² parameters. - MLP test RMSE remains ~10⁻⁴, indicating poor scalability. --- ### Bottom Row 11. **sph_harm_m_0_n_1** - KAN test RMSE drops to ~10⁻⁶ at 10² parameters. - MLP test RMSE plateaus at ~10⁻⁴, with no improvement. 12. **sph_harm_m_1_n_1** - KAN test RMSE reaches ~10⁻⁶ at 10² parameters. - MLP test RMSE remains ~10⁻⁴, showing limited scalability. 13. **sph_harm_m_0_n_2** - KAN test RMSE declines to ~10⁻⁶ at 10² parameters. - MLP test RMSE plateaus at ~10⁻⁴, with no further gains. 14. **sph_harm_m_1_n_2** - KAN test RMSE drops to ~10⁻⁶ at 10² parameters. - MLP test RMSE remains ~10⁻⁴, indicating poor scalability. 15. **sph_harm_m_2_n_2** - KAN test RMSE reaches ~10⁻⁶ at 10² parameters. - MLP test RMSE stabilizes at ~10⁻⁴, with no improvement. --- ## Key Trends 1. **KAN Models**: - Consistently outperform MLPs across all datasets. - RMSE decreases sharply with parameter growth, often reaching <10⁻⁶. - Training and test performance align closely, suggesting minimal overfitting. 2. **MLP Models**: - RMSE plateaus at ~10⁻⁴ for most datasets, regardless of parameter count. - Limited scalability compared to KAN models. 3. **Exceptions**: - In `sph_harm_m_2_n_2`, KAN test RMSE briefly spikes (~10⁻⁵) at 10¹ parameters before declining. --- ## Conclusion KAN models demonstrate superior performance in reducing RMSE as parameter count increases, while MLP models exhibit diminishing returns. This trend is consistent across all 15 datasets analyzed. </details> Figure 3.2: Fitting special functions. We show the Pareto Frontier of KANs and MLPs in the plane spanned by the number of model parameters and RMSE loss. Consistently accross all special functions, KANs have better Pareto Frontiers than MLPs. The definitions of these special functions are in Table 1. One caveat for the above results is that we assume knowledge of the “true” KAN shape. In practice, we do not know the existence of KA representations. Even when we are promised that such a KA representation exists, we do not know the KAN shape a priori. Special functions in more than one variables are such cases, because it would be (mathematically) surprising if multivariate special functions (e.g., a Bessel function $f(\nu,x)=J_{\nu}(x)$ ) could be written in KA represenations, involving only univariate functions and sums). We show below that: 1. Finding (approximate) compact KA representations of special functions is possible, revealing novel mathematical properties of special functions from the perspective of Kolmogorov-Arnold representations. 1. KANs are more efficient and accurate in representing special functions than MLPs. We collect 15 special functions common in math and physics, summarized in Table 1. We choose MLPs with fixed width 5 or 100 and depths swept in $\{2,3,4,5,6\}$ . We run KANs both with and without pruning. KANs without pruning: We fix the shape of KAN, whose width are set to 5 and depths are swept in {2,3,4,5,6}. KAN with pruning. We use the sparsification $(\lambda=10^{-2}\ {\rm or}\ 10^{-3})$ and pruning technique in Section 2.5.1 to obtain a smaller KAN pruned from a fixed-shape KAN. Each KAN is initialized to have $G=3$ , trained with LBFGS, with increasing number of grid points every 200 steps to cover $G=\{3,5,10,20,50,100,200\}$ . For each hyperparameter combination, we run 3 random seeds. For each dataset and each model family (KANs or MLPs), we plot the Pareto frontier Pareto frontier is defined as fits that are optimal in the sense of no other fit being both simpler and more accurate., in the (number of parameters, RMSE) plane, shown in Figure 3.2. KANs’ performance is shown to be consistently better than MLPs, i.e., KANs can achieve lower training/test losses than MLPs, given the same number of parameters. Moreover, we report the (surprisingly compact) shapes of our auto-discovered KANs for special functions in Table 1. On one hand, it is interesting to interpret what these compact representations mean mathematically (we include the KAN illustrations in Figure F.1 and F.2 in Appendix F). On the other hand, these compact representations imply the possibility of breaking down a high-dimensional lookup table into several 1D lookup tables, which can potentially save a lot of memory, with the (almost negligible) overhead to perform a few additions at inference time. | Name | scipy.special API | Minimal KAN shape test RMSE $<10^{-2}$ | Minimal KAN test RMSE | Best KAN shape | Best KAN test RMSE | MLP test RMSE | | --- | --- | --- | --- | --- | --- | --- | | Jacobian elliptic functions | ${\rm ellipj}(x,y)$ | [2,2,1] | $7.29× 10^{-3}$ | [2,3,2,1,1,1] | ${\bf 1.33× 10^{-4}}$ | $6.48× 10^{-4}$ | | Incomplete elliptic integral of the first kind | ${\rm ellipkinc}(x,y)$ | [2,2,1,1] | $1.00× 10^{-3}$ | [2,2,1,1,1] | ${\bf 1.24× 10^{-4}}$ | $5.52× 10^{-4}$ | | Incomplete elliptic integral of the second kind | ${\rm ellipeinc}(x,y)$ | [2,2,1,1] | $8.36× 10^{-5}$ | [2,2,1,1] | ${\bf 8.26× 10^{-5}}$ | $3.04× 10^{-4}$ | | Bessel function of the first kind | ${\rm jv}(x,y)$ | [2,2,1] | $4.93× 10^{-3}$ | [2,3,1,1,1] | ${\bf 1.64× 10^{-3}}$ | $5.52× 10^{-3}$ | | Bessel function of the second kind | ${\rm yv}(x,y)$ | [2,3,1] | $1.89× 10^{-3}$ | [2,2,2,1] | ${\bf 1.49× 10^{-5}}$ | $3.45× 10^{-4}$ | | Modified Bessel function of the second kind | ${\rm kv}(x,y)$ | [2,1,1] | $4.89× 10^{-3}$ | [2,2,1] | ${\bf 2.52× 10^{-5}}$ | $1.67× 10^{-4}$ | | Modified Bessel function of the first kind | ${\rm iv}(x,y)$ | [2,4,3,2,1,1] | $9.28× 10^{-3}$ | [2,4,3,2,1,1] | ${\bf 9.28× 10^{-3}}$ | $1.07× 10^{-2}$ | | Associated Legendre function $(m=0)$ | ${\rm lpmv}(0,x,y)$ | [2,2,1] | $5.25× 10^{-5}$ | [2,2,1] | ${\bf 5.25× 10^{-5}}$ | $1.74× 10^{-2}$ | | Associated Legendre function $(m=1)$ | ${\rm lpmv}(1,x,y)$ | [2,4,1] | $6.90× 10^{-4}$ | [2,4,1] | ${\bf 6.90× 10^{-4}}$ | $1.50× 10^{-3}$ | | Associated Legendre function $(m=2)$ | ${\rm lpmv}(2,x,y)$ | [2,2,1] | $4.88× 10^{-3}$ | [2,3,2,1] | ${\bf 2.26× 10^{-4}}$ | $9.43× 10^{-4}$ | | spherical harmonics $(m=0,n=1)$ | ${\rm sph\_harm}(0,1,x,y)$ | [2,1,1] | $2.21× 10^{-7}$ | [2,1,1] | ${\bf 2.21× 10^{-7}}$ | $1.25× 10^{-6}$ | | spherical harmonics $(m=1,n=1)$ | ${\rm sph\_harm}(1,1,x,y)$ | [2,2,1] | $7.86× 10^{-4}$ | [2,3,2,1] | ${\bf 1.22× 10^{-4}}$ | $6.70× 10^{-4}$ | | spherical harmonics $(m=0,n=2)$ | ${\rm sph\_harm}(0,2,x,y)$ | [2,1,1] | $1.95× 10^{-7}$ | [2,1,1] | ${\bf 1.95× 10^{-7}}$ | $2.85× 10^{-6}$ | | spherical harmonics $(m=1,n=2)$ | ${\rm sph\_harm}(1,2,x,y)$ | [2,2,1] | $4.70× 10^{-4}$ | [2,2,1,1] | ${\bf 1.50× 10^{-5}}$ | $1.84× 10^{-3}$ | | spherical harmonics $(m=2,n=2)$ | ${\rm sph\_harm}(2,2,x,y)$ | [2,2,1] | $1.12× 10^{-3}$ | [2,2,3,2,1] | ${\bf 9.45× 10^{-5}}$ | $6.21× 10^{-4}$ | Table 1: Special functions 3.3 Feynman datasets The setup in Section 3.1 is when we clearly know “true” KAN shapes. The setup in Section 3.2 is when we clearly do not know “true” KAN shapes. This part investigates a setup lying in the middle: Given the structure of the dataset, we may construct KANs by hand, but we are not sure if they are optimal. In this regime, it is interesting to compare human-constructed KANs and auto-discovered KANs via pruning (techniques in Section 2.5.1). | Feynman Eq. | Original Formula | Dimensionless formula | Variables | Human-constructed KAN shape | Pruned KAN shape (smallest shape that achieves RMSE < $10^{-2}$ ) | Pruned KAN shape (lowest loss) | Human-constructed KAN loss (lowest test RMSE) | Pruned KAN loss (lowest test RMSE) | Unpruned KAN loss (lowest test RMSE) | MLP loss (lowest test RMSE) | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | I.6.2 | ${\rm exp}(-\frac{\theta^{2}}{2\sigma^{2}})/\sqrt{2\pi\sigma^{2}}$ | ${\rm exp}(-\frac{\theta^{2}}{2\sigma^{2}})/\sqrt{2\pi\sigma^{2}}$ | $\theta,\sigma$ | [2,2,1,1] | [2,2,1] | [2,2,1,1] | $7.66× 10^{-5}$ | ${\bf 2.86× 10^{-5}}$ | $4.60× 10^{-5}$ | $1.45× 10^{-4}$ | | I.6.2b | ${\rm exp}(-\frac{(\theta-\theta_{1})^{2}}{2\sigma^{2}})/\sqrt{2\pi\sigma^{2}}$ | ${\rm exp}(-\frac{(\theta-\theta_{1})^{2}}{2\sigma^{2}})/\sqrt{2\pi\sigma^{2}}$ | $\theta,\theta_{1},\sigma$ | [3,2,2,1,1] | [3,4,1] | [3,2,2,1,1] | $1.22× 10^{-3}$ | ${\bf 4.45× 10^{-4}}$ | $1.25× 10^{-3}$ | $7.40× 10^{-4}$ | | I.9.18 | $\frac{Gm_{1}m_{2}}{(x_{2}-x_{1})^{2}+(y_{2}-y_{1})^{2}+(z_{2}-z_{1})^{2}}$ | $\frac{a}{(b-1)^{2}+(c-d)^{2}+(e-f)^{2}}$ | $a,b,c,d,e,f$ | [6,4,2,1,1] | [6,4,1,1] | [6,4,1,1] | ${\bf 1.48× 10^{-3}}$ | $8.62× 10^{-3}$ | $6.56× 10^{-3}$ | $1.59× 10^{-3}$ | | I.12.11 | $q(E_{f}+Bv{\rm sin}\theta)$ | $1+a{\rm sin}\theta$ | $a,\theta$ | [2,2,2,1] | [2,2,1] | [2,2,1] | $2.07× 10^{-3}$ | $1.39× 10^{-3}$ | $9.13× 10^{-4}$ | ${\bf 6.71× 10^{-4}}$ | | I.13.12 | $Gm_{1}m_{2}(\frac{1}{r_{2}}-\frac{1}{r_{1}})$ | $a(\frac{1}{b}-1)$ | $a,b$ | [2,2,1] | [2,2,1] | [2,2,1] | $7.22× 10^{-3}$ | $4.81× 10^{-3}$ | $2.72× 10^{-3}$ | ${\bf 1.42× 10^{-3}}$ | | I.15.3x | $\frac{x-ut}{\sqrt{1-(\frac{u}{c})^{2}}}$ | $\frac{1-a}{\sqrt{1-b^{2}}}$ | $a,b$ | [2,2,1,1] | [2,1,1] | [2,2,1,1,1] | $7.35× 10^{-3}$ | $1.58× 10^{-3}$ | $1.14× 10^{-3}$ | ${\bf 8.54× 10^{-4}}$ | | I.16.6 | $\frac{u+v}{1+\frac{uv}{c^{2}}}$ | $\frac{a+b}{1+ab}$ | $a,b$ | [2,2,2,2,2,1] | [2,2,1] | [2,2,1] | $1.06× 10^{-3}$ | $1.19× 10^{-3}$ | $1.53× 10^{-3}$ | ${\bf 6.20× 10^{-4}}$ | | I.18.4 | $\frac{m_{1}r_{1}+m_{2}r_{2}}{m_{1}+m_{2}}$ | $\frac{1+ab}{1+a}$ | $a,b$ | [2,2,2,1,1] | [2,2,1] | [2,2,1] | $3.92× 10^{-4}$ | ${\bf 1.50× 10^{-4}}$ | $1.32× 10^{-3}$ | $3.68× 10^{-4}$ | | I.26.2 | ${\rm arcsin}(n{\rm sin}\theta_{2})$ | ${\rm arcsin}(n{\rm sin}\theta_{2})$ | $n,\theta_{2}$ | [2,2,2,1,1] | [2,2,1] | [2,2,2,1,1] | $1.22× 10^{-1}$ | ${\bf 7.90× 10^{-4}}$ | $8.63× 10^{-4}$ | $1.24× 10^{-3}$ | | I.27.6 | $\frac{1}{\frac{1}{d_{1}}+\frac{n}{d_{2}}}$ | $\frac{1}{1+ab}$ | $a,b$ | [2,2,1,1] | [2,1,1] | [2,1,1] | $2.22× 10^{-4}$ | ${\bf 1.94× 10^{-4}}$ | $2.14× 10^{-4}$ | $2.46× 10^{-4}$ | | I.29.16 | $\sqrt{x_{1}^{2}+x_{2}^{2}-2x_{1}x_{2}{\rm cos}(\theta_{1}-\theta_{2})}$ | $\sqrt{1+a^{2}-2a{\rm cos}(\theta_{1}-\theta_{2})}$ | $a,\theta_{1},\theta_{2}$ | [3,2,2,3,2,1,1] | [3,2,2,1] | [3,2,3,1] | $2.36× 10^{-1}$ | $3.99× 10^{-3}$ | ${\bf 3.20× 10^{-3}}$ | $4.64× 10^{-3}$ | | I.30.3 | $I_{*,0}\frac{{\rm sin}^{2}(\frac{n\theta}{2})}{{\rm sin}^{2}(\frac{\theta}{2})}$ | $\frac{{\rm sin}^{2}(\frac{n\theta}{2})}{{\rm sin}^{2}(\frac{\theta}{2})}$ | $n,\theta$ | [2,3,2,2,1,1] | [2,4,3,1] | [2,3,2,3,1,1] | $3.85× 10^{-1}$ | ${\bf 1.03× 10^{-3}}$ | $1.11× 10^{-2}$ | $1.50× 10^{-2}$ | | I.30.5 | ${\rm arcsin}(\frac{\lambda}{nd})$ | ${\rm arcsin}(\frac{a}{n})$ | $a,n$ | [2,1,1] | [2,1,1] | [2,1,1,1,1,1] | $2.23× 10^{-4}$ | ${\bf 3.49× 10^{-5}}$ | $6.92× 10^{-5}$ | $9.45× 10^{-5}$ | | I.37.4 | $I_{*}=I_{1}+I_{2}+2\sqrt{I_{1}I_{2}}{\rm cos}\delta$ | $1+a+2\sqrt{a}{\rm cos}\delta$ | $a,\delta$ | [2,3,2,1] | [2,2,1] | [2,2,1] | $7.57× 10^{-5}$ | ${\bf 4.91× 10^{-6}}$ | $3.41× 10^{-4}$ | $5.67× 10^{-4}$ | | I.40.1 | $n_{0}{\rm exp}(-\frac{mgx}{k_{b}T})$ | $n_{0}e^{-a}$ | $n_{0},a$ | [2,1,1] | [2,2,1] | [2,2,1,1,1,2,1] | $3.45× 10^{-3}$ | $5.01× 10^{-4}$ | ${\bf 3.12× 10^{-4}}$ | $3.99× 10^{-4}$ | | I.44.4 | $nk_{b}T{\rm ln}(\frac{V_{2}}{V_{1}})$ | $n{\rm ln}a$ | $n,a$ | [2,2,1] | [2,2,1] | [2,2,1] | ${\bf 2.30× 10^{-5}}$ | $2.43× 10^{-5}$ | $1.10× 10^{-4}$ | $3.99× 10^{-4}$ | | I.50.26 | $x_{1}({\rm cos}(\omega t)+\alpha{\rm cos}^{2}(wt))$ | ${\rm cos}a+\alpha{\rm cos}^{2}a$ | $a,\alpha$ | [2,2,3,1] | [2,3,1] | [2,3,2,1] | ${\bf 1.52× 10^{-4}}$ | $5.82× 10^{-4}$ | $4.90× 10^{-4}$ | $1.53× 10^{-3}$ | | II.2.42 | $\frac{k(T_{2}-T_{1})A}{d}$ | $(a-1)b$ | $a,b$ | [2,2,1] | [2,2,1] | [2,2,2,1] | $8.54× 10^{-4}$ | $7.22× 10^{-4}$ | $1.22× 10^{-3}$ | ${\bf 1.81× 10^{-4}}$ | | II.6.15a | $\frac{3}{4\pi\epsilon}\frac{p_{d}z}{r^{5}}\sqrt{x^{2}+y^{2}}$ | $\frac{1}{4\pi}c\sqrt{a^{2}+b^{2}}$ | $a,b,c$ | [3,2,2,2,1] | [3,2,1,1] | [3,2,1,1] | $2.61× 10^{-3}$ | $3.28× 10^{-3}$ | $1.35× 10^{-3}$ | ${\bf 5.92× 10^{-4}}$ | | II.11.7 | $n_{0}(1+\frac{p_{d}E_{f}{\rm cos}\theta}{k_{b}T})$ | $n_{0}(1+a{\rm cos}\theta)$ | $n_{0},a,\theta$ | [3,3,3,2,2,1] | [3,3,1,1] | [3,3,1,1] | $7.10× 10^{-3}$ | $8.52× 10^{-3}$ | $5.03× 10^{-3}$ | ${\bf 5.92× 10^{-4}}$ | | II.11.27 | $\frac{n\alpha}{1-\frac{n\alpha}{3}}\epsilon E_{f}$ | $\frac{n\alpha}{1-\frac{n\alpha}{3}}$ | $n,\alpha$ | [2,2,1,2,1] | [2,1,1] | [2,2,1] | $2.67× 10^{-5}$ | $4.40× 10^{-5}$ | ${\bf 1.43× 10^{-5}}$ | $7.18× 10^{-5}$ | | II.35.18 | $\frac{n_{0}}{{\rm exp}(\frac{\mu_{m}B}{k_{b}T})+{\rm exp}(-\frac{\mu_{m}B}{k_{% b}T})}$ | $\frac{n_{0}}{{\rm exp}(a)+{\rm exp}(-a)}$ | $n_{0},a$ | [2,1,1] | [2,1,1] | [2,1,1,1] | $4.13× 10^{-4}$ | $1.58× 10^{-4}$ | ${\bf 7.71× 10^{-5}}$ | $7.92× 10^{-5}$ | | II.36.38 | $\frac{\mu_{m}B}{k_{b}T}+\frac{\mu_{m}\alpha M}{\epsilon c^{2}k_{b}T}$ | $a+\alpha b$ | $a,\alpha,b$ | [3,3,1] | [3,2,1] | [3,2,1] | $2.85× 10^{-3}$ | ${\bf 1.15× 10^{-3}}$ | $3.03× 10^{-3}$ | $2.15× 10^{-3}$ | | II.38.3 | $\frac{YAx}{d}$ | $\frac{a}{b}$ | $a,b$ | [2,1,1] | [2,1,1] | [2,2,1,1,1] | $1.47× 10^{-4}$ | ${\bf 8.78× 10^{-5}}$ | $6.43× 10^{-4}$ | $5.26× 10^{-4}$ | | III.9.52 | $\frac{p_{d}E_{f}}{h}\frac{{\rm sin}^{2}((\omega-\omega_{0})t/2)}{((\omega-% \omega_{0})t/2)^{2}}$ | $a\frac{{\rm sin}^{2}(\frac{b-c}{2})}{(\frac{b-c}{2})^{2}}$ | $a,b,c$ | [3,2,3,1,1] | [3,3,2,1] | [3,3,2,1,1,1] | $4.43× 10^{-2}$ | $3.90× 10^{-3}$ | $2.11× 10^{-2}$ | ${\bf 9.07× 10^{-4}}$ | | III.10.19 | $\mu_{m}\sqrt{B_{x}^{2}+B_{y}^{2}+B_{z}^{2}}$ | $\sqrt{1+a^{2}+b^{2}}$ | $a,b$ | [2,1,1] | [2,1,1] | [2,1,2,1] | $2.54× 10^{-3}$ | $1.18× 10^{-3}$ | $8.16× 10^{-4}$ | ${\bf 1.67× 10^{-4}}$ | | III.17.37 | $\beta(1+\alpha{\rm cos}\theta)$ | $\beta(1+\alpha{\rm cos}\theta)$ | $\alpha,\beta,\theta$ | [3,3,3,2,2,1] | [3,3,1] | [3,3,1] | $1.10× 10^{-3}$ | $5.03× 10^{-4}$ | ${\bf 4.12× 10^{-4}}$ | $6.80× 10^{-4}$ | Table 2: Feynman dataset Feynman dataset. The Feynman dataset collects many physics equations from Feynman’s textbooks [36, 37]. For our purpose, we are interested in problems in the Feynman_no_units dataset that have at least 2 variables, since univariate problems are trivial for KANs (they simplify to 1D splines). A sample equation from the Feynman dataset is the relativisic velocity addition formula $$ \displaystyle f(u,v)=(u+v)/(1+uv). $$ The dataset can be constructed by randomly drawing $u_{i}∈(-1,1)$ , $v_{i}∈(-1,1)$ , and computing $f_{i}=f(u_{i},v_{i})$ . Given many tuples $(u_{i},v_{i},f_{i})$ , a neural network is trained and aims to predict $f$ from $u$ and $v$ . We are interested in (1) how well a neural network can perform on test samples; (2) how much we can learn about the structure of the problem from neural networks. We compare four kinds of neural networks: 1. Human-constructued KAN. Given a symbolic formula, we rewrite it in Kolmogorov-Arnold representations. For example, to multiply two numbers $x$ and $y$ , we can use the identity $xy=\frac{(x+y)^{2}}{4}-\frac{(x-y)^{2}}{4}$ , which corresponds to a $[2,2,1]$ KAN. The constructued shapes are listed in the “Human-constructed KAN shape” in Table 2. 1. KANs without pruning. We fix the KAN shape to width 5 and depths are swept over {2,3,4,5,6}. 1. KAN with pruning. We use the sparsification $(\lambda=10^{-2}\ {\rm or}\ 10^{-3})$ and the pruning technique from Section 2.5.1 to obtain a smaller KAN from a fixed-shape KAN from (2). 1. MLPs with fixed width 5, depths swept in $\{2,3,4,5,6\}$ , and activations chosen from $\{{\rm Tanh},{\rm ReLU},{\rm SiLU}\}$ . Each KAN is initialized to have $G=3$ , trained with LBFGS, with increasing number of grid points every 200 steps to cover $G=\{3,5,10,20,50,100,200\}$ . For each hyperparameter combination, we try 3 random seeds. For each dataset (equation) and each method, we report the results of the best model (minimal KAN shape, or lowest test loss) over random seeds and depths in Table 2. We find that MLPs and KANs behave comparably on average. For each dataset and each model family (KANs or MLPs), we plot the Pareto frontier in the plane spanned by the number of parameters and RMSE losses, shown in Figure D.1 in Appendix D. We conjecture that the Feynman datasets are too simple to let KANs make further improvements, in the sense that variable dependence is usually smooth or monotonic, which is in contrast to the complexity of special functions which often demonstrate oscillatory behavior. Auto-discovered KANs are smaller than human-constructed ones. We report the pruned KAN shape in two columns of Table 2; one column is for the minimal pruned KAN shape that can achieve reasonable loss (i.e., test RMSE smaller than $10^{-2}$ ); the other column is for the pruned KAN that achieves lowest test loss. For completeness, we visualize all 54 pruned KANs in Appendix D (Figure D.2 and D.3). It is interesting to observe that auto-discovered KAN shapes (for both minimal and best) are usually smaller than our human constructions. This means that KA representations can be more efficient than we imagine. At the same time, this may make interpretability subtle because information is being squashed into a smaller space than what we are comfortable with. Consider the relativistic velocity composition $f(u,v)=\frac{u+v}{1+uv}$ , for example. Our construction is quite deep because we were assuming that multiplication of $u,v$ would use two layers (see Figure 4.1 (a)), inversion of $1+uv$ would use one layer, and multiplication of $u+v$ and $1/(1+uv)$ would use another two layers Note that we cannot use the logarithmic construction for division, because $u$ and $v$ here might be negative numbers., resulting a total of 5 layers. However, the auto-discovered KANs are only 2 layers deep! In hindsight, this is actually expected if we recall the rapidity trick in relativity: define the two “rapidities” $a\equiv{\rm arctanh}\ u$ and $b\equiv{\rm arctanh}\ v$ . The relativistic composition of velocities are simple additions in rapidity space, i.e., $\frac{u+v}{1+uv}={\rm tanh}({\rm arctanh}\ u+{\rm arctanh}\ v)$ , which can be realized by a two-layer KAN. Pretending we do not know the notion of rapidity in physics, we could potentially discover this concept right from KANs without trial-and-error symbolic manipulations. The interpretability of KANs which can facilitate scientific discovery is the main topic in Section 4. 3.4 Solving partial differential equations <details> <summary>2404.19756v5/x5.png Details</summary> ![6d1342fc076a8d7aac3b95b0913df3303669bbfc55142dbab480c564a21d5f1b](http://localhost:8000/v1/image/6d1342fc076a8d7aac3b95b0913df3303669bbfc55142dbab480c564a21d5f1b) ### Visual Description # Technical Document Extraction: Chart Analysis ## Chart 1: L2 Error Squared vs. Step - **X-axis**: `step` (0 to 250) - **Y-axis**: `L2 error squared` (10⁻¹ to 10⁻⁷) - **Legend**: - `KAN [2,10,1]` (blue line) - `MLP [2,10,1]` (orange line) - `MLP [2,100,100,100,1]` (green line) - **Key Trends**: - `KAN [2,10,1]` starts at ~10⁻¹, drops sharply to ~10⁻⁷ by step 250. - `MLP [2,10,1]` remains flat at ~10⁻¹. - `MLP [2,100,100,100,1]` starts at ~10⁻¹, drops to ~10⁻⁵ by step 50, then stabilizes. ## Chart 2: H1 Error Squared vs. Step - **X-axis**: `step` (0 to 250) - **Y-axis**: `H1 error squared` (10⁰ to 10⁻⁴) - **Legend**: - `KAN [2,10,1]` (blue line) - `MLP [2,10,1]` (orange line) - `MLP [2,100,100,100,1]` (green line) - **Key Trends**: - `KAN [2,10,1]` starts at ~10⁰, drops to ~10⁻⁴ by step 250. - `MLP [2,10,1]` remains flat at ~10⁰. - `MLP [2,100,100,100,1]` starts at ~10⁰, drops to ~10⁻³ by step 50, then stabilizes. ## Chart 3: L2 Error Squared vs. Number of Parameters - **X-axis**: `Number of parameters` (log scale: 10¹ to 10⁵) - **Y-axis**: `L2 error squared` (10⁰ to 10⁻⁷) - **Legend**: - `KAN [2,5,1]` (blue line) - `KAN [2,7,1]` (orange line) - `KAN [2,10,1]` (green line) - `MLP (depth 2)` (red line) - `MLP (depth 3)` (purple line) - `MLP (depth 4)` (brown line) - `MLP (depth 5)` (pink line) - `N⁻⁴` (dashed black line) - **Key Trends**: - `KAN` lines (blue, orange, green) show steep declines, outperforming `MLP` lines. - `MLP (depth 5)` (pink) and `MLP (depth 4)` (brown) lines are closest to `N⁻⁴`. - `KAN [2,10,1]` (green) intersects `N⁻⁴` at ~10³ parameters. ## Chart 4: H1 Error Squared vs. Number of Parameters - **X-axis**: `Number of parameters` (log scale: 10¹ to 10⁵) - **Y-axis**: `H1 error squared` (10⁰ to 10⁻³) - **Legend**: - `KAN [2,5,1]` (blue line) - `KAN [2,7,1]` (orange line) - `KAN [2,10,1]` (green line) - `MLP (depth 2)` (red line) - `MLP (depth 3)` (purple line) - `MLP (depth 4)` (brown line) - `MLP (depth 5)` (pink line) - `N⁻⁴` (dashed black line) - **Key Trends**: - `KAN` lines (blue, orange, green) show steep declines, outperforming `MLP` lines. - `MLP (depth 5)` (pink) and `MLP (depth 4)` (brown) lines are closest to `N⁻⁴`. - `KAN [2,10,1]` (green) intersects `N⁻⁴` at ~10³ parameters. ## Cross-Reference Validation - **Legend Colors/Labels**: - All `KAN` lines (blue, orange, green) match their respective datasets across charts. - `MLP` lines (red, purple, brown, pink) consistently represent their depth configurations. - `N⁻⁴` (dashed black) is present in Charts 3 and 4 as a reference baseline. - **Consistency Checks**: - `KAN [2,10,1]` (green) appears in Charts 1, 3, and 4 with identical trends. - `MLP [2,100,100,100,1]` (green) in Charts 1 and 2 aligns with `MLP (depth 5)` (pink) in Charts 3 and 4. </details> Figure 3.3: The PDE example. We plot L2 squared and H1 squared losses between the predicted solution and ground truth solution. First and second: training dynamics of losses. Third and fourth: scaling laws of losses against the number of parameters. KANs converge faster, achieve lower losses, and have steeper scaling laws than MLPs. We consider a Poisson equation with zero Dirichlet boundary data. For $\Omega=[-1,1]^{2}$ , consider the PDE $$ \displaystyle u_{xx}+u_{yy} \displaystyle=f\quad\text{in}\,\,\Omega\,, \displaystyle u \displaystyle=0\quad\text{on}\,\,\partial\Omega\,. $$ We consider the data $f=-\pi^{2}(1+4y^{2})\sin(\pi x)\sin(\pi y^{2})+2\pi\sin(\pi x)\cos(\pi y^{2})$ for which $u=\sin(\pi x)\sin(\pi y^{2})$ is the true solution. We use the framework of physics-informed neural networks (PINNs) [38, 39] to solve this PDE, with the loss function given by $$ \text{loss}_{\text{pde}}=\alpha\text{loss}_{i}+\text{loss}_{b}\coloneqq\alpha% \frac{1}{n_{i}}\sum_{i=1}^{n_{i}}|u_{xx}(z_{i})+u_{yy}(z_{i})-f(z_{i})|^{2}+% \frac{1}{n_{b}}\sum_{i=1}^{n_{b}}u^{2}\,, $$ where we use $\text{loss}_{i}$ to denote the interior loss, discretized and evaluated by a uniform sampling of $n_{i}$ points $z_{i}=(x_{i},y_{i})$ inside the domain, and similarly we use $\text{loss}_{b}$ to denote the boundary loss, discretized and evaluated by a uniform sampling of $n_{b}$ points on the boundary. $\alpha$ is the hyperparameter balancing the effect of the two terms. We compare the KAN architecture with that of MLPs using the same hyperparameters $n_{i}=10000$ , $n_{b}=800$ , and $\alpha=0.01$ . We measure both the error in the $L^{2}$ norm and energy ( $H^{1}$ ) norm and see that KAN achieves a much better scaling law with a smaller error, using smaller networks and fewer parameters; see Figure 3.3. A 2-Layer width-10 KAN is 100 times more accurate than a 4-Layer width-100 MLP ( $10^{-7}$ vs $10^{-5}$ MSE) and 100 times more parameter efficient ( $10^{2}$ vs $10^{4}$ parameters). Therefore we speculate that KANs might have the potential of serving as a good neural network representation for model reduction of PDEs. However, we want to note that our implementation of KANs are typically 10x slower than MLPs to train. The ground truth being a symbolic formula might be an unfair comparison for MLPs since KANs are good at representing symbolic formulas. In general, KANs and MLPs are good at representing different function classes of PDE solutions, which needs detailed future study to understand their respective boundaries. 3.5 Continual Learning <details> <summary>2404.19756v5/x6.png Details</summary> ![e02293a83774255a0efed3745aa3505a8175f97d295d2e17cbf1aefd95da2ee0](http://localhost:8000/v1/image/e02293a83774255a0efed3745aa3505a8175f97d295d2e17cbf1aefd95da2ee0) ### Visual Description # Technical Document Extraction: Phase Analysis Chart ## Image Structure - **Grid Layout**: 5 columns (Phase 1–5) × 3 rows (Data, KAN, MLP) - **Axes**: - **X-axis**: Ranges from -1 to 1 (labeled "Data" in all subplots) - **Y-axis**: Ranges from 0 to 1 (labeled "Data", "KAN", "MLP" per row) ## Key Components ### Phase Labels - **Columns**: - Phase 1 (leftmost) - Phase 2 - Phase 3 - Phase 4 - Phase 5 (rightmost) ### Subplot Labels - **Rows**: - **Top Row**: "Data" (black line) - **Middle Row**: "KAN" (dark gray line) - **Bottom Row**: "MLP" (light gray line) ## Data Trends ### Data Subplots (Top Row) - **Visual Pattern**: - Each phase shows a **central peak** at x=0. - Smaller symmetric peaks at x≈±0.5 and x≈±1. - Peaks decrease in amplitude with distance from x=0. - **Trend Verification**: - All phases exhibit identical symmetric peak distributions. - No phase-specific variations observed. ### KAN Subplots (Middle Row) - **Visual Pattern**: - **Rising trend** with oscillations. - Oscillations increase in amplitude toward the right edge of each phase. - Final peak at x=1 for all phases. - **Trend Verification**: - Consistent upward trajectory across all phases. - Oscillation frequency matches phase boundaries. ### MLP Subplots (Bottom Row) - **Visual Pattern**: - **Sharp peak** at x=-1 (left edge) for all phases. - **Secondary peak** at x=1 (right edge) for all phases. - Flat baseline (y=0) between peaks. - **Trend Verification**: - Peaks are phase-agnostic; identical across all phases. - No intermediate activity (y=0) between x=-1 and x=1. ## Color Legend & Spatial Grounding - **Legend**: **Not explicitly present** in the image. - **Color Mapping**: - **Black**: Data subplots (top row). - **Dark Gray**: KAN subplots (middle row). - **Light Gray**: MLP subplots (bottom row). - **Spatial Confirmation**: - All phase columns share identical subplot structures. - No cross-phase data alignment observed. ## Component Isolation 1. **Header**: Phase labels (Phase 1–5) at the top of each column. 2. **Main Chart**: - Each phase contains three independent subplots. - No inter-phase dependencies in data representation. 3. **Footer**: No additional annotations or labels. ## Data Table Reconstruction | Phase | Data Subplot | KAN Subplot | MLP Subplot | |-------|--------------|-------------|-------------| | 1 | Central peak at 0 | Rising trend with oscillations | Peaks at -1 and 1 | | 2 | Central peak at 0 | Rising trend with oscillations | Peaks at -1 and 1 | | 3 | Central peak at 0 | Rising trend with oscillations | Peaks at -1 and 1 | | 4 | Central peak at 0 | Rising trend with oscillations | Peaks at -1 and 1 | | 5 | Central peak at 0 | Rising trend with oscillations | Peaks at -1 and 1 | ## Observations - **Consistency**: All phases exhibit identical subplot patterns. - **Phase-Specific Behavior**: No unique trends or deviations between phases. - **Axis Consistency**: Uniform x-axis (-1 to 1) and y-axis (0 to 1) across all subplots. ## Conclusion The chart visualizes three metrics (Data, KAN, MLP) across five identical phases. No phase-specific variations are observed, suggesting a controlled or standardized experimental setup. The absence of a legend implies color coding is implicit via subplot labels. </details> Figure 3.4: A toy continual learning problem. The dataset is a 1D regression task with 5 Gaussian peaks (top row). Data around each peak is presented sequentially (instead of all at once) to KANs and MLPs. KANs (middle row) can perfectly avoid catastrophic forgetting, while MLPs (bottom row) display severe catastrophic forgetting. Catastrophic forgetting is a serious problem in current machine learning [40]. When a human masters a task and switches to another task, they do not forget how to perform the first task. Unfortunately, this is not the case for neural networks. When a neural network is trained on task 1 and then shifted to being trained on task 2, the network will soon forget about how to perform task 1. A key difference between artificial neural networks and human brains is that human brains have functionally distinct modules placed locally in space. When a new task is learned, structure re-organization only occurs in local regions responsible for relevant skills [41, 42], leaving other regions intact. Most artificial neural networks, including MLPs, do not have this notion of locality, which is probably the reason for catastrophic forgetting. We show that KANs have local plasticity and can avoid catastrophic forgetting by leveraging the locality of splines. The idea is simple: since spline bases are local, a sample will only affect a few nearby spline coefficients, leaving far-away coefficients intact (which is desirable since far-away regions may have already stored information that we want to preserve). By contrast, since MLPs usually use global activations, e.g., ReLU/Tanh/SiLU etc., any local change may propagate uncontrollably to regions far away, destroying the information being stored there. We use a toy example to validate this intuition. The 1D regression task is composed of 5 Gaussian peaks. Data around each peak is presented sequentially (instead of all at once) to KANs and MLPs, as shown in Figure 3.4 top row. KAN and MLP predictions after each training phase are shown in the middle and bottom rows. As expected, KAN only remodels regions where data is present on in the current phase, leaving previous regions unchanged. By contrast, MLPs remodels the whole region after seeing new data samples, leading to catastrophic forgetting. Here we simply present our preliminary results on an extremely simple example, to demonstrate how one could possibly leverage locality in KANs (thanks to spline parametrizations) to reduce catastrophic forgetting. However, it remains unclear whether our method can generalize to more realistic setups, especially in high-dimensional cases where it is unclear how to define “locality”. In future work, We would also like to study how our method can be connected to and combined with SOTA methods in continual learning [43, 44]. 4 KANs are interpretable In this section, we show that KANs are interpretable and interactive thanks to the techniques we developed in Section 2.5. We want to test the use of KANs not only on synthetic tasks (Section 4.1 and 4.2), but also in real-life scientific research. We demonstrate that KANs can (re)discover both highly non-trivial relations in knot theory (Section 4.3) and phase transition boundaries in condensed matter physics (Section 4.4). KANs could potentially be the foundation model for AI + Science due to their accuracy (last section) and interpretability (this section). 4.1 Supervised toy datasets <details> <summary>2404.19756v5/extracted/6190555/figs/interpretable_examples.png Details</summary> ![1f32ed887c9c64e6a25d55238a18d6a29ead94058dbb5d428f58ed87acf6aee9](http://localhost:8000/v1/image/1f32ed887c9c64e6a25d55238a18d6a29ead94058dbb5d428f58ed87acf6aee9) ### Visual Description # Technical Diagram Analysis ## Diagram Structure The image contains a 2x3 grid of technical diagrams with mathematical notation and symbolic representations. Each diagram is labeled with a specific operation or function type. --- ### Diagram (a): Multiplication **Title:** (a) multiplication **Nodes:** - Input nodes: `x`, `y` (represented as black dots) - Output node: `xy` (represented as a black dot with a cross) **Connections:** - Direct edges between `x` and `y` - Crossed edges between `x` and `xy`, `y` and `xy` **Mathematical Representation:** `xy` (implied multiplication) --- ### Diagram (b): Division **Title:** (b) division **Nodes:** - Input nodes: `x`, `y` (black dots) - Output node: `x/y` (black dot with cross) **Connections:** - Direct edges between `x` and `y` - Crossed edges between `x` and `x/y`, `y` and `x/y` **Mathematical Representation:** `x/y` --- ### Diagram (c): Numerical to Category **Title:** (c) numerical to category **Nodes:** - Input node: `X` (black dot) - Output nodes: `e_{argmin_i|x - x_i|}` (multiple nodes with arrows) **Connections:** - Single input node `X` connected to multiple output nodes via arrows **Mathematical Representation:** `e_{argmin_i|x - x_i|}` (vector notation) --- ### Diagram (d): Special Function **Title:** (d) special function **Nodes:** - Input nodes: `x`, `y` (black dots) - Function node: `exp(J₀(20x) + y²)` (black dot with cross) **Connections:** - Direct edges between `x` and `y` - Crossed edges between `x` and function node, `y` and function node **Mathematical Representation:** `exp(J₀(20x) + y²)` --- ### Diagram (e): Phase Transition **Title:** (e) phase transition **Nodes:** - Input nodes: `X₁`, `X₂`, `X₃` (black dots) - Function node: `tanh(5(x₁⁴ + x₂⁴ + x₃⁴ - 1))` (black dot with cross) **Connections:** - Direct edges between `X₁`, `X₂`, `X₃` - Crossed edges between input nodes and function node **Mathematical Representation:** `tanh(5(x₁⁴ + x₂⁴ + x₃⁴ - 1))` --- ### Diagram (f): Deeper Compositions **Title:** (f) deeper compositions **Nodes:** - Input nodes: `X₁`, `X₂`, `X₃`, `X₄` (black dots) - Function node: `√((x₁ - x₂)² + (x₃ - x₄)²)` (black dot with cross) **Connections:** - Complex network with multiple crossed edges - Hierarchical connections between input nodes and function node **Mathematical Representation:** `√((x₁ - x₂)² + (x₃ - x₄)²)` --- ## Key Observations 1. **Symbolic Representation:** - Black dots represent input variables - Crossed dots represent function outputs - Arrows indicate directional relationships 2. **Mathematical Operations:** - Basic arithmetic (multiplication/division) - Special functions (exponential, hyperbolic tangent) - Composition of functions with multiple variables 3. **Architectural Patterns:** - Feedforward structures in (a)-(e) - Hierarchical composition in (f) - Vectorized operations in (c) 4. **Notation Consistency:** - Subscript notation for categorical outputs (`argmin_i`) - Superscript notation for variable powers (`x₁⁴`) - Square root symbol for compositional functions --- ## Technical Implications These diagrams appear to represent: 1. **Neural Network Architectures** - Feedforward structures in (a)-(e) - Compositional layers in (f) 2. **Mathematical Function Approximation** - Use of special functions (Bessel function `J₀`, hyperbolic tangent) - Composition of operations for complex transformations 3. **Categorical Mapping** - Numerical-to-categorical conversion in (c) - Distance-based assignment (`argmin_i|x - x_i|`) --- ## Limitations - No explicit data values or numerical results provided - No color-coded data series or heatmaps present - No temporal or sequential dimensions represented --- ## Language Analysis All text is in English. No non-English content detected. </details> Figure 4.1: KANs are interepretable for simple symbolic tasks We first examine KANs’ ability to reveal the compositional structures in symbolic formulas. Six examples are listed below and their KANs are visualized in Figure 4.1. KANs are able to reveal the compositional structures present in these formulas, as well as learn the correct univariate functions. 1. Multiplication $f(x,y)=xy$ . A $[2,5,1]$ KAN is pruned to a $[2,2,1]$ KAN. The learned activation functions are linear and quadratic. From the computation graph, we see that the way it computes $xy$ is leveraging $2xy=(x+y)^{2}-(x^{2}+y^{2})$ . 1. Division of positive numbers $f(x,y)=x/y$ . A $[2,5,1]$ KAN is pruned to a $[2,1,1]$ KAN. The learned activation functions are logarithmic and exponential functions, and the KAN is computing $x/y$ by leveraging the identity $x/y={\exp}({\log}x-{\log}y)$ . 1. Numerical to categorical. The task is to convert a real number in $[0,1]$ to its first decimal digit (as one hots), e.g., $0.0618→[1,0,0,0,0,·s]$ , $0.314→[0,0,0,1,0,·s]$ . Notice that activation functions are learned to be spikes located around the corresponding decimal digits. 1. Special function $f(x,y)={\rm exp}(J_{0}(20x)+y^{2})$ . One limitation of symbolic regression is that it will never find the correct formula of a special function if the special function is not provided as prior knowledge. KANs can learn special functions – the highly wiggly Bessel function $J_{0}(20x)$ is learned (numerically) by KAN. 1. Phase transition $f(x_{1},x_{2},x_{3})={\rm tanh}(5(x_{1}^{4}+x_{2}^{4}+x_{3}^{4}-1))$ . Phase transitions are of great interest in physics, so we want KANs to be able to detect phase transitions and to identify the correct order parameters. We use the tanh function to simulate the phase transition behavior, and the order parameter is the combination of the quartic terms of $x_{1},x_{2},x_{3}$ . Both the quartic dependence and tanh dependence emerge after KAN training. This is a simplified case of a localization phase transition discussed in Section 4.4. 1. Deeper compositions $f(x_{1},x_{2},x_{3},x_{4})=\sqrt{(x_{1}-x_{2})^{2}+(x_{3}-x_{4})^{2}}$ . To compute this, we would need the identity function, squared function, and square root, which requires at least a three-layer KAN. Indeed, we find that a $[4,3,3,1]$ KAN can be auto-pruned to a $[4,2,1,1]$ KAN, which exactly corresponds to the computation graph we would expect. More examples from the Feynman dataset and the special function dataset are visualized in Figure D.2, D.3, F.1, F.2 in Appendices D and F. 4.2 Unsupervised toy dataset Often, scientific discoveries are formulated as supervised learning problems, i.e., given input variables $x_{1},x_{2},·s,x_{d}$ and output variable(s) $y$ , we want to find an interpretable function $f$ such that $y≈ f(x_{1},x_{2},·s,x_{d})$ . However, another type of scientific discovery can be formulated as unsupervised learning, i.e., given a set of variables $(x_{1},x_{2},·s,x_{d})$ , we want to discover a structural relationship between the variables. Specifically, we want to find a non-zero $f$ such that $$ \displaystyle f(x_{1},x_{2},\cdots,x_{d})\approx 0. $$ For example, consider a set of features $(x_{1},x_{2},x_{3})$ that satisfies $x_{3}={\rm exp}({\rm sin}(\pi x_{1})+x_{2}^{2})$ . Then a valid $f$ is $f(x_{1},x_{2},x_{3})={\rm sin}(\pi x_{1})+x_{2}^{2}-{\rm log}(x_{3})=0$ , implying that points of $(x_{1},x_{2},x_{3})$ form a 2D submanifold specified by $f=0$ instead of filling the whole 3D space. If an algorithm for solving the unsupervised problem can be devised, it has a considerable advantage over the supervised problem, since it requires only the sets of features $S=(x_{1},x_{2},·s,x_{d})$ . The supervised problem, on the other hand, tries to predict subsets of features in terms of the others, i.e. it splits $S=S_{\text{in}}\cup S_{\text{out}}$ into input and output features of the function to be learned. Without domain expertise to advise the splitting, there are $2^{d}-2$ possibilities such that $|S_{\text{in}}|>0$ and $|S_{\text{out}}|>0$ . This exponentially large space of supervised problems can be avoided by using the unsupervised approach. This unsupervised learning approach will be valuable to the knot dataset in Section 4.3. A Google Deepmind team [45] manually chose signature to be the target variable, otherwise they would face this combinatorial problem described above. This raises the question whether we can instead tackle the unsupervised learning directly. We present our method and a toy example below. We tackle the unsupervised learning problem by turning it into a supervised learning problem on all of the $d$ features, without requiring the choice of a splitting. The essential idea is to learn a function $f(x_{1},...,x_{d})=0$ such that $f$ is not the $0 0$ -function. To do this, similar to contrastive learning, we define positive samples and negative samples: positive samples are feature vectors of real data. Negative samples are constructed by feature corruption. To ensure that the overall feature distribution for each topological invariant stays the same, we perform feature corruption by random permutation of each feature across the entire training set. Now we want to train a network $g$ such that $g(\mathbf{x}_{\rm real})=1$ and $g(\mathbf{x}_{\rm fake})=0$ which turns the problem into a supervised problem. However, remember that we originally want $f(\mathbf{x}_{\rm real})=0$ and $f(\mathbf{x}_{\rm fake})≠ 0$ . We can achieve this by having $g=\sigma\circ f$ where $\sigma(x)={\rm exp}(-\frac{x^{2}}{2w^{2}})$ is a Gaussian function with a small width $w$ , which can be conveniently realized by a KAN with shape $[...,1,1]$ whose last activation is set to be the Gaussian function $\sigma$ and all previous layers form $f$ . Except for the modifications mentioned above, everything else is the same for supervised training. <details> <summary>2404.19756v5/extracted/6190555/figs/unsupervised_toy.png Details</summary> ![9bbccc5677eca18f11850d8910945359211844869a0c134aee251a3b1337179c](http://localhost:8000/v1/image/9bbccc5677eca18f11850d8910945359211844869a0c134aee251a3b1337179c) ### Visual Description # Technical Document Extraction: Diagram Analysis ## Diagram Structure Overview The image contains **two side-by-side diagrams** labeled with distinct seed values. Both diagrams share a consistent structure but differ in seed initialization and node connectivity. --- ### **Left Diagram: `seed = 0`** #### Header Text - **Seed Label**: `seed = 0` - **Data/Random Indicator**: `1 (data) or 0 (random)` #### Central Node - A **black dot** connected to a **vertical red line** with a **red box** around its midpoint. - The red line extends upward from the central node, terminating at the top of the diagram. #### X Nodes (X₁–X₆) - Six nodes labeled **X₁** to **X₆**, arranged horizontally below the central node. - Each node has a **black dot** and a **square** containing a waveform (graph-like line). - **Connectivity**: - **X₁**: Waveform with a **sinusoidal curve** (peaks and troughs). - **X₂**: Waveform with a **parabolic curve** (U-shaped). - **X₃**: Waveform with a **linear decline** (downward slope). - **X₄**: Waveform with a **noisy, irregular curve**. - **X₅**: Waveform with a **linear increase** (upward slope). - **X₆**: Empty square (no waveform). #### Connections - Lines connect the central node to all X nodes (X₁–X₆). - Additional lines connect: - **X₄** to **X₅** (dashed line). - **X₅** to **X₆** (dashed line). --- ### **Right Diagram: `seed = 2024`** #### Header Text - **Seed Label**: `seed = 2024` - **Data/Random Indicator**: `1 (data) or 0 (random)` #### Central Node - Identical to the left diagram: **black dot** with a **vertical red line** and **red box**. #### X Nodes (X₁–X₆) - Six nodes labeled **X₁** to **X₆**, arranged horizontally below the central node. - **Waveform Differences**: - **X₁**: Empty square. - **X₂**: Empty square. - **X₃**: Empty square. - **X₄**: Waveform with a **noisy, irregular curve** (same as left diagram). - **X₅**: Waveform with a **linear increase** (same as left diagram). - **X₆**: Waveform with a **linear increase** (new addition). #### Connections - Lines connect the central node to all X nodes (X₁–X₆). - Additional lines connect: - **X₄** to **X₅** (dashed line). - **X₅** to **X₆** (dashed line). --- ### Key Observations 1. **Seed Dependency**: - The left diagram (`seed = 0`) shows more populated waveforms (X₁–X₅), while the right diagram (`seed = 2024`) has fewer waveforms (X₄–X₆). - Seed value appears to influence the initialization or activation of nodes. 2. **Waveform Patterns**: - Waveforms in squares likely represent **data signals** or **random noise** (as indicated by the `1 (data) or 0 (random)` label). - Patterns vary by node (e.g., sinusoidal, parabolic, linear). 3. **Connectivity**: - Dashed lines between X₄–X₅–X₆ suggest **conditional or secondary relationships** between nodes. 4. **Red Line/Box**: - The red line and box around the central node may denote a **critical threshold** or **activation point** in the system. --- ### Technical Implications - The diagrams likely model a **probabilistic or algorithmic process** where: - Seed values determine initial conditions. - Nodes (X₁–X₆) represent variables or features. - Waveforms indicate data presence/absence or signal strength. - Connectivity defines dependencies or transitions between variables. --- ### Missing Elements - No explicit **legend** or **axis titles** are present. - No numerical data or quantitative labels are provided (e.g., time, magnitude). - The purpose of the red line/box and dashed connections is inferred but not explicitly labeled. --- ### Summary The diagrams illustrate a **seed-dependent system** with nodes (X₁–X₆) that exhibit varying data patterns based on initialization. The red line/box and connectivity suggest a hierarchical or conditional structure, possibly for data processing or simulation. </details> Figure 4.2: Unsupervised learning of a toy task. KANs can identify groups of dependent variables, i.e., $(x_{1},x_{2},x_{3})$ and $(x_{4},x_{5})$ in this case. Now we demonstrate that the unsupervised paradigm works for a synthetic example. Let us consider a 6D dataset, where $(x_{1},x_{2},x_{3})$ are dependent variables such that $x_{3}=\exp(\sin(x_{1})+x_{2}^{2})$ ; $(x_{4},x_{5})$ are dependent variables with $x_{5}=x_{4}^{3}$ ; $x_{6}$ is independent of the other variables. In Figure 4.2, we show that for seed = 0, KAN reveals the functional dependence among $x_{1}$ , $x_{2}$ , and $x_{3}$ ; for another seed = 2024, KAN reveals the functional dependence between $x_{4}$ and $x_{5}$ . Our preliminary results rely on randomness (different seeds) to discover different relations; in the future we would like to investigate a more systematic and more controlled way to discover a complete set of relations. Even so, our tool in its current status can provide insights for scientific tasks. We present our results with the knot dataset in Section 4.3. 4.3 Application to Mathematics: Knot Theory Knot theory is a subject in low-dimensional topology that sheds light on topological aspects of three-manifolds and four-manifolds and has a variety of applications, including in biology and topological quantum computing. Mathematically, a knot $K$ is an embedding of $S^{1}$ into $S^{3}$ . Two knots $K$ and $K^{\prime}$ are topologically equivalent if one can be deformed into the other via deformation of the ambient space $S^{3}$ , in which case we write $[K]=[K^{\prime}]$ . Some knots are topologically trivial, meaning that they can be smoothly deformed to a standard circle. Knots have a variety of deformation-invariant features $f$ called topological invariants, which may be used to show that two knots are topologically inequivalent, $[K]≠[K^{\prime}]$ if $f(K)≠ f(K^{\prime})$ . In some cases the topological invariants are geometric in nature. For instance, a hyperbolic knot $K$ has a knot complement $S^{3}\setminus K$ that admits a canonical hyperbolic metric $g$ such that $\text{vol}_{g}(K)$ is a topological invariant known as the hyperbolic volume. Other topological invariants are algebraic in nature, such as the Jones polynomial. Given the fundamental nature of knots in mathematics and the importance of its applications, it is interesting to study whether ML can lead to new results. For instance, in [46] reinforcement learning was utilized to establish ribbonness of certain knots, which ruled out many potential counterexamples to the smooth 4d Poincaré conjecture. Supervised learning In [45], supervised learning and human domain experts were utilized to arrive at a new theorem relating algebraic and geometric knot invariants. In this case, gradient saliency identified key invariants for the supervised problem, which led the domain experts to make a conjecture that was subsequently refined and proven. We study whether a KAN can achieve good interpretable results on the same problem, which predicts the signature of a knot. Their main results from studying the knot theory dataset are: 1. They use network attribution methods to find that the signature $\sigma$ is mostly dependent on meridinal distance $\mu$ (real $\mu_{r}$ , imag $\mu_{i}$ ) and longitudinal distance $\lambda$ . 1. Human scientists later identified that $\sigma$ has high correlation with the ${\rm slope}\equiv{\rm Re}(\frac{\lambda}{\mu})=\frac{\lambda\mu_{r}}{\mu_{r}^{% 2}+\mu_{i}^{2}}$ and derived a bound for $|2\sigma-{\rm slope}|$ . We show below that KANs not only rediscover these results with much smaller networks and much more automation, but also present some interesting new results and insights. <details> <summary>2404.19756v5/extracted/6190555/figs/math.png Details</summary> ![08027c5dc7d865831e493c1a43c5954eb602a3cd81b0c3499d5d31f0c5ac2432](http://localhost:8000/v1/image/08027c5dc7d865831e493c1a43c5954eb602a3cd81b0c3499d5d31f0c5ac2432) ### Visual Description # Technical Document Extraction ## Diagram (a): All 17 Variables (Test Accuracy: 81.6%) - **Title**: "All 17 variables (test acc: 81.6%)" - **X-axis**: Numerical range from -12 to 14 (inclusive), labeled with integer ticks. - **Variables (17 total)**: 1. hyperbolic_torsion_degree 2. short_geodesic_real_part 3. short_geodesic_imag_part 4. injectivity_radius 5. chem_simons 6. cusp_volume 7. longitudinal_translation 8. meridional_translation_real 9. meridional_translation_imag 10. volume 11. symmetry_0 12. symmetry_D3 13. symmetry_D4 14. symmetry_D6 15. symmetry_D8 16. symmetry_Z/2 + Z/2 17. meridional_translation_real (repeated in diagram) ## Diagram (b): 3 Important Variables (Test Accuracy: 78.2%) - **Title**: "3 important variables (test acc: 78.2%)" - **X-axis**: Numerical range from -12 to 14 (inclusive), labeled with integer ticks. - **Variables (3 total)**: 1. longitudinal_translation 2. meridional_translation_imag 3. meridional_translation_real ## Diagram (c): Dependence of Test Accuracy on 3 Variables - **Title**: "Dependence of test acc on 3 variables" - **Venn Diagram Components**: - **meridional_translation_real** (Red): - Unique: 65.0% - Overlap with meridional_translation_imag: 67.3% - Overlap with longitudinal_translation: 77.5% - Full overlap (all three): 78.2% - **meridional_translation_imag** (Blue): - Unique: 43.8% - Overlap with meridional_translation_real: 67.3% - Overlap with longitudinal_translation: 50.8% - Full overlap (all three): 78.2% - **longitudinal_translation** (Green): - Unique: 36.9% - Overlap with meridional_translation_real: 77.5% - Overlap with meridional_translation_imag: 50.8% - Full overlap (all three): 78.2% ## Key Observations 1. **Test Accuracy**: - Using all 17 variables: 81.6% - Using 3 key variables: 78.2% 2. **Variable Importance**: - `meridional_translation_real` contributes most to test accuracy (77.5% overlap with longitudinal_translation). - `meridional_translation_imag` has the lowest unique contribution (43.8%). 3. **Interdependencies**: - Full overlap of all three variables accounts for 78.2% of test accuracy. - Unique contributions decrease as variables are combined. </details> Figure 4.3: Knot dataset, supervised mode. With KANs, we rediscover Deepmind’s results that signature is mainly dependent on meridinal translation (real and imaginary parts). To investigate (1), we treat 17 knot invariants as inputs and signature as outputs. Similar to the setup in [45], signatures (which are even numbers) are encoded as one-hot vectors and networks are trained with cross-entropy loss. We find that an extremely small $[17,1,14]$ KAN is able to achieve $81.6\%$ test accuracy (while Deepmind’s 4-layer width-300 MLP achieves 78% test accuracy). The $[17,1,14]$ KAN ( $G=3$ , $k=3$ ) has $≈ 200$ parameters, while the MLP has $≈ 3× 10^{5}$ parameters, shown in Table 3. It is remarkable that KANs can be both more accurate and much more parameter efficient than MLPs at the same time. In terms of interpretability, we scale the transparency of each activation according to its magnitude, so it becomes immediately clear which input variables are important without the need for feature attribution (see Figure 4.3 left): signature is mostly dependent on $\mu_{r}$ , and slightly dependent on $\mu_{i}$ and $\lambda$ , while dependence on other variables is small. We then train a $[3,1,14]$ KAN on the three important variables, obtaining test accuracy $78.2\%$ . Our results have one subtle difference from results in [45]: they find that signature is mostly dependent on $\mu_{i}$ , while we find that signature is mostly dependent on $\mu_{r}$ . This difference could be due to subtle algorithmic choices, but has led us to carry out the following experiments: (a) ablation studies. We show that $\mu_{r}$ contributes more to accuracy than $\mu_{i}$ (see Figure 4.3): for example, $\mu_{r}$ alone can achieve $65.0\%$ accuracy, while $\mu_{i}$ alone can only achieve $43.8\%$ accuracy. (b) We find a symbolic formula (in Table 4) which only involves $\mu_{r}$ and $\lambda$ , but can achieve $77.8\%$ test accuracy. | Method | Architecture | Parameter Count | Accuracy | | --- | --- | --- | --- | | Deepmind’s MLP | 4 layer, width-300 | $3× 10^{5}$ | $78.0\%$ | | KANs | 2 layer, $[17,1,14]$ ( $G=3$ , $k=3$ ) | $2× 10^{2}$ | $81.6\%$ | Table 3: KANs can achieve better accuracy than MLPs with much fewer parameters in the signature classification problem. Soon after our preprint was first released, Prof. Shi Lab from Georgia tech discovered that an MLP with only 60 parameters is sufficient to achieve 80% accuracy (public but unpublished results). This is good news for AI + Science because this means perhaps many AI + Science tasks are not that computationally demanding than we might think (either with MLPs or with KANs), hence many new scientific discoveries are possible even on personal laptops. To investigate (2), i.e., obtain the symbolic form of $\sigma$ , we formulate the problem as a regression task. Using auto-symbolic regression introduced in Section 2.5.1, we can convert a trained KAN into symbolic formulas. We train KANs with shapes $[3,1]$ , $[3,1,1]$ , $[3,2,1]$ , whose corresponding symbolic formulas are displayed in Table 4 B-D. It is clear that by having a larger KAN, both accuracy and complexity increase. So KANs provide not just a single symbolic formula, but a whole Pareto frontier of formulas, trading off simplicity and accuracy. However, KANs need additional inductive biases to further simplify these equations to rediscover the formula from [45] (Table 4 A). We have tested two scenarios: (1) in the first scenario, we assume the ground truth formula has a multi-variate Pade representation (division of two multi-variate Taylor series). We first train $[3,2,1]$ and then fit it to a Pade representation. We can obtain Formula E in Table 4, which bears similarity with Deepmind’s formula. (2) We hypothesize that the division is not very interpretable for KANs, so we train two KANs (one for the numerator and the other for the denominator) and divide them manually. Surprisingly, we end up with the formula F (in Table 4) which only involves $\mu_{r}$ and $\lambda$ , although $\mu_{i}$ is also provided but ignored by KANs. | Id | Formula | Discovered by | test acc | $r^{2}$ with Signature | $r^{2}$ with DM formula | | --- | --- | --- | --- | --- | --- | | A | $\frac{\lambda\mu_{r}}{(\mu_{r}^{2}+\mu_{i}^{2})}$ | Human (DM) | 83.1% | 0.946 | 1 | | B | $-0.02{\rm sin}(4.98\mu_{i}+0.85)+0.08|4.02\mu_{r}+6.28|-0.52-0.04e^{-0.88(1-0.% 45\lambda)^{2}}$ | $[3,1]$ KAN | 62.6% | 0.837 | 0.897 | | C | $0.17{\rm tan}(-1.51+0.1e^{-1.43(1-0.4\mu_{i})^{2}+0.09e^{-0.06(1-0.21\lambda)^% {2}}}+1.32e^{-3.18(1-0.43\mu_{r})^{2}})$ | $[3,1,1]$ KAN | 71.9% | 0.871 | 0.934 | | D | $-0.09+1.04{\rm exp}(-9.59(-0.62{\rm sin}(0.61\mu_{r}+7.26))-0.32{\rm tan}(0.03% \lambda-6.59)+1-0.11e^{-1.77(0.31-\mu_{i})^{2})^{2}}-1.09e^{-7.6(0.65(1-0.01% \lambda)^{3}}+0.27{\rm atan}(0.53\mu_{i}-0.6)+0.09+{\rm exp}(-2.58(1-0.36\mu_{% r})^{2}))$ | $[3,2,1]$ KAN | 84.0% | 0.947 | 0.997 | | E | $\frac{4.76\lambda\mu_{r}}{3.09\mu_{i}+6.05\mu_{r}^{2}+3.54\mu_{i}^{2}}$ | [3,2,1] KAN + Pade approx | $82.8\%$ | 0.946 | 0.997 | | F | $\frac{2.94-2.92(1-0.10\mu_{r})^{2}}{0.32(0.18-\mu_{r})^{2}+5.36(1-0.04\lambda)% ^{2}+0.50}$ | $[3,1]$ KAN/ $[3,1]$ KAN | 77.8% | 0.925 | 0.977 | Table 4: Symbolic formulas of signature as a function of meridinal translation $\mu$ (real $\mu_{r}$ , imag $\mu_{i}$ ) and longitudinal translation $\lambda$ . In [45], formula A was discovered by human scientists inspired by neural network attribution results. Formulas B-F are auto-discovered by KANs. KANs can trade-off between simplicity and accuracy (B, C, D). By adding more inductive biases, KAN is able to discover formula E which is not too dissimilar from formula A. KANs also discovered a formula F which only involves two variables ( $\mu_{r}$ and $\lambda$ ) instead of all three variables, with little sacrifice in accuracy. So far, we have rediscovered the main results from [45]. It is remarkable to see that KANs made this discovery very intuitive and convenient. Instead of using feature attribution methods (which are great methods), one can instead simply stare at visualizations of KANs. Moreover, automatic symbolic regression also makes the discovery of symbolic formulas much easier. In the next part, we propose a new paradigm of “AI for Math” not included in the Deepmind paper, where we aim to use KANs’ unsupervised learning mode to discover more relations (besides signature) in knot invariants. <details> <summary>2404.19756v5/extracted/6190555/figs/knot_unsupervised.png Details</summary> ![9ace4c03c5532c23de5eee52788df9d0cb1d17482bd31e41be58f3e7716b12ac](http://localhost:8000/v1/image/9ace4c03c5532c23de5eee52788df9d0cb1d17482bd31e41be58f3e7716b12ac) ### Visual Description # Technical Document Extraction ## Panel (a): rediscover signature dependence ### Tree Diagram - **Root Node**: - Loss: `2.11e-01` - Label: `1 (data) or 0 (random)` - **Branches**: - `hyperbolic_geodesic` - `short_geodesic` - `objectivity` - `dem_volumes` - `metric_translation` - `longitudinal_translation` - `cusp_volume` - `symmetry_D3` - `symmetry_D4` - `symmetry_D6` - `symmetry_D8` - `signature` ### Scatter Plot - **Axes**: - X-axis: `meridional_translation_real` - Y-axis: `longitudinal_translation` - **Colorbar**: - Label: `signature` - Range: `-10` (purple) to `10` (yellow) - **Legend**: - Position: Top-right (not visible in image) - Colors: Match colorbar gradient ## Panel (b): rediscover cusp_volume definition ### Tree Diagram - **Root Node**: - Loss: `2.05e-01` - Label: `1 (data) or 0 (random)` - **Branches**: - `hyperbolic_geodesic` - `short_geodesic` - `objectivity` - `dem_volumes` - `cusp_volume` - `metric_translation` - `longitudinal_translation` - `symmetry_D3` - `symmetry_D4` - `symmetry_D6` - `symmetry_D8` - `signature` ### Scatter Plot - **Axes**: - X-axis: `longitudinal_translation*meridional_translation_imag` - Y-axis: `cusp_volume` - **Line**: - Equation: `y = x` (red dashed line) - **Legend**: - Position: Top-right (not visible in image) ## Panel (c): rediscover an inequality ### Tree Diagram - **Root Node**: - Loss: `2.62e-01` - Label: `1 (data) or 0 (random)` - **Branches**: - `hyperbolic_geodesic` - `short_geodesic` - `objectivity` - `dem_volumes` - `cusp_volume` - `metric_translation` - `longitudinal_translation` - `symmetry_D3` - `symmetry_D4` - `symmetry_D6` - `symmetry_D8` - `signature` ### Scatter Plot - **Axes**: - X-axis: `short_geodesic_real_part` - Y-axis: `injectivity_radius` - **Line**: - Equation: `y = x/2` (red dashed line) - **Legend**: - Position: Top-right (not visible in image) ## Common Elements - **Colorbar**: - Label: `signature` - Range: `-10` (purple) to `10` (yellow) - Position: Right side of all scatter plots - **Legend**: - Position: Top-right corner (not visible in image) - Colors: Match colorbar gradient ## Key Observations 1. **Loss Values**: - Panel (a): `2.11e-01` - Panel (b): `2.05e-01` - Panel (c): `2.62e-01` 2. **Scatter Plot Trends**: - Panel (a): Data points form a U-shaped distribution with high signature values (yellow) in the upper-right quadrant. - Panel (b): Linear relationship (`y=x`) between `longitudinal_translation*meridional_translation_imag` and `cusp_volume`. - Panel (c): Linear relationship (`y=x/2`) between `short_geodesic_real_part` and `injectivity_radius`. ## Spatial Grounding - **Colorbar**: Right-aligned across all panels. - **Legend**: Top-right corner (not visible in image). - **Tree Diagrams**: Positioned above scatter plots in each panel. ## Language Notes - All text is in English. No other languages detected. </details> Figure 4.4: Knot dataset, unsupervised mode. With KANs, we rediscover three mathematical relations in the knot dataset. Unsupervised learning As we mentioned in Section 4.2, unsupervised learning is the setup that is more promising since it avoids manual partition of input and output variables which have combinatorially many possibilities. In the unsupervised learning mode, we treat all 18 variables (including signature) as inputs such that they are on the same footing. Knot data are positive samples, and we randomly shuffle features to obtain negative samples. An $[18,1,1]$ KAN is trained to classify whether a given feature vector belongs to a positive sample (1) or a negative sample (0). We manually set the second layer activation to be the Gaussian function with a peak one centered at zero, so positive samples will have activations at (around) zero, implicitly giving a relation among knot invariants $\sum_{i=1}^{18}g_{i}(x_{i})=0$ where $x_{i}$ stands for a feature (invariant), and $g_{i}$ is the corresponding activation function which can be readily read off from KAN diagrams. We train the KANs with $\lambda=\{10^{-2},10^{-3}\}$ to favor sparse combination of inputs, and ${\rm seed}=\{0,1,·s,99\}$ . All 200 networks can be grouped into three clusters, with representative KANs displayed in Figure 4.4. These three groups of dependent variables are: 1. The first group of dependent variables is signature, real part of meridinal distance, and longitudinal distance (plus two other variables which can be removed because of (3)). This is the signature dependence studied above, so it is very interesting to see that this dependence relation is rediscovered again in the unsupervised mode. 1. The second group of variables involve cusp volume $V$ , real part of meridinal translation $\mu_{r}$ and longitudinal translation $\lambda$ . Their activations all look like logarithmic functions (which can be verified by the implied symbolic functionality in Section 2.5.1). So the relation is $-\log V+\log\mu_{r}+\log\lambda=0$ which is equivalent to $V=\mu_{r}\lambda$ , which is true by definition. It is, however, reassuring that we discover this relation without any prior knowledge. 1. The third group of variables includes the real part of short geodesic $g_{r}$ and injectivity radius. Their activations look qualitatively the same but differ by a minus sign, so it is conjectured that these two variables have a linear correlation. We plot 2D scatters, finding that $2r$ upper bounds $g_{r}$ , which is also a well-known relation [47]. It is interesting that KANs’ unsupervised mode can rediscover several known mathematical relations. The good news is that the results discovered by KANs are probably reliable; the bad news is that we have not discovered anything new yet. It is worth noting that we have chosen a shallow KAN for simple visualization, but deeper KANs can probably find more relations if they exist. We would like to investigate how to discover more complicated relations with deeper KANs in future work. 4.4 Application to Physics: Anderson localization Anderson localization is the fundamental phenomenon in which disorder in a quantum system leads to the localization of electronic wave functions, causing all transport to be ceased [48]. In one and two dimensions, scaling arguments show that all electronic eigenstates are exponentially localized for an infinitesimal amount of random disorder [49, 50]. In contrast, in three dimensions, a critical energy forms a phase boundary that separates the extended states from the localized states, known as a mobility edge. The understanding of these mobility edges is crucial for explaining various fundamental phenomena such as the metal-insulator transition in solids [51], as well as localization effects of light in photonic devices [52, 53, 54, 55, 56]. It is therefore necessary to develop microscopic models that exhibit mobility edges to enable detailed investigations. Developing such models is often more practical in lower dimensions, where introducing quasiperiodicity instead of random disorder can also result in mobility edges that separate localized and extended phases. Furthermore, experimental realizations of analytical mobility edges can help resolve the debate on localization in interacting systems [57, 58]. Indeed, several recent studies have focused on identifying such models and deriving exact analytic expressions for their mobility edges [59, 60, 61, 62, 63, 64, 65]. Here, we apply KANs to numerical data generated from quasiperiodic tight-binding models to extract their mobility edges. In particular, we examine three classes of models: the Mosaic model (MM) [63], the generalized Aubry-André model (GAAM) [62] and the modified Aubry-André model (MAAM) [60]. For the MM, we testify KAN’s ability to accurately extract mobility edge as a 1D function of energy. For the GAAM, we find that the formula obtained from a KAN closely matches the ground truth. For the more complicated MAAM, we demonstrate yet another example of the symbolic interpretability of this framework. A user can simplify the complex expression obtained from KANs (and corresponding symbolic formulas) by means of a “collaboration” where the human generates hypotheses to obtain a better match (e.g., making an assumption of the form of certain activation function), after which KANs can carry out quick hypotheses testing. To quantify the localization of states in these models, the inverse participation ratio (IPR) is commonly used. The IPR for the $k^{th}$ eigenstate, $\psi^{(k)}$ , is given by $$ \displaystyle\text{IPR}_{k}=\frac{\sum_{n}|\psi^{(k)}_{n}|^{4}}{\left(\sum_{n}% |\psi^{(k)}_{n}|^{2}\right)^{2}} $$ where the sum runs over the site index. Here, we use the related measure of localization – the fractal dimension of the states, given by $$ \displaystyle D_{k}=-\frac{\log(\text{IPR}_{k})}{\log(N)} $$ where $N$ is the system size. $D_{k}=0(1)$ indicates localized (extended) states. Mosaic Model (MM) We first consider a class of tight-binding models defined by the Hamiltonian [63] $$ \displaystyle H=t\sum_{n}\left(c^{\dagger}_{n+1}c_{n}+\text{H.c.}\right)+\sum_% {n}V_{n}(\lambda,\phi)c^{\dagger}_{n}c_{n}, $$ where $t$ is the nearest-neighbor coupling, $c_{n}(c^{\dagger}_{n})$ is the annihilation (creation) operator at site $n$ and the potential energy $V_{n}$ is given by $$ \displaystyle V_{n}(\lambda,\phi)=\begin{cases}\lambda\cos(2\pi nb+\phi)&j=m% \kappa\\ 0,&\text{otherwise,}\end{cases} $$ To introduce quasiperiodicity, we set $b$ to be irrational (in particular, we choose $b$ to be the golden ratio $\frac{1+\sqrt{5}}{2}$ ). $\kappa$ is an integer and the quasiperiodic potential occurs with interval $\kappa$ . The energy ( $E$ ) spectrum for this model generically contains extended and localized regimes separated by a mobility edge. Interestingly, a unique feature found here is that the mobility edges are present for an arbitrarily strong quasiperiodic potential (i.e. there are always extended states present in the system that co-exist with localized ones). The mobility edge can be described by $g(\lambda,E)\equiv\lambda-|f_{\kappa}(E)|=0$ . $g(\lambda,E)$ > 0 and $g(\lambda,E)$ < 0 correspond to localized and extended phases, respectively. Learning the mobility edge therefore hinges on learning the “order parameter” $g(\lambda,E)$ . Admittedly, this problem can be tackled by many other theoretical methods for this class of models [63], but we will demonstrate below that our KAN framework is ready and convenient to take in assumptions and inductive biases from human users. Let us assume a hypothetical user Alice, who is a new PhD student in condensed matter physics, and she is provided with a $[2,1]$ KAN as an assistant for the task. Firstly, she understands that this is a classification task, so it is wise to set the activation function in the second layer to be sigmoid by using the fix_symbolic functionality. Secondly, she realizes that learning the whole 2D function $g(\lambda,E)$ is unnecessary because in the end she only cares about $\lambda=\lambda(E)$ determined by $g(\lambda,E)=0$ . In so doing, it is reasonable to assume $g(\lambda,E)=\lambda-h(E)=0$ . Alice simply sets the activation function of $\lambda$ to be linear by again using the fix_symbolic functionality. Now Alice trains the KAN network and conveniently obtains the mobility edge, as shown in Figure 4.5. Alice can get both intuitive qualitative understanding (bottom) and quantitative results (middle), which well match the ground truth (top). <details> <summary>2404.19756v5/extracted/6190555/figs/mosaic_results.png Details</summary> ![ceaec8613f91d251f051b3a064b84c18229302f30e18d71aecb14993a0cddb65](http://localhost:8000/v1/image/ceaec8613f91d251f051b3a064b84c18229302f30e18d71aecb14993a0cddb65) ### Visual Description # Technical Document Extraction ## Overview The image contains four comparative analysis sections for κ values 2–5, each comprising: 1. Heatmap visualization 2. Line plot comparison 3. Component diagram 4. Color scale legend --- ## Heatmap Analysis (Top Row) ### General Structure - **Axes**: - X-axis: λ (0–5) - Y-axis: E (-10–10) - **Color Scale**: - Right-side gradient from purple (0.2) to yellow (1.0) - Represents "fractal dimension" - **Key Features**: - Red dashed lines demarcate regions where fractal dimension > 0.5 - Vertical yellow bands indicate transition zones ### κ-Specific Observations | κ Value | Color Intensity Pattern | Red Line Position | |---------|-------------------------|-------------------| | 2 | Broad yellow band | Near λ=0 | | 3 | Narrower yellow band | λ=0.5–1.5 | | 4 | Concentrated yellow | λ=1–2 | | 5 | Thin yellow strip | λ=1.5–2.5 | --- ## Line Plot Analysis (Middle Row) ### General Structure - **Axes**: - X-axis: λ (0–5) - Y-axis: E (-2–2) - **Legend Elements**: - Solid blue: KAN function - Dashed black: True λ function - **Equations**: - κ=2: - KAN: `1/(1+|E|)` - True λ: `1/|E|` - κ=3: - KAN: `1/(1+|E|^(1/2))` - True λ: `1/|E|^(1/2)` - κ=4: - KAN: `1/(1+|E|^(1/2)+√2)` - True λ: `1/|E|^(1/2)` - κ=5: - KAN: `1/(1+|E|+φ(|E|-1))` - True λ: `1/|E|^(1/2)` ### Trend Verification - All KAN curves asymptotically approach True λ as λ increases - κ=2 shows largest divergence at λ=0 - κ=5 demonstrates closest convergence near λ=2 --- ## Component Diagrams (Bottom Row) ### General Structure - **Elements**: - Red boxes: Input/output relationships - Black boxes: Distribution functions - Arrows: Functional dependencies - **Labels**: - λ (input) - E (output) - "Fractal dimension > 0.5" annotation ### κ-Specific Variations | κ Value | Black Box Contents | Red Box Connections | |---------|--------------------|---------------------| | 2 | Single peak | Direct λ→E | | 3 | Double peak | λ→E with intermediate | | 4 | Triple peak | Branching pathways | | 5 | Quadruple peak | Complex interdependencies | --- ## Color Scale Legend - **Position**: Right of each heatmap - **Gradient**: - Purple (0.2) → Yellow (1.0) - **Key**: "fractal dimension" --- ## Spatial Grounding 1. **Heatmaps**: Top row, left-to-right κ=2→5 2. **Line Plots**: Middle row, aligned with heatmaps 3. **Diagrams**: Bottom row, corresponding to upper plots 4. **Legends**: Right-aligned with respective plots --- ## Data Table Reconstruction | κ | KAN Equation | True λ Equation | Fractal Threshold | |---|--------------|-----------------|-------------------| | 2 | 1/(1+|E|) | 1/|E| | >0.5 | | 3 | 1/(1+|E|^½) | 1/|E|^½ | >0.5 | | 4 | 1/(1+|E|^½+√2)| 1/|E|^½ | >0.5 | | 5 | 1/(1+|E|+φ(|E|-1)) | 1/|E|^½ | >0.5 | --- ## Critical Observations 1. Increasing κ correlates with: - Narrower high-fractal-dimension regions - Closer convergence between KAN and True λ 2. Red dashed lines in heatmaps consistently mark: - λ=0 for κ=2 - λ=0.5–1.5 for κ=3 - λ=1–2 for κ=4 - λ=1.5–2.5 for κ=5 3. Component diagrams show increasing complexity with κ: - κ=2: Simple linear relationship - κ=5: Multi-path interdependencies --- ## Language Note All text appears in English. No non-English content detected. </details> Figure 4.5: Results for the Mosaic Model. Top: phase diagram. Middle and Bottom: KANs can obtain both qualitative intuition (bottom) and extract quantitative results (middle). $\varphi=\frac{1+\sqrt{5}}{2}$ is the golden ratio. Generalized Andre-Aubry Model (GAAM) We next consider a class of tight-binding models defined by the Hamiltonian [62] $$ \displaystyle H=t\sum_{n}\left(c^{\dagger}_{n+1}c_{n}+\text{H.c.}\right)+\sum_% {n}V_{n}(\alpha,\lambda,\phi)c^{\dagger}_{n}c_{n}, $$ where $t$ is the nearest-neighbor coupling, $c_{n}(c^{\dagger}_{n})$ is the annihilation (creation) operator at site $n$ and the potential energy $V_{n}$ is given by $$ \displaystyle V_{n}(\alpha,\lambda,\phi)=2\lambda\frac{\cos(2\pi nb+\phi)}{1-% \alpha\cos(2\pi nb+\phi)}, $$ which is smooth for $\alpha∈(-1,1)$ . To introduce quasiperiodicity, we again set $b$ to be irrational (in particular, we choose $b$ to be the golden ratio). As before, we would like to obtain an expression for the mobility edge. For these models, the mobility edge is given by the closed form expression [62, 64], $$ \displaystyle\alpha E=2(t-\lambda). $$ We randomly sample the model parameters: $\phi$ , $\alpha$ and $\lambda$ (setting the energy scale $t=1$ ) and calculate the energy eigenvalues as well as the fractal dimension of the corresponding eigenstates, which forms our training dataset. | System | Origin | Mobility Edge Formula | Accuracy | | --- | --- | --- | --- | | GAAM | Theory | $\alpha E+2\lambda-2=0$ | 99.2% | | KAN auto | $\cancel{1.52E^{2}}+21.06\alpha E+\cancel{0.66E}+\cancel{3.55\alpha^{2}}+% \cancel{0.91\alpha}+45.13\lambda-54.45=0$ | 99.0% | | | MAAM | Theory | $E+{\rm exp}(p)-\lambda{\rm cosh}p=0$ | 98.6% | | KAN auto | $13.99{\rm sin}(0.28{\rm sin}(0.87\lambda+2.22)-0.84{\rm arctan}(0.58E-0.26)+0.% 85{\rm arctan}(0.94p+0.13)-8.14)-16.74+43.08{\rm exp}(-0.93(0.06(0.13-p)^{2}-0% .27{\rm tanh}(0.65E+0.25)+0.63{\rm arctan}(0.54\lambda-0.62)+1)^{2})=0$ | 97.1% | | | KAN man (step 2) + auto | $4.19(0.28{\rm sin}(0.97\lambda+2.17)-0.77{\rm arctan}(0.83E-0.19)+{\rm arctan}% (0.97p+0.15)-0.35)^{2}-28.93+39.27{\rm exp}(-0.6(0.28{\rm cosh}^{2}(0.49p-0.16% )-0.34{\rm arctan}(0.65E+0.51)+0.83{\rm arctan}(0.54\lambda-0.62)+1)^{2})=0$ | 97.7% | | | KAN man (step 3) + auto | $-4.63E-10.25(-0.94{\rm sin}(0.97\lambda-6.81)+{\rm tanh}(0.8p-0.45)+0.09)^{2}+% 11.78{\rm sin}(0.76p-1.41)+22.49{\rm arctan}(1.08\lambda-1.32)+31.72=0$ | 97.7% | | | KAN man (step 4A) | $6.92E-6.23(-0.92\lambda-1)^{2}+2572.45(-0.05\lambda+0.95{\rm cosh}(0.11p+0.4)-% 1)^{2}-12.96{\rm cosh}^{2}(0.53p+0.16)+19.89=0$ | 96.6% | | | KAN man (step 4B) | $7.25E-8.81(-0.83\lambda-1)^{2}-4.08(-p-0.04)^{2}+12.71(-0.71\lambda+(0.3p+1)^{% 2}-0.86)^{2}+10.29=0$ | 95.4% | | Table 5: Symbolic formulas for two systems GAAM and MAAM, ground truth ones and KAN-discovered ones. Here the “order parameter” to be learned is $g(\alpha,E,\lambda,\phi)=\alpha E+2(\lambda-1)$ and mobility edge corresponds to $g=0$ . Let us again assume that Alice wants to figure out the mobility edge but only has access to IPR or fractal dimension data, so she decides to use KAN to help her with the task. Alice wants the model to be as small as possible, so she could either start from a large model and use auto-pruning to get a small model, or she could guess a reasonable small model based on her understanding of the complexity of the given problem. Either way, let us assume she arrives at a $[4,2,1,1]$ KAN. First, she sets the last activation to be sigmoid because this is a classification problem. She trains her KAN with some sparsity regularization to accuracy 98.7% and visualizes the trained KAN in Figure 4.6 (a) step 1. She observes that $\phi$ is not picked up on at all, which makes her realize that the mobility edge is independent of $\phi$ (agreeing with Eq. (4.8)). In addition, she observes that almost all other activation functions are linear or quadratic, so she turns on automatic symbolic snapping, constraining the library to be only linear or quadratic. After that, she immediately gets a network which is already symbolic (shown in Figure 4.6 (a) step 2), with comparable (even slightly better) accuracy 98.9%. By using symbolic_formula functionality, Alice conveniently gets the symbolic form of $g$ , shown in Table 5 GAAM-KAN auto (row three). Perhaps she wants to cross out some small terms and snap coefficient to small integers, which takes her close to the true answer. This hypothetical story for Alice would be completely different if she is using a symbolic regression method. If she is lucky, SR can return the exact correct formula. However, the vast majority of the time SR does not return useful results and it is impossible for Alice to “debug” or interact with the underlying process of symbolic regression. Furthermore, Alice may feel uncomfortable/inexperienced to provide a library of symbolic terms as prior knowledge to SR before SR is run. By constrast in KANs, Alice does not need to put any prior information to KANs. She can first get some clues by staring at a trained KAN and only then it is her job to decide which hypothesis she wants to make (e.g., “all activations are linear or quadratic”) and implement her hypothesis in KANs. Although it is not likely for KANs to return the correct answer immediately, KANs will always return something useful, and Alice can collaborate with it to refine the results. Modified Andre-Aubry Model (MAAM) The last class of models we consider is defined by the Hamiltonian [60] $$ \displaystyle H=\sum_{n\neq n^{\prime}}te^{-p|n-n^{\prime}|}\left(c^{\dagger}_% {n}c_{n^{\prime}}+\text{H.c.}\right)+\sum_{n}V_{n}(\lambda,\phi)c^{\dagger}_{n% }c_{n}, $$ where $t$ is the strength of the exponentially decaying coupling in space, $c_{n}(c^{\dagger}_{n})$ is the annihilation (creation) operator at site $n$ and the potential energy $V_{n}$ is given by $$ \displaystyle V_{n}(\lambda,\phi)=\lambda\cos(2\pi nb+\phi), $$ As before, to introduce quasiperiodicity, we set $b$ to be irrational (the golden ratio). For these models, the mobility edge is given by the closed form expression [60], $$ \displaystyle\lambda\cosh(p)=E+t=E+t_{1}{\rm exp}(p) $$ where we define $t_{1}\equiv t{\rm exp}(-p)$ as the nearest neighbor hopping strength, and we set $t_{1}=1$ below. Let us assume Alice wants to figure out the mobility edge for MAAM. This task is more complicated and requires more human wisdom. As in the last example, Alice starts from a $[4,2,1,1]$ KAN and trains it but gets an accuracy around 75% which is less than acceptable. She then chooses a larger $[4,3,1,1]$ KAN and successfully gets 98.4% which is acceptable (Figure 4.6 (b) step 1). Alice notices that $\phi$ is not picked up on by KANs, which means that the mobility edge is independent of the phase factor $\phi$ (agreeing with Eq. (4.11)). If Alice turns on the automatic symbolic regression (using a large library consisting of exp, tanh etc.), she would get a complicated formula in Tabel 5 -MAAM-KAN auto, which has 97.1% accuracy. However, if Alice wants to find a simpler symbolic formula, she will want to use the manual mode where she does the symbolic snapping by herself. Before that she finds that the $[4,3,1,1]$ KAN after training can then be pruned to be $[4,2,1,1]$ , while maintaining $97.7\%$ accuracy (Figure 4.6 (b)). Alice may think that all activation functions except those dependent on $p$ are linear or quadratic and snap them to be either linear or quadratic manually by using fix_symbolic. After snapping and retraining, the updated KAN is shown in Figure 4.6 (c) step 3, maintaining $97.7\%$ accuracy. From now on, Alice may make two different choices based on her prior knowledge. In one case, Alice may have guessed that the dependence on $p$ is ${\rm cosh}$ , so she sets the activations of $p$ to be ${\rm cosh}$ function. She retrains KAN and gets 96.9% accuracy (Figure 4.6 (c) Step 4A). In another case, Alice does not know the ${\rm cosh}\ p$ dependence, so she pursues simplicity and again assumes the functions of $p$ to be quadratic. She retrains KAN and gets 95.4% accuracy (Figure 4.6 (c) Step 4B). If she tried both, she would realize that ${\rm cosh}$ is better in terms of accuracy, while quadratic is better in terms of simplicity. The formulas corresponding to these steps are listed in Table 5. It is clear that the more manual operations are done by Alice, the simpler the symbolic formula is (which slight sacrifice in accuracy). KANs have a “knob" that a user can tune to trade-off between simplicity and accuracy (sometimes simplicity can even lead to better accuracy, as in the GAAM case). <details> <summary>2404.19756v5/extracted/6190555/figs/al_complex.png Details</summary> ![e23e257916fe067adaecd988401213cdecf5ec9bcad761da1e2d86f1847442fc](http://localhost:8000/v1/image/e23e257916fe067adaecd988401213cdecf5ec9bcad761da1e2d86f1847442fc) ### Visual Description # Technical Document Extraction: Fractal Dimension Analysis and Symbolic Regression Models ## Section (a): GAAM, Automatic Mode ### Step 1: Training - **Diagram Components**: - **Nodes**: Labeled `α`, `λ`, `φ`, `E` (Greek letters and variables). - **Connections**: - Red lines (Fractal Dimension > 0.5) connect: - Top node to middle nodes (`α`, `λ`). - Middle nodes to bottom nodes (`φ`, `E`). - Black lines connect remaining nodes. - **Accuracy**: **98.7%** (bottom of diagram). - **Legend**: - Red = Fractal Dimension > 0.5 (top-right corner). ### Step 2: Automatic Symbolic Regression - **Diagram Components**: - **Nodes**: Same labels (`α`, `λ`, `φ`, `E`). - **Connections**: - Red lines (Fractal Dimension > 0.5) form a denser network: - Top node connects to all middle nodes. - Middle nodes interconnect and connect to bottom nodes. - **Accuracy**: **98.9%** (bottom of diagram). - **Legend**: Same as Step 1. --- ## Section (b): MAAM, Automatic Mode ### Step 1: Training - **Diagram Components**: - **Nodes**: Labeled `λ`, `ρ`, `φ`, `E`. - **Connections**: - Red lines (Fractal Dimension > 0.5) connect: - Top node to middle nodes (`λ`, `ρ`). - Middle nodes to bottom nodes (`φ`, `E`). - **Accuracy**: **98.4%** (bottom of diagram). - **Legend**: Same as Section (a). ### Step 2: Automatic Symbolic Regression - **Diagram Components**: - **Nodes**: Same labels (`λ`, `ρ`, `φ`, `E`). - **Connections**: - Red lines (Fractal Dimension > 0.5) form a dense network: - Top node connects to all middle nodes. - Middle nodes interconnect and connect to bottom nodes. - **Accuracy**: **97.1%** (bottom of diagram). - **Legend**: Same as Section (a). --- ## Section (c): MAAM, Manual Mode ### Step 1: Training - **Diagram Components**: - **Nodes**: Labeled `λ`, `ρ`, `φ`, `E`. - **Connections**: - Red lines (Fractal Dimension > 0.5) connect: - Top node to middle nodes (`λ`, `ρ`). - Middle nodes to bottom nodes (`φ`, `E`). - **Accuracy**: **98.4%** (bottom of diagram). - **Legend**: Same as Section (a). ### Step 2: Pruning - **Diagram Components**: - **Nodes**: Same labels (`λ`, `ρ`, `φ`, `E`). - **Connections**: - Red lines (Fractal Dimension > 0.5) reduced: - Top node connects to middle nodes. - Middle nodes connect to bottom nodes with fewer interconnections. - **Accuracy**: **97.7%** (bottom of diagram). - **Legend**: Same as Section (a). ### Step 3: Fixing Symbolic and Training - **Diagram Components**: - **Nodes**: Same labels (`λ`, `ρ`, `φ`, `E`). - **Connections**: - Red lines (Fractal Dimension > 0.5) form a simplified network: - Top node connects to middle nodes. - Middle nodes connect to bottom nodes with minimal interconnections. - **Accuracy**: **97.7%** (bottom of diagram). - **Legend**: Same as Section (a). ### Step 4A: Fixing Symbolic and Training (snap p to cosh) - **Diagram Components**: - **Nodes**: Same labels (`λ`, `ρ`, `φ`, `E`). - **Connections**: - Red lines (Fractal Dimension > 0.5) form a sparse network: - Top node connects to middle nodes. - Middle nodes connect to bottom nodes with no interconnections. - **Accuracy**: **96.6%** (bottom of diagram). - **Legend**: Same as Section (a). ### Step 4B: Fixing Symbolic and Training (snap p to quadratic) - **Diagram Components**: - **Nodes**: Same labels (`λ`, `ρ`, `φ`, `E`). - **Connections**: - Red lines (Fractal Dimension > 0.5) form a fragmented network: - Top node connects to middle nodes. - Middle nodes connect to bottom nodes with partial interconnections. - **Accuracy**: **95.4%** (bottom of diagram). - **Legend**: Same as Section (a). --- ## Key Observations 1. **Legend Consistency**: - Red lines in all diagrams correspond to "Fractal Dimension > 0.5" (top-right legend). - No other colors are defined in the legend; black lines are unclassified. 2. **Accuracy Trends**: - **GAAM Automatic Mode**: Highest accuracy (98.7–98.9%) with dense red connections. - **MAAM Automatic Mode**: Slightly lower accuracy (98.4–97.1%) with similar dense networks. - **MAAM Manual Mode**: Accuracy decreases (98.4% → 95.4%) as steps progress from training to symbolic fixes. 3. **Diagram Complexity**: - Red lines (Fractal Dimension > 0.5) correlate with higher accuracy. - Manual mode steps show reduced connectivity and lower accuracy, suggesting pruning/symbolic fixes degrade performance. 4. **Spatial Grounding**: - All accuracies are positioned at the bottom of their respective diagrams. - Legend is consistently placed in the top-right corner across all sections. --- ## Notes - No other languages or data tables are present. - All textual information is in English. - Diagrams use geometric shapes (squares, circles) to represent nodes and connections. </details> Figure 4.6: Human-KAN collaboration to discover mobility edges of GAAM and MAAM. The human user can choose to be lazy (using the auto mode) or more involved (using the manual mode). More details in text. 5 Related works Kolmogorov-Arnold theorem and neural networks. The connection between the Kolmogorov-Arnold theorem (KAT) and neural networks is not new in the literature [66, 67, 9, 10, 11, 12, 13, 14, 68, 69], but the pathological behavior of inner functions makes KAT appear unpromising in practice [66]. Most of these prior works stick to the original 2-layer width-( $2n+1$ ) networks, which were limited in expressive power and many of them are even predating back-propagation. Therefore, most studies were built on theories with rather limited or artificial toy experiments. More broadly speaking, KANs are also somewhat related to generalized additive models (GAMs) [70], graph neural networks [71] and kernel machines [72]. The connections are intriguing and fundamental but might be out of the scope of the current paper. Our contribution lies in generalizing the Kolmogorov network to arbitrary widths and depths, revitalizing and contexualizing them in today’s deep learning stream, as well as highlighting its potential role as a foundation model for AI + Science. Neural Scaling Laws (NSLs). NSLs are the phenomena where test losses behave as power laws against model size, data, compute etc [73, 74, 75, 76, 24, 77, 78, 79]. The origin of NSLs still remains mysterious, but competitive theories include intrinsic dimensionality [73], quantization of tasks [78], resource theory [79], random features [77], compositional sparsity [66], and maximu arity [25]. This paper contributes to this space by showing that a high-dimensional function can surprisingly scale as a 1D function (which is the best possible bound one can hope for) if it has a smooth Kolmogorov-Arnold representation. Our paper brings fresh optimism to neural scaling laws, since it promises the fastest scaling exponent ever. We have shown in our experiments that this fast neural scaling law can be achieved on synthetic datasets, but future research is required to address the question whether this fast scaling is achievable for more complicated tasks (e.g., language modeling): Do KA representations exist for general tasks? If so, does our training find these representations in practice? Mechanistic Interpretability (MI). MI is an emerging field that aims to mechanistically understand the inner workings of neural networks [80, 81, 82, 83, 84, 85, 86, 87, 5]. MI research can be roughly divided into passive and active MI research. Most MI research is passive in focusing on understanding existing neural networks trained with standard methods. Active MI research attempts to achieve interpretability by designing intrinsically interpretable architectures or developing training methods to explicitly encourage interpretability [86, 87]. Our work lies in the second category, where the model and training method are by design interpretable. Learnable activations. The idea of learnable activations in neural networks is not new in machine learning. Trainable activations functions are learned in a differentiable way [88, 14, 89, 90] or searched in a discrete way [91]. Activation function are parametrized as polynomials [88], splines [14, 92, 93], sigmoid linear unit [89], or neural networks [90]. KANs use B-splines to parametrize their activation functions. We also present our preliminary results on learnable activation networks (LANs), whose properties lie between KANs and MLPs and their results are deferred to Appendix B to focus on KANs in the main paper. Symbolic Regression. There are many off-the-shelf symbolic regression methods based on genetic algorithms (Eureka [94], GPLearn [95], PySR [96]), neural-network based methods (EQL [97], OccamNet [98]), physics-inspired method (AI Feynman [36, 37]), and reinforcement learning-based methods [99]. KANs are most similar to neural network-based methods, but differ from previous works in that our activation functions are continuously learned before symbolic snapping rather than manually fixed [94, 98]. Physics-Informed Neural Networks (PINNs) and Physics-Informed Neural Operators (PINOs). In Subsection 3.4, we demonstrate that KANs can replace the paradigm of using MLPs for imposing PDE loss when solving PDEs. We refer to Deep Ritz Method [100], PINNs [38, 39, 101] for PDE solving, and Fourier Neural operator [102], PINOs [103, 104, 105], DeepONet [106] for operator learning methods learning the solution map. There is potential to replace MLPs with KANs in all the aforementioned networks. AI for Mathematics. As we saw in Subsection 4.3, AI has recently been applied to several problems in Knot theory, including detecting whether a knot is the unknot [107, 108] or a ribbon knot [46], and predicting knot invariants and uncovering relations among them [109, 110, 111, 45]. For a summary of data science applications to datasets in mathematics and theoretical physics see e.g. [112, 113], and for ideas how to obtain rigorous results from ML techniques in these fields, see [114]. 6 Discussion In this section, we discuss KANs’ limitations and future directions from the perspective of mathematical foundation, algorithms and applications. Mathematical aspects: Although we have presented preliminary mathematical analysis of KANs (Theorem 2.1), our mathematical understanding of them is still very limited. The Kolmogorov-Arnold representation theorem has been studied thoroughly in mathematics, but the theorem corresponds to KANs with shape $[n,2n+1,1]$ , which is a very restricted subclass of KANs. Does our empirical success with deeper KANs imply something fundamental in mathematics? An appealing generalized Kolmogorov-Arnold theorem could define “deeper” Kolmogorov-Arnold representations beyond depth-2 compositions, and potentially relate smoothness of activation functions to depth. Hypothetically, there exist functions which cannot be represented smoothly in the original (depth-2) Kolmogorov-Arnold representations, but might be smoothly represented with depth-3 or beyond. Can we use this notion of “Kolmogorov-Arnold depth” to characterize function classes? Algorithmic aspects: We discuss the following: 1. Accuracy. Multiple choices in architecture design and training are not fully investigated so alternatives can potentially further improve accuracy. For example, spline activation functions might be replaced by radial basis functions or other local kernels. Adaptive grid strategies can be used. 1. Efficiency. One major reason why KANs run slowly is because different activation functions cannot leverage batch computation (large data through the same function). Actually, one can interpolate between activation functions being all the same (MLPs) and all different (KANs), by grouping activation functions into multiple groups (“multi-head”), where members within a group share the same activation function. 1. Hybrid of KANs and MLPs. KANs have two major differences compared to MLPs: 1. activation functions are on edges instead of on nodes, 1. activation functions are learnable instead of fixed. Which change is more essential to explain KAN’s advantage? We present our preliminary results in Appendix B where we study a model which has (ii), i.e., activation functions are learnable (like KANs), but not (i), i.e., activation functions are on nodes (like MLPs). Moreover, one can also construct another model with fixed activations (like MLPs) but on edges (like KANs). 1. Adaptivity. Thanks to the intrinsic locality of spline basis functions, we can introduce adaptivity in the design and training of KANs to enhance both accuracy and efficiency: see the idea of multi-level training like multigrid methods as in [115, 116], or domain-dependent basis functions like multiscale methods as in [117]. Application aspects: We have presented some preliminary evidences that KANs are more effective than MLPs in science-related tasks, e.g., fitting physical equations and PDE solving. We would like to apply KANs to solve Navier-Stokes equations, density functional theory, or any other tasks that can be formulated as regression or PDE solving. We would also like to apply KANs to machine-learning-related tasks, which would require integrating KANs into current architectures, e.g., transformers – one may propose “kansformers” which replace MLPs by KANs in transformers. KAN as a “language model” for AI + Science The reason why large language models are so transformative is because they are useful to anyone who can speak natural language. The language of science is functions. KANs are composed of interpretable functions, so when a human user stares at a KAN, it is like communicating with it using the language of functions. This paragraph aims to promote the AI-Scientist-Collaboration paradigm rather than our specific tool KANs. Just like people use different languages to communicate, we expect that in the future KANs will be just one of the languages for AI + Science, although KANs will be one of the very first languages that would enable AI and human to communicate. However, enabled by KANs, the AI-Scientist-Collaboration paradigm has never been this easy and convenient, which leads us to rethink the paradigm of how we want to approach AI + Science: Do we want AI scientists, or do we want AI that helps scientists? The intrinsic difficulty of (fully automated) AI scientists is that it is hard to make human preferences quantitative, which would codify human preferences into AI objectives. In fact, scientists in different fields may feel differently about which functions are simple or interpretable. As a result, it is more desirable for scientists to have an AI that can speak the scientific language (functions) and can conveniently interact with inductive biases of individual scientist(s) to adapt to a specific scientific domain. Final takeaway: Should I use KANs or MLPs? Currently, the biggest bottleneck of KANs lies in its slow training. KANs are usually 10x slower than MLPs, given the same number of parameters. We should be honest that we did not try hard to optimize KANs’ efficiency though, so we deem KANs’ slow training more as an engineering problem to be improved in the future rather than a fundamental limitation. If one wants to train a model fast, one should use MLPs. In other cases, however, KANs should be comparable or better than MLPs, which makes them worth trying. The decision tree in Figure 6.1 can help decide when to use a KAN. In short, if you care about interpretability and/or accuracy, and slow training is not a major concern, we suggest trying KANs, at least for small-scale AI + Science problems. <details> <summary>2404.19756v5/extracted/6190555/figs/decision_tree.png Details</summary> ![e1f7d6aafc24ec2f858a49ded81cf968e9549411248ce57d7a717eb8793e5ce1](http://localhost:8000/v1/image/e1f7d6aafc24ec2f858a49ded81cf968e9549411248ce57d7a717eb8793e5ce1) ### Visual Description # Technical Document: Decision Flowchart for KANs vs. MLPs ## Overview The image presents a decision flowchart titled **"Should I use KANs or MLPs?"** designed to guide users in selecting between Kernel Approximate Neural Networks (KANs) and Multilayer Perceptrons (MLPs) based on three criteria: **Accuracy**, **Interpretability**, and **Efficiency**. The flowchart uses a tree-like structure with binary decision nodes (diamonds) and terminal outcomes (rectangles). --- ## Key Components ### 1. **Main Title** - **"Should I use KANs or MLPs?"** - Positioned at the top center, serving as the root question for the flowchart. ### 2. **Primary Criteria** Three high-level decision branches: 1. **Accuracy** 2. **Interpretability** 3. **Efficiency** Each branch splits into sub-questions and outcomes. --- ## Detailed Flowchart Analysis ### **Accuracy Branch** #### Sub-Questions: 1. **Compositional structure** - **Yes**: KAN - **No**: Either is fine 2. **Complicated function** - **Yes**: KAN - **No**: Either is fine 3. **Continual learning** - **Yes**: KAN - **No**: Either is fine --- ### **Interpretability Branch** #### Sub-Questions: 1. **Dimension** - **High**: Either is hard - **Low**: KAN 2. **Level** - **Quantitative**: KAN - **Qualitative**: Either is fine --- ### **Efficiency Branch** #### Sub-Questions: 1. **Want small models** - **Yes**: KAN - **No**: Either is fine 2. **Want fast training** - **Yes**: MLP - **No**: Either is fine --- ## Terminal Outcomes The flowchart concludes with three possible outcomes: 1. **KAN** (Kernel Approximate Neural Network) 2. **MLP** (Multilayer Perceptron) 3. **Either is fine** (No strong preference based on criteria) --- ## Visual Elements - **Neural Network Diagrams**: - Two schematic diagrams at the top right: - Left: Likely represents a KAN (structured as a graph with interconnected nodes). - Right: Likely represents an MLP (structured as a layered network with colored edges). - No explicit labels for these diagrams in the image. - **Icons**: - A seated figure with a question mark (`?`) at the top left, symbolizing decision-making uncertainty. --- ## Key Trends and Data Points - **KANs** are recommended when: - The problem involves compositional structure, complicated functions, or continual learning. - Interpretability requirements are low-dimensional or quantitative. - Small models or fast training are not prioritized. - **MLPs** are recommended when: - Fast training is prioritized. - Other criteria (accuracy, interpretability) do not strongly favor KANs. - **Neutral Outcomes**: - "Either is fine" appears frequently, indicating scenarios where both KANs and MLPs are viable. --- ## Notes - The flowchart assumes binary decisions at each node (Yes/No). - No numerical data or statistical trends are present; the flowchart is purely logical. - The neural network diagrams are illustrative but lack explicit labels for layers or activation functions. --- ## Conclusion This flowchart provides a structured decision-making framework for choosing between KANs and MLPs based on problem-specific requirements. Users should evaluate their priorities in **Accuracy**, **Interpretability**, and **Efficiency** to determine the optimal architecture. </details> Figure 6.1: Should I use KANs or MLPs? Acknowledgement We would like to thank Mikail Khona, Tomaso Poggio, Pingchuan Ma, Rui Wang, Di Luo, Sara Beery, Catherine Liang, Yiping Lu, Nicholas H. Nelsen, Nikola Kovachki, Jonathan W. Siegel, Hongkai Zhao, Juncai He, Shi Lab (Humphrey Shi, Steven Walton, Chuanhao Yan) and Matthieu Darcy for fruitful discussion and constructive suggestions. Z.L., F.R., J.H., M.S. and M.T. are supported by IAIFI through NSF grant PHY-2019786. The work of FR is in addition supported by the NSF grant PHY-2210333 and by startup funding from Northeastern University. Y.W and T.H are supported by the NSF Grant DMS-2205590 and the Choi Family Gift Fund. S. V. and M. S. acknowledge support from the U.S. Office of Naval Research (ONR) Multidisciplinary University Research Initiative (MURI) under Grant No. N00014-20-1-2325 on Robust Photonic Materials with Higher-Order Topological Protection. References - [1] Simon Haykin. Neural networks: a comprehensive foundation. Prentice Hall PTR, 1994. - [2] George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303–314, 1989. - [3] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989. - [4] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. - [5] Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023. - [6] A.N. Kolmogorov. On the representation of continuous functions of several variables as superpositions of continuous functions of a smaller number of variables. Dokl. Akad. Nauk, 108(2), 1956. - [7] Andrei Nikolaevich Kolmogorov. On the representation of continuous functions of many variables by superposition of continuous functions of one variable and addition. In Doklady Akademii Nauk, volume 114, pages 953–956. Russian Academy of Sciences, 1957. - [8] Jürgen Braun and Michael Griebel. On a constructive proof of kolmogorov’s superposition theorem. Constructive approximation, 30:653–675, 2009. - [9] David A Sprecher and Sorin Draghici. Space-filling curves and kolmogorov superposition-based neural networks. Neural Networks, 15(1):57–67, 2002. - [10] Mario Köppen. On the training of a kolmogorov network. In Artificial Neural Networks—ICANN 2002: International Conference Madrid, Spain, August 28–30, 2002 Proceedings 12, pages 474–479. Springer, 2002. - [11] Ji-Nan Lin and Rolf Unbehauen. On the realization of a kolmogorov network. Neural Computation, 5(1):18–20, 1993. - [12] Ming-Jun Lai and Zhaiming Shen. The kolmogorov superposition theorem can break the curse of dimensionality when approximating high dimensional functions. arXiv preprint arXiv:2112.09963, 2021. - [13] Pierre-Emmanuel Leni, Yohan D Fougerolle, and Frédéric Truchetet. The kolmogorov spline network for image processing. In Image Processing: Concepts, Methodologies, Tools, and Applications, pages 54–78. IGI Global, 2013. - [14] Daniele Fakhoury, Emanuele Fakhoury, and Hendrik Speleers. Exsplinet: An interpretable and expressive spline-based neural network. Neural Networks, 152:332–346, 2022. - [15] Hadrien Montanelli and Haizhao Yang. Error bounds for deep relu networks using the kolmogorov–arnold superposition theorem. Neural Networks, 129:1–6, 2020. - [16] Juncai He. On the optimal expressive power of relu dnns and its application in approximation with kolmogorov superposition theorem. arXiv preprint arXiv:2308.05509, 2023. - [17] Juncai He, Lin Li, Jinchao Xu, and Chunyue Zheng. Relu deep neural networks and linear finite elements. arXiv preprint arXiv:1807.03973, 2018. - [18] Juncai He and Jinchao Xu. Deep neural networks and finite elements of any order on arbitrary dimensions. arXiv preprint arXiv:2312.14276, 2023. - [19] Tomaso Poggio, Andrzej Banburski, and Qianli Liao. Theoretical issues in deep networks. Proceedings of the National Academy of Sciences, 117(48):30039–30045, 2020. - [20] Federico Girosi and Tomaso Poggio. Representation properties of networks: Kolmogorov’s theorem is irrelevant. Neural Computation, 1(4):465–469, 1989. - [21] Henry W Lin, Max Tegmark, and David Rolnick. Why does deep and cheap learning work so well? Journal of Statistical Physics, 168:1223–1247, 2017. - [22] Hongyi Xu, Funshing Sin, Yufeng Zhu, and Jernej Barbič. Nonlinear material design using principal stretches. ACM Transactions on Graphics (TOG), 34(4):1–11, 2015. - [23] Carl De Boor. A practical guide to splines, volume 27. springer-verlag New York, 1978. - [24] Utkarsh Sharma and Jared Kaplan. A neural scaling law from the dimension of the data manifold. arXiv preprint arXiv:2004.10802, 2020. - [25] Eric J Michaud, Ziming Liu, and Max Tegmark. Precision machine learning. Entropy, 25(1):175, 2023. - [26] Joel L Horowitz and Enno Mammen. Rate-optimal estimation for a general class of nonparametric regression models with unknown link functions. 2007. - [27] Michael Kohler and Sophie Langer. On the rate of convergence of fully connected deep neural network regression estimates. The Annals of Statistics, 49(4):2231–2249, 2021. - [28] Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with relu activation function. 2020. - [29] Ronald A DeVore, Ralph Howard, and Charles Micchelli. Optimal nonlinear approximation. Manuscripta mathematica, 63:469–478, 1989. - [30] Ronald A DeVore, George Kyriazis, Dany Leviatan, and Vladimir M Tikhomirov. Wavelet compression and nonlinear n-widths. Adv. Comput. Math., 1(2):197–214, 1993. - [31] Jonathan W Siegel. Sharp lower bounds on the manifold widths of sobolev and besov spaces. arXiv preprint arXiv:2402.04407, 2024. - [32] Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks, 94:103–114, 2017. - [33] Peter L Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks. Journal of Machine Learning Research, 20(63):1–17, 2019. - [34] Jonathan W Siegel. Optimal approximation rates for deep relu neural networks on sobolev and besov spaces. Journal of Machine Learning Research, 24(357):1–52, 2023. - [35] Yongji Wang and Ching-Yao Lai. Multi-stage neural networks: Function approximator of machine precision. Journal of Computational Physics, page 112865, 2024. - [36] Silviu-Marian Udrescu and Max Tegmark. Ai feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16):eaay2631, 2020. - [37] Silviu-Marian Udrescu, Andrew Tan, Jiahai Feng, Orisvaldo Neto, Tailin Wu, and Max Tegmark. Ai feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity. Advances in Neural Information Processing Systems, 33:4860–4871, 2020. - [38] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686–707, 2019. - [39] George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed machine learning. Nature Reviews Physics, 3(6):422–440, 2021. - [40] Ronald Kemker, Marc McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. - [41] Bryan Kolb and Ian Q Whishaw. Brain plasticity and behavior. Annual review of psychology, 49(1):43–64, 1998. - [42] David Meunier, Renaud Lambiotte, and Edward T Bullmore. Modular and hierarchically modular organization of brain networks. Frontiers in neuroscience, 4:7572, 2010. - [43] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017. - [44] Aojun Lu, Tao Feng, Hangjie Yuan, Xiaotian Song, and Yanan Sun. Revisiting neural networks for continual learning: An architectural perspective, 2024. - [45] Alex Davies, Petar Veličković, Lars Buesing, Sam Blackwell, Daniel Zheng, Nenad Tomašev, Richard Tanburn, Peter Battaglia, Charles Blundell, András Juhász, et al. Advancing mathematics by guiding human intuition with ai. Nature, 600(7887):70–74, 2021. - [46] Sergei Gukov, James Halverson, Ciprian Manolescu, and Fabian Ruehle. Searching for ribbons with machine learning, 2023. - [47] P. Petersen. Riemannian Geometry. Graduate Texts in Mathematics. Springer New York, 2006. - [48] Philip W Anderson. Absence of diffusion in certain random lattices. Physical review, 109(5):1492, 1958. - [49] David J Thouless. A relation between the density of states and range of localization for one dimensional random systems. Journal of Physics C: Solid State Physics, 5(1):77, 1972. - [50] Elihu Abrahams, PW Anderson, DC Licciardello, and TV Ramakrishnan. Scaling theory of localization: Absence of quantum diffusion in two dimensions. Physical Review Letters, 42(10):673, 1979. - [51] Ad Lagendijk, Bart van Tiggelen, and Diederik S Wiersma. Fifty years of anderson localization. Physics today, 62(8):24–29, 2009. - [52] Mordechai Segev, Yaron Silberberg, and Demetrios N Christodoulides. Anderson localization of light. Nature Photonics, 7(3):197–204, 2013. - [53] Z Valy Vardeny, Ajay Nahata, and Amit Agrawal. Optics of photonic quasicrystals. Nature photonics, 7(3):177–187, 2013. - [54] Sajeev John. Strong localization of photons in certain disordered dielectric superlattices. Physical review letters, 58(23):2486, 1987. - [55] Yoav Lahini, Rami Pugatch, Francesca Pozzi, Marc Sorel, Roberto Morandotti, Nir Davidson, and Yaron Silberberg. Observation of a localization transition in quasiperiodic photonic lattices. Physical review letters, 103(1):013901, 2009. - [56] Sachin Vaidya, Christina Jörg, Kyle Linn, Megan Goh, and Mikael C Rechtsman. Reentrant delocalization transition in one-dimensional photonic quasicrystals. Physical Review Research, 5(3):033170, 2023. - [57] Wojciech De Roeck, Francois Huveneers, Markus Müller, and Mauro Schiulaz. Absence of many-body mobility edges. Physical Review B, 93(1):014203, 2016. - [58] Xiaopeng Li, Sriram Ganeshan, JH Pixley, and S Das Sarma. Many-body localization and quantum nonergodicity in a model with a single-particle mobility edge. Physical review letters, 115(18):186601, 2015. - [59] Fangzhao Alex An, Karmela Padavić, Eric J Meier, Suraj Hegde, Sriram Ganeshan, JH Pixley, Smitha Vishveshwara, and Bryce Gadway. Interactions and mobility edges: Observing the generalized aubry-andré model. Physical review letters, 126(4):040603, 2021. - [60] J Biddle and S Das Sarma. Predicted mobility edges in one-dimensional incommensurate optical lattices: An exactly solvable model of anderson localization. Physical review letters, 104(7):070601, 2010. - [61] Alexander Duthie, Sthitadhi Roy, and David E Logan. Self-consistent theory of mobility edges in quasiperiodic chains. Physical Review B, 103(6):L060201, 2021. - [62] Sriram Ganeshan, JH Pixley, and S Das Sarma. Nearest neighbor tight binding models with an exact mobility edge in one dimension. Physical review letters, 114(14):146601, 2015. - [63] Yucheng Wang, Xu Xia, Long Zhang, Hepeng Yao, Shu Chen, Jiangong You, Qi Zhou, and Xiong-Jun Liu. One-dimensional quasiperiodic mosaic lattice with exact mobility edges. Physical Review Letters, 125(19):196604, 2020. - [64] Yucheng Wang, Xu Xia, Yongjian Wang, Zuohuan Zheng, and Xiong-Jun Liu. Duality between two generalized aubry-andré models with exact mobility edges. Physical Review B, 103(17):174205, 2021. - [65] Xin-Chi Zhou, Yongjian Wang, Ting-Fung Jeffrey Poon, Qi Zhou, and Xiong-Jun Liu. Exact new mobility edges between critical and localized states. Physical Review Letters, 131(17):176401, 2023. - [66] Tomaso Poggio. How deep sparse networks avoid the curse of dimensionality: Efficiently computable functions are compositionally sparse. CBMM Memo, 10:2022, 2022. - [67] Johannes Schmidt-Hieber. The kolmogorov–arnold representation theorem revisited. Neural networks, 137:119–126, 2021. - [68] Aysu Ismayilova and Vugar E Ismailov. On the kolmogorov neural networks. Neural Networks, page 106333, 2024. - [69] Michael Poluektov and Andrew Polar. A new iterative method for construction of the kolmogorov-arnold representation. arXiv preprint arXiv:2305.08194, 2023. - [70] Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, and Geoffrey E Hinton. Neural additive models: Interpretable machine learning with neural nets. Advances in neural information processing systems, 34:4699–4711, 2021. - [71] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems, 30, 2017. - [72] Huan Song, Jayaraman J Thiagarajan, Prasanna Sattigeri, and Andreas Spanias. Optimizing kernel machines using deep learning. IEEE transactions on neural networks and learning systems, 29(11):5528–5540, 2018. - [73] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. - [74] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020. - [75] Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In ACL Rolling Review - May 2021, 2021. - [76] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017. - [77] Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2102.06701, 2021. - [78] Eric J Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. - [79] Jinyeop Song, Ziming Liu, Max Tegmark, and Jeff Gore. A resource model for neural scaling law. arXiv preprint arXiv:2402.05164, 2024. - [80] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022. - [81] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359–17372, 2022. - [82] Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023. - [83] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652, 2022. - [84] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, 2023. - [85] Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. - [86] Ziming Liu, Eric Gan, and Max Tegmark. Seeing is believing: Brain-inspired modular training for mechanistic interpretability. Entropy, 26(1):41, 2023. - [87] Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer ElShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Kamal Ndousse, Andy Jones, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hatfield-Dodds, Jackson Kernion, Tom Conerly, Shauna Kravec, Stanislav Fort, Saurav Kadavath, Josh Jacobson, Eli Tran-Johnson, Jared Kaplan, Jack Clark, Tom Brown, Sam McCandlish, Dario Amodei, and Christopher Olah. Softmax linear units. Transformer Circuits Thread, 2022. https://transformer-circuits.pub/2022/solu/index.html. - [88] Mohit Goyal, Rajan Goyal, and Brejesh Lall. Learning activation functions: A new paradigm for understanding neural networks. arXiv preprint arXiv:1906.09529, 2019. - [89] Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017. - [90] Shijun Zhang, Zuowei Shen, and Haizhao Yang. Neural network architecture beyond width and depth. Advances in Neural Information Processing Systems, 35:5669–5681, 2022. - [91] Garrett Bingham and Risto Miikkulainen. Discovering parametric activation functions. Neural Networks, 148:48–65, 2022. - [92] Pakshal Bohra, Joaquim Campos, Harshit Gupta, Shayan Aziznejad, and Michael Unser. Learning activation functions in deep (spline) neural networks. IEEE Open Journal of Signal Processing, 1:295–309, 2020. - [93] Shayan Aziznejad and Michael Unser. Deep spline networks with control of lipschitz regularity. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3242–3246. IEEE, 2019. - [94] Renáta Dubcáková. Eureqa: software review. Genetic Programming and Evolvable Machines, 12:173–178, 2011. - [95] Gplearn. https://github.com/trevorstephens/gplearn. Accessed: 2024-04-19. - [96] Miles Cranmer. Interpretable machine learning for science with pysr and symbolicregression. jl. arXiv preprint arXiv:2305.01582, 2023. - [97] Georg Martius and Christoph H Lampert. Extrapolation and learning equations. arXiv preprint arXiv:1610.02995, 2016. - [98] Owen Dugan, Rumen Dangovski, Allan Costa, Samuel Kim, Pawan Goyal, Joseph Jacobson, and Marin Soljačić. Occamnet: A fast neural model for symbolic regression at scale. arXiv preprint arXiv:2007.10784, 2020. - [99] Terrell N. Mundhenk, Mikel Landajuela, Ruben Glatt, Claudio P. Santiago, Daniel faissol, and Brenden K. Petersen. Symbolic regression via deep reinforcement learning enhanced genetic programming seeding. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. - [100] Bing Yu et al. The deep ritz method: a deep learning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics, 6(1):1–12, 2018. - [101] Junwoo Cho, Seungtae Nam, Hyunmo Yang, Seok-Bae Yun, Youngjoon Hong, and Eunbyung Park. Separable physics-informed neural networks. Advances in Neural Information Processing Systems, 36, 2024. - [102] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020. - [103] Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-informed neural operator for learning partial differential equations. ACM/JMS Journal of Data Science, 2021. - [104] Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Research, 24(89):1–97, 2023. - [105] Haydn Maust, Zongyi Li, Yixuan Wang, Daniel Leibovici, Oscar Bruno, Thomas Hou, and Anima Anandkumar. Fourier continuation for exact derivative computation in physics-informed neural operators. arXiv preprint arXiv:2211.15960, 2022. - [106] Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature machine intelligence, 3(3):218–229, 2021. - [107] Sergei Gukov, James Halverson, Fabian Ruehle, and Piotr Sułkowski. Learning to Unknot. Mach. Learn. Sci. Tech., 2(2):025035, 2021. - [108] L. H. Kauffman, N. E. Russkikh, and I. A. Taimanov. Rectangular knot diagrams classification with deep learning, 2020. - [109] Mark C Hughes. A neural network approach to predicting and computing knot invariants. Journal of Knot Theory and Its Ramifications, 29(03):2050005, 2020. - [110] Jessica Craven, Vishnu Jejjala, and Arjun Kar. Disentangling a deep learned volume formula. JHEP, 06:040, 2021. - [111] Jessica Craven, Mark Hughes, Vishnu Jejjala, and Arjun Kar. Illuminating new and known relations between knot invariants. 11 2022. - [112] Fabian Ruehle. Data science applications to string theory. Phys. Rept., 839:1–117, 2020. - [113] Y.H. He. Machine Learning in Pure Mathematics and Theoretical Physics. G - Reference,Information and Interdisciplinary Subjects Series. World Scientific, 2023. - [114] Sergei Gukov, James Halverson, and Fabian Ruehle. Rigor with machine learning from field theory to the poincaréconjecture. Nature Reviews Physics, 2024. - [115] Shumao Zhang, Pengchuan Zhang, and Thomas Y Hou. Multiscale invertible generative networks for high-dimensional bayesian inference. In International Conference on Machine Learning, pages 12632–12641. PMLR, 2021. - [116] Jinchao Xu and Ludmil Zikatanov. Algebraic multigrid methods. Acta Numerica, 26:591–721, 2017. - [117] Yifan Chen, Thomas Y Hou, and Yixuan Wang. Exponentially convergent multiscale finite element method. Communications on Applied Mathematics and Computation, pages 1–17, 2023. - [118] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances in neural information processing systems, 33:7462–7473, 2020. Appendix Appendix A KAN Functionalities Table 6 includes common functionalities that users may find useful. | Functionality | Descriptions | | --- | --- | | model.train(dataset) | training model on dataset | | model.plot() | plotting | | model.prune() | pruning | | model.fix_symbolic(l,i,j,fun) | fix the activation function $\phi_{l,i,j}$ to be the symbolic function fun | | model.suggest_symbolic(l,i,j) | suggest symbolic functions that match the numerical value of $\phi_{l,i,j}$ | | model.auto_symbolic() | use top 1 symbolic suggestions from suggest_symbolic to replace all activation functions | | model.symbolic_formula() | return the symbolic formula | Table 6: KAN functionalities Appendix B Learnable activation networks (LANs) B.1 Architecture Besides KAN, we also proposed another type of learnable activation networks (LAN), which are almost MLPs but with learnable activation functions parametrized as splines. KANs have two main changes to standard MLPs: (1) the activation functions become learnable rather than being fixed; (2) the activation functions are placed on edges rather than nodes. To disentangle these two factors, we also propose learnable activation networks (LAN) which only has learnable activations but still on nodes, illustrated in Figure B.1. For a LAN with width $N$ , depth $L$ , and grid point number $G$ , the number of parameters is $N^{2}L+NLG$ where $N^{2}L$ is the number of parameters for weight matrices and $NLG$ is the number of parameters for spline activations, which causes little overhead in addition to MLP since usually $G\ll N$ so $NLG\ll N^{2}L$ . LANs are similar to MLPs so they can be initialized from pretrained MLPs and fine-tuned by allowing learnable activation functions. An example is to use LAN to improve SIREN, presented in Section B.3. Comparison of LAN and KAN. Pros of LANs: 1. LANs are conceptually simpler than KANs. They are closer to standard MLPs (the only change is that activation functions become learnable). 1. LANs scale better than KANs. LANs/KANs have learnable activation functions on nodes/edges, respectively. So activation parameters in LANs/KANs scale as $N$ / $N^{2}$ , where $N$ is model width. Cons of LANs: 1. LANs seem to be less interpretable (weight matrices are hard to interpret, just like in MLPs); 1. LANs also seem to be less accurate than KANs, but still more accurate than MLPs. Like KANs, LANs also admit grid extension if theLANs’ activation functions are parametrized by splines. B.2 LAN interpretability results <details> <summary>2404.19756v5/extracted/6190555/figs/lan_toy_interpretability_evolution.png Details</summary> ![c574ce558bc5f5869a519253fd322518a6eb910f22060f407e90aab612199544](http://localhost:8000/v1/image/c574ce558bc5f5869a519253fd322518a6eb910f22060f407e90aab612199544) ### Visual Description # Technical Diagram Analysis ## Diagram Overview The image depicts a sequence of interconnected diagrams illustrating a computational process involving variables **x** and **y** across multiple steps. Each diagram represents a stage in the process, with mathematical expressions and graphical components. --- ## Step-by-Step Breakdown ### Step 0 - **Label**: `step=0` - **Mathematical Expression**: `exp(sin(πx) + y²)` - **Components**: - **Nodes**: Two black circles labeled **x** and **y**. - **Connections**: Multiple red, blue, and gray lines connecting **x** and **y** to intermediate nodes. - **Intermediate Nodes**: 5 small squares containing diagonal lines (likely representing linear functions or placeholders). ### Step 20 - **Label**: `step=20` - **Mathematical Expression**: `exp(sin(πx) + y²)` - **Components**: - **Nodes**: Same **x** and **y** nodes as Step 0. - **Connections**: Simplified network with fewer lines (red and blue dominate). - **Intermediate Nodes**: 5 small squares with wavy lines (non-linear functions or dynamic data). ### Step 50 - **Label**: `step=50` - **Mathematical Expression**: `exp(sin(πx) + y²)` - **Components**: - **Nodes**: **x** and **y** nodes persist. - **Connections**: Further reduced complexity; red lines dominate. - **Intermediate Nodes**: 4 small squares with varying waveforms (e.g., sinusoidal, parabolic). ### Step 100 - **Label**: `step=100` - **Mathematical Expression**: `exp(sin(πx) + y²)` - **Components**: - **Nodes**: **x** and **y** nodes remain. - **Connections**: Minimal lines (red and blue only). - **Intermediate Nodes**: 3 small squares with simplified waveforms. ### Pruning After Step 100 - **Label**: `Pruning after step 100` - **Mathematical Expression**: `exp(sin(πx) + y²)` - **Components**: - **Nodes**: **x** and **y** nodes. - **Connections**: Single red line directly connecting **x** to **y**. - **Intermediate Nodes**: 2 small squares with basic waveforms (likely final output or critical data points). --- ## Key Observations 1. **Mathematical Consistency**: The expression `exp(sin(πx) + y²)` is constant across all steps, suggesting it is the core function being iteratively processed. 2. **Node Evolution**: The **x** and **y** nodes remain central throughout, acting as input/output anchors. 3. **Line Color Significance**: - **Red Lines**: Likely represent primary data flow or critical connections. - **Blue Lines**: May indicate secondary or auxiliary pathways. - **Gray Lines**: Possibly transitional or deprecated connections. 4. **Pruning Effect**: After Step 100, the diagram simplifies to a direct **x**→**y** connection, implying optimization or elimination of redundant pathways. --- ## Diagram Flow 1. **Initialization (Step 0)**: Complex network with multiple intermediate nodes. 2. **Iteration (Steps 20–100)**: Gradual reduction in complexity, with intermediate nodes evolving from linear to non-linear representations. 3. **Finalization (Pruning)**: Streamlined to a minimal structure, retaining only essential connections. --- ## Notes - No explicit legends or axis titles are present in the image. - The small squares with graphs likely represent intermediate computational results or function evaluations. - The process appears to model a dynamic system where complexity decreases over time, culminating in a simplified, optimized state. </details> Figure B.1: Training of a learnable activation network (LAN) on the toy example $f(x,y)={\rm exp}({\rm sin}(\pi x)+y^{2})$ . <details> <summary>2404.19756v5/extracted/6190555/figs/lan_interpretable_examples.png Details</summary> ![266d38d7c9feb0a90f590fba70551ad1f84d68f35cbf9d1cde3701b254c57628](http://localhost:8000/v1/image/266d38d7c9feb0a90f590fba70551ad1f84d68f35cbf9d1cde3701b254c57628) ### Visual Description # Technical Diagram Analysis ## Diagram Structure The image contains a 2x3 grid of neural network diagrams, each labeled (a) through (f). Each diagram illustrates different mathematical operations or transformations using node-based architectures. --- ### (a) Multiplication **Title:** multiplication **Equation:** $ xy $ **Components:** - **Input Nodes:** $ X $, $ Y $ (black circles) - **Hidden Layer:** Single square node with waveform pattern - **Output Node:** $ xy $ (black circle) - **Connections:** - Red: $ X \rightarrow \text{Hidden} $ - Blue: $ Y \rightarrow \text{Hidden} $ - Purple: $ \text{Hidden} \rightarrow xy $ **Legend:** - Red: Input-to-hidden connections - Blue: Input-to-hidden connections - Purple: Hidden-to-output connections --- ### (b) Division **Title:** division **Equation:** $ x/y $ **Components:** - **Input Nodes:** $ X $, $ Y $ (black circles) - **Hidden Layer:** Single square node with waveform pattern - **Output Node:** $ x/y $ (black circle) - **Connections:** - Red: $ X \rightarrow \text{Hidden} $ - Blue: $ Y \rightarrow \text{Hidden} $ - Light Blue: $ \text{Hidden} \rightarrow x/y $ **Legend:** - Red: Input-to-hidden connections - Blue: Input-to-hidden connections - Light Blue: Hidden-to-output connections --- ### (c) Numerical to Category **Title:** numerical to category **Equation:** $ \vec{e}_{\text{argmin}_i |x - x_i|} $ **Components:** - **Input Nodes:** 10 waveform nodes (black circles) - **Hidden Layer:** Single square node with waveform pattern - **Output Node:** $ X $ (black circle) - **Connections:** - Red: Input-to-hidden - Blue: Input-to-hidden - Purple: Hidden-to-output **Legend:** - Red: Input-to-hidden connections - Blue: Input-to-hidden connections - Purple: Hidden-to-output connections --- ### (d) Special Function **Title:** special function **Equation:** $ \exp(J_0(20x) + y^2) $ **Components:** - **Input Nodes:** $ X $, $ Y $ (black circles) - **Hidden Layer:** 5 waveform nodes - **Output Node:** $ xy $ (black circle) - **Connections:** - Red: $ X \rightarrow \text{Hidden} $ - Blue: $ Y \rightarrow \text{Hidden} $ - Purple: $ \text{Hidden} \rightarrow xy $ **Legend:** - Red: Input-to-hidden connections - Blue: Input-to-hidden connections - Purple: Hidden-to-output connections --- ### (e) Phase Transition **Title:** phase transition **Equation:** $ \tanh(5(x_1^4 + x_2^4 + x_3^4 - 1)) $ **Components:** - **Input Nodes:** $ X_1 $, $ X_2 $, $ X_3 $ (black circles) - **Hidden Layer:** 5 waveform nodes - **Output Node:** $ X_1, X_2, X_3 $ (black circles) - **Connections:** - Red: $ X_1 \rightarrow \text{Hidden} $ - Blue: $ X_2 \rightarrow \text{Hidden} $ - Purple: $ X_3 \rightarrow \text{Hidden} $ **Legend:** - Red: Input-to-hidden connections - Blue: Input-to-hidden connections - Purple: Hidden-to-output connections --- ### (f) Compositions **Title:** compositions **Equation:** $ \sqrt{(x_1 - x_2)^2 + (x_3 - x_4)^2} $ **Components:** - **Input Nodes:** $ X_1 $, $ X_2 $, $ X_3 $, $ X_4 $ (black circles) - **Hidden Layer:** 2 waveform nodes - **Output Node:** $ X_1, X_2, X_3, X_4 $ (black circles) - **Connections:** - Red: $ X_1 \rightarrow \text{Hidden} $ - Blue: $ X_2 \rightarrow \text{Hidden} $ - Purple: $ X_3 \rightarrow \text{Hidden} $ **Legend:** - Red: Input-to-hidden connections - Blue: Input-to-hidden connections - Purple: Hidden-to-output connections --- ## Key Observations 1. **Color Coding Consistency:** - Red: Input-to-hidden layer connections - Blue: Input-to-hidden layer connections - Purple: Hidden-to-output connections 2. **Node Types:** - Black circles: Input/output nodes - Squares: Hidden layer nodes with waveform patterns 3. **Mathematical Operations:** - Diagrams encode operations like multiplication, division, and phase transitions using neural network architectures. - Equations are explicitly labeled for each diagram. 4. **Legend Placement:** - Legends are positioned to the right of each diagram, with colors matching connection types. 5. **Spatial Grounding:** - Input nodes are consistently placed at the bottom of diagrams. - Output nodes are positioned at the top or center-right. --- ## Missing Elements - No numerical data or trends (e.g., heatmaps, line charts) are present. - No textual blocks requiring translation; all labels are in English. </details> Figure B.2: LANs on synthetic examples. LANs do not appear to be very interpretable. We conjecture that the weight matrices leave too many degree of freedoms. We present preliminary interpretabilty results of LANs in Figure B.2. With the same examples in Figure 4.1 for which KANs are perfectly interpretable, LANs seem much less interpretable due to the existence of weight matrices. First, weight matrices are less readily interpretable than learnable activation functions. Second, weight matrices bring in too many degrees of freedom, making learnable activation functions too unconstrained. Our preliminary results with LANs seem to imply that getting rid of linear weight matrices (by having learnable activations on edges, like KANs) is necessary for interpretability. B.3 Fitting Images (LAN) <details> <summary>2404.19756v5/extracted/6190555/figs/siren.png Details</summary> ![1fca4126855698573a210cbe464fd27f3bd0abf5edc4c9cd8abcb533598396f4](http://localhost:8000/v1/image/1fca4126855698573a210cbe464fd27f3bd0abf5edc4c9cd8abcb533598396f4) ### Visual Description # Technical Document Extraction: PSNR vs Training Steps ## Chart Overview The image presents a line chart comparing the performance of three models (SIREN, LAN trained from scratch, LAN fine-tuned on SIREN) across training steps, measured by Peak Signal-to-Noise Ratio (PSNR). Three reference images labeled "Ground Truth" are spatially anchored to specific points on the chart. --- ### **Axis Labels & Scales** - **X-axis**: `training steps` (0 to 20,000 in increments of 2,500) - **Y-axis**: `PSNR` (25 to 31 in increments of 1) - **Legend**: Located in the bottom-right corner, with three entries: - **Blue**: SIREN - **Orange**: LAN (train from scratch) - **Green**: LAN (fine tune SIREN) --- ### **Key Trends & Data Points** 1. **SIREN (Blue Line)** - **Trend**: Gradual upward slope with minor fluctuations. - **Key Points**: - Starts at ~25.2 PSNR at 0 steps. - Reaches ~29.0 PSNR by 10,000 steps. - Plateaus near ~29.2 PSNR after 15,000 steps. 2. **LAN (Train from Scratch) (Orange Line)** - **Trend**: Steeper initial ascent, surpassing SIREN around 5,000 steps, then plateaus. - **Key Points**: - Starts at ~25.2 PSNR at 0 steps. - Peaks at ~29.8 PSNR at ~7,500 steps (marked by an arrow pointing to the orange line). - Declines slightly to ~29.6 PSNR by 10,000 steps. 3. **LAN (Fine Tune SIREN) (Green Line)** - **Trend**: Initial dip, followed by rapid improvement after 10,000 steps, surpassing all other lines. - **Key Points**: - Starts at ~25.2 PSNR at 0 steps. - Dips to ~27.5 PSNR at ~10,000 steps. - Rises sharply to ~30.8 PSNR by 17,500 steps (marked by an arrow pointing to the green line). - Maintains ~30.8 PSNR through 20,000 steps. --- ### **Embedded Images & Spatial Grounding** 1. **Ground Truth Image** - Placed at the origin (x=0, y=25.2). - Depicts a person operating a theodolite in an open field. 2. **Image at ~7,500 Steps** - Positioned at x=7,500, y=29.8 (orange line peak). - Shows the same individual with a theodolite, emphasizing the model's performance at this training step. 3. **Image at ~17,500 Steps** - Positioned at x=17,500, y=30.8 (green line peak). - Repeats the theodolite operator, highlighting the fine-tuned model's superior performance. --- ### **Legend Cross-Reference** - **Blue (SIREN)**: Matches the gradual blue line. - **Orange (LAN Train)**: Matches the orange line peaking at 7,500 steps. - **Green (LAN Fine-Tune)**: Matches the green line peaking at 17,500 steps. --- ### **Component Isolation** 1. **Header**: No explicit header text; title inferred from axis labels. 2. **Main Chart**: Dominates the image, with three lines and three embedded images. 3. **Footer**: Legend occupies the bottom-right corner. --- ### **Textual Content** - **Legend Labels**: - SIREN - LAN (train from scratch) - LAN (fine tune SIREN) - **Axis Titles**: - `training steps` (x-axis) - `PSNR` (y-axis) - **Embedded Image Labels**: - "Ground Truth" (top-left image) --- ### **Conclusion** The chart demonstrates that fine-tuning LAN on SIREN (green line) achieves the highest PSNR (~30.8) by 17,500 steps, outperforming both SIREN (~29.2) and LAN trained from scratch (~29.6). The "Ground Truth" images contextualize the models' performance at critical training milestones. </details> Figure B.3: A SIREN network (fixed sine activations) can be adapted to LANs (learnable activations) to improve image representations. Implicit neural representations view images as 2D functions $f(x,y)$ , where the pixel value $f$ is a function of two coordinates of the pixel $x$ and $y$ . To compress an image, such an implicit neural representation ( $f$ is a neural network) can achieve impressive compression of parameters while maintaining almost original image quality. SIREN [118] proposed to use MLPs with periodic activation functions to fit the function $f$ . It is natural to consider other activation functions, which are allowed in LANs. However, since we initialize LAN activations to be smooth but SIREN requires high-frequency features, LAN does not work immediately. Note that each activation function in LANs is a sum of the base function and the spline function, i.e., $\phi(x)=b(x)+{\rm spline}(x)$ , we set $b(x)$ to sine functions, the same setup as in SIREN but let ${\rm spline}(x)$ be trainable. For both MLP and LAN, the shape is [2,128,128,128,128,128,1]. We train them with the Adam optimizer, batch size 4096, for 5000 steps with learning rate $10^{-3}$ and 5000 steps with learning rate $10^{-4}$ . As shown in Figure B.3, the LAN (orange) can achieve higher PSNR than the MLP (blue) due to the LAN’s flexibility to fine tune activation functions. We show that it is also possible to initialize a LAN from an MLP and further fine tune the LAN (green) for better PSNR. We have chosen $G=5$ in our experiments, so the additional parameter increase is roughly $G/N=5/128≈ 4\%$ over the original parameters. Appendix C Dependence on hyperparameters We show the effects of hyperparamters on the $f(x,y)={\rm exp}({\rm sin}(\pi x)+y^{2})$ case in Figure C.1. To get an interpretable graph, we want the number of active activation functions to be as small (ideally 3) as possible. 1. We need entropy penalty to reduce the number of active activation functions. Without entropy penalty, there are many duplicate functions. 1. Results can depend on random seeds. With some unlucky seed, the pruned network could be larger than needed. 1. The overall penalty strength $\lambda$ effectively controls the sparsity. 1. The grid number $G$ also has a subtle effect on interpretability. When $G$ is too small, because each one of activation function is not very expressive, the network tends to use the ensembling strategy, making interpretation harder. 1. The piecewise polynomial order $k$ only has a subtle effect on interpretability. However, it behaves a bit like the random seeds which do not display any visible pattern in this toy example. <details> <summary>2404.19756v5/extracted/6190555/figs/interpretability_hyperparameters.png Details</summary> ![7b8bd8e9d3892b20bee793f9a85bc7e7c08ff8a5c86d0ab8d420f3958da1b86f](http://localhost:8000/v1/image/7b8bd8e9d3892b20bee793f9a85bc7e7c08ff8a5c86d0ab8d420f3958da1b86f) ### Visual Description # Technical Document Extraction: Network Structure Analysis Diagram ## Diagram Overview The image presents a comparative analysis of network structures under varying parameters, organized into five panels (a)-(e). Each panel demonstrates how specific variables influence node connections and network topology. --- ### Panel (a): Effect of Entropy Regularization **Title**: Effect of entropy regularization **Sub-labels**: - Left box: `w/ entropy` - Equation: `exp(sin(πx) + y²)` - Diagram: Sparse connections between nodes `x` and `y` - Right box: `w/o entropy` - Equation: `exp(sin(πx) + y²)` - Diagram: Dense interconnections between nodes **Key Observations**: - Entropy regularization reduces network connectivity density. - Node labels: `x` (input), `y` (output). --- ### Panel (b): Effect of Random Seeds **Title**: Effect of random seeds **Sub-labels**: - `seed = 0` - Equation: `exp(sin(πx) + y²)` - Diagram: Moderate connectivity - `seed = 1` - Equation: `exp(sin(πx) + y²)` - Diagram: Increased branching - `seed = 2` - Equation: `exp(sin(πx) + y²)` - Diagram: Complex inter-node paths - `seed = 3` - Equation: `exp(sin(πx) + y²)` - Diagram: Highly optimized topology **Key Observations**: - Random seeds introduce variability in network architecture. - All configurations share the same base equation but differ in structural outcomes. --- ### Panel (c): Effect of Lambda (Overall Penalty Strength) **Title**: Effect of lambda (overall penalty strength) **Sub-labels**: - `λ = 0` - Equation: `exp(sin(πx) + y²)` - Diagram: Fully connected network - `λ = 0.0001` - Equation: `exp(sin(πx) + y²)` - Diagram: Slight reduction in connections - `λ = 0.01` - Equation: `exp(sin(πx) + y²)` - Diagram: Significant sparsity **Key Observations**: - Higher λ values enforce stricter penalties, reducing network density. - Node labels remain consistent (`x`, `y`). --- ### Panel (d): Effect of G (Number of Grid Points) **Title**: Effect of G (number of grid points) **Sub-labels**: - `G = 2` - Equation: `exp(sin(πx) + y²)` - Diagram: Minimal grid-based connections - `G = 3` - Equation: `exp(sin(πx) + y²)` - Diagram: Intermediate grid complexity - `G = 5` (Standard setup) - Equation: `exp(sin(πx) + y²)` - Diagram: Balanced grid structure **Key Observations**: - `G = 5` is marked as the "Standard setup" with optimal grid resolution. - Increasing G enhances spatial resolution in the network. --- ### Panel (e): Effect of k (Piecewise Polynomial Order) **Title**: Effect of k (piecewise polynomial order) **Sub-labels**: - `k = 1` - Equation: `exp(sin(πx) + y²)` - Diagram: Linear node relationships - `k = 2` - Equation: `exp(sin(πx) + y²)` - Diagram: Quadratic interactions - `k = 3` - Equation: `exp(sin(πx) + y²)` - Diagram: Cubic complexity - `k = 4` - Equation: `exp(sin(πx) + y²)` - Diagram: Higher-order polynomial paths - `k = 5` - Equation: `exp(sin(πx) + y²)` - Diagram: Advanced polynomial modeling - `k = 6` - Equation: `exp(sin(πx) + y²)` - Diagram: Maximum polynomial flexibility **Key Observations**: - Higher k values enable more complex node interactions. - All configurations use the same base equation but vary in functional approximation depth. --- ### Cross-Referenced Elements - **Equations**: All panels use `exp(sin(πx) + y²)` as the base function, with structural differences arising from parameter adjustments. - **Node Labels**: Consistent use of `x` (input) and `y` (output) across all diagrams. - **Standard Setup**: Explicitly marked in panel (d) for `G = 5`. --- ### Summary This diagram systematically explores how entropy regularization, random seeds, penalty strength (λ), grid resolution (G), and polynomial order (k) influence network topology. Each parameter adjustment alters connectivity patterns while maintaining the core mathematical formulation. The "Standard setup" (`G = 5`) serves as a reference point for optimal grid-based performance. </details> Figure C.1: Effects of hyperparameters on interpretability results. Appendix D Feynman KANs We include more results on the Feynman dataset (Section 3.3). Figure D.1 shows the pareto frontiers of KANs and MLPs for each Feynman dataset. Figure D.3 and D.2 visualize minimal KANs (under the constraint test RMSE $<10^{-2}$ ) and best KANs (with the lowest test RMSE loss) for each Feynman equation fitting task. <details> <summary>2404.19756v5/x7.png Details</summary> ![c5c809d9a0d82177a6d01a6c2cf028ffeba0b2f778af3756bcb465b426838c79](http://localhost:8000/v1/image/c5c809d9a0d82177a6d01a6c2cf028ffeba0b2f778af3756bcb465b426838c79) ### Visual Description # Technical Document Extraction: RMSE vs. Number of Parameters ## Chart Structure - **Grid Layout**: 4 columns × 7 rows of subplots (28 total charts). - **X-Axis**: "number of parameters" (log scale: 10¹ to 10⁴). - **Y-Axis**: "RMSE" (log scale: 10⁻¹ to 10⁻⁷). ## Legend (Consistent Across All Charts) | Model/Type | Color | Line Style | |---------------------|--------|------------| | KAN train | Blue | Solid | | KAN test | Blue | Dashed | | ReLU MLP train | Orange | Dashed | | ReLU MLP test | Orange | Solid | | Tanh MLP train | Green | Solid | | Tanh MLP test | Green | Dashed | | SiLU MLP train | Purple | Solid | | SiLU MLP test | Purple | Dashed | ## Key Trends 1. **KAN Models**: - **Train/Test Lines**: Solid/Dashed blue lines. - **Behavior**: Sharp RMSE reduction at low parameter counts (10¹–10³), followed by plateauing. Test RMSE often outperforms training RMSE after parameter scaling. 2. **ReLU MLP**: - **Train/Test Lines**: Dashed/Solid orange lines. - **Behavior**: Gradual RMSE decline, plateauing at higher parameter counts. Test RMSE converges closely with training RMSE. 3. **Tanh MLP**: - **Train/Test Lines**: Solid/Dashed green lines. - **Behavior**: Similar to ReLU MLP but with slightly higher RMSE values. Test RMSE shows minimal divergence from training RMSE. 4. **SiLU MLP**: - **Train/Test Lines**: Solid/Purple dashed lines. - **Behavior**: Smooth RMSE reduction, often outperforming ReLU/Tanh MLPs at mid-to-high parameter counts. Test RMSE closely tracks training RMSE. ## Notable Observations - **Parameter Efficiency**: KAN models achieve lower RMSE with fewer parameters compared to MLPs. - **Test vs. Train**: Test RMSE generally improves relative to training RMSE as parameters increase, except for KAN models where test RMSE plateaus earlier. - **Model Comparison**: SiLU MLP often exhibits the best performance at higher parameter counts, followed by Tanh, ReLU, and KAN. ## Subplot Labels Each chart is labeled with a unique identifier (e.g., "I.6.2", "II.38.3") at the top, likely representing dataset/configuration variations. No additional textual annotations are present in the charts. </details> Figure D.1: The Pareto Frontiers of KANs and MLPs for Feynman datasets. <details> <summary>2404.19756v5/x8.png Details</summary> ![2a0b7cabcfce41f911cb139a172e9a7f622615fc3d39e8fe9935042e1fc49a03](http://localhost:8000/v1/image/2a0b7cabcfce41f911cb139a172e9a7f622615fc3d39e8fe9935042e1fc49a03) ### Visual Description # Technical Document Extraction: Flowchart Diagram Analysis ## 1. Diagram Structure Overview The image contains a 5x5 grid of flowchart diagrams, each labeled with a unique identifier in the format: - **I.x.x (best)** for primary diagrams - **II.x.x (best)** for secondary diagrams - **III.x.x (best)** for tertiary diagrams Each diagram consists of: - Nodes (squares, circles, and diamonds) - Directed arrows indicating flow - Nested sub-diagrams within larger structures ## 2. Label Inventory All textual labels extracted from the diagrams: ### Row 1 (Top) 1. I.6.2 (best) 2. I.6.2b (best) 3. I.9.18 (best) 4. I.12.11 (best) 5. I.26.2 (best) ### Row 2 6. I.13.12 (best) 7. I.15.3x (best) 8. I.16.6 (best) 9. I.18.4 (best) 10. I.27.6 (best) ### Row 3 11. I.30.3 (best) 12. I.44.4 (best) 13. I.37.4 (best) 14. I.30.5 (best) 15. I.50.26 (best) ### Row 4 16. II.6.15b (best) 17. II.11.27 (best) 18. II.2.42 (best) 19. II.38.3 (best) 20. II.35.18 (best) ### Row 5 21. II.11.7 (best) 22. II.10.19 (best) 23. III.17.37 (best) 24. III.36.38 (best) ## 3. Diagram Component Analysis ### Node Types - **Squares**: Primary decision points - **Circles**: Process steps - **Diamonds**: Evaluation/checkpoints - **Sub-diagrams**: Nested within larger structures ### Connection Patterns - **Solid arrows**: Primary flow - **Dashed arrows**: Alternative paths - **Bidirectional arrows**: Feedback loops ## 4. Spatial Grounding All labels positioned in the **top-left corner** of their respective diagrams. No legend or color-coding system observed in the image. ## 5. Trend Verification No numerical data or quantitative trends present. All diagrams represent structural relationships rather than statistical information. ## 6. Component Isolation Each diagram operates as an independent unit within the grid. No cross-diagram dependencies or shared elements identified. ## 7. Final Notes - All text is in English - No data tables or heatmaps present - Diagrams appear to represent algorithmic flowcharts or process optimization models - "(best)" suffix suggests these are optimal configurations among multiple possibilities This structured analysis provides complete textual extraction and contextual interpretation of the flowchart diagrams for technical documentation purposes. </details> Figure D.2: Best Feynman KANs <details> <summary>2404.19756v5/x9.png Details</summary> ![de614ea12026581071d3e5ba6572e7e8bb65624d47b8037df8303155684f48e6](http://localhost:8000/v1/image/de614ea12026581071d3e5ba6572e7e8bb65624d47b8037df8303155684f48e6) ### Visual Description # Technical Diagram Analysis ## Diagram Labels and Structure The image contains a grid of **35 minimal diagrams** organized into three main sections (I, II, III) with hierarchical numbering. Each diagram is labeled with a unique identifier and "(minimal)" in parentheses. Below is the complete list of labels: ### Section I - I.6.2 (minimal) - I.6.2b (minimal) - I.9.18 (minimal) - I.12.11 (minimal) - I.13.12 (minimal) - I.15.3x (minimal) - I.26.2 (minimal) - I.16.6 (minimal) - I.18.4 (minimal) - I.27.6 (minimal) - I.37.4 (minimal) - I.29.16 (minimal) - I.30.3 (minimal) - I.40.1 (minimal) - I.50.26 (minimal) - I.30.5 (minimal) ### Section II - II.2.42 (minimal) - II.2.42 (minimal) - II.11.7 (minimal) - II.11.27 (minimal) - II.38.3 (minimal) - II.35.18 (minimal) - II.36.38 (minimal) - II.9.52 (minimal) ### Section III - III.17.37 (minimal) - III.10.19 (minimal) ## Diagram Components Each diagram consists of: 1. **Nodes**: Represented as circles or squares. 2. **Connections**: Lines (solid or dashed) linking nodes. 3. **Shaded Squares**: Some nodes contain diagonal lines or patterns (e.g., `\\`, `/`, `\\/`). 4. **Arrows**: Directional indicators on certain connections (e.g., `→`). ## Key Observations - **Hierarchical Structure**: Diagrams in Section I are more complex, with dense interconnections. Sections II and III show simpler, sparser layouts. - **Recurring Patterns**: - Diagrams like II.2.42 and I.6.2b use triangular node arrangements. - Shaded squares with diagonal lines appear in I.16.6 and III.10.19. - **Minimalism**: All diagrams are labeled "(minimal)", suggesting optimized or reduced configurations. ## Notes - No axis titles, legends, or numerical data are present in the image. - Textual information is limited to the labels and embedded patterns in shaded squares. </details> Figure D.3: Minimal Feynman KANs Appendix E Remark on grid size For both PDE and regression tasks, when we choose the training data on uniform grids, we witness a sudden increase in training loss (i.e., sudden drop in performance) when the grid size is updated to a large level, comparable to the different training points in one spatial direction. This could be due to implementation of B-spline in higher dimensions and needs further investigation. Appendix F KANs for special functions We include more results on the special function dataset (Section 3.2). Figure F.2 and F.1 visualize minimal KANs (under the constraint test RMSE $<10^{-2}$ ) and best KANs (with the lowest test RMSE loss) for each special function fitting task. <details> <summary>2404.19756v5/x10.png Details</summary> ![b96c6e6eef7146f944d4d676c12310debe864e030b5e320735bfe52239d57a74](http://localhost:8000/v1/image/b96c6e6eef7146f944d4d676c12310debe864e030b5e320735bfe52239d57a74) ### Visual Description # Technical Document Extraction: Diagram Analysis ## Diagram Labels and Structures ### Row 1 1. **ellipj (minimal)** - Tree structure with 2 primary branches. - Root node splits into two nodes, each with two sub-nodes. - Squares at nodes contain curves (likely function representations). 2. **ellipkinc (minimal)** - Root node with 3 branches. - Top branch has 1 sub-node; middle and bottom branches each have 2 sub-nodes. - Curves in squares suggest varying function behaviors. 3. **ellipeinc (minimal)** - Root node with 3 branches. - Top branch has 1 sub-node; middle and bottom branches each have 2 sub-nodes. - Curves in squares show distinct patterns compared to ellipkinc. ### Row 2 4. **iv (minimal)** - Balanced binary tree with 2 levels. - Root splits into two nodes, each with two sub-nodes. - Curves in squares exhibit smooth transitions. 5. **jv (minimal)** - Root node with 2 branches. - Top branch has 1 sub-node; bottom branch has 2 sub-nodes. - Curves in squares show sharper inflection points. 6. **kv (minimal)** - Root node with 2 branches. - Top branch has 1 sub-node; bottom branch has 2 sub-nodes. - Curves in squares display linear trends. ### Row 3 7. **yv (minimal)** - Root node with 3 branches. - Top branch has 1 sub-node; middle and bottom branches each have 2 sub-nodes. - Curves in squares show oscillatory behavior. 8. **lpmv_m_1 (minimal)** - Root node with 2 branches. - Top branch has 3 sub-nodes; bottom branch has 2 sub-nodes. - Curves in squares indicate polynomial approximations. 9. **lpmv_m_2 (minimal)** - Root node with 2 branches. - Top branch has 2 sub-nodes; bottom branch has 3 sub-nodes. - Curves in squares show higher-degree polynomial fits. ### Row 4 10. **lpmv_m_0 (minimal)** - Root node with 2 branches. - Top branch has 1 sub-node; bottom branch has 2 sub-nodes. - Curves in squares exhibit baseline behavior. 11. **sph_harm_0_n_1 (minimal)** - Root node with 2 branches. - Top branch has 1 sub-node; bottom branch has 2 sub-nodes. - Curves in squares suggest spherical harmonic components. 12. **sph_harm_m_0_n_2 (minimal)** - Root node with 2 branches. - Top branch has 1 sub-node; bottom branch has 2 sub-nodes. - Curves in squares display higher-order spherical harmonics. ### Row 5 13. **sph_harm_m_1_n_1 (minimal)** - Root node with 2 branches. - Top branch has 1 sub-node; bottom branch has 2 sub-nodes. - Curves in squares indicate mixed spherical harmonic terms. 14. **sph_harm_m_1_n_2 (minimal)** - Root node with 2 branches. - Top branch has 1 sub-node; bottom branch has 2 sub-nodes. - Curves in squares show increased complexity in harmonic terms. 15. **sph_harm_m_2_n_2 (minimal)** - Root node with 2 branches. - Top branch has 1 sub-node; bottom branch has 2 sub-nodes. - Curves in squares represent highest-order spherical harmonics in the set. ## Key Observations - **Label Patterns**: - Terms like `ellip`, `iv`, `jv`, `kv`, `yv` likely denote elliptic integrals or related functions. - `lpmv_m_*` and `sph_harm_m_*_n_*` suggest polynomial and spherical harmonic expansions with parameters `m` and `n`. - "(minimal)" suffix implies base cases or simplified configurations. - **Tree Structures**: - Diagrams use hierarchical trees to represent function decomposition or algorithmic steps. - Nodes with curves may symbolize function evaluations or intermediate results. - Branch asymmetry (e.g., 1 vs. 2 sub-nodes) could reflect computational efficiency or mathematical constraints. - **Curve Characteristics**: - Smooth curves: Likely represent continuous functions (e.g., integrals). - Sharp inflections: May indicate discontinuities or piecewise definitions. - Oscillatory patterns: Suggest periodic or wave-like behavior (e.g., spherical harmonics). ## Conclusion The diagrams encode hierarchical relationships for mathematical functions, with labels indicating specific cases (e.g., minimal configurations). The tree structures and curve patterns provide insights into function behavior, decomposition strategies, or algorithmic flow. No numerical data or legends are present; analysis is based on structural and symbolic cues. </details> Figure F.1: Best special KANs <details> <summary>2404.19756v5/x11.png Details</summary> ![6085685559b6accdb5ce79aac3ce1911482b842b2a705b08e8d064f35b5dbc87](http://localhost:8000/v1/image/6085685559b6accdb5ce79aac3ce1911482b842b2a705b08e8d064f35b5dbc87) ### Visual Description # Technical Document Extraction: Hierarchical Algorithm Diagrams ## Extracted Textual Information ### Labels and Diagram Identifiers All diagrams are labeled with algorithm names followed by "(best)" to denote optimal configurations. The complete list of labels is as follows: 1. **ellipj (best)** 2. **ellipeinc (best)** 3. **ellipkinc (best)** 4. **iv (best)** 5. **kv (best)** 6. **jv (best)** 7. **lpmv_m_0 (best)** 8. **lpmv_m_1 (best)** 9. **sph_harm_m_0_n_1 (best)** 10. **sph_harm_m_1_n_1 (best)** 11. **sph_harm_m_2_n_2 (best)** 12. **sph_harm_m_0_n_2 (best)** --- ## Diagram Structure and Components ### General Description The image contains **12 hierarchical diagrams** arranged in a 4x3 grid. Each diagram represents a computational algorithm or method, with nodes and edges illustrating relationships or workflows. Key observations: - **Nodes**: Represented as squares or circles, often containing mathematical symbols (e.g., `√`, `∫`, `Σ`). - **Edges**: Lines connecting nodes, some with directional arrows. - **Hierarchical Layout**: Top-down structure with branching nodes, suggesting decision trees or dependency graphs. ### Spatial Grounding - **Legend**: No explicit legend is present in the image. All labels are directly embedded as diagram titles. - **Color Coding**: No color differentiation is used; all diagrams are in black-and-white line art. --- ## Key Trends and Data Points - **Algorithm Complexity**: Diagrams vary in complexity, with some (e.g., `iv (best)`) showing dense interconnections, while others (e.g., `sph_harm_m_0_n_2 (best)`) have simpler structures. - **Mathematical Symbols**: Nodes frequently include integrals (`∫`), summations (`Σ`), and square roots (`√`), indicating mathematical operations or constraints. - **Best Configuration**: The "(best)" suffix suggests these diagrams represent optimized or most efficient implementations of the algorithms. --- ## Component Isolation ### Header - No explicit header is present. Diagrams are standalone with labels as identifiers. ### Main Chart - Each diagram is a self-contained hierarchical structure. Examples: - **ellipj (best)**: A linear chain of nodes with branching at the base. - **iv (best)**: A highly interconnected network with multiple feedback loops. - **sph_harm_m_0_n_2 (best)**: A minimal structure with a single branching node. ### Footer - No footer is present. All information is contained within the diagrams and their labels. --- ## Transcribed Text Embedded in Diagrams No additional text is embedded within the diagrams beyond the labels. All mathematical symbols (e.g., `√`, `∫`, `Σ`) are part of node annotations. --- ## Conclusion The image provides a visual representation of 12 optimized algorithms, each depicted as a hierarchical graph. The labels explicitly identify the algorithms and their "best" configurations. No numerical data, trends, or legends are present, as the focus is on structural and symbolic representation of the algorithms. </details> Figure F.2: Minimal special KANs

Rendering Paper...