2510.13999

# REAP the Experts: Why Pruning Prevails for One-Shot MoE compression **Authors**: - Yani Ioannou2, Vithursan Thangarasa (1Cerebras Systems Inc., 2Schulich School of Engineering, University of Calgary) Abstract Sparsely-activated Mixture-of-Experts (SMoE) models offer efficient pre-training and low latency but their large parameter counts create significant memory overhead, motivating research into expert compression. Contrary to recent findings favouring expert merging on discriminative benchmarks, we demonstrate that expert pruning is a superior strategy for generative tasks. We prove that merging introduces an irreducible error by causing a “functional subspace collapse”, due to the loss of the router’s independent, input-dependent control over experts. Leveraging this insight, we propose Router-weighted Expert Activation Pruning (REAP), a novel pruning criterion that considers both router gate-values and expert activation norms. Across a diverse set of SMoE models ranging from 20B to 1T parameters, REAP consistently outperforms merging and other pruning methods on generative benchmarks, especially at 50% compression. Notably, our method achieves near-lossless compression on code generation and tool-calling tasks with Qwen3-Coder-480B and Kimi-K2, even after pruning 50% of experts. Correspondence to mklasby@ucalgary.ca & vithu@cerebras.net $\dagger$ Work completed while on internship at Cerebras | <details> <summary>2510.13999v1/x1.png Details</summary>  ### Visual Description Icon/Small Image (25x24) </details> | https://github.com/CerebrasResearch/reap | | --- | --- | | <details> <summary>2510.13999v1/x2.png Details</summary>  ### Visual Description Icon/Small Image (26x24) </details> | https://hf.co/cerebras/Qwen3-Coder-REAP-363B-A35B-FP8 | | <details> <summary>2510.13999v1/x3.png Details</summary>  ### Visual Description Icon/Small Image (26x24) </details> | https://hf.co/cerebras/Qwen3-Coder-REAP-246B-A35B-FP8 | 1 Introduction Interest in the Sparsely-activated Mixture-of-Experts (SMoE) architecture for Large Language Models (LLMs) surged following the release of DeepSeek-V3 (DeepSeek-AI et al., 2024) and other high-quality open-weight SMoE LLMs (Jiang et al., 2024; Meta AI Team, 2025; Yang et al., 2025a; Zeng et al., 2025; Baidu, 2025; Kimi Team et al., 2025). Compared to dense models, the SMoEs offer lower latency and more efficient pre-training (Fedus et al., 2022). However, SMoEs require more parameters than dense models to achieve similar accuracy, resulting in significant memory overhead. Further, expert usage imbalance during inference causes poor accelerator utilization, leading to increased latency or compromises such as dropped tokens (Balmau et al., 2025). Expert usage imbalance also represents an opportunity, motivating prior work which investigates whether experts can be compressed without negatively impairing accuracy (Li et al., 2023; Lu et al., 2024). By eliminating or compressing redundant experts, memory overhead is reduced. A more uniform distribution of expert usage would also improve hardware utilization. Expert compression is particularly valuable for use cases which feature small batch sizes such as local deployments and academic research. Initial expert compression efforts focused on expert pruning, the removal of experts in their entirety. However, expert pruning is a strong intervention on the model’s weights. Techniques such as quantization, low-rank compression, and expert merging also offer memory savings but maintain a lossy representation of the less important experts. Crucially, expert merging has recently been demonstrated to outperform expert pruning when evaluated with perplexity and on Multiple Choice (MC) question answering benchmarks (Li et al., 2023; Liu et al., 2024b). However, an evaluation comparing these methods on generative benchmarks has yet to be conducted. In this work, we demonstrate that — when paired with a suitable saliency criterion — expert pruning outperforms expert merging, particularly on generative benchmark tasks such as code generation, creative writing, and mathematical reasoning. Specifically, our main contributions are as follows: - We prove that expert merging introduces irreducible error due to the loss of the router’s independent, input-dependant modulation of the expert outputs resulting in functional subspace collapse, substantially reducing the functional output space of the compressed SMoE layer. In contrast, in expert pruned SMoEs the router maintains independent control over the remaining experts; - We introduce Router-weighted Expert Activation Pruning (REAP), a novel expert pruning saliency criterion, which selects experts to prune which contribute minimally to the layer output by considering both the router gate-values and average activation norm of the experts; - Across diverse SMoE architectures ranging from 20B to 1T parameters and a suite of generative evaluations, we demonstrate the significant and consistent advantage of REAP over existing expert pruning and merging approaches, particularly at 50% compression. Notably, our method achieves near-lossless compression on code generation tasks after pruning 50% of experts from Qwen3-Coder-480B and Kimi-K2; - We open-source our code and select compressed model checkpoints to facilitate further research on compressed SMoEs and their applications. 2 Related Work Sparsely activated SMoE architecture. A Mixture-of-Experts (MoE) layer is comprised of multiple, specialized feed-forward subnetworks known as experts and a router which produces gate-values (i.e., gates) to dynamically modulate the output of the experts based on the input. The architecture was revived in the deep learning era by the introduction of the SMoE by Shazeer et al. (2017). SMoEs layers only select a subset of experts to use for each input, enabling massive scaling of model parameters without a commensurate increase in computational cost (Lepikhin et al., 2021; Fedus et al., 2022). In transformer-based LLMs, SMoE layers are integrated by replacing the traditional feed-forward layers. Further innovations such as auxiliary-loss-free load balancing (DeepSeek-AI et al., 2024), shared experts, and fined-grained experts (Dai et al., 2024) have propelled SMoE architectures to become the de facto standard for LLMs in recent months. Expert pruning. Although SMoE layers effectively decouple total model parameters from inference costs, the memory overhead of storing large SMoEs restricts their deployment in resourced-constrained environments, motivating research in expert pruning to reduce total number of parameters. Early efforts demonstrated that progressively pruning experts based on router weights during fine-tuning until a single expert remained could preserve model quality in task-specific settings (Chen et al., 2022). Koishekenov et al. (2023) found expert pruning to be effective without further fine-tuning despite aggressively pruning up to 80% of experts. Muzio et al. (2024) found that global pruning using gate-values as a saliency criterion was more effective than uniform, layer-wise frequency-based pruning. Other sophisticated pruning criteria have been proposed: Lu et al. (2024) introduced an exhaustive search strategy which prunes experts that minimize the reconstruction loss between the original and pruned layer outputs; Liu et al. (2024a) used a gradient-free evolutionary algorithm to prune experts. Both of these works demonstrated significant improvements over naive frequency-based pruning. A comprehensive evaluation of 16 diverse pruning criteria was conducted by Jaiswal et al. (2025). Expert Activation Norm (EAN) was empirically found to be the highest performing criterion and the benefits of iterative pruning were presented. Expert merging. While the above-noted works prove that expert compression is feasible via pruning, an alternative compression technique is to merge experts. Generally, merging requires both a clustering algorithm and a merging technique. Li et al. (2023) introduced Merge Sparse Mixture of Experts (M-SMoE) which first initializes expert cluster centres by identifying the dominant experts with the highest usage frequency globally across all layers. The remaining non-dominant experts are clustered based on the cosine similarity of router logits. Finally, experts weights are aligned via permutation with the weight matching algorithm (Ainsworth et al., 2023) and merged using frequency-weighted parameter averaging. Li et al. (2023) found that their technique outperformed Chen et al. ’s (2022) pruning method on MC benchmarks. Chen et al. (2025) proposed Hierarchical Clustering for Sparsely activated Mixture of Experts (HC-SMoE). HC-SMoE clusters experts based on the euclidean similarity of their representative vectors — the average activation of each expert measured on every token in a calibration dataset — using hierarchical agglomerative clustering. Similar to M-SMoE, HC-SMoE uses frequency-weighted parameter averaging to merge clusters into a single merged expert. Without any fine-tuning, Chen et al. (2025) found that their technique outperformed expert pruning based on router logits (He et al., 2025a), frequency, and Lu et al. ’s (2024) method when benchmarked on a suite of MC question answering tasks. Other compression techniques. In addition to pruning and merging, experts may be compressed through quantization (Huang et al., 2025), low-rank decomposition (Yang et al., 2024a; Gu et al., 2025; He et al., 2025b), weight sparsity (He et al., 2025a), or a combination of any of the above techniques (Liu et al., 2025). These other approaches are orthogonal to expert pruning and merging; however, note that expert merging necessitates re-quantization for block quantization formats that share common scaling coefficients across a group of weights. Model merging. Model merging aims to combine parameters from multiple trained neural networks and has been rapidly adopted as a cost-effective way to improve model quality across diverse domains. The initial motivation for merging was based on the finding that mode connectivity exists between the loss landscapes of two or more trained neural networks, enabling interpolation of their parameters without incurring an increase in loss (Garipov et al., 2018; Ainsworth et al., 2023; Ito et al., 2024). Simple parameter averaging remains an effective technique; however, more sophisticated strategies based on task vectors have also been proposed to minimize interference in the merged model parameters (Ilharco et al., 2023; Yadav et al., 2023; Yu et al., 2024). Much of the existing literature focuses on the setting in which multiple fine-tunes of a single checkpoint are merged. Non-local merging in which the models do not share a common checkpoint is more closely related to expert merging. Sharma et al. (2024) found that re-scaling of model activations was necessary to achieve high-quality non-local merging. LLM evaluation. Evaluating LLMs is challenging; prior work demonstrated that simple metrics such as perplexity can be misleading when used to evaluate compressed LLMs (Jaiswal et al., 2024). MC benchmarks typically measure the log-likelihood of answer tokens to determine a model’s response to a question (Gao et al., 2023; Chandak et al., 2025). As such, each response choice is evaluated in a single forward pass, without any tokens being generated by the model. Perplexity and MC accuracy can therefore be viewed as discriminative metrics. In contrast, generative benchmarks require the model to output a response, more closely corresponding with real-world use-cases of LLMs. Tasks such as code generation, mathematical reasoning with structured outputs, and creative writing are examples of generative benchmarks. 3 Merging Experts Causes Functional Subspace Collapse Setup. To motivate our proposed expert pruning method, we first formally develop the expected errors of both expert merging and pruning. Consider a SMoE layer with $K$ experts $f_{1},...,f_{K}$ , each a function $f_{k}:\mathbb{R}^{d}→\mathbb{R}^{d}$ , and a router producing non-negative gates $\mathbf{g(x)}=(g_{1}(x),...,g_{K}(x))∈\Delta^{K-1}$ . Top- $k$ routing is achieved by zeroing all but the largest $k$ gates. The output of the original layer is $$ h(x):=\sum_{k=1}^{K}g_{k}(x)\,f_{k}(x). \tag{1} $$ Two operations at fixed compression. To analyse the fundamental difference between compression operations, we focus on the elementary case of reducing two experts, $(f_{i},f_{j})$ , to one. This pairwise analysis is the building block for any larger merge within a cluster. Pruning removes expert $j$ and re-normalizes the router outputs over the remaining $K-1$ experts, producing a new set of gates $\bar{g}(x)$ . Merging replaces $(f_{i},f_{j})$ with a new expert $\tilde{f}$ . Existing one-shot expert merging methods such as HC-SMoE and M-SMoE sum the gates for the original experts $g_{i}(x)+g_{j}(x)$ . The pruned, $\bar{h}(x)$ , and merged, $\tilde{h}(x)$ , layer outputs are $$ \bar{h}(x):=\sum_{k\neq j}\bar{g}_{k}(x)\,f_{k}(x), \tag{2} $$ $$ \tilde{h}(x):=\sum_{k\neq i,j}g_{k}(x)f_{k}(x)+\big(g_{i}(x)+g_{j}(x)\big)\tilde{f}(x). \tag{3} $$ 3.1 Merging induces an input-dependent target a single expert cannot realize Define the router’s input-dependent mixing ratio $r(x):=\frac{g_{i}(x)}{g_{i}(x)+g_{j}(x)}∈[0,1]$ on the set where $g_{i}+g_{j}>0$ . Substituting $g_{i}(x)$ and $g_{j}(x)$ in terms of $r(x)$ , the original contribution of the pair $(i,j)$ can be written as $$ \displaystyle g_{i}(x)f_{i}(x)+g_{j}(x)f_{j}(x) \displaystyle=\big[r(x)(g_{i}(x)+g_{j}(x))\big]f_{i}(x)+\big[(1-r(x))(g_{i}(x)+g_{j}(x))\big]f_{j}(x) \displaystyle=\big(g_{i}(x)+g_{j}(x)\big)\underbrace{\Big(r(x)f_{i}(x)+\big(1-r(x)\big)f_{j}(x)\Big)}_{\text{The ideal, input-dependent target expert}}. \tag{4} $$ After merging, the router must apply the summed gate, $g_{i}(x)+g_{j}(x)$ , to a constant convex combination of the constituent experts which is independent of $x$ . The core issue is that the merged model is forced to approximate the dynamic, input-dependent target expert with a static one. The following theorem quantifies this unavoidable approximation error. **Theorem 1 (Irreducible error of merging)** *Let $\tilde{f}_{\alpha}(x)=\alpha f_{i}(x)+(1-\alpha)f_{j}(x)$ with a constant $\alpha∈[0,1]$ and define $\Delta_{ij}:=f_{i}(x)-f_{j}(x)$ . The $L^{2}$ error of the merged pair is minimized when $\alpha$ is chosen to be the expected mixing ratio, $\alpha^{\star}:=\mathbb{E}[r(x)]$ . Omitting the argument $(x)$ for brevity, this minimal error is $$ \displaystyle\big\|\,\big(g_{i}{+}g_{j}\big)\!\big(rf_{i}{+}(1{-}r)f_{j}\big)\;-\;\big(g_{i}{+}g_{j}\big)\!\big(\alpha f_{i}{+}(1{-}\alpha)f_{j}\big)\big\|^{2} \displaystyle=\underbrace{\mathbb{E}\!\left[(g_{i}{+}g_{j})^{2}\right]}_{\text{router scale}}\cdot\underbrace{\mathrm{Var}[r(x)]}_{\text{policy variability}}\cdot\underbrace{\|\Delta_{ij}\|^{2}}_{\text{expert gap}}. \tag{5} $$ In particular, if the router’s policy is not constant ( $\mathrm{Var}[r(x)]>0$ ) and the experts are not functionally identical ( $\|\Delta_{ij}\|>0$ ), then every constant- $\alpha$ merge incurs strictly positive excess risk.* * Proof* The error term simplifies to $\big\|\big(g_{i}{+}g_{j}\big)\big(r-\alpha\big)\Delta_{ij}\big\|^{2}$ . Assuming independence between the router policy and expert functions, this is proportional to $\mathbb{E}[(r-\alpha)^{2}]$ . This is a standard least-squares problem minimized when $\alpha=\mathbb{E}[r]$ , and the minimal value is $\mathrm{Var}[r]$ . ∎ Consequences. Theorem 1 illustrates that merging with summed gates is fundamentally flawed whenever (i) the router has learned an input-dependent policy for mixing two experts ( $\mathrm{Var}[r]>0$ ), and (ii) the experts are themselves distinct ( $\|\Delta_{ij}\|>0$ ). Any fixed $\alpha$ cannot overcome the irreducible error bound established in equation 5. 3.2 Pruning preserves independent control Pruning removes one function but importantly does not tie the remaining gates. The router still modulates each surviving expert independently. In contrast, merging removes a degree of freedom in the policy by replacing individual experts with their mergers. For a direct comparison under no fine-tuning, pruning expert $j$ and reallocating its gate-value to expert $i$ produces the error $$ \big\|(g_{i}(x)f_{i}(x)+g_{j}(x)f_{j}(x))-(g_{i}(x){+}g_{j}(x))f_{i}(x)\big\|^{2}\;=\;\mathbb{E}\!\left[g_{j}(x)^{2}\,\|\Delta_{ij}(x)\|_{2}^{2}\right]. \tag{6} $$ Unlike equation 5, equation 6 does not penalize policy variability, the router still controls surviving experts independently. Whenever the router substantially mixes $i$ and $j$ (large $\mathrm{Var}[r]$ ) while the pruned expert $j$ has a small average gate-value ( $\mathbb{E}[g_{j}^{2}]$ ), pruning admits a strictly smaller error than merging. Synthesis. Theorem 1 establishes that summed gate merging incurs an irreducible error directly proportional to the router’s policy variability ( $\mathrm{Var}[r(x)]$ ). In contrast, the error from pruning a low-usage expert (Eq. 6) is proportional to its gate-value ( $\mathbb{E}[g_{j}^{2}]$ ) and is insensitive to policy variability. Therefore, when the router actively mixes between two distinct experts, merging is fundamentally disadvantaged. Remarks. (i) The constant-mixture model $\tilde{f}_{\alpha}$ is mathematically related to the frequency weighted parameter averaging merge used in practice. (ii) Even if $\tilde{f}$ was dependent on $x$ , the router after merging cannot independently modulate the two latent functions, so the original policy is invalidated. (iii) With top-k routers, the specific irreducible error from policy variability ( $\mathrm{Var}[r(x)]$ ) is generated exclusively on the support where both experts are selected. Outside that support, this component vanishes, leaving only a static error term that depends on the functional expert gap. (iv) See Appendix ˜ A for an extension of the above analysis to hierarchical clustering. 3.3 Empirical evidence for loss of independent control Setup. We analyse the functional subspaces of expert outputs across four diverse state-of-the-art SMoE architectures by recording mean expert activations from 32 samples of 2048 tokens from the c4 dataset (Raffel et al., 2020). By projecting expert activations onto their first two principal components, we visualize how pruning and merging affect the learned representations. See Appendix ˜ B for additional discussion. Early vs. late behaviour. Figures ˜ 1 and A4 demonstrate a striking progression of functional collapse from early to late layers across all architectures. In early layers, the original experts form relatively compact manifolds with moderate spread. After pruning, the surviving experts maintain their positions on the original manifold, preserving its geometric structure with reduced density. In contrast, merging produces a visible contraction toward the manifold’s centre. The contrast becomes dramatic in late layers, where experts are more specialized, and in high granularity architectures with many experts per layer. The progression from early to late layers validates our theoretical prediction that the irreducible error is proportional to $\mathrm{Var}[r(x)]$ . Early layers, which typically learn more generic features, exhibit lower policy variability and thus less dramatic collapse. Late layers, where experts have specialized for distinct computational roles, demonstrate high policy variability, resulting in the severe functional collapse observed when these specialized experts are merged into static averages. Synthesis across architectures. The consistency of these patterns across architectures with vastly different expert counts (8 to 128), sparsity levels (6.25% to 25% active), and parameter scales (21.9B to 109B) demonstrates that functional collapse under merging is a fundamental property of the operation rather than an artifact of specific implementations. These visualizations reveal that the core issue is not the reduction in the number of experts per se, but rather the qualitative change in the router’s control structure. <details> <summary>2510.13999v1/x4.png Details</summary>  ### Visual Description ## Technical Description of Scatter Plots This image presents a series of three scatter plots arranged horizontally, illustrating the transformation of data points across different stages. The plots share common axes and a legend. ### Legend and Axes * **Legend:** Located at the top of the image, the legend defines three categories of data points: * **Original Experts:** Represented by pink circles. * **Surviving:** Represented by blue circles. * **Merged:** Represented by green 'x' markers. * **Vertical Axis (Y-axis):** Labeled "PC2", with tick marks at -1.0, -0.5, and 0.0. * **Horizontal Axis (X-axis):** Labeled "PC1", with tick marks at -0.5, 0.0, and 0.5. * **Grid:** A faint grid is present in all three plots, corresponding to the axis tick marks. ### Plot 1: Original Experts * **Title:** "Original Experts" is positioned above the first plot. * **Data Representation:** This plot shows the distribution of the "Original Experts" data points (pink circles). * **Observations:** * The majority of the points are clustered around the origin (PC1=0, PC2=0), extending along a positive diagonal trend from approximately (-0.2, -0.2) to (0.2, 0.2). * There are several outliers, including a notable point near (0.3, -0.7) and another near (0.9, -1.0). * The overall spread is relatively broad compared to the subsequent plots. ### Plot 2: Surviving * **Title:** "Surviving" is positioned above the second plot. * **Data Representation:** This plot shows the "Surviving" data points (blue circles). The "Original Experts" points are also visible but are rendered in faint grey, indicating they are background context. * **Observations:** * The "Surviving" points form a tighter cluster compared to the "Original Experts," concentrated around the origin and following the same positive diagonal trend. * The outliers observed in the "Original Experts" plot (e.g., near (0.3, -0.7) and (0.9, -1.0)) are not part of the "Surviving" group, as indicated by their faint grey appearance. * The "Surviving" group appears to be a subset of the "Original Experts," excluding the most extreme outliers. ### Plot 3: Merged * **Title:** "Merged" is positioned above the third plot. * **Data Representation:** This plot shows the "Merged" data points (green 'x' markers). The "Original Experts" points are again visible in faint grey as background. * **Observations:** * The "Merged" points form the tightest cluster among the three plots. * This cluster is centered approximately around (PC1=0.1, PC2=0.1). * The "Merged" group represents a highly concentrated subset of the original data, occupying a smaller region in the PC1-PC2 space compared to both the "Original Experts" and "Surviving" groups. </details> (a) Qwen3-30B Layer 0 <details> <summary>2510.13999v1/x5.png Details</summary>  ### Visual Description ## Technical Description of Scatter Plots This image presents three side-by-side scatter plots, all sharing the same axes and grid. The plots illustrate the distribution of data points in a two-dimensional principal component space defined by PC1 and PC2. ### Common Elements * **Y-axis:** Labeled "PC2", with major tick marks and labels at -150, -100, -50, 0, and 50. * **X-axis:** Labeled "PC1", with major tick marks and labels at -100, 0, 100, and 200. * **Grid:** A light gray grid is overlaid on all three plots, corresponding to the major tick marks. ### Plot 1: Original Experts (Left) * **Title:** "Original Experts" * **Legend:** A pink circle marker is associated with this title. * **Data Distribution:** The plot displays a large number of pink circle markers. The points are widely scattered across the PC1-PC2 plane. There is a concentration of points around the origin (PC1=0, PC2=0), with a spread extending approximately from PC1=-150 to PC1=200 and from PC2=-150 to PC2=70. ### Plot 2: Surviving (Middle) * **Title:** "Surviving" * **Legend:** A blue circle marker is associated with this title. * **Data Distribution:** This plot shows a subset of the points from the "Original Experts" plot, represented by blue circle markers. These "Surviving" points are primarily located in the lower half of the plot (PC2 < 0), with a few points slightly above PC2=0. They are spread along the PC1 axis, roughly from -100 to 180. Faint gray points in the background represent the complete set of "Original Experts" data for context. ### Plot 3: Merged (Right) * **Title:** "Merged" * **Legend:** A green 'x' marker is associated with this title. * **Data Distribution:** This plot displays a distinct cluster of green 'x' markers. These "Merged" points are tightly grouped in the upper-left quadrant of the plot, approximately centered around PC1=-20 and PC2=30. Similar to the middle plot, faint gray points representing the "Original Experts" data are visible in the background. </details> (b) Qwen3-30B Layer 47 Figure 1: (1(a)) Functional subspace (PCA) for early SMoE layers in Qwen3-30B. Pruning (blue) preserves the manifold geometry; merging (green) collapses it toward the centre. (1(b)) Functional subspace (PCA) for late MoE layers. The contraction under merging is dramatically more pronounced, with up to 100 $×$ reduction in spread for models with many experts. See Figure ˜ A4 for results from other models. 4 Router-weighted Expert Activation Pruning (REAP) The above analysis demonstrates that the functional output space of a SMoE layer is defined by the coordinated behaviour of the router and experts. An expert’s total contribution to its layer’s output is determined by both its gate-value, $g_{k}(x)$ , and the magnitude of its output vector, $\big\|f_{k}(x)\big\|_{2}$ . However, naive frequency-based pruning fails to consider these properties. Intuitively, pruning experts which contribute minimally to the layer output minimizes the difference between the original and pruned layer outputs. Let $h(x)$ be the original output and $\bar{h}_{\setminus j}(x)$ be the output after pruning expert $j$ and re-normalizing the remaining router weights. The error induced by pruning expert $j$ is $$ \Delta\bar{h}_{\setminus j}(x):=h(x)-\bar{h}_{\setminus j}(x)=\sum_{k}g_{k}(x)f_{k}(x)-\sum_{k\neq j}\frac{g_{k}(x)}{1-g_{j}(x)}f_{k}(x). \tag{7} $$ Re-normalization of the router weights after pruning expert $j$ modulates all other remaining expert outputs, making direct minimization of $\Delta h_{j}$ complex. However, since our goal is to prune unimportant experts, we can reasonably assume their gate-values are small when active $\mathbb{E}_{x\sim\mathcal{X}}[g_{j}(x)]\ll 1$ . Under this assumption, the weight re-normalization factor is negligible, i.e., $1-g_{j}(x)≈ 1$ , and the error induced by pruning expert $j$ is approximately equal to the expert’s direct contribution to the layer output $$ \Delta\bar{h}_{\setminus j}(x)\approx\sum_{k}g_{k}(x)f_{k}(x)-\sum_{k\neq j}g_{k}(x)f_{k}(x))=g_{j}(x)f_{j}(x). \tag{8} $$ To select which experts to prune, we propose a novel saliency criterion, REAP, which approximates an expert’s importance by measuring its direct contribution to the layer’s output magnitude. Specifically, the saliency score, $S_{j}$ , is defined as the average of this contribution over tokens for which the expert is active where $S_{j}$ is the saliency of expert $f_{j}$ and $\mathcal{X}_{j}$ is the set of inputs where $g_{j}(x)∈ TopK(\mathbf{g(x))}$ . $$ S_{j}=\frac{1}{|\mathcal{X}_{j}|}\sum_{x\in\mathcal{X}_{j}}g_{j}(x)\cdot\big\|f_{j}(x)\big\|_{2}, \tag{9} $$ where $\mathcal{X}_{j}$ is the set of tokens where expert $j$ is active (i.e., $\mathcal{X}_{j}=\{x\mid j∈\text{TopK}(\mathbf{g}(x))\}$ ). The experts with the minimum saliency score are selected for pruning. REAP is robust to outlier activations and has a direct, intuitive interpretation by quantifying the average magnitude an expert adds to the output vector when it is selected by the router. Pruning experts with the lowest $S_{j}$ removes those with the least impactful contribution. 5 Experiments Setup. We implement REAP and other expert compression baselines in PyTorch (Ansel et al., 2024). We collect router logits and expert activation data to calibrate the compression algorithms using a variety of general pre-training and domain-specific Supervised Fine-Tuning (SFT) datasets. For calibration, 1,024 samples are randomly selected and packed to 2,048 sequence length for models with $≤$ 110B parameters. For models with $≥$ 110B parameters, we select 12,228 samples with a maximum sequence length of 16,384 tokens without truncation or packing. We compress models by pruning or merging 25% or 50% of experts in each layer, except for M-SMoE which determines the number of clusters per layer based on global expert usage frequency. When evaluating models with $≤$ 50B parameters on coding and MC, we calibrate and compress the models using three different seeds and report the mean. Larger models, creative writing, and mathematical reasoning evaluations are reported using a single seed, except where explicitly noted otherwise. All models are evaluated in the one-shot setting, with no additional fine-tuning after compression. Models and data. We evaluate the expert compression algorithms on a diverse set of six SMoE architectures covering model sizes from 21B to 1T with varying degrees of sparsity and expert granularity, see Table ˜ 1 for details. For MC question answering and code generation benchmarks, we use c4 (Raffel et al., 2020; Allen Institute for AI, 2024) and evol-codealpaca (Chaudhary, 2023; Luo et al., 2024; Tam, 2023) datasets to assess both general and domain-specific calibration. Models with $≥$ 110B parameters are additionally calibrated with data from xlam-function-calling (Liu et al., 2024c; Salesforce, 2025) and SWE-smith-trajectories (Yang et al., 2025c; b) datasets. For creative writing and math benchmarks we employ WritingPrompts curated (Pritsker, 2024) and tulu-3-sft-personas-math (Lambert et al., 2025; Allen Institute for AI, 2025), respectively. The default chat template is applied to all SFT datasets and </think> tags are explicitly closed to disable reasoning in hybrid reasoning models. Table 1: Comparison of SMoE models included in our study. | Model | Routed Experts | Shared Experts | Top-K | Sparsity | Parameters (1e9) | Active Params. (1e9) | First layer dense | | --- | --- | --- | --- | --- | --- | --- | --- | | ERNIE-4.5-21B-A3B-PT | 64 | 2 | 6 | 87.88% | 21.9 | 3 | Yes | | Qwen3-30B-A3B | 128 | 0 | 8 | 93.75% | 30.5 | 3 | No | | Mixtral-8x7B-Instruct-v0.1 | 8 | 0 | 2 | 75.00% | 46.7 | 13 | No | | GLM-4.5-Air | 128 | 1 | 8 | 93.02% | 106.9 | 12 | Yes | | Llama-4-Scout-17B-16E-Instruct | 16 | 1 | 1 | 88.24% | 107.8 | 17 | No | | Qwen3-Coder-480B-A35B-Instruct-FP8 | 160 | 0 | 8 | 95.00% | 480.2 | 35 | No | | Kimi-K2-Instruct-W4A16 (RedHatAI, 2025) | 384 | 1 | 8 | 97.66% | 1026.4 | 32 | Yes | Evaluation. Compressed SMoE models are evaluated on a suite of benchmarks including MC question answering, code generation, mathematical reasoning, creative writing, and tool calling. See Appendix ˜ C for details. We implement HC-SMoE and M-SMoE as expert merging baselines. Average linkage criterion is used for HC-SMoE. M-SMoE does not include low-rank compression from the complete MC-SMoE method. Pruning baselines consist of frequency-based pruning and EAN. See Appendix ˜ D for formal definitions. 5.1 Results In Tables ˜ 2 and 2 code generation, creative writing, math reasoning, and MC results are presented for Qwen3-30B and GLM-4.5-Air after calibration with the evol-codealpaca dataset. Table ˜ 3 contains results for large-scale SMoE pruned models on code generation, tool calling, and MC benchmarks. See Table ˜ A4 and Table ˜ A5 for detailed MC and code generation results, respectively. Figure ˜ A5 depicts coding generation and MC accuracy verses model parameters. See Appendix ˜ E for additional results. Zero-shot MC question answering. Both merging and pruning are capable of producing accurate compressed SMoE models for MC question answering. HC-SMoE and REAP have a mean decrease in accuracy of approximately 4% and 13% for compression ratios of 25% and 50%, respectively, excluding large-scale SMoEs. REAP achieves first or second rank among all methods, models and compression ratios, suggesting strong consistency regardless of specific model architecture. When calibrated on c4, we find slightly improved accuracies for all compression methods with similar rankings as noted above, see Table ˜ A6. Generative benchmarks. Compared to MC, generative benchmarks are more representative of real-world use cases of LLMs. In this setting, pruning emerges as the clearly superior compression method on the generative task benchmarks. Excluding large-scale SMoEs, REAP achieves a mean decrease in accuracy of 2.8% and 8.0% at 25% and 50% compression ratios, respectively, on coding. In comparison, both HC-SMoE and M-SMoE produce mean decreases in accuracy >5% at 25% compression and >20% at 50% compression. Notably, REAP maintains significantly higher accuracy at 50% compression than other pruning methods. On creative writing, REAP and EAN are near-lossless at 25% compression with REAP offering improved quality at 50% compression. Merging methods are less consistent across various model architectures and compression ratios. For example, M-SMoE is the best method for Qwen3-30B at 50% compression, but the worst on GLM-4.5-Air. REAP attains the best mathematical reasoning results with a remarkable mean decrease in accuracy of just 1.1% at 50% compression. HC-SMoE and M-SMoE offer high accuracy at 25% compression but are significantly less accurate than pruning at 50% compression. Table 2: MC and generative benchmark results for Qwen3-30B and GLM-4.5-Air. | | Coding | Creative Writing | Math | MC | | | | | | | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Model | Compression | Technique | Method | Eval+ | LiveCode | Code Avg | WildBench | GSM8K | MATH-500 | Math Avg | MC Avg | | Qwen3-30B-A3B | Baseline | 0.859 | 0.302 | 0.581 | 0.811 | 0.903 | 0.872 | 0.887 | 0.721 | | | | 25% | Merging | M-SMoE | 0.822 | 0.293 | 0.558 | 0.805 | 0.901 | 0.872 | 0.886 | 0.558 | | | HC-SMoE | 0.800 | 0.258 | 0.529 | 0.497 | 0.864 | 0.834 | 0.849 | 0.674 | | | | | Pruning | Frequency | 0.849 | 0.302 | 0.576 | 0.807 | 0.905 | 0.864 | 0.885 | 0.600 | | | | EAN | 0.840 | 0.311 | 0.576 | 0.811 | 0.895 | 0.866 | 0.881 | 0.603 | | | | | REAP | 0.843 | 0.308 | 0.575 | 0.804 | 0.892 | 0.864 | 0.878 | 0.669 | | | | | 50% | Merging | M-SMoE | 0.621 | 0.205 | 0.413 | 0.725 | 0.824 | 0.838 | 0.831 | 0.451 | | | HC-SMoE | 0.574 | 0.185 | 0.379 | 0.008 | 0.760 | 0.696 | 0.728 | 0.542 | | | | | Pruning | Frequency | 0.704 | 0.236 | 0.470 | 0.677 | 0.882 | 0.860 | 0.871 | 0.483 | | | | EAN | 0.798 | 0.306 | 0.552 | 0.702 | 0.886 | 0.842 | 0.864 | 0.493 | | | | | REAP | 0.821 | 0.293 | 0.557 | 0.718 | 0.878 | 0.872 | 0.875 | 0.518 | | | | | GLM-4.5-Air | Baseline | 0.820 | 0.374 | 0.597 | 0.839 | 0.846 | 0.918 | 0.882 | 0.747 | | | | 25% | Merging | M-SMoE | 0.781 | 0.330 | 0.555 | 0.781 | 0.848 | 0.880 | 0.864 | 0.596 | | | HC-SMoE | 0.793 | 0.363 | 0.578 | 0.788 | 0.842 | 0.908 | 0.875 | 0.704 | | | | | Pruning | Frequency | 0.805 | 0.341 | 0.573 | 0.793 | 0.832 | 0.908 | 0.870 | 0.648 | | | | EAN | 0.821 | 0.374 | 0.597 | 0.824 | 0.839 | 0.908 | 0.874 | 0.637 | | | | | REAP | 0.794 | 0.390 | 0.592 | 0.831 | 0.835 | 0.926 | 0.880 | 0.678 | | | | | 50% | Merging | M-SMoE | 0.493 | 0.099 | 0.296 | 0.391 | 0.465 | 0.466 | 0.465 | 0.444 | | | HC-SMoE | 0.662 | 0.220 | 0.441 | 0.593 | 0.667 | 0.732 | 0.700 | 0.564 | | | | | Pruning | Frequency | 0.546 | 0.104 | 0.325 | 0.604 | 0.615 | 0.612 | 0.613 | 0.521 | | | | EAN | 0.773 | 0.253 | 0.513 | 0.702 | 0.781 | 0.838 | 0.809 | 0.511 | | | | | REAP | 0.755 | 0.352 | 0.553 | 0.754 | 0.820 | 0.926 | 0.873 | 0.559 | | | | Table 3: Large-scale pruned SMoEs on agentic, non-agentic coding, tool-use tasks, and MC benchmarks. | | Non-Agentic Coding | Agentic Coding | Tool-Use (BFCLv3) | MC | | | | | | | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Model | Compression | Method | Eval+ | LiveCode | Code Avg | SWE-Bench-Verified | Non-Live | Live | Multi-Turn | Overall | MC Avg | | Qwen3-Coder- 480B-A35B- Instruct-FP8 | Baseline | 0.889 | 0.431 | 0.660 | 0.540 | 0.866 | 0.825 | 0.380 | 0.690 | 0.750 | | | 25% | Frequency | 0.792 | 0.296 | 0.544 | 0.378 | 0.844 | 0.763 | 0.355 | 0.654 | 0.606 | | | EAN | 0.876 | 0.419 | 0.647 | 0.534 | 0.831 | 0.813 | 0.384 | 0.676 | 0.702 | | | | REAP | 0.884 | 0.416 | 0.650 | 0.540 | 0.878 | 0.823 | 0.392 | 0.698 | 0.748 | | | | 50% | Frequency | 0.011 | 0.012 | 0.011 | 0.000 | 0.200 | 0.392 | 0.000 | 0.197 | 0.506 | | | EAN | 0.831 | 0.382 | 0.607 | 0.536 | 0.822 | 0.774 | 0.383 | 0.659 | 0.591 | | | | REAP | 0.873 | 0.415 | 0.644 | 0.522 | 0.849 | 0.801 | 0.371 | 0.674 | 0.692 | | | | Kimi-K2- Instruct- W4A16 | Baseline | 0.883 | 0.434 | 0.659 | 0.554 | 0.840 | 0.802 | 0.355 | 0.666 | 0.780 | | | 25% | Frequency | 0.524 | 0.082 | 0.303 | 0.000 | 0.644 | 0.603 | 0.045 | 0.431 | 0.604 | | | EAN | 0.831 | 0.379 | 0.605 | 0.562 | 0.819 | 0.802 | 0.335 | 0.652 | 0.703 | | | | REAP | 0.889 | 0.440 | 0.664 | 0.580 | 0.842 | 0.801 | 0.263 | 0.635 | 0.773 | | | | 50% | Frequency | 0.124 | 0.000 | 0.062 | 0.000 | 0.255 | 0.397 | 0.003 | 0.218 | 0.439 | | | EAN | 0.772 | 0.253 | 0.513 | 0.576 | 0.778 | 0.767 | 0.173 | 0.573 | 0.587 | | | | REAP | 0.863 | 0.429 | 0.646 | 0.576 | 0.785 | 0.743 | 0.164 | 0.564 | 0.643 | | | <details> <summary>2510.13999v1/x6.png Details</summary>  ### Visual Description ## Chart Analysis: Mean Accuracy of Pruning and Merging Methods This image presents two grouped bar charts comparing the mean accuracy of different pruning and merging methods at 50% and 25% compression ratios against a 0% baseline. The comparison is conducted across four tasks: Coding, Math, Creative Writing, and MC. ### Chart 1: GLM – 4.5 – Air **Y-Axis:** Mean Accuracy (%) (Range: 0-90) **X-Axis:** Tasks (Coding, Math, Creative Writing, MC) **Legend:** * **Compression Ratio:** * 0%: Dashed gray line * 50%: Solid bar * 25%: Hatched bar * **Pruning methods:** * REAP (ours): Blue * EAN: Red * Frequency: Green * **Merging methods:** * HC-SMoE: Yellow * M-SMoE: Light Blue **Data Table for GLM – 4.5 – Air:** | Task | Method | 0% Baseline | 50% Compression | 25% Compression | | :--- | :--- | :--- | :--- | :--- | | **Coding** | REAP (ours) | ~60% | ~60% | ~51% | | | EAN | ~60% | ~59% | ~50% | | | Frequency | ~60% | ~57% | ~32% | | | HC-SMoE | ~60% | ~57% | ~44% | | | M-SMoE | ~60% | ~55% | ~30% | | **Math** | REAP (ours) | ~88% | ~88% | ~86% | | | EAN | ~88% | ~87% | ~81% | | | Frequency | ~88% | ~87% | ~61% | | | HC-SMoE | ~88% | ~87% | ~70% | | | M-SMoE | ~88% | ~86% | ~46% | | **Creative Writing** | REAP (ours) | ~84% | ~83% | ~75% | | | EAN | ~84% | ~82% | ~70% | | | Frequency | ~84% | ~79% | ~60% | | | HC-SMoE | ~84% | ~78% | ~59% | | | M-SMoE | ~84% | ~78% | ~39% | | **MC** | REAP (ours) | ~75% | ~68% | ~56% | | | EAN | ~75% | ~63% | ~51% | | | Frequency | ~75% | ~65% | ~52% | | | HC-SMoE | ~75% | ~70% | ~56% | | | M-SMoE | ~75% | ~59% | ~44% | --- ### Chart 2: Qwen3 – 30B – A3B **Y-Axis:** Mean Accuracy (%) (Range: 0-90) **X-Axis:** Tasks (Coding, Math, Creative Writing, MC) **Legend:** (Same as Chart 1) **Data Table for Qwen3 – 30B – A3B:** | Task | Method | 0% Baseline | 50% Compression | 25% Compression | | :--- | :--- | :--- | :--- | :--- | | **Coding** | REAP (ours) | ~58% | ~58% | ~56% | | | EAN | ~58% | ~58% | ~55% | | | Frequency | ~58% | ~58% | ~47% | | | HC-SMoE | ~58% | ~58% | ~38% | | | M-SMoE | ~58% | ~56% | ~41% | | **Math** | REAP (ours) | ~89% | ~89% | ~88% | | | EAN | ~89% | ~88% | ~86% | | | Frequency | ~89% | ~88% | ~87% | | | HC-SMoE | ~89% | ~85% | ~73% | | | M-SMoE | ~89% | ~84% | ~83% | | **Creative Writing** | REAP (ours) | ~81% | ~81% | ~72% | | | EAN | ~81% | ~81% | ~70% | | | Frequency | ~81% | ~81% | ~68% | | | HC-SMoE | ~81% | ~81% | ~50% | | | M-SMoE | ~81% | ~81% | ~73% | | **MC** | REAP (ours) | ~72% | ~68% | ~55% | | | EAN | ~72% | ~63% | ~53% | | | Frequency | ~72% | ~63% | ~52% | | | HC-SMoE | ~72% | ~67% | ~54% | | | M-SMoE | ~72% | ~56% | ~45% | </details> Figure 2: GLM-4.5-Air and Qwen3-30B accuracy vs. task type. REAP offers significant improvements compared to other methods at 50% compression. Note the significant performance drop for merging methods on generative tasks (Coding, Math, Creative Writing) compared to their relative strength on MC. Expert pruning at scale. To asses whether pruning remains viable at scale, we prune Qwen3-Coder-480B and Kimi-K2-Instruct. On MC questions, REAP outperforms other pruning methods. On non-agentic coding tasks, REAP achieves near-lossless accuracy with a 0.20% and 1.4% mean decrease in accuracy compared to baseline at 25% and 50%, respectively, outperforming EAN and frequency-based pruning, particularly at 50% compression. On the challenging SWE-Bench task, both REAP and EAN maintain high accuracy at 25% and 50% compression, with some scores slightly exceeding the baseline. On tool use, EAN and REAP are comparable, with REAP slightly outperforming at 50% compression with a mean decrease in accuracy of 5.9% versus 6.2% for EAN. Frequency-based pruning suffers from a sharp degradation in quality at 50% compression, highlighting the importance of pruning saliency criteria which consider expert activations. Scaling the pruning methods is relatively trivial. Unlike HC-SMoE, calibration for pruning does not require recording activations from every expert for every token, facilitating efficient calibration. Further, pruning can be easily applied to quantized models without any additional steps required to reconcile block scales or re-quantize following compression. <details> <summary>2510.13999v1/x7.png Details</summary>  ### Visual Description ### Technical Description of the Box Plot This image is a box plot illustrating the distribution of "N-gram diversity" across different "N-gram size" categories for four distinct models or methods. #### Axis Information * **Y-axis:** Labeled "N-gram diversity". The axis is marked with numerical values at intervals of 0.2, specifically: 0.2, 0.4, 0.6, 0.8, and 1.0. * **X-axis:** Labeled "N-gram size". The axis is marked with three discrete categories: 2, 3, and 4. #### Legend Information A legend is located in the bottom center-right of the plot, defining the color coding for the four models: * **Gray:** Baseline * **Dark Blue:** REAP * **Light Blue:** M-SMoE * **Gold/Yellow:** HC-SMoE #### Data Analysis by N-gram Size For each N-gram size (2, 3, and 4), four box plots are presented, corresponding to the four models defined in the legend. Each box plot displays the median, interquartile range (IQR), whiskers (representing the range of the data within 1.5 times the IQR from the quartiles), and individual outlier points (represented by open circles). **1. N-gram size: 2** * **Baseline (Gray):** The median is approximately 0.83. The IQR ranges from ~0.79 to ~0.86. Whiskers extend from ~0.68 to ~0.91. Several outliers are present below the lower whisker, down to ~0.59. * **REAP (Dark Blue):** The median is approximately 0.82. The IQR ranges from ~0.78 to ~0.84. Whiskers extend from ~0.69 to ~0.90. Outliers are present below the lower whisker, down to ~0.61. * **M-SMoE (Light Blue):** The median is approximately 0.78. The IQR ranges from ~0.70 to ~0.83. Whiskers extend from ~0.52 to ~0.91. Outliers are present below the lower whisker, down to ~0.23. * **HC-SMoE (Gold):** The median is approximately 0.74. The IQR ranges from ~0.66 to ~0.80. Whiskers extend from ~0.48 to ~0.88. Outliers are present below the lower whisker, down to ~0.28. **2. N-gram size: 3** * **Baseline (Gray):** The median is approximately 0.93. The IQR ranges from ~0.90 to ~0.95. Whiskers extend from ~0.84 to ~0.98. Outliers are present below the lower whisker, down to ~0.69. * **REAP (Dark Blue):** The median is approximately 0.92. The IQR ranges from ~0.89 to ~0.94. Whiskers extend from ~0.84 to ~0.98. Outliers are present below the lower whisker, down to ~0.72. * **M-SMoE (Light Blue):** The median is approximately 0.89. The IQR ranges from ~0.85 to ~0.93. Whiskers extend from ~0.72 to ~0.98. Outliers are present below the lower whisker, down to ~0.51. * **HC-SMoE (Gold):** The median is approximately 0.87. The IQR ranges from ~0.78 to ~0.92. Whiskers extend from ~0.58 to ~0.98. Outliers are present below the lower whisker, down to ~0.46. **3. N-gram size: 4** * **Baseline (Gray):** The median is approximately 0.97. The IQR ranges from ~0.95 to ~0.98. Whiskers extend from ~0.89 to ~1.0. Outliers are present below the lower whisker, down to ~0.75. * **REAP (Dark Blue):** The median is approximately 0.96. The IQR ranges from ~0.94 to ~0.98. Whiskers extend from ~0.89 to ~1.0. Outliers are present below the lower whisker, down to ~0.79. * **M-SMoE (Light Blue):** The median is approximately 0.94. The IQR ranges from ~0.91 to ~0.97. Whiskers extend from ~0.80 to ~1.0. Outliers are present below the lower whisker, down to ~0.26. * **HC-SMoE (Gold):** The median is approximately 0.92. The IQR ranges from ~0.84 to ~0.96. Whiskers extend from ~0.68 to ~1.0. Outliers are present below the lower whisker, down to ~0.36. #### Overall Trends * N-gram diversity generally increases as the N-gram size increases from 2 to 4 for all models. * For a given N-gram size, the Baseline and REAP models tend to have higher median N-gram diversity compared to M-SMoE and HC-SMoE. * M-SMoE and HC-SMoE models show a wider range of diversity values (larger IQR and whisker span) and more pronounced outliers, particularly at the lower end of the diversity scale, compared to Baseline and REAP. </details> (a) N-Gram diversity <details> <summary>2510.13999v1/x8.png Details</summary>  ### Visual Description # Technical Description of the Violin Plot ## Chart Overview The image is a violin plot comparing the distribution of "Cross perplexity" across three different "Generator model" categories. The plot combines a box plot with a kernel density estimation to show the distribution shape of the data. ## Axes and Labels * **Y-axis:** The vertical axis is labeled "Cross perplexity". It has a logarithmic scale with major tick marks at **10⁰** and **10¹**. Horizontal gridlines are present at these major ticks and at intermediate logarithmic intervals (e.g., 2, 3, 4, 5, 6, 7, 8, 9, 20, 30, etc., though not explicitly labeled). * **X-axis:** The horizontal axis is labeled "Generator model". It displays three categorical labels: **REAP**, **M-SMoE**, and **HC-SMoE**. ## Data Analysis by Category The plot shows three violin shapes, each corresponding to a generator model. Inside each violin is a box plot with a white dot representing the median, a thick bar representing the interquartile range (IQR), and thin lines (whiskers) extending to the rest of the distribution, excluding outliers. ### 1. REAP (Dark Blue Violin) * **Distribution Shape:** The distribution is relatively narrow and centered at a lower cross perplexity value compared to the other two models. * **Median:** The median cross perplexity is approximately **2**. * **Interquartile Range (IQR):** The IQR spans from roughly **1.5** to **3**. * **Range:** The whiskers extend from a minimum of about **1** to a maximum of about **6**. ### 2. M-SMoE (Light Blue Violin) * **Distribution Shape:** The distribution is wider than REAP and centered at a higher value. It shows a broader range of cross perplexity values. * **Median:** The median cross perplexity is approximately **3**. * **Interquartile Range (IQR):** The IQR spans from roughly **2** to **5**. * **Range:** The whiskers extend from a minimum of about **0.6** to a maximum of about **10**. ### 3. HC-SMoE (Gold/Yellow Violin) * **Distribution Shape:** The distribution is similar in width to M-SMoE but is centered at a slightly higher value. * **Median:** The median cross perplexity is approximately **3.5**. * **Interquartile Range (IQR):** The IQR spans from roughly **2.5** to **6**. * **Range:** The whiskers extend from a minimum of about **0.7** to a maximum of about **15**. ## Key Trends * **Median Cross Perplexity:** The median cross perplexity increases in the order: REAP < M-SMoE < HC-SMoE. * **Distribution Spread:** REAP has the narrowest distribution of cross perplexity values, indicating more consistent performance. M-SMoE and HC-SMoE have wider distributions, showing greater variability in their cross perplexity. * **Overall Range:** HC-SMoE has the highest maximum cross perplexity value, while M-SMoE has the lowest minimum value. REAP's range is the most contained. </details> (b) Cross perplexity <details> <summary>2510.13999v1/x9.png Details</summary>  ### Visual Description ## Chart Description This is a line chart plotting **JSD** on the y-axis against **Token Position** on the x-axis. The chart compares three different models: **REAP**, **M-SMoE**, and **HC-SMoE**. Each model is represented by a colored line with a corresponding shaded area indicating the confidence interval or standard deviation. ### Axes * **Y-axis:** Labeled "JSD". The scale ranges from 0.0 to 0.7, with major tick marks at 0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, and 0.7. * **X-axis:** Labeled "Token Position". The scale ranges from 0 to 25, with major tick marks at 0, 10, and 20. ### Legend The legend is located in the bottom right corner of the chart and defines the colors for each model: * **REAP:** Dark blue line. * **M-SMoE:** Light blue line. * **HC-SMoE:** Gold/yellow line. ### Data Trends All three models show an increasing trend in JSD as the Token Position increases, starting from a JSD of 0.0 at Token Position 0. 1. **REAP (Dark Blue):** * The line starts at (0, 0) and rises steeply to approximately 0.3 at Token Position 2. * The increase becomes more gradual, reaching a JSD of approximately 0.6 at Token Position 25. * The shaded area around the line indicates a moderate level of variability, roughly ±0.05 JSD units. 2. **M-SMoE (Light Blue):** * The line starts at (0, 0) and shows the steepest initial rise among the three models, reaching approximately 0.45 at Token Position 2. * It continues to rise, reaching a JSD of approximately 0.62 at Token Position 25. * The shaded area is wider than REAP's, indicating higher variability, roughly ±0.05-0.08 JSD units. 3. **HC-SMoE (Gold/Yellow):** * The line starts at (0, 0) and rises steeply to approximately 0.35 at Token Position 2. * It continues to rise, generally staying between the REAP and M-SMoE lines, and reaches the highest JSD of approximately 0.65 at Token Position 25. * The shaded area is similar in width to M-SMoE's, indicating comparable variability. Overall, M-SMoE and HC-SMoE show a faster initial increase in JSD compared to REAP. HC-SMoE achieves the highest final JSD value at Token Position 25, followed by M-SMoE and then REAP. </details> (c) Completion logit JSD <details> <summary>2510.13999v1/x10.png Details</summary>  ### Visual Description Here is a detailed technical description of the image. ### Chart Overview The image is a line chart plotting two different metrics, "SV Align." (Singular-vector alignment) and "L2 Distance," across different layers of a model. The chart includes a legend specifying the line styles for the metrics and colors for different "Expert clusters." ### Axes and Labels * **X-Axis:** * **Label:** "Layer" * **Markers:** 0, 8, 16, 24, 32, 40. The axis extends from 0 to approximately 48. * **Y-Axis:** * **Label:** "SV Align. ↑ / L2 Distance ↓" * The upward arrow (↑) next to "SV Align." indicates that higher values are better or represent greater alignment. * The downward arrow (↓) next to "L2 Distance" indicates that lower values are better or represent smaller distance. * **Markers:** 0.0, 0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.4. The axis range is from 0.0 to 1.5. ### Legend The legend is positioned within the chart area, approximately in the center. It is divided into two sections: 1. **Dist. Type** (defines line style): * **Solid Line (—):** "Singular-vector alignment" * **Dashed Line (---):** "L2 Distance" 2. **Expert clusters** (defines line color): * **Grey:** "Base to IFT" * **Yellow/Gold:** "HC-SMoE" * **Light Blue:** "M-SMoE" * **Purple/Magenta:** "M-SMoE - permuted" ### Data Series Analysis The chart plots several lines representing combinations of "Dist. Type" and "Expert clusters." #### Singular-vector alignment (Solid Lines) * **Base to IFT (Grey, Solid):** This line is perfectly flat and constant at a value of **1.0** across all layers from 0 to ~48. * **M-SMoE (Light Blue, Solid):** This line is perfectly flat and constant at a value of **0.0** across all layers. * **M-SMoE - permuted (Purple, Solid):** This line is relatively flat, maintaining a low value of approximately **0.02** across all layers. * **HC-SMoE (Yellow, Solid):** A solid yellow line is not visible on the chart, suggesting it is either not plotted or overlaps perfectly with another line (likely the M-SMoE line at 0.0). #### L2 Distance (Dashed Lines) * **Base to IFT (Grey, Dashed):** This line starts at a low value of approximately **0.05** at Layer 0 and shows a gradual, slight upward trend, reaching approximately **0.10** by Layer 48. * **The three "SMoE" cluster variants show high and relatively stable L2 distances, with slight fluctuations and shaded regions indicating variability:** * **M-SMoE (Light Blue, Dashed):** This line is the highest among the dashed lines, fluctuating around a value of approximately **1.41** across all layers. * **HC-SMoE (Yellow, Dashed):** This line is situated just below the M-SMoE line, fluctuating around a value of approximately **1.40**. * **M-SMoE - permuted (Purple, Dashed):** This line is the lowest of the three high-value dashed lines, fluctuating around a value of approximately **1.38**. ### Summary of Trends * For **Singular-vector alignment (higher is better)**, the "Base to IFT" model shows perfect alignment (1.0), while the "M-SMoE" and "M-SMoE - permuted" models show near-zero alignment. * For **L2 Distance (lower is better)**, the "Base to IFT" model shows a very low distance that slightly increases with depth. In contrast, all "SMoE" variants exhibit a consistently high L2 distance across all layers, with "M-SMoE" having the highest distance and "M-SMoE - permuted" having the lowest among them. </details> (d) Expert distance Figure 3: (3(a)) & (3(b)) N-Gram diversity and cross-perplexity of compressed Qwen3-30B-A3B models at 50% compression, respectively. (3(c)) JSD of compressed and baseline model logits vs. completion token position for Qwen3-30B-A3B at 50% compression. Initially, all compressed models share close alignment with the baseline model. However, as the completion token position increases the merged models diverge from the baseline more rapidly than the REAP pruned model. (3(d)) The mean relative L2-distance and singular-vector alignment between Qwen3-30B expert weights at 50% compression. Expert merging is more challenging than model merging due to large distances between experts in weight space and low singular-vector alignment. Quantifying merged MoE generation quality. While merged expert SMoEs offer reasonable quality for discriminative tasks such as MC question and answering, they fail to remain competitive on generative tasks. To help explain the performance gap of merged models between discriminative and generative tasks, we perform an analysis of the compressed model outputs and compare with REAP pruned models. We prompt 50% compressed Qwen3-30B models with 100 questions randomly sampled from the evol-codealpaca dataset and record their outputs. In Figure ˜ 3(a), we measure the N-gram diversity and find that the merged models have significantly lower diversity across all N-gram sizes measured. In contrast, the REAP pruned model remains similar to the base model, albeit slightly less diverse. In Figure ˜ 3(b), we measure the perplexity of the text generated by the compressed models with the original baseline model. The text generated by the merged models has both a higher mean and higher variance than the pruned model generations, suggesting that the REAP pruned model outputs are more closely aligned to the original model. The alignment between the baseline and REAP pruned SMoEs is further supported by Figure ˜ 3(c), which plots the JSD of the compressed and baseline logits vs. output token position. The merged model logits diverge from the baseline more rapidly than the pruned model. The challenges of expert merging. Model merging has been widely adopted to facilitate LLM fine-tuning. Why does expert merging miss the mark? In addition to the loss of the router’s input-dependent modulation of experts explored in Section ˜ 3, we argue that the non-local nature of expert merging and high cardinality of expert clusters pose significant unresolved challenges. In Figure ˜ 3(d), we plot the mean relative L2-distance between experts clustered by HC-SMoE or M-SMoE and compare with the distance between expert weights from the pretrained to Instruct Fine-Tuned (IFT) checkpoints. We find that the distance between clustered experts within the same layer greatly exceeds that of experts in the IFT checkpoint after fine-tuning. Ito et al. (2024) found that weight matching permutations improved alignment of parameters’ singular vectors. Following their approach, we decompose expert weights with Singular Value Decomposition (SVD) and plot the singular-vector alignment in Figure ˜ 3(d). Even after applying weight matching permutations, the M-SMoE expert clusters remain far apart both in weight space and singular-vector alignment. The relatively poorly aligned experts highlight the considerable challenge of coherently merging their parameters. When merging works well, it’s more closely related to pruning than one might expect. In Figure ˜ 6(a), we depict the frequency of singleton clusters — clusters containing a single expert — for both HC-SMoE and M-SMoE. A singleton cluster is directly analogous to an expert that remains after pruning. We find that HC-SMoE in particular has a high prevalence of singleton clusters, leaving important experts unadulterated and compressing the rest into a few mega -clusters containing tens of experts. This is particularly true of the high granularity models which contain more experts per layer. We hypothesize that the cardinality of these mega-clusters poses a challenge for existing merging algorithms and test this intuition in Figure ˜ 6(b). Unfortunately, even modest restrictions of the maximum cluster size to 32 — half the number of experts to compress — results in large decreases in model quality on coding tasks. The importance of domain-specific calibration. In Figure ˜ A7, we plot the code generation accuracy of the various compression methods and models when calibrated on either c4 or evol-codealpaca. The difference is stark, c4 calibration results in a collapse in accuracy, with several compressed model instances failing to produce coherent outputs, resulting in 0% accuracy. In Figure ˜ A8, we compare the accuracy of compressed Qwen3-30B models calibrated with either domain-specific data or the combined calibration data across all generative tasks. The domain-specific calibrated models achieve significantly higher accuracy, especially at 50% compression. 6 Discussion Similar to prior work, we find that expert merging performs reasonably well on MC benchmarks. This may be because MC tasks only require a discriminative function that can be approximated by an average expert. In contrast, merging fails to maintain model quality on generative tasks, particularly at 50% compression. Generative tasks require auto-regressive generation, a capability that is lost when the router’s fine-grained control is removed. Compared to expert pruning, merging is less consistent, exhibiting higher variance across models and compression ratios. The outputs of expert merged models are more repetitive and less closely aligned with the base model compared with pruned models. Taken together, these observations are direct evidence of alterations to the functional manifold of the SMoE layers discussed in Section ˜ 3.3 stemming from the loss of the router’s input-dependent control over the experts and subsequent introduction of novel functions due to tying of the merged expert gates. Overall, expert pruned models offer consistently higher accuracy than merged models on generative tasks. REAP is a robust pruning criterion that generalizes across a wide array of SMoE architectures, compression ratios, and generative tasks. By taking into consideration both the router gate-values and expert activation norms, REAP prunes the experts which contribute the least to each layers output on a per-token average, regardless of usage frequency. REAP is scalable, achieving near-lossless compression on coding tasks with Qwen3-Coder-480B and Kimi-K2. The successes of REAP highlights the crucial importance of preserving coordination between the router and experts. Compression methods which impair the router’s ability to independently modulate expert outputs or tie gate-values are less likely to succeed. Finally, this work highlights the importance of comprehensive downstream evaluations and the significant challenges involved with evaluating LLMs. Discriminative metrics such as perplexity and log-likelihood based MC benchmarks are not necessarily good proxies for generative model quality. 7 Conclusion Our analysis of current SMoE expert merging techniques finds that the router’s loss of independent control over experts results in functional subspace collapse. In contrast, expert pruning produces a coordinate subspace of the original layer which maintains the topology of the functional manifold. Based on our findings that the coordination between the router and experts is fundamental, we introduce REAP, a novel expert pruning method which prunes experts that contribute the least to the layer’s output. Empirically, we demonstrate that REAP retains remarkably high accuracy on an wide array of generative tasks across a diverse set of model architectures. We hope that this work inspires further compression techniques for SMoEs and facilitates the deployment of accurate, domain-specific models in resource constrained settings. Acknowledgments We would like to acknowledge the helpful feedback of Mohammed Adnan and Rohan Jain. ML and YI gratefully acknowledge the support of Alberta Innovates (ALLRP-577350-22, ALLRP-222301502), the Natural Sciences and Engineering Research Council of Canada (NSERC) (RGPIN-2022-03120, DGECR-2022-00358), and Defence Research and Development Canada (DGDND-2022-03120). ML and YI are grateful for computational resources made available to us by the Digital Research Alliance of Canada. YI is supported by a Schulich Research Chair. Ethics Statement This work research focused on the algorithmic compression of SMoE models and does not involve the use of human subjects, personally identifiable information, or sensitive data. The datasets used for calibration and evaluation (e.g., c4, evol-codealpaca) are publicly available. Our aim is enable the use of large-scale SMoE models in resource constrained settings. However, we acknowledge that compression techniques such as REAP could potentially facilitate deployment of models for malicious purposes. Further, our compression methods are applied to pre-trained models and any biases related to fairness, discrimination, or representation inherent in the original models may be present in their compressed versions. We make no attempt in this work to mitigate these potential biases. The primary contribution of this paper is technical, and we do not foresee any new, direct ethical concerns arising from our proposed methodology beyond those already associated with the deployment of large language models. Reproducibility Statement We are committed to ensuring the reproducibility of our research. We have open-sourced our code and released select compressed model checkpoints to facilitate further research on compressed SMoEs. REAP is formally described in Section 4. The baseline methods we compare against, including frequency-based pruning, EAN, M-SMoE, and HC-SMoE, are formally defined in Appendix D. Section 5 provides a detailed description of our experimental setup, including the specific models used, the calibration and evaluation datasets, and the implementation details for all compression experiments. Further evaluation details are provided in Appendix C. References - Ainsworth et al. (2023) Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In Proceedings of the Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=CQsmMYmlP5T. - Allen Institute for AI (2024) Allen Institute for AI. allenai/c4 · Datasets at Hugging Face, August 2024. URL https://huggingface.co/datasets/allenai/c4. - Allen Institute for AI (2025) Allen Institute for AI. allenai/tulu-3-sft-personas-math · Datasets at Hugging Face, 2025. URL https://huggingface.co/datasets/allenai/tulu-3-sft-personas-math. - Ansel et al. (2024) Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, CK Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Michael Suo, Phil Tillet, Eikan Wang, Xiaodong Wang, William Wen, Shunting Zhang, Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. In 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’24). ACM, April 2024. doi: 10.1145/3620665.3640366. URL https://docs.pytorch.org/assets/pytorch2-2.pdf. - Artificial Analysis (2025) Artificial Analysis. Artificial analysis intelligence benchmarking methodology. https://artificialanalysis.ai/methodology/intelligence-benchmarking, September 2025. Version 3.0. - Baidu (2025) Baidu. Ernie 4.5 technical report, 2025. URL https://yiyan.baidu.com/blog/publication/ERNIE_Technical_Report.pdf. - Balmau et al. (2025) Oana Balmau, Anne-Marie Kermarrec, Rafael Pires, André Loureiro Espírito Santo, Martijn de Vos, and Milos Vujasinovic. Accelerating moe model inference with expert sharding. In Proceedings of the 5th Workshop on Machine Learning and Systems, pp. 192–199, 2025. - Barres et al. (2025) Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. $\tau^{2}$ -bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982, 2025. - Bentivogli et al. (2009) Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. TAC, 7(8):1, 2009. - Chandak et al. (2025) Nikhil Chandak, Shashwat Goel, Ameya Prabhu, Moritz Hardt, and Jonas Geiping. Answer Matching Outperforms Multiple Choice for Language Model Evaluation, July 2025. URL http://arxiv.org/abs/2507.02856. arXiv:2507.02856 [cs]. - Chaudhary (2023) Sahil Chaudhary. Code alpaca: An instruction-following llama model for code generation. https://github.com/sahil280114/codealpaca, 2023. - Chen et al. (2025) I-Chun Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chao, Yen-Chang Hsu, and Chun-Yi Lee. Retraining-free merging of sparse moe via hierarchical clustering. In Proceedings of the Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=hslOzRxzXL. - Chen et al. (2022) Tianyu Chen, Shaohan Huang, Yuan Xie, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, and Furu Wei. Task-Specific Expert Pruning for Sparse Mixture-of-Experts, June 2022. URL http://arxiv.org/abs/2206.00277. arXiv:2206.00277 [cs]. - Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2924–2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1300. URL https://aclanthology.org/N19-1300/. - Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https://arxiv.org/abs/1803.05457. arXiv:1803.05457 [cs]. - Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168. arXiv:2110.14168 [cs]. - Dai et al. (2024) Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models, January 2024. URL http://arxiv.org/abs/2401.06066. arXiv:2401.06066 [cs] version: 1. - DeepSeek-AI et al. (2024) DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. DeepSeek-V3 Technical Report, December 2024. URL http://arxiv.org/abs/2412.19437. arXiv:2412.19437 [cs] version: 1. - Fedus et al. (2022) William Fedus, Barret Zoph, and Noam Shazeer. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, June 2022. URL http://arxiv.org/abs/2101.03961. arXiv:2101.03961 [cs]. - Gao (2021) Leo Gao. Multiple Choice Normalization in LM Evaluation, October 2021. URL https://blog.eleuther.ai/multiple-choice-normalization/. - Gao et al. (2023) Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL https://github.com/EleutherAI/lm-evaluation-harness. - Garipov et al. (2018) Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018. - Gu et al. (2025) Hao Gu, Wei Li, Lujun Li, Zhu Qiyuan, Mark G. Lee, Shengjie Sun, Wei Xue, and Yike Guo. Delta decompression for moe-based LLMs compression. In Proceedings of the Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=ziezViPoN1. - He et al. (2025a) Shwai He, Daize Dong, Liang Ding, and Ang Li. Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques, March 2025a. URL http://arxiv.org/abs/2406.02500. arXiv:2406.02500 [cs] version: 3. - He et al. (2025b) Yifei He, Yang Liu, Chen Liang, and Hany Hassan Awadalla. Efficiently Editing Mixture-of-Experts Models with Compressed Experts, March 2025b. URL http://arxiv.org/abs/2503.00634. arXiv:2503.00634 [cs]. - Hendrycks et al. (2021a) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In Proceedings of the Ninth International Conference on Learning Representations, 2021a. URL https://openreview.net/forum?id=d7KBjmI3GmQ. - Hendrycks et al. (2021b) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021b. URL https://openreview.net/forum?id=7Bywt2mQsCe. - Huang et al. (2025) Wei Huang, Yue Liao, Jianhui Liu, Ruifei He, Haoru Tan, Shiming Zhang, Hongsheng Li, Si Liu, and XIAOJUAN QI. Mixture compressor for mixture-of-experts LLMs gains more. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=hheFYjOsWO. - Ilharco et al. (2023) Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=6t0Kwf8-jrj. - Ito et al. (2024) Akira Ito, Masanori Yamada, and Atsutoshi Kumagai. Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methods. In Proceedings of the Forty-Second International Conference on Machine Learning, October 2024. URL https://openreview.net/forum?id=lYRkGZZi9D. - Jain et al. (2025) Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In Proceedings of the Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=chfJJYC3iL. - Jaiswal et al. (2025) Ajay Jaiswal, Jianyu Wang, Yixiao Li, Pingzhi Li, Tianlong Chen, Zhangyang Wang, Chong Wang, Ruoming Pang, and Xianzhi Du. Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations, April 2025. URL http://arxiv.org/abs/2504.05586. arXiv:2504.05586 [cs]. - Jaiswal et al. (2024) Ajay Kumar Jaiswal, Zhe Gan, Xianzhi Du, Bowen Zhang, Zhangyang Wang, and Yinfei Yang. Compressing llms: The truth is rarely pure and never simple. In Proceedings of the Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=B9klVS7Ddk. - Jiang et al. (2024) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of Experts, January 2024. URL http://arxiv.org/abs/2401.04088. arXiv:2401.04088 [cs]. - Jimenez et al. (2024) Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66. - Kimi Team et al. (2025) Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Hongcheng Gao, Peizhong Gao, Tong Gao, Xinran Gu, Longyu Guan, Haiqing Guo, Jianhang Guo, Hao Hu, Xiaoru Hao, Tianhong He, Weiran He, Wenyang He, Chao Hong, Yangyang Hu, Zhenxing Hu, Weixiao Huang, Zhiqi Huang, Zihao Huang, Tao Jiang, Zhejun Jiang, Xinyi Jin, Yongsheng Kang, Guokun Lai, Cheng Li, Fang Li, Haoyang Li, Ming Li, Wentao Li, Yanhao Li, Yiwei Li, Zhaowei Li, Zheming Li, Hongzhan Lin, Xiaohan Lin, Zongyu Lin, Chengyin Liu, Chenyu Liu, Hongzhang Liu, Jingyuan Liu, Junqi Liu, Liang Liu, Shaowei Liu, T. Y. Liu, Tianwei Liu, Weizhou Liu, Yangyang Liu, Yibo Liu, Yiping Liu, Yue Liu, Zhengying Liu, Enzhe Lu, Lijun Lu, Shengling Ma, Xinyu Ma, Yingwei Ma, Shaoguang Mao, Jie Mei, Xin Men, Yibo Miao, Siyuan Pan, Yebo Peng, Ruoyu Qin, Bowen Qu, Zeyu Shang, Lidong Shi, Shengyuan Shi, Feifan Song, Jianlin Su, Zhengyuan Su, Xinjie Sun, Flood Sung, Heyi Tang, Jiawen Tao, Qifeng Teng, Chensi Wang, Dinglu Wang, Feng Wang, Haiming Wang, Jianzhou Wang, Jiaxing Wang, Jinhong Wang, Shengjie Wang, Shuyi Wang, Yao Wang, Yejie Wang, Yiqin Wang, Yuxin Wang, Yuzhi Wang, Zhaoji Wang, Zhengtao Wang, Zhexu Wang, Chu Wei, Qianqian Wei, Wenhao Wu, Xingzhe Wu, Yuxin Wu, Chenjun Xiao, Xiaotong Xie, Weimin Xiong, Boyu Xu, Jing Xu, Jinjing Xu, L. H. Xu, Lin Xu, Suting Xu, Weixin Xu, Xinran Xu, Yangchuan Xu, Ziyao Xu, Junjie Yan, Yuzi Yan, Xiaofei Yang, Ying Yang, Zhen Yang, Zhilin Yang, Zonghan Yang, Haotian Yao, Xingcheng Yao, Wenjie Ye, Zhuorui Ye, Bohong Yin, Longhui Yu, Enming Yuan, Hongbang Yuan, Mengjie Yuan, Haobing Zhan, Dehao Zhang, Hao Zhang, Wanlu Zhang, Xiaobin Zhang, Yangkun Zhang, Yizhi Zhang, Yongting Zhang, Yu Zhang, Yutao Zhang, Yutong Zhang, Zheng Zhang, Haotian Zhao, Yikai Zhao, Huabin Zheng, Shaojie Zheng, Jianren Zhou, Xinyu Zhou, Zaida Zhou, Zhen Zhu, Weiyu Zhuang, and Xinxing Zu. Kimi K2: Open Agentic Intelligence, July 2025. URL http://arxiv.org/abs/2507.20534. arXiv:2507.20534 [cs]. - Koishekenov et al. (2023) Yeskendir Koishekenov, Alexandre Berard, and Vassilina Nikoulina. Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Model. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3567–3585, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.198. URL https://aclanthology.org/2023.acl-long.198/. - Lambert et al. (2025) Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Validad Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Christopher Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers in open language model post-training. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=i1uGbfHHpH. - Lepikhin et al. (2021) Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In Proceedings of the Ninth International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=qrwe7XHTmYb. - Li et al. (2023) Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy. In Proceedings of the Twelfth International Conference on Learning Representations, October 2023. URL https://openreview.net/forum?id=eFWG9Cy3WK. - Lightman et al. (2023) Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2023. - Lin et al. (2024) Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild. In Proceedings of the Thirteenth International Conference on Learning Representations, October 2024. URL https://openreview.net/forum?id=MKEHCx25xp. - Liu et al. (2024a) Enshu Liu, Junyi Zhu, Zinan Lin, Xuefei Ning, Matthew B. Blaschko, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs, July 2024a. URL http://arxiv.org/abs/2407.00945. arXiv:2407.00945 [cs]. - Liu et al. (2024b) James Liu, Pragaash Ponnusamy, Tianle Cai, Han Guo, Yoon Kim, and Ben Athiwaratkun. Training-Free Activation Sparsity in Large Language Models, October 2024b. URL http://arxiv.org/abs/2408.14690. arXiv:2408.14690 [cs]. - Liu et al. (2025) Jiacheng Liu, Peng Tang, Wenfeng Wang, Yuhang Ren, Xiaofeng Hou, Pheng-Ann Heng, Minyi Guo, and Chao Li. A Survey on Inference Optimization Techniques for Mixture of Experts Models, January 2025. URL http://arxiv.org/abs/2412.14219. arXiv:2412.14219 [cs] version: 2. - Liu et al. (2023) Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558–21572, 2023. - Liu et al. (2024c) Zuxin Liu, Thai Quoc Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh R N, Liangwei Yang, Silvio Savarese, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, and Caiming Xiong. APIGen: Automated PIpeline for generating verifiable and diverse function-calling datasets. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024c. URL https://openreview.net/forum?id=Jfg3vw2bjx. - Lu et al. (2024) Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6159–6172, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.334. URL https://aclanthology.org/2024.acl-long.334/. - Luo et al. (2024) Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. In Proceedings of the Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=UnUwSIgK5W. - Meta AI Team (2025) Meta AI Team. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation, 2025. URL https://ai.meta.com/blog/llama-4-multimodal-intelligence/. - Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018. - ModelScope Team (2024) ModelScope Team. EvalScope: Evaluation framework for large models, 2024. URL https://github.com/modelscope/evalscope. - Muzio et al. (2024) Alexandre Muzio, Alex Sun, and Churan He. SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts, April 2024. URL http://arxiv.org/abs/2404.05089. arXiv:2404.05089 [cs]. - Neil Chowdhury et al. (2024) Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry. Introducing SWE-bench Verified, August 2024. URL https://openai.com/index/introducing-swe-bench-verified/. - Patil et al. (2024) Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive APIs. In Proceedings of the Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=tBRNC6YemY. - Patil et al. (2025) Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=2GmDdhBdDk. - Pritsker (2024) Jade Pritsker. euclaise/WritingPrompts_curated · Datasets at Hugging Face, December 2024. URL https://huggingface.co/datasets/euclaise/WritingPrompts_curated. - Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020. - RedHatAI (2025) RedHatAI. RedHatAI/Kimi-K2-Instruct-quantized.w4a16 · Hugging Face, September 2025. URL https://huggingface.co/RedHatAI/Kimi-K2-Instruct-quantized.w4a16. - Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. URL https://dl.acm.org/doi/abs/10.1145/3474381. - Salesforce (2025) Salesforce. Salesforce/xlam-function-calling-60k · Datasets at Hugging Face, May 2025. URL https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k. - Sharma et al. (2024) Ekansh Sharma, Daniel M. Roy, and Gintare Karolina Dziugaite. The Non-Local Model Merging Problem: Permutation Symmetries and Variance Collapse, October 2024. URL http://arxiv.org/abs/2410.12766. arXiv:2410.12766 [cs]. - Shazeer (2020) Noam Shazeer. GLU Variants Improve Transformer, February 2020. URL http://arxiv.org/abs/2002.05202. arXiv:2002.05202 [cs]. - Shazeer et al. (2017) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, January 2017. URL http://arxiv.org/abs/1701.06538. arXiv:1701.06538 [cs, stat]. - Tam (2023) Zhi Rui Tam. theblackcat102/evol-codealpaca-v1 · Datasets at Hugging Face, July 2023. URL https://huggingface.co/datasets/theblackcat102/evol-codealpaca-v1. - Yadav et al. (2023) Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES-merging: Resolving interference when merging models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=xtaX3WyCj1. - Yang et al. (2025a) An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 Technical Report, May 2025a. URL http://arxiv.org/abs/2505.09388. arXiv:2505.09388 [cs]. - Yang et al. (2024a) Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Yuanlin Duan, Wenqi Jia, Miao Yin, Yu Cheng, and Bo Yuan. MoE-i 2: Compressing mixture of experts models through inter-expert pruning and intra-expert low-rank decomposition. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 10456–10466, Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.612. URL https://aclanthology.org/2024.findings-emnlp.612/. - Yang et al. (2024b) John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In Proceedings of the Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL https://openreview.net/forum?id=mXpq6ut8J3. - Yang et al. (2025b) John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-bench/SWE-smith-trajectories · Datasets at Hugging Face, May 2025b. URL https://huggingface.co/datasets/SWE-bench/SWE-smith-trajectories. - Yang et al. (2025c) John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. SWE-smith: Scaling Data for Software Engineering Agents, May 2025c. URL http://arxiv.org/abs/2504.21798. arXiv:2504.21798 [cs]. - Yu et al. (2024) Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning, 2024. - Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Màrquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791–4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1472. URL https://aclanthology.org/P19-1472/. - Zeng et al. (2025) Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, Yean Cheng, Yifan An, Yilin Niu, Yuanhao Wen, Yushi Bai, Zhengxiao Du, Zihan Wang, Zilin Zhu, Bohan Zhang, Bosi Wen, Bowen Wu, Bowen Xu, Can Huang, Casey Zhao, Changpeng Cai, Chao Yu, Chen Li, Chendi Ge, Chenghua Huang, Chenhui Zhang, Chenxi Xu, Chenzheng Zhu, Chuang Li, Congfeng Yin, Daoyan Lin, Dayong Yang, Dazhi Jiang, Ding Ai, Erle Zhu, Fei Wang, Gengzheng Pan, Guo Wang, Hailong Sun, Haitao Li, Haiyang Li, Haiyi Hu, Hanyu Zhang, Hao Peng, Hao Tai, Haoke Zhang, Haoran Wang, Haoyu Yang, He Liu, He Zhao, Hongwei Liu, Hongxi Yan, Huan Liu, Huilong Chen, Ji Li, Jiajing Zhao, Jiamin Ren, Jian Jiao, Jiani Zhao, Jianyang Yan, Jiaqi Wang, Jiayi Gui, Jiayue Zhao, Jie Liu, Jijie Li, Jing Li, Jing Lu, Jingsen Wang, Jingwei Yuan, Jingxuan Li, Jingzhao Du, Jinhua Du, Jinxin Liu, Junkai Zhi, Junli Gao, Ke Wang, Lekang Yang, Liang Xu, Lin Fan, Lindong Wu, Lintao Ding, Lu Wang, Man Zhang, Minghao Li, Minghuan Xu, Mingming Zhao, Mingshu Zhai, Pengfan Du, Qian Dong, Shangde Lei, Shangqing Tu, Shangtong Yang, Shaoyou Lu, Shijie Li, Shuang Li, Shuang-Li, Shuxun Yang, Sibo Yi, Tianshu Yu, Wei Tian, Weihan Wang, Wenbo Yu, Weng Lam Tam, Wenjie Liang, Wentao Liu, Xiao Wang, Xiaohan Jia, Xiaotao Gu, Xiaoying Ling, Xin Wang, Xing Fan, Xingru Pan, Xinyuan Zhang, Xinze Zhang, Xiuqing Fu, Xunkai Zhang, Yabo Xu, Yandong Wu, Yida Lu, Yidong Wang, Yilin Zhou, Yiming Pan, Ying Zhang, Yingli Wang, Yingru Li, Yinpei Su, Yipeng Geng, Yitong Zhu, Yongkun Yang, Yuhang Li, Yuhao Wu, Yujiang Li, Yunan Liu, Yunqing Wang, Yuntao Li, Yuxuan Zhang, Zezhen Liu, Zhen Yang, Zhengda Zhou, Zhongpei Qiao, Zhuoer Feng, Zhuorui Liu, Zichen Zhang, Zihan Wang, Zijun Yao, Zikang Wang, Ziqiang Liu, Ziwei Chai, Zixuan Li, Zuodong Zhao, Wenguang Chen, Jidong Zhai, Bin Xu, Minlie Huang, Hongning Wang, Juanzi Li, Yuxiao Dong, and Jie Tang. GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models, August 2025. URL http://arxiv.org/abs/2508.06471. arXiv:2508.06471 [cs]. Appendix A Extension to Hierarchical Clustering While Theorem 1 analyses pairwise merging, practical implementations often employ hierarchical clustering to form groups of experts. Consider a cluster $C=\{f_{i_{1}},...,f_{i_{k}}\}$ of $k$ experts merged into a single representative $\tilde{f}_{C}$ . The original contribution of this cluster can be decomposed as: $$ \sum_{j\in C}g_{i_{j}}(x)f_{i_{j}}(x)=\left(\sum_{j\in C}g_{i_{j}}(x)\right)\cdot\underbrace{\sum_{j\in C}w_{j}(x)f_{i_{j}}(x)}_{\text{Dynamic, input-dependent mixture}} \tag{10} $$ where $w_{j}(x)=\frac{g_{i_{j}}(x)}{\sum_{l∈ C}g_{i_{l}}(x)}$ are the within-cluster mixing ratios that sum to 1. After hierarchical merging, the router must apply the summed gate $\sum_{j∈ C}g_{i_{j}}$ to a single, static cluster representative $\tilde{f}_{C}$ , typically computed as a weighted average of the cluster members based on calibration data. This induces an irreducible error: **Theorem 2 (Hierarchical clustering error)** *For a cluster $C$ merged into $\tilde{f}_{C}=\sum_{j∈ C}\alpha_{j}f_{i_{j}}$ with fixed weights $\alpha_{j}≥ 0$ , $\sum_{j}\alpha_{j}=1$ , the minimal $L^{2}$ error is: $$ \min_{\{\alpha_{j}\}}\left\|\sum_{j\in C}g_{i_{j}}f_{i_{j}}-\left(\sum_{j\in C}g_{i_{j}}\right)\tilde{f}_{C}\right\|^{2}=\mathbb{E}\left[\left(\sum_{j\in C}g_{i_{j}}\right)^{2}\right]\cdot\mathrm{Var}_{x}\left[\sum_{j\in C}w_{j}(x)f_{i_{j}}(x)\right] \tag{11} $$ The error grows with both the cluster’s total gate-value and the variance of the dynamic mixture that the cluster must approximate with a static representative.* Implications for cluster formation. The hierarchical error bound reveals a fundamental tension: - Large clusters ( $|C|$ large) aggregate more gate-value $\sum_{j∈ C}g_{i_{j}}$ , amplifying any approximation error - Diverse clusters (high $\|\Delta_{ij}\|$ for $i,j∈ C$ ) increase the variance term, as the static representative must approximate a wider range of functions - Imbalanced clustering (many singletons, few mega-clusters) combines the worst aspects: mega-clusters suffer severe collapse while singletons provide minimal compression Distance metrics like Euclidean distance that consider magnitude can exacerbate these issues by creating clusters based on norm similarity rather than functional role, potentially grouping experts with different specializations but similar scales. The resulting mega-clusters force the router to apply a single control signal to what were previously dozens of independently modulated experts, explaining the catastrophic functional collapse observed empirically in late layers where $\mathrm{Var}[w_{j}(x)]$ is highest. Appendix B Additional empirical evidence for loss of independent control In Figure ˜ 1(a), Qwen3’s layer 0 exemplifies the contraction of the functional output space by merging in early layers. The original 128 experts span from $-0.4$ to $1.0$ along PC1, pruning maintains this full range with 64 experts, while merging contracts the distribution to approximately $[-0.2,0.3]$ , a 5-fold reduction. This contraction is dramatic in late layers, where experts are more specialized. As depicted in Figure ˜ 4(f), the original 15 experts of Llama-4’s layer 47 occupy a vast, multi-modal space spanning PC1 coordinates from $-800$ to $600$ . Pruning preserves this remarkable diversity, with the 8 surviving experts distributed across the same multi-modal regions. However, merging induces a catastrophic collapse to a tiny cluster around coordinates $(200,0)$ , representing nearly two orders of magnitude reduction in functional diversity. This pattern intensifies with the number of experts: Qwen3’s layer 47 (Figure ˜ 1(b)) shows the most severe collapse, with 128 original experts spanning PC1 from $-200$ to $300$ reduced to a minute region after merging, while its 64 pruned experts maintain the original distribution’s full breadth. Manifold geometry preservation Across all models and layers, we observe a fundamental geometric principle: pruning preserves the topology of the functional manifold while merging fundamentally alters it. This distinction is most clearly visible in ERNIE’s representations (Figures 4(a) and 4(b)). In layer 1, the original 64 routed experts plus 2 shared experts form a characteristic curved structure with several outliers representing specialized experts. After pruning, the red points precisely overlay the gray ghost of the original distribution, including the outlier positions, demonstrating that each surviving expert maintains its exact functional role. The merged configuration, however, shows all experts collapsed into a tight cluster at the distribution’s centre, eliminating both the outliers and the manifold’s curvature. The preservation of manifold geometry under pruning reflects the mathematical structure of the operation: the pruned hypothesis class is a coordinate subspace of the original, with the router maintaining independent control over each surviving expert. The geometric collapse under merging visualizes the loss of independent control when gates $g_{i}$ and $g_{j}$ are tied into their sum $(g_{i}+g_{j})$ , the router can no longer independently modulate the two underlying functions, forcing the model to approximate the dynamic mixture $r(x)f_{i}(x)+(1-r(x))f_{j}(x)$ with a static expert $\tilde{f}_{\alpha}$ . Mixtral, with only 8 experts, provides an interesting edge case (Figures 4(c) and 4(d)). Even with fewer experts, the same geometric principles apply. Pruning maintains the convex hull of the original distribution while merging contracts it. The less dramatic collapse compared to models with more experts suggests that with fewer experts, each must remain more general, leading to lower $\|\Delta_{ij}\|^{2}$ (expert gap) and lower $\mathrm{Var}[r(x)]$ (policy variability), both factors in our irreducible error bound. <details> <summary>2510.13999v1/x11.png Details</summary>  ### Visual Description ## Technical Description of Scatter Plots The image presents a series of three scatter plots arranged horizontally, sharing a common legend and axes. The plots illustrate the distribution of data points across two principal components, labeled "PC1" and "PC2". ### Legend A legend is positioned at the top center of the image, defining three categories of data points: * **Original Experts:** Represented by red circular markers. * **Surviving:** Represented by blue circular markers. * **Merged:** Represented by green 'x' markers. ### Axes All three plots share the same axes and scales: * **Vertical Axis (Y-axis):** Labeled "PC2". Tick marks are placed at intervals of 1, specifically at -2, -1, 0, and 1. * **Horizontal Axis (X-axis):** Labeled "PC1". Tick marks are placed at intervals of 2, specifically at 0, 2, and 4. * A grid is overlaid on each plot, corresponding to the major tick marks. ### Plot 1 (Left Panel): Original Experts This plot displays the distribution of the "Original Experts" data points (red circles). * **Cluster:** A dense cluster of points is centered around the origin (PC1 ≈ 0, PC2 ≈ 0). * **Outliers:** Several outlier points are scattered away from the central cluster: * One point at approximately (PC1=2.2, PC2=1.6). * One point at approximately (PC1=0.8, PC2=0.6). * One point at approximately (PC1=-0.6, PC2=-1.8). * One point at approximately (PC1=3.9, PC2=-1.2). ### Plot 2 (Middle Panel): Surviving This plot shows the distribution of the "Surviving" data points (blue circles). * **Cluster:** Similar to the first plot, a dense cluster of points is centered around the origin. * **Outliers:** The outlier points from the "Original Experts" plot are also present, now colored blue, with some slight variations in position or additional points: * One point at approximately (PC1=2.2, PC2=1.6). * One point at approximately (PC1=0.8, PC2=0.6). * One point at approximately (PC1=-0.6, PC2=-1.8). * One point at approximately (PC1=3.8, PC2=-1.6). * An additional outlier is visible at approximately (PC1=-0.8, PC2=-2.5). ### Plot 3 (Right Panel): Merged This plot illustrates the "Merged" data points (green 'x' markers). * **Cluster:** A dense cluster of green 'x' markers is centered around the origin, corresponding to the central clusters in the previous two plots. * **Outliers:** * One green 'x' marker is located at approximately (PC1=0.6, PC2=-0.4). * The other outlier points seen in the "Original Experts" and "Surviving" plots are shown as faint, light grey circles, indicating they are not part of the "Merged" category. </details> (a) ERNIE-4.5-21B Layer 1 <details> <summary>2510.13999v1/x12.png Details</summary>  ### Visual Description Here is a detailed description of the information contained in the image. ### Overview The image presents three side-by-side scatter plots, sharing a common legend and axes. The plots illustrate the distribution of data points, labeled as different types of "Experts," in a two-dimensional space defined by principal components. ### Legend The legend is located at the top of the image and defines the markers used in the plots: * **Red Circle:** "Original Experts" * **Blue Circle:** "Surviving" * **Green 'x':** "Merged" ### Axes All three plots share the same axes and scales: * **Vertical Axis (Y-axis):** Labeled "PC2". The axis has markers at 0, 20, 40, 60, and 80. * **Horizontal Axis (X-axis):** Labeled "PC1". The axis has markers at -50 and 0. ### Plot Descriptions #### 1. Left Plot: "Original Experts" * **Data Displayed:** This plot shows the distribution of "Original Experts" using red circle markers. * **Key Trends & Data Points:** * The majority of the points are clustered tightly around the origin (PC1 ≈ 0, PC2 ≈ 0), roughly within the range of PC1 from -20 to +20 and PC2 from -10 to +10. * There are several notable outliers: * One point is located at approximately PC1 = 40, PC2 = 80. * Another point is at approximately PC1 = -10, PC2 = 30. * A point is situated around PC1 = 20, PC2 = 20. * A small group of points is located around PC1 = -70, PC2 = 20. #### 2. Middle Plot: "Surviving" * **Data Displayed:** This plot shows the "Surviving" experts using blue circle markers. The "Original Experts" from the left plot are shown in the background as faint grey circles for context. * **Key Trends & Data Points:** * The "Surviving" points are a subset of the "Original Experts." * They form a tight cluster around the origin, similar to the main cluster in the first plot, but with a slightly smaller spread (roughly PC1 from -10 to +20 and PC2 from -5 to +5). * Crucially, the outliers observed in the "Original Experts" plot (e.g., at PC2=80, PC2=30, etc.) are not present in the "Surviving" set. #### 3. Right Plot: "Merged" * **Data Displayed:** This plot shows the "Merged" experts using green 'x' markers. Similar to the middle plot, the "Original Experts" are shown in the background as faint grey circles. * **Key Trends & Data Points:** * The "Merged" points are also a subset of the "Original Experts." * They form a tight cluster around the origin, very similar in location and spread to the "Surviving" points (roughly PC1 from -5 to +15 and PC2 from -5 to +5). * Unlike the "Surviving" set, the "Merged" set includes one distinct outlier point located at approximately PC1 = 30, PC2 = 0. </details> (b) ERNIE-4.5-21B Layer 27 <details> <summary>2510.13999v1/x13.png Details</summary>  ### Visual Description ## Technical Description of Scatter Plots This image presents a series of three scatter plots arranged horizontally, sharing common axes and a legend. The plots visualize data points in a two-dimensional space defined by principal components. ### Legend and Axes * **Legend:** Located at the top of the figure, the legend defines three categories of data points: * **Original Experts:** Represented by red circles. * **Surviving:** Represented by blue circles. * **Merged:** Represented by green 'x' markers. * **Y-axis:** The vertical axis is labeled "PC2". It has major tick marks and labels at -0.1, 0.0, 0.1, and 0.2. * **X-axis:** The horizontal axis for each of the three plots is labeled "PC1". It has major tick marks and labels at -0.2, 0.0, and 0.2. * **Grid:** A faint grid is present in all three plots, corresponding to the major tick marks on both axes. ### Plot 1: Original Experts (Left) This plot displays the distribution of the "Original Experts" data points. * **Title:** The plot is implicitly titled "Original Experts" based on the legend and the data points shown. * **Data Points:** The plot contains 8 red circles representing the "Original Experts". Their approximate coordinates (PC1, PC2) are: * (-0.23, 0.00) * (-0.13, 0.04) * (-0.02, -0.11) * (-0.01, 0.00) * (0.01, -0.05) * (0.08, 0.21) * (0.09, -0.03) * (0.21, -0.06) ### Plot 2: Surviving (Middle) This plot highlights the "Surviving" data points. * **Title:** The plot is implicitly titled "Surviving". * **Data Points:** The plot shows 4 blue circles representing the "Surviving" experts. Faint grey circles are also visible, showing the positions of the other "Original Experts" for context. The approximate coordinates (PC1, PC2) of the blue "Surviving" points are: * (-0.01, -0.01) * (0.00, -0.06) * (0.11, -0.04) * (0.17, -0.06) ### Plot 3: Merged (Right) This plot highlights the "Merged" data points. * **Title:** The plot is implicitly titled "Merged". * **Data Points:** The plot shows 4 green 'x' markers representing the "Merged" experts. Faint grey circles are also visible, showing the positions of the "Original Experts" for context. The approximate coordinates (PC1, PC2) of the green "Merged" points are: * (-0.09, 0.00) * (0.06, -0.08) * (0.13, -0.03) * (0.23, 0.00) </details> (c) Mixtral-8x7B Layer 0 <details> <summary>2510.13999v1/x14.png Details</summary>  ### Visual Description Here is a detailed technical description of the image. ### **Overview** The image presents a series of three scatter plots arranged horizontally, sharing common X and Y axes. The plots illustrate the distribution of data points in a two-dimensional principal component space, labeled "PC1" and "PC2". Each plot shows a different subset or transformation of the data, distinguished by title, marker color, and marker type. ### **Common Chart Elements** * **Y-axis:** Labeled "**PC2**". The axis has major tick marks and labels at **-60**, **-40**, **-20**, **0**, and **20**. * **X-axis:** Labeled "**PC1**". The axis has major tick marks and labels at **-100**, **0**, and **100**. * **Grid:** A light gray grid is overlaid on all three plots, corresponding to the major tick marks of both axes. --- ### **Plot 1: Original Experts** * **Title:** "**Original Experts**" is displayed at the top center of the plot. * **Legend:** A red circle marker is shown to the left of the title. * **Data Points:** The plot contains several red circle markers representing the "Original Experts". * **Estimated Coordinates (PC1, PC2):** * (-170, 32) * (-75, -48) * (20, 22) * (28, 2) * (35, 6) * (22, -57) * (60, 25) * (90, 15) --- ### **Plot 2: Surviving** * **Title:** "**Surviving**" is displayed at the top center of the plot. * **Legend:** A blue circle marker is shown to the left of the title. * **Data Points:** The plot features blue circle markers representing the "Surviving" points. Faint gray circle markers are also visible in the background, corresponding to the positions of the "Original Experts" from the first plot, providing context. * **Estimated Coordinates of "Surviving" Points (Blue Circles) (PC1, PC2):** * (-85, -47) * (-20, -5) * (-5, 17) * (15, 2) --- ### **Plot 3: Merged** * **Title:** "**Merged**" is displayed at the top center of the plot. * **Legend:** A green 'x' marker is shown to the left of the title. * **Data Points:** The plot displays green 'x' markers representing the "Merged" points. Similar to the second plot, faint gray circle markers are visible in the background, showing the positions of the "Original Experts". * **Estimated Coordinates of "Merged" Points (Green 'x's) (PC1, PC2):** * (-75, 0) * (-25, -41) * (8, 7) * (25, 14) </details> (d) Mixtral-8x7B Layer 31 <details> <summary>2510.13999v1/x15.png Details</summary>  ### Visual Description Here is a detailed technical description of the image. ### **Overview** The image presents a series of three scatter plots arranged horizontally, sharing a common coordinate system. The plots illustrate the distribution of data points, likely representing "experts," in a two-dimensional principal component space defined by **PC1** (x-axis) and **PC2** (y-axis). ### **Common Plot Elements** * **Y-axis (PC2):** The vertical axis is labeled "PC2". Major tick marks and grid lines are present at **-0.01**, **0.00**, and **0.01**. The axis range extends approximately from -0.015 to 0.015. This label is explicitly shown only on the leftmost plot but applies to all three. * **X-axis (PC1):** The horizontal axis is labeled "PC1" for all three plots. Major tick marks and grid lines are present at **-0.05**, **0.00**, and **0.05**. The axis range extends approximately from -0.06 to 0.06. * **Grid:** A light gray grid is overlaid on each plot, corresponding to the major tick marks on both axes. --- ### **Plot 1: Original Experts** * **Title:** The plot is titled "**Original Experts**". * **Legend/Marker:** A sample pink circle marker is displayed next to the title. * **Data Representation:** The plot shows a distribution of **pink circle markers**. * **Data Points (Approximate Coordinates [PC1, PC2]):** * [-0.045, 0.006] * [-0.035, 0.010] * [-0.030, -0.012] * [-0.018, -0.001] * [-0.010, 0.003] * [-0.008, -0.002] * [-0.005, -0.002] * [-0.003, -0.012] * [0.000, -0.004] * [0.005, 0.002] * [0.015, -0.008] * [0.018, 0.003] * [0.020, -0.001] * [0.030, 0.010] * [0.035, 0.012] * [0.045, -0.010] --- ### **Plot 2: Surviving** * **Title:** The plot is titled "**Surviving**". * **Legend/Marker:** A sample blue circle marker is displayed next to the title. * **Data Representation:** The plot shows a subset of the original points as **blue circle markers**. Faint gray circle markers are also visible, representing the positions of all "Original Experts" from the first plot for context. * **Data Points (Surviving - Blue Circles, Approximate Coordinates [PC1, PC2]):** * [-0.040, 0.007] * [-0.035, 0.010] * [-0.025, -0.011] * [-0.015, -0.001] * [-0.008, 0.004] * [-0.005, -0.002] * [-0.002, -0.001] * [0.000, -0.002] --- ### **Plot 3: Merged** * **Title:** The plot is titled "**Merged**". * **Legend/Marker:** A sample green 'x' marker is displayed next to the title. * **Data Representation:** The plot shows a new set of points as **green 'x' markers**. Similar to the second plot, faint gray circle markers are visible in the background, showing the positions of the "Original Experts". * **Data Points (Merged - Green 'x' Markers, Approximate Coordinates [PC1, PC2]):** * [-0.048, 0.009] * [-0.025, -0.006] * [-0.005, 0.007] * [-0.002, 0.004] * [0.005, -0.001] * [0.010, 0.001] * [0.025, 0.013] * [0.055, 0.004] </details> (e) Llama-4 Layer 0 <details> <summary>2510.13999v1/x16.png Details</summary>  ### Visual Description ## Technical Description of Scatter Plots The image presents three side-by-side scatter plots, all sharing the same axes and scale. The plots visualize data points in a two-dimensional principal component space defined by PC1 and PC2. ### Common Axis Information * **Y-axis:** Labeled "PC2". Major tick marks and labels are present at -200, 0, and 200. The visible range extends from approximately -350 to 350. * **X-axis:** Labeled "PC1". Major tick marks and labels are present at -500, 0, and 500. The visible range extends from approximately -750 to 750. * **Grid:** A light gray grid is overlaid on all three plots, corresponding to the major tick marks. ### Plot 1: Original Experts (Left) * **Title:** "Original Experts" * **Legend:** A red circle marker is associated with this title. * **Data Points:** The plot displays a set of red circle markers scattered across the PC1-PC2 plane. The points are distributed in several clusters and isolated locations: * A cluster of points is located in the upper-left quadrant, roughly between PC1 -300 to 0 and PC2 150 to 250. * A single point is near PC1 -100, PC2 0. * A cluster is in the lower-left quadrant, around PC1 -300 to -100 and PC2 -300 to -250. * An isolated point is at the far left, near PC1 -700, PC2 -180. * A cluster is in the lower-right quadrant, roughly between PC1 100 to 500 and PC2 -200 to -100. * An isolated point is in the upper-right quadrant, near PC1 200, PC2 300. ### Plot 2: Surviving (Middle) * **Title:** "Surviving" * **Legend:** A blue circle marker is associated with this title. * **Data Points:** * **Foreground (Blue Circles):** A subset of the points from the "Original Experts" plot are highlighted in blue. These "Surviving" points are primarily concentrated in the central region of the plot, roughly between PC1 -100 to 200 and PC2 -220 to 200. * **Background (Gray Circles):** The remaining points from the "Original Experts" set, which are not part of the "Surviving" group, are shown as faint gray circles in the background, providing context. ### Plot 3: Merged (Right) * **Title:** "Merged" * **Legend:** A green 'x' marker is associated with this title. * **Data Points:** * **Foreground (Green 'x's):** A different set of points are shown with green 'x' markers. These "Merged" points are located in the right half of the plot, roughly between PC1 100 to 700 and PC2 -280 to 100. * **Background (Gray Circles):** Similar to the middle plot, the "Original Experts" data points are shown as faint gray circles in the background for context. The green 'x' markers appear to be in different locations than the original gray points, suggesting they represent new or modified data points rather than a simple subset. </details> (f) Llama-4 Layer 47 Figure A4: (4(a), 4(c), 4(e)) Functional subspace (PCA) for early SMoE layers. Pruning (blue) preserves the manifold geometry; merging (green) collapses it toward the centre. (4(b), 4(d), 4(f)) Functional subspace (PCA) for late MoE layers. Appendix C Evaluation details Multiple choice (MC) evaluation. Following Chen et al. (2025), our MC benchmarks include: AI2 Reasoning Challenge (ARC-c & ARC-e) (Clark et al., 2018), BoolQ (Clark et al., 2019), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2021a), OpenBookQA (OBQA) (Mihaylov et al., 2018), Recognizing Textual Entailment Challenge (RTE) (Bentivogli et al., 2009), and WinoGrande (WinoG.) (Sakaguchi et al., 2021). We evaluate the models in the zero-shot setting using the standard log-likelihood approach with lm-eval-harness (Gao et al., 2023). We report byte-length normalized accuracies for ARC-c, ARC-e, HellaSwag, and OBQA Reported as the acc_norm field in the EleutherAI evaluation harness outputs. See Gao (2021) for details. . Coding evaluation. For code generation, all models are evaluated on EvalPlus (Liu et al., 2023) and 182 LiveCodeBench (Jain et al., 2025) questions collected between January and April 2025. We extend the original source code for these benchmarks to evaluate our models. We additionally evaluate Kimi-K2-Instruct-W4A16 and Qwen3-Coder-480B on the agentic coding benchmark SWE-Bench (Jimenez et al., 2024) and tool-calling benchmark BFCLv3 (Patil et al., 2025). For BFCLv3, we use the original Gorilla framework for evaluating our models (Patil et al., 2024). For SWE-Bench evaluation, we run our compressed models with the mini-SWE-agent scaffolding (Yang et al., 2024b) and report the score on the SWE-Bench Verified test set (Neil Chowdhury et al., 2024). We use 4,096 and 16,384 as the maximum number of output tokens for evaluating Qwen3-Coder-480B and Kimi-K2-Instruct-W4A16 on SWE-Bench, respectively. The input context length for both models is limited to 65,536. We do not limit the number of turns in mini-SWE-agent flow, but restart the rollout in cases where the model could not generate a valid patch (that is, in the case when the output of the final turn does not contain a diff --git substring). We set the maximum number of restarts to 20, which we found to be sufficient to generate patches for all samples with pruned models, unless the model produces degenerate responses like repeating strings. We use the cloud-based evaluation provided with the sb-cli tool to get the final scores for all evaluated models. For $\tau^{2}$ -bench Barres et al. (2025), we use greedy decoding and 4,096 as the maximum number of output tokens for each LLM call. For user simulation, we use the gpt-4.1-2025-04-14 model; maximum number of steps is 100 and number of trials is set to three for each domain. Following Artificial Analysis (2025), we additionally implement an LLM-based repetition checking step. Every 30 steps of the simulation, a model (in our case, gpt-4.1-mini-2025-04-14) is given the past 30 episodes of the conversation trajectory with a repetition checking prompt to determine whether the agent is stuck in the loop or making meaningful progress. This allows early task termination if the agent is stuck. We use the same decoding parameters for the repetition model as for the user and assistant models. Math and creative writing evaluation. Mathematical reasoning is assessed on GSM8K (Cobbe et al., 2021) and MATH-500 (Hendrycks et al., 2021b; Lightman et al., 2023) benchmarks using the evalscope (ModelScope Team, 2024) framework. To assess creative writing, we use 146 creative writing prompts sampled from WildBench (Lin et al., 2024) with GPT-4o used as the judge to evaluate the model responses. We report normalized scores using the WildBench rubric. Generation configuration. For models with $≤$ 110B parameters, we use greedy sampling (i.e, temperature = 0.0) to evaluate code generation and math reasoning. For creative writing we use the default temperature, top-P, and top-K settings for each respective model. The maximum number of output tokens is extended to 16,384 for all generative tasks to account for the verbosity of some models. For hybrid reasoning models such as Qwen3-30B-A3B, we disable reasoning on all tasks by setting enable $\_$ thinking=False in the chat template. For larger models with $≥$ 110B parameters, we use greedy sampling for EvalPlus, SWE-Bench, and BFCLv3. On LiveCodeBench, Qwen3-Coder-480B and Kimi-K2 are evaluated with default sampling parameters and greedy sampling, respectively. We report the mean and standard deviation for Qwen3-Coder-480B on LiveCodeBench over five random seeds. We use a repetition penalty of 1.05 for all large model evaluations. For EvalPlus we use 768 as the maximum number of output tokens and 16,384 for LiveCodeBench. For BFCLv3 we set the maximum number of output tokens to 4,096. Model details. The Kimi-K2-Instruct-W4A16 model used throughout this study is an INT4 weight-quantized version of Kimi-K2-Instruct released by RedHatAI (2025). Appendix D Baseline methods The following formally describes the baselines compression methods we consider. Notation. Let $\mathcal{X}_{cal}$ be a calibration dataset. Consider a SMoE model with $n$ layers, $L_{n}$ , $K$ experts per layer $f_{1},...,f_{K}$ , each a function $f_{k}:\mathbb{R}^{d}→\mathbb{R}^{d}$ , and a router producing non-negative gates $\mathbf{g}(x)=(g_{1}(x),...,g_{K}(x))∈\Delta^{K-1}$ . The output of layer $L_{n}$ is $$ h_{n}=\sum_{i}^{K}g_{i}(x)f_{i}(x). $$ The expert usage frequency, $\nu_{i}$ , for expert $f_{i}$ is the number of tokens in $\mathcal{X}_{cal}$ for which $f_{i}$ is activated $$ \nu_{i}=|\mathcal{X}_{i}|, $$ where $\mathcal{X}_{i}=\{x∈\mathcal{X}_{cal}\mid i∈\text{TopK}(\mathbf{g}(x))\}$ . Given saliency scores, $\mathbf{S}∈\mathbb{R}^{K}$ , pruning removes experts with the minimum saliency score. For merging, we first cluster experts based on their pairwise distances, $\mathbf{D}∈\mathbb{R}^{K× K}$ , and then merge the parameters of experts contained within each cluster. Frequency-based pruning. The frequency-based pruning saliency criterion prunes experts with the lowest usage frequency across the calibration dataset. The saliency of $f_{i}$ is simply $S_{i}=\nu_{i}$ . EAN pruning. EAN pruning introduced by Jaiswal et al. (2025) accumulates the activation norm of each expert across tokens for which the expert is activated. The saliency of $f_{i}$ is $$ S_{i}=\sum_{x\in\mathcal{X}_{i}}\|f_{i}(x)\|_{2}. \tag{12} $$ M-SMoE merging. Proposed by Li et al. (2023), M-SMoE first uses weight-matching (Ainsworth et al., 2023) to find a permutation matrix $\mathbf{P_{j}}$ which aligns expert $f_{j}$ to expert $f_{i}$ . In the models we study, each expert is a two-layer feed-forward SwiGLU block (Shazeer, 2020) with up, gate, and down projections: $f_{j}=\{W_{up}^{(j)},W_{gate}^{(j)},W_{down}^{(j)}\}$ . The permutation matrix is applied to the intermediate dimension of the experts such that the expert outputs are invariant to the transformation | | $\displaystyle W^{\prime(j)}_{up}=W^{(j)}_{up}\mathbf{P}_{j},$ | $\displaystyle W^{\prime(j)}_{gate}=W^{(j)}_{gate}\mathbf{P}_{j},$ | $\displaystyle W^{\prime(j)}_{down}=\mathbf{P}_{j}^{T}W^{(j)}_{down}.$ | | | --- | --- | --- | --- | --- | The permuted expert is defined as $\tilde{f}_{j}=\{W^{\prime(j)}_{up},W^{\prime(j)}_{gate},W^{\prime(j)}_{down}\}$ . To initialize the expert clusters, M-SMoE identifies the set of $m$ dominant experts $\mathbb{F}_{dom}$ , as the experts across all layers with the highest usage frequency $\nu$ . The pairwise expert distance is based on the cosine distance of the router gate-values measured on the calibration dataset $$ D_{i,j}=\frac{1}{|\mathcal{X}_{cal}|}\sum_{x\in\mathcal{X}_{cal}}1-\frac{g_{i}(x)\cdot g_{j}(x)}{\|g_{i}(x)\|\|g_{j}(x)\|}. \tag{13} $$ Non-dominant expert $j$ is clustered by selecting the dominant expert with the smallest pairwise distance $$ i^{*}=\operatorname*{arg\,min}_{i\in\mathbb{F}_{dom}}D_{i,j}. $$ The merged expert $f_{\alpha}$ is created by calculating the frequency-weighted average of the permuted parameters, $W^{\prime}$ , of all experts in the cluster $\mathbb{C}_{\alpha}$ $$ \tilde{W}_{a}=\frac{\sum_{i\in\mathbb{C}_{\alpha}}\nu_{i}W^{\prime}_{i}}{\sum_{i\in\mathbb{C}_{\alpha}\nu_{i}}}. \tag{14} $$ HC-SMoE merging. Chen et al. (2025) clusters experts based on their representative vectors, $A_{i}$ , defined as the average activation across every token in the calibration dataset $$ A_{i}:=\mathbb{E}_{x\sim\mathcal{X}_{cal}}[f_{i}(x)]=\frac{1}{|\mathcal{X}_{cal}|}\sum_{x\in\mathcal{X}_{cal}}f_{i}(x). $$ The expert pairwise distance is defined as the cosine distance between representative vectors $$ D_{i,j}=1-\frac{A_{i}\cdot A_{j}}{\|A_{i}\|\|A_{j}\|}. \tag{15} $$ Clusters are formed using hierarchical agglomerative clustering with average linkage criterion. We start by initializing each expert as a singleton cluster. At every iteration, the closest pair of clusters, $\mathbb{C}_{i}^{*},\mathbb{C}_{j}^{*}$ are joined and the pairwise distances updated as the average of the constituents | | $\displaystyle i^{*},j^{*}=\operatorname*{arg\,min}_{i,j}D_{i,j},$ | $\displaystyle\mathbb{C}_{\alpha}=\mathbb{C}_{i^{*}}\cup\mathbb{C}_{j^{*}},$ | $\displaystyle D_{a,k}=\frac{\sum_{i∈\mathbb{C}_{\alpha}}D_{i,k}}{|\mathbb{C}_{\alpha}|}.$ | | | --- | --- | --- | --- | --- | The clusters are merged with equation 14. Appendix E Additional results Table ˜ A4 shows the full suite of MC question answering benchmarks and the average result across all models and methods. Table ˜ A5 tabulates code generation accuracy of compressed SMoE models calibrated on evol-codealpaca. Eval+ is the average of MBPP, MBPP+, HumanEval (HE), HE+. The Code Avg column is the average of Eval+ and LiveCodeBench (LiveCode). Table ˜ A6 summarizes the accuracy of the various compression methods studied when calibrated with the c4 dataset on coding and MC benchmarks. Notably, while the MC performance is generally slightly higher than models calibrated on evol-codealpaca, the resulting code generation quality is abysmal, with most models failing to generate coherent output. Table A4: Detailed benchmark results for multiple-choice QA tasks. | Model | Compression | Technique | Method | ARC-c | ARC-e | BoolQ | Hellaswag | MMLU | OBQA | RTE | WinoG. | MC Avg | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | ERNIE-4.5-21B- A3B-PT | Baseline | 0.564 | 0.782 | 0.873 | 0.813 | 0.737 | 0.462 | 0.812 | 0.724 | 0.721 | | | | 25% | Merging | M-SMoE | 0.434 $±$ 0.006 | 0.652 $±$ 0.008 | 0.846 $±$ 0.001 | 0.597 $±$ 0.002 | 0.591 $±$ 0.001 | 0.350 $±$ 0.006 | 0.819 $±$ 0.010 | 0.655 $±$ 0.003 | 0.618 $±$ 0.002 | | | HC-SMoE | 0.506 $±$ 0.000 | 0.717 $±$ 0.001 | 0.849 $±$ 0.001 | 0.714 $±$ 0.001 | 0.652 $±$ 0.002 | 0.371 $±$ 0.002 | 0.799 $±$ 0.002 | 0.674 $±$ 0.004 | 0.660 $±$ 0.001 | | | | | Pruning | Frequency | 0.486 $±$ 0.004 | 0.711 $±$ 0.000 | 0.852 $±$ 0.004 | 0.675 $±$ 0.003 | 0.628 $±$ 0.003 | 0.373 $±$ 0.003 | 0.780 $±$ 0.006 | 0.676 $±$ 0.005 | 0.648 $±$ 0.001 | | | | EAN | 0.498 $±$ 0.005 | 0.713 $±$ 0.002 | 0.863 $±$ 0.002 | 0.717 $±$ 0.004 | 0.625 $±$ 0.001 | 0.405 $±$ 0.011 | 0.811 $±$ 0.009 | 0.702 $±$ 0.005 | 0.667 $±$ 0.000 | | | | | REAP | 0.527 $±$ 0.004 | 0.759 $±$ 0.002 | 0.857 $±$ 0.003 | 0.717 $±$ 0.003 | 0.644 $±$ 0.001 | 0.409 $±$ 0.009 | 0.756 $±$ 0.008 | 0.690 $±$ 0.001 | 0.670 $±$ 0.002 | | | | | 50% | Merging | M-SMoE | 0.294 $±$ 0.033 | 0.452 $±$ 0.040 | 0.764 $±$ 0.010 | 0.341 $±$ 0.011 | 0.385 $±$ 0.001 | 0.270 $±$ 0.004 | 0.687 $±$ 0.017 | 0.529 $±$ 0.010 | 0.465 $±$ 0.012 | | | HC-SMoE | 0.411 $±$ 0.003 | 0.641 $±$ 0.002 | 0.822 $±$ 0.001 | 0.523 $±$ 0.001 | 0.495 $±$ 0.002 | 0.330 $±$ 0.005 | 0.742 $±$ 0.011 | 0.587 $±$ 0.009 | 0.569 $±$ 0.001 | | | | | Pruning | Frequency | 0.400 $±$ 0.002 | 0.584 $±$ 0.006 | 0.830 $±$ 0.001 | 0.522 $±$ 0.003 | 0.506 $±$ 0.006 | 0.303 $±$ 0.004 | 0.758 $±$ 0.004 | 0.625 $±$ 0.004 | 0.566 $±$ 0.002 | | | | EAN | 0.417 $±$ 0.005 | 0.633 $±$ 0.005 | 0.830 $±$ 0.003 | 0.572 $±$ 0.001 | 0.509 $±$ 0.002 | 0.336 $±$ 0.003 | 0.785 $±$ 0.014 | 0.626 $±$ 0.003 | 0.589 $±$ 0.003 | | | | | REAP | 0.417 $±$ 0.009 | 0.626 $±$ 0.007 | 0.803 $±$ 0.006 | 0.556 $±$ 0.003 | 0.505 $±$ 0.003 | 0.325 $±$ 0.006 | 0.775 $±$ 0.014 | 0.623 $±$ 0.008 | 0.579 $±$ 0.002 | | | | | Qwen3-30B-A3B | Baseline | 0.563 | 0.790 | 0.887 | 0.778 | 0.779 | 0.454 | 0.816 | 0.702 | 0.721 | | | | 25% | Merging | M-SMoE | 0.357 $±$ 0.006 | 0.519 $±$ 0.003 | 0.843 $±$ 0.006 | 0.529 $±$ 0.002 | 0.536 $±$ 0.004 | 0.310 $±$ 0.005 | 0.735 $±$ 0.027 | 0.635 $±$ 0.005 | 0.558 $±$ 0.003 | | | HC-SMoE | 0.478 $±$ 0.006 | 0.722 $±$ 0.006 | 0.863 $±$ 0.003 | 0.714 $±$ 0.000 | 0.684 $±$ 0.002 | 0.417 $±$ 0.001 | 0.805 $±$ 0.004 | 0.710 $±$ 0.004 | 0.674 $±$ 0.001 | | | | | Pruning | Frequency | 0.401 $±$ 0.011 | 0.600 $±$ 0.016 | 0.847 $±$ 0.003 | 0.593 $±$ 0.005 | 0.600 $±$ 0.004 | 0.342 $±$ 0.012 | 0.781 $±$ 0.002 | 0.637 $±$ 0.005 | 0.600 $±$ 0.005 | | | | EAN | 0.406 $±$ 0.007 | 0.603 $±$ 0.014 | 0.847 $±$ 0.005 | 0.607 $±$ 0.006 | 0.600 $±$ 0.002 | 0.337 $±$ 0.003 | 0.764 $±$ 0.002 | 0.660 $±$ 0.009 | 0.603 $±$ 0.004 | | | | | REAP | 0.481 $±$ 0.005 | 0.720 $±$ 0.005 | 0.852 $±$ 0.003 | 0.706 $±$ 0.006 | 0.674 $±$ 0.002 | 0.405 $±$ 0.005 | 0.813 $±$ 0.006 | 0.701 $±$ 0.008 | 0.669 $±$ 0.003 | | | | | 50% | Merging | M-SMoE | 0.278 $±$ 0.003 | 0.402 $±$ 0.003 | 0.753 $±$ 0.004 | 0.399 $±$ 0.002 | 0.366 $±$ 0.004 | 0.278 $±$ 0.002 | 0.586 $±$ 0.014 | 0.546 $±$ 0.004 | 0.451 $±$ 0.002 | | | HC-SMoE | 0.368 $±$ 0.002 | 0.593 $±$ 0.003 | 0.740 $±$ 0.003 | 0.473 $±$ 0.002 | 0.516 $±$ 0.003 | 0.301 $±$ 0.007 | 0.724 $±$ 0.004 | 0.620 $±$ 0.005 | 0.542 $±$ 0.001 | | | | | Pruning | Frequency | 0.285 $±$ 0.001 | 0.424 $±$ 0.002 | 0.779 $±$ 0.003 | 0.458 $±$ 0.003 | 0.397 $±$ 0.002 | 0.286 $±$ 0.004 | 0.659 $±$ 0.012 | 0.570 $±$ 0.009 | 0.483 $±$ 0.001 | | | | EAN | 0.296 $±$ 0.006 | 0.426 $±$ 0.009 | 0.759 $±$ 0.007 | 0.471 $±$ 0.002 | 0.443 $±$ 0.001 | 0.291 $±$ 0.009 | 0.668 $±$ 0.020 | 0.589 $±$ 0.009 | 0.493 $±$ 0.003 | | | | | REAP | 0.344 $±$ 0.004 | 0.504 $±$ 0.008 | 0.745 $±$ 0.005 | 0.489 $±$ 0.013 | 0.507 $±$ 0.005 | 0.311 $±$ 0.003 | 0.625 $±$ 0.031 | 0.623 $±$ 0.007 | 0.518 $±$ 0.004 | | | | | Mixtral-8x7B- Instruct-v0.1 | Baseline | 0.650 | 0.842 | 0.887 | 0.861 | 0.691 | 0.496 | 0.722 | 0.740 | 0.736 | | | | 25% | Merging | M-SMoE | 0.532 $±$ 0.004 | 0.769 $±$ 0.007 | 0.847 $±$ 0.001 | 0.747 $±$ 0.002 | 0.553 $±$ 0.001 | 0.429 $±$ 0.008 | 0.632 $±$ 0.010 | 0.656 $±$ 0.004 | 0.646 $±$ 0.001 | | | HC-SMoE | 0.590 $±$ 0.004 | 0.797 $±$ 0.004 | 0.869 $±$ 0.003 | 0.835 $±$ 0.002 | 0.626 $±$ 0.000 | 0.482 $±$ 0.004 | 0.703 $±$ 0.012 | 0.731 $±$ 0.007 | 0.704 $±$ 0.001 | | | | | Pruning | Frequency | 0.616 $±$ 0.014 | 0.826 $±$ 0.007 | 0.875 $±$ 0.001 | 0.825 $±$ 0.002 | 0.637 $±$ 0.003 | 0.451 $±$ 0.003 | 0.706 $±$ 0.017 | 0.692 $±$ 0.005 | 0.704 $±$ 0.002 | | | | EAN | 0.607 $±$ 0.004 | 0.831 $±$ 0.001 | 0.884 $±$ 0.001 | 0.836 $±$ 0.001 | 0.646 $±$ 0.002 | 0.484 $±$ 0.005 | 0.700 $±$ 0.004 | 0.732 $±$ 0.004 | 0.715 $±$ 0.000 | | | | | REAP | 0.611 $±$ 0.003 | 0.825 $±$ 0.001 | 0.874 $±$ 0.002 | 0.830 $±$ 0.002 | 0.643 $±$ 0.001 | 0.475 $±$ 0.006 | 0.761 $±$ 0.002 | 0.718 $±$ 0.001 | 0.717 $±$ 0.001 | | | | | 50% | Merging | M-SMoE | 0.446 $±$ 0.005 | 0.700 $±$ 0.001 | 0.788 $±$ 0.003 | 0.630 $±$ 0.002 | 0.430 $±$ 0.001 | 0.386 $±$ 0.003 | 0.570 $±$ 0.000 | 0.596 $±$ 0.005 | 0.568 $±$ 0.001 | | | HC-SMoE | 0.539 $±$ 0.003 | 0.759 $±$ 0.000 | 0.851 $±$ 0.001 | 0.791 $±$ 0.001 | 0.543 $±$ 0.000 | 0.442 $±$ 0.000 | 0.700 $±$ 0.004 | 0.712 $±$ 0.002 | 0.667 $±$ 0.001 | | | | | Pruning | Frequency | 0.541 $±$ 0.004 | 0.781 $±$ 0.003 | 0.824 $±$ 0.013 | 0.759 $±$ 0.002 | 0.516 $±$ 0.002 | 0.411 $±$ 0.006 | 0.708 $±$ 0.023 | 0.650 $±$ 0.005 | 0.649 $±$ 0.004 | | | | EAN | 0.551 $±$ 0.014 | 0.774 $±$ 0.008 | 0.859 $±$ 0.004 | 0.794 $±$ 0.002 | 0.550 $±$ 0.006 | 0.452 $±$ 0.014 | 0.717 $±$ 0.023 | 0.693 $±$ 0.008 | 0.674 $±$ 0.005 | | | | | REAP | 0.544 $±$ 0.005 | 0.785 $±$ 0.005 | 0.837 $±$ 0.003 | 0.778 $±$ 0.002 | 0.554 $±$ 0.001 | 0.462 $±$ 0.005 | 0.715 $±$ 0.013 | 0.679 $±$ 0.005 | 0.669 $±$ 0.001 | | | | | Llama-4-Scout- 17B-16E- Instruct | Baseline | 0.627 | 0.848 | 0.879 | 0.823 | 0.803 | 0.462 | 0.765 | 0.692 | 0.738 | | | | 25% | Merging | M-SMoE | 0.573 | 0.802 | 0.872 | 0.752 | 0.719 | 0.434 | 0.769 | 0.671 | 0.699 | | | HC-SMoE | 0.588 | 0.814 | 0.876 | 0.779 | 0.720 | 0.424 | 0.729 | 0.695 | 0.703 | | | | | Pruning | Frequency | 0.584 | 0.817 | 0.876 | 0.779 | 0.733 | 0.438 | 0.773 | 0.691 | 0.711 | | | | EAN | 0.582 | 0.816 | 0.872 | 0.777 | 0.735 | 0.446 | 0.791 | 0.679 | 0.712 | | | | | REAP | 0.594 | 0.830 | 0.872 | 0.788 | 0.756 | 0.452 | 0.769 | 0.683 | 0.718 | | | | | 50% | Merging | M-SMoE | 0.498 | 0.717 | 0.856 | 0.676 | 0.609 | 0.388 | 0.787 | 0.665 | 0.649 | | | HC-SMoE | 0.526 | 0.781 | 0.862 | 0.718 | 0.628 | 0.386 | 0.726 | 0.660 | 0.661 | | | | | Pruning | Frequency | 0.518 | 0.734 | 0.860 | 0.704 | 0.652 | 0.398 | 0.765 | 0.657 | 0.661 | | | | EAN | 0.510 | 0.750 | 0.857 | 0.712 | 0.650 | 0.398 | 0.762 | 0.662 | 0.663 | | | | | REAP | 0.561 | 0.802 | 0.869 | 0.745 | 0.682 | 0.432 | 0.762 | 0.664 | 0.689 | | | | | GLM-4.5-Air | Baseline | 0.619 | 0.825 | 0.882 | 0.858 | 0.789 | 0.478 | 0.747 | 0.776 | 0.747 | | | | 25% | Merging | M-SMoE | 0.429 | 0.651 | 0.808 | 0.671 | 0.578 | 0.362 | 0.578 | 0.695 | 0.596 | | | HC-SMoE | 0.577 | 0.782 | 0.860 | 0.815 | 0.722 | 0.458 | 0.668 | 0.755 | 0.704 | | | | | Pruning | Frequency | 0.493 | 0.715 | 0.827 | 0.732 | 0.653 | 0.422 | 0.614 | 0.725 | 0.648 | | | | EAN | 0.492 | 0.705 | 0.805 | 0.736 | 0.656 | 0.368 | 0.603 | 0.730 | 0.637 | | | | | REAP | 0.555 | 0.756 | 0.813 | 0.796 | 0.701 | 0.434 | 0.643 | 0.724 | 0.678 | | | | | 50% | Merging | M-SMoE | 0.291 | 0.452 | 0.693 | 0.433 | 0.382 | 0.266 | 0.484 | 0.551 | 0.444 | | | HC-SMoE | 0.428 | 0.671 | 0.761 | 0.590 | 0.524 | 0.318 | 0.603 | 0.613 | 0.564 | | | | | Pruning | Frequency | 0.334 | 0.535 | 0.767 | 0.566 | 0.478 | 0.288 | 0.567 | 0.635 | 0.521 | | | | EAN | 0.358 | 0.530 | 0.682 | 0.573 | 0.489 | 0.300 | 0.516 | 0.635 | 0.511 | | | | | REAP | 0.427 | 0.604 | 0.662 | 0.642 | 0.569 | 0.318 | 0.606 | 0.640 | 0.559 | | | | | Qwen3-Coder- 480B-A35B- Instruct-FP8 | Baseline | 0.644 | 0.822 | 0.906 | 0.841 | 0.850 | 0.468 | 0.751 | 0.717 | 0.750 | | | | 25% | Pruning | Frequency | 0.443 | 0.673 | 0.845 | 0.651 | 0.621 | 0.280 | 0.704 | 0.632 | 0.606 | | | EAN | 0.555 | 0.766 | 0.891 | 0.769 | 0.795 | 0.404 | 0.747 | 0.691 | 0.702 | | | | | REAP | 0.635 | 0.824 | 0.900 | 0.841 | 0.836 | 0.466 | 0.754 | 0.725 | 0.748 | | | | | 50% | Pruning | Frequency | 0.314 | 0.470 | 0.791 | 0.502 | 0.451 | 0.262 | 0.679 | 0.580 | 0.506 | | | EAN | 0.402 | 0.596 | 0.858 | 0.629 | 0.615 | 0.216 | 0.744 | 0.666 | 0.591 | | | | | REAP | 0.546 | 0.772 | 0.872 | 0.756 | 0.696 | 0.430 | 0.762 | 0.701 | 0.692 | | | | | Kimi-K2- Instruct- W4A16 | Baseline | 0.712 | 0.879 | 0.913 | 0.765 | 0.872 | 0.504 | 0.783 | 0.811 | 0.780 | | | | 25% | Pruning | Frequency | 0.518 | 0.771 | 0.825 | 0.787 | 0.242 | 0.420 | 0.653 | 0.613 | 0.604 | | | EAN | 0.615 | 0.819 | 0.893 | 0.843 | 0.500 | 0.446 | 0.762 | 0.743 | 0.703 | | | | | REAP | 0.671 | 0.854 | 0.907 | 0.860 | 0.809 | 0.470 | 0.805 | 0.809 | 0.773 | | | | | 50% | Pruning | Frequency | 0.285 | 0.498 | 0.620 | 0.436 | 0.241 | 0.314 | 0.617 | 0.500 | 0.439 | | | EAN | 0.426 | 0.682 | 0.863 | 0.663 | 0.324 | 0.356 | 0.726 | 0.659 | 0.587 | | | | | REAP | 0.476 | 0.661 | 0.883 | 0.643 | 0.636 | 0.350 | 0.816 | 0.681 | 0.643 | | | | Table A5: Detailed benchmark results for non-agentic code generation tasks. Eval+ is the average of MBPP, MBPP+, HE, HE+. The Code Avg column is the average of Eval+ and LiveCodeBench (LiveCode). | Model | Compression | Technique | Method | HE | HE+ | MBPP | MBPP+ | Eval+ | LiveCode | Code Avg | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | ERNIE-4.5-21B- A3B-PT | Baseline | 0.902 | 0.866 | 0.910 | 0.765 | 0.861 | 0.231 | 0.546 | | | | 25% | Merging | M-SMoE | 0.774 $±$ 0.011 | 0.730 $±$ 0.009 | 0.768 $±$ 0.015 | 0.647 $±$ 0.017 | 0.730 $±$ 0.005 | 0.194 $±$ 0.022 | 0.462 $±$ 0.011 | | | HC-SMoE | 0.837 $±$ 0.007 | 0.805 $±$ 0.000 | 0.827 $±$ 0.003 | 0.696 $±$ 0.008 | 0.791 $±$ 0.004 | 0.207 $±$ 0.008 | 0.499 $±$ 0.003 | | | | | Pruning | Frequency | 0.890 $±$ 0.006 | 0.846 $±$ 0.009 | 0.837 $±$ 0.010 | 0.709 $±$ 0.010 | 0.820 $±$ 0.006 | 0.151 $±$ 0.096 | 0.486 $±$ 0.045 | | | | EAN | 0.890 $±$ 0.006 | 0.848 $±$ 0.011 | 0.840 $±$ 0.006 | 0.727 $±$ 0.004 | 0.826 $±$ 0.004 | 0.161 $±$ 0.111 | 0.494 $±$ 0.054 | | | | | REAP | 0.892 $±$ 0.009 | 0.854 $±$ 0.012 | 0.876 $±$ 0.000 | 0.738 $±$ 0.003 | 0.840 $±$ 0.005 | 0.167 $±$ 0.124 | 0.504 $±$ 0.060 | | | | | 50% | Merging | M-SMoE | 0.104 $±$ 0.022 | 0.100 $±$ 0.029 | 0.239 $±$ 0.036 | 0.207 $±$ 0.040 | 0.162 $±$ 0.012 | 0.024 $±$ 0.008 | 0.093 $±$ 0.008 | | | HC-SMoE | 0.425 $±$ 0.004 | 0.404 $±$ 0.007 | 0.608 $±$ 0.018 | 0.511 $±$ 0.011 | 0.487 $±$ 0.008 | 0.082 $±$ 0.015 | 0.285 $±$ 0.009 | | | | | Pruning | Frequency | 0.699 $±$ 0.031 | 0.640 $±$ 0.022 | 0.696 $±$ 0.014 | 0.584 $±$ 0.006 | 0.655 $±$ 0.015 | 0.083 $±$ 0.066 | 0.369 $±$ 0.025 | | | | EAN | 0.675 $±$ 0.019 | 0.642 $±$ 0.009 | 0.713 $±$ 0.015 | 0.591 $±$ 0.016 | 0.655 $±$ 0.014 | 0.112 $±$ 0.064 | 0.384 $±$ 0.035 | | | | | REAP | 0.797 $±$ 0.009 | 0.764 $±$ 0.007 | 0.767 $±$ 0.017 | 0.644 $±$ 0.013 | 0.743 $±$ 0.008 | 0.137 $±$ 0.119 | 0.440 $±$ 0.064 | | | | | Qwen3-30B-A3B | Baseline | 0.927 | 0.884 | 0.881 | 0.743 | 0.859 | 0.302 | 0.581 | | | | 25% | Merging | M-SMoE | 0.878 $±$ 0.012 | 0.833 $±$ 0.007 | 0.849 $±$ 0.007 | 0.728 $±$ 0.007 | 0.822 $±$ 0.004 | 0.293 $±$ 0.017 | 0.558 $±$ 0.006 | | | HC-SMoE | 0.866 $±$ 0.011 | 0.805 $±$ 0.016 | 0.832 $±$ 0.006 | 0.698 $±$ 0.005 | 0.800 $±$ 0.004 | 0.258 $±$ 0.000 | 0.529 $±$ 0.002 | | | | | Pruning | Frequency | 0.921 $±$ 0.006 | 0.874 $±$ 0.007 | 0.868 $±$ 0.000 | 0.735 $±$ 0.003 | 0.849 $±$ 0.004 | 0.302 $±$ 0.011 | 0.576 $±$ 0.004 | | | | EAN | 0.909 $±$ 0.006 | 0.864 $±$ 0.004 | 0.859 $±$ 0.009 | 0.729 $±$ 0.008 | 0.840 $±$ 0.004 | 0.311 $±$ 0.018 | 0.576 $±$ 0.010 | | | | | REAP | 0.917 $±$ 0.007 | 0.876 $±$ 0.004 | 0.853 $±$ 0.002 | 0.727 $±$ 0.006 | 0.843 $±$ 0.002 | 0.308 $±$ 0.015 | 0.575 $±$ 0.008 | | | | | 50% | Merging | M-SMoE | 0.687 $±$ 0.013 | 0.638 $±$ 0.004 | 0.618 $±$ 0.004 | 0.541 $±$ 0.007 | 0.621 $±$ 0.006 | 0.205 $±$ 0.019 | 0.413 $±$ 0.007 | | | HC-SMoE | 0.577 $±$ 0.023 | 0.541 $±$ 0.013 | 0.631 $±$ 0.010 | 0.546 $±$ 0.004 | 0.574 $±$ 0.010 | 0.185 $±$ 0.018 | 0.379 $±$ 0.005 | | | | | Pruning | Frequency | 0.787 $±$ 0.016 | 0.756 $±$ 0.022 | 0.692 $±$ 0.016 | 0.579 $±$ 0.016 | 0.704 $±$ 0.017 | 0.236 $±$ 0.025 | 0.470 $±$ 0.021 | | | | EAN | 0.886 $±$ 0.025 | 0.837 $±$ 0.020 | 0.798 $±$ 0.006 | 0.669 $±$ 0.008 | 0.798 $±$ 0.013 | 0.306 $±$ 0.003 | 0.552 $±$ 0.005 | | | | | REAP | 0.919 $±$ 0.007 | 0.870 $±$ 0.004 | 0.805 $±$ 0.009 | 0.692 $±$ 0.008 | 0.821 $±$ 0.003 | 0.293 $±$ 0.003 | 0.557 $±$ 0.001 | | | | | Mixtral-8x7B- Instruct-v0.1 | Baseline | 0.524 | 0.476 | 0.556 | 0.463 | 0.505 | 0.123 | 0.314 | | | | 25% | Merging | M-SMoE | 0.315 $±$ 0.007 | 0.270 $±$ 0.015 | 0.446 $±$ 0.007 | 0.380 $±$ 0.015 | 0.353 $±$ 0.008 | 0.033 $±$ 0.010 | 0.193 $±$ 0.008 | | | HC-SMoE | 0.439 $±$ 0.028 | 0.386 $±$ 0.020 | 0.530 $±$ 0.022 | 0.441 $±$ 0.007 | 0.449 $±$ 0.005 | 0.110 $±$ 0.010 | 0.279 $±$ 0.002 | | | | | Pruning | Frequency | 0.400 $±$ 0.034 | 0.358 $±$ 0.035 | 0.541 $±$ 0.006 | 0.453 $±$ 0.012 | 0.438 $±$ 0.018 | 0.099 $±$ 0.014 | 0.269 $±$ 0.004 | | | | EAN | 0.413 $±$ 0.027 | 0.366 $±$ 0.024 | 0.477 $±$ 0.009 | 0.409 $±$ 0.013 | 0.416 $±$ 0.015 | 0.111 $±$ 0.006 | 0.264 $±$ 0.006 | | | | | REAP | 0.439 $±$ 0.018 | 0.370 $±$ 0.007 | 0.535 $±$ 0.011 | 0.452 $±$ 0.011 | 0.449 $±$ 0.002 | 0.102 $±$ 0.010 | 0.275 $±$ 0.005 | | | | | 50% | Merging | M-SMoE | 0.085 $±$ 0.026 | 0.076 $±$ 0.022 | 0.139 $±$ 0.121 | 0.118 $±$ 0.102 | 0.091 $±$ 0.079 | 0.004 $±$ 0.006 | 0.047 $±$ 0.037 | | | HC-SMoE | 0.175 $±$ 0.015 | 0.146 $±$ 0.000 | 0.335 $±$ 0.026 | 0.282 $±$ 0.031 | 0.235 $±$ 0.018 | 0.013 $±$ 0.008 | 0.124 $±$ 0.008 | | | | | Pruning | Frequency | 0.187 $±$ 0.015 | 0.148 $±$ 0.007 | 0.342 $±$ 0.016 | 0.287 $±$ 0.012 | 0.241 $±$ 0.007 | 0.023 $±$ 0.004 | 0.132 $±$ 0.003 | | | | EAN | 0.220 $±$ 0.006 | 0.189 $±$ 0.006 | 0.375 $±$ 0.020 | 0.325 $±$ 0.015 | 0.277 $±$ 0.005 | 0.031 $±$ 0.011 | 0.154 $±$ 0.007 | | | | | REAP | 0.232 $±$ 0.018 | 0.193 $±$ 0.013 | 0.274 $±$ 0.106 | 0.241 $±$ 0.087 | 0.235 $±$ 0.056 | 0.035 $±$ 0.003 | 0.135 $±$ 0.027 | | | | | Llama-4-Scout- 17B-16E- Instruct | Baseline | 0.829 | 0.768 | 0.788 | 0.640 | 0.757 | 0.341 | 0.549 | | | | 25% | Merging | M-SMoE | 0.823 | 0.762 | 0.786 | 0.635 | 0.752 | 0.324 | 0.538 | | | HC-SMoE | 0.787 | 0.738 | 0.735 | 0.587 | 0.712 | 0.148 | 0.430 | | | | | Pruning | Frequency | 0.835 | 0.768 | 0.788 | 0.630 | 0.755 | 0.317 | 0.536 | | | | EAN | 0.823 | 0.762 | 0.804 | 0.648 | 0.759 | 0.328 | 0.544 | | | | | REAP | 0.829 | 0.787 | 0.788 | 0.622 | 0.756 | 0.242 | 0.499 | | | | | 50% | Merging | M-SMoE | 0.787 | 0.732 | 0.762 | 0.614 | 0.723 | 0.187 | 0.455 | | | HC-SMoE | 0.604 | 0.530 | 0.500 | 0.399 | 0.508 | 0.077 | 0.293 | | | | | Pruning | Frequency | 0.823 | 0.756 | 0.751 | 0.595 | 0.731 | 0.223 | 0.477 | | | | EAN | 0.805 | 0.744 | 0.754 | 0.601 | 0.726 | 0.209 | 0.468 | | | | | REAP | 0.841 | 0.768 | 0.762 | 0.624 | 0.749 | 0.248 | 0.499 | | | | | GLM-4.5-Air | Baseline | 0.848 | 0.829 | 0.860 | 0.743 | 0.820 | 0.374 | 0.597 | | | | 25% | Merging | M-SMoE | 0.866 | 0.793 | 0.807 | 0.659 | 0.781 | 0.330 | 0.555 | | | HC-SMoE | 0.872 | 0.805 | 0.825 | 0.669 | 0.793 | 0.363 | 0.578 | | | | | Pruning | Frequency | 0.848 | 0.811 | 0.854 | 0.706 | 0.805 | 0.341 | 0.573 | | | | EAN | 0.872 | 0.817 | 0.876 | 0.720 | 0.821 | 0.374 | 0.597 | | | | | REAP | 0.866 | 0.805 | 0.828 | 0.677 | 0.794 | 0.390 | 0.592 | | | | | 50% | Merging | M-SMoE | 0.518 | 0.500 | 0.519 | 0.437 | 0.493 | 0.099 | 0.296 | | | HC-SMoE | 0.707 | 0.659 | 0.706 | 0.577 | 0.662 | 0.220 | 0.441 | | | | | Pruning | Frequency | 0.628 | 0.573 | 0.534 | 0.450 | 0.546 | 0.104 | 0.325 | | | | EAN | 0.841 | 0.780 | 0.807 | 0.661 | 0.773 | 0.253 | 0.513 | | | | | REAP | 0.878 | 0.841 | 0.712 | 0.587 | 0.755 | 0.352 | 0.553 | | | | | Qwen3-Coder- 480B-A35B- Instruct-FP8 | Baseline | 0.951 | 0.890 | 0.923 | 0.791 | 0.889 | 0.431 $±$ 0.011 | 0.660 | | | | 25% | Pruning | Frequency | 0.884 | 0.805 | 0.810 | 0.669 | 0.792 | 0.296 $±$ 0.017 | 0.544 | | | EAN | 0.939 | 0.878 | 0.911 | 0.775 | 0.876 | 0.419 $±$ 0.015 | 0.647 | | | | | REAP | 0.957 | 0.890 | 0.917 | 0.772 | 0.884 | 0.416 $±$ 0.013 | 0.650 | | | | | 50% | Pruning | Frequency | 0.020 | 0.012 | 0.007 | 0.003 | 0.011 | 0.012 $±$ 0.001 | 0.011 | | | EAN | 0.915 | 0.841 | 0.854 | 0.714 | 0.831 | 0.382 $±$ 0.012 | 0.607 | | | | | REAP | 0.939 | 0.872 | 0.910 | 0.772 | 0.873 | 0.415 $±$ 0.015 | 0.644 | | | | | Kimi-K2- Instruct- W4A16 | Baseline | 0.963 | 0.921 | 0.913 | 0.735 | 0.883 | 0.434 | 0.659 | | | | 25% | Pruning | Frequency | 0.530 | 0.463 | 0.595 | 0.508 | 0.524 | 0.082 | 0.303 | | | EAN | 0.909 | 0.860 | 0.857 | 0.698 | 0.831 | 0.379 | 0.605 | | | | | REAP | 0.957 | 0.921 | 0.918 | 0.759 | 0.889 | 0.440 | 0.664 | | | | | 50% | Pruning | Frequency | 0.098 | 0.079 | 0.175 | 0.146 | 0.124 | 0.000 | 0.062 | | | EAN | 0.866 | 0.811 | 0.780 | 0.632 | 0.772 | 0.253 | 0.513 | | | | | REAP | 0.915 | 0.884 | 0.899 | 0.754 | 0.863 | 0.429 | 0.646 | | | | Table A6: C4 calibrated results for coding and MC tasks. | | Coding | MC | | | | | | | | | | | | | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Model | Compression | Technique | Method | Eval+ | LiveCode | Code Avg | ARC-c | ARC-e | BoolQ | Hellaswag | MMLU | OBQA | RTE | WinoG. | MC Avg | | ERNIE-4.5-21B- A3B-PT | Baseline | 0.861 | 0.231 | 0.546 | 0.564 | 0.782 | 0.873 | 0.813 | 0.737 | 0.462 | 0.812 | 0.724 | 0.721 | | | | 25% | Merging | M-SMoE | 0.065 | 0.016 | 0.041 | 0.497 | 0.729 | 0.860 | 0.723 | 0.602 | 0.424 | 0.801 | 0.699 | 0.667 | | | HC-SMoE | 0.403 | 0.099 | 0.251 | 0.515 | 0.728 | 0.860 | 0.745 | 0.649 | 0.428 | 0.794 | 0.694 | 0.677 | | | | | Pruning | Frequency | 0.274 | 0.000 | 0.137 | 0.515 | 0.735 | 0.841 | 0.719 | 0.588 | 0.382 | 0.791 | 0.683 | 0.657 | | | | EAN | 0.282 | 0.000 | 0.141 | 0.528 | 0.750 | 0.853 | 0.790 | 0.558 | 0.442 | 0.783 | 0.706 | 0.676 | | | | | REAP | 0.242 | 0.023 | 0.133 | 0.490 | 0.716 | 0.855 | 0.783 | 0.656 | 0.452 | 0.809 | 0.723 | 0.685 | | | | | 50% | Merging | M-SMoE | 0.000 | 0.000 | 0.000 | 0.297 | 0.460 | 0.674 | 0.449 | 0.312 | 0.280 | 0.671 | 0.575 | 0.465 | | | HC-SMoE | 0.000 | 0.000 | 0.000 | 0.409 | 0.615 | 0.666 | 0.515 | 0.489 | 0.290 | 0.632 | 0.580 | 0.524 | | | | | Pruning | Frequency | 0.000 | 0.000 | 0.000 | 0.393 | 0.625 | 0.717 | 0.569 | 0.496 | 0.324 | 0.758 | 0.619 | 0.563 | | | | EAN | 0.007 | 0.003 | 0.005 | 0.451 | 0.676 | 0.742 | 0.687 | 0.474 | 0.398 | 0.736 | 0.691 | 0.607 | | | | | REAP | 0.033 | 0.000 | 0.016 | 0.406 | 0.612 | 0.754 | 0.654 | 0.468 | 0.396 | 0.718 | 0.656 | 0.583 | | | | | Qwen3-30B-A3B | Baseline | 0.859 | 0.302 | 0.581 | 0.563 | 0.790 | 0.887 | 0.778 | 0.779 | 0.454 | 0.816 | 0.702 | 0.721 | | | | 25% | Merging | M-SMoE | 0.000 | 0.000 | 0.000 | 0.551 | 0.768 | 0.883 | 0.761 | 0.733 | 0.418 | 0.848 | 0.701 | 0.708 | | | HC-SMoE | 0.831 | 0.269 | 0.550 | 0.470 | 0.713 | 0.833 | 0.622 | 0.646 | 0.376 | 0.805 | 0.665 | 0.641 | | | | | Pruning | Frequency | 0.000 | 0.000 | 0.000 | 0.548 | 0.789 | 0.889 | 0.775 | 0.735 | 0.438 | 0.801 | 0.694 | 0.709 | | | | EAN | 0.000 | 0.000 | 0.000 | 0.569 | 0.802 | 0.889 | 0.774 | 0.735 | 0.438 | 0.801 | 0.697 | 0.713 | | | | | REAP | 0.735 | 0.227 | 0.481 | 0.557 | 0.781 | 0.872 | 0.746 | 0.718 | 0.436 | 0.794 | 0.704 | 0.701 | | | | | 50% | Merging | M-SMoE | 0.000 | 0.000 | 0.000 | 0.262 | 0.348 | 0.693 | 0.479 | 0.237 | 0.290 | 0.523 | 0.542 | 0.422 | | | HC-SMoE | 0.728 | 0.209 | 0.468 | 0.316 | 0.495 | 0.715 | 0.354 | 0.422 | 0.282 | 0.603 | 0.536 | 0.465 | | | | | Pruning | Frequency | 0.000 | 0.000 | 0.000 | 0.349 | 0.488 | 0.782 | 0.672 | 0.503 | 0.364 | 0.588 | 0.619 | 0.545 | | | | EAN | 0.000 | 0.000 | 0.000 | 0.480 | 0.736 | 0.876 | 0.760 | 0.607 | 0.424 | 0.762 | 0.694 | 0.667 | | | | | REAP | 0.006 | 0.000 | 0.003 | 0.421 | 0.640 | 0.837 | 0.653 | 0.495 | 0.388 | 0.704 | 0.635 | 0.596 | | | | | Mixtral-8x7B- Instruct-v0.1 | Baseline | 0.505 | 0.123 | 0.314 | 0.650 | 0.842 | 0.887 | 0.861 | 0.691 | 0.496 | 0.722 | 0.740 | 0.736 | | | | 25% | Merging | M-SMoE | 0.320 | 0.044 | 0.182 | 0.532 | 0.775 | 0.828 | 0.746 | 0.529 | 0.424 | 0.603 | 0.632 | 0.634 | | | HC-SMoE | 0.420 | 0.121 | 0.271 | 0.608 | 0.811 | 0.876 | 0.838 | 0.631 | 0.484 | 0.736 | 0.726 | 0.714 | | | | | Pruning | Frequency | 0.396 | 0.070 | 0.233 | 0.612 | 0.816 | 0.868 | 0.836 | 0.593 | 0.482 | 0.675 | 0.739 | 0.703 | | | | EAN | 0.399 | 0.092 | 0.246 | 0.613 | 0.814 | 0.875 | 0.842 | 0.613 | 0.498 | 0.690 | 0.733 | 0.710 | | | | | REAP | 0.415 | 0.077 | 0.246 | 0.606 | 0.807 | 0.875 | 0.835 | 0.633 | 0.486 | 0.791 | 0.709 | 0.718 | | | | | 50% | Merging | M-SMoE | 0.000 | 0.000 | 0.000 | 0.260 | 0.460 | 0.614 | 0.395 | 0.240 | 0.302 | 0.527 | 0.526 | 0.416 | | | HC-SMoE | 0.174 | 0.033 | 0.103 | 0.540 | 0.764 | 0.862 | 0.795 | 0.544 | 0.448 | 0.675 | 0.709 | 0.667 | | | | | Pruning | Frequency | 0.173 | 0.008 | 0.090 | 0.504 | 0.739 | 0.793 | 0.771 | 0.463 | 0.426 | 0.675 | 0.646 | 0.627 | | | | EAN | 0.139 | 0.008 | 0.074 | 0.550 | 0.756 | 0.842 | 0.804 | 0.529 | 0.460 | 0.726 | 0.716 | 0.673 | | | | | REAP | 0.167 | 0.012 | 0.089 | 0.525 | 0.774 | 0.856 | 0.794 | 0.533 | 0.454 | 0.751 | 0.688 | 0.672 | | | | Table A7: $\tau^{2}$ -bench results with REAP compression across different benchmark domains on Qwen3-480B-A35B-Coder-FP8. | Dataset | Compression | Method | passˆ1 | passˆ2 | passˆ3 | | --- | --- | --- | --- | --- | --- | | Retail | Baseline | 0.643 | 0.544 | 0.500 | | | 25% | REAP | 0.661 | 0.535 | 0.465 | | | 50% | REAP | 0.632 | 0.515 | 0.456 | | | Airline | Baseline | 0.460 | 0.340 | 0.280 | | | 25% | REAP | 0.487 | 0.367 | 0.320 | | | 50% | REAP | 0.447 | 0.333 | 0.280 | | | Telecom | Baseline | 0.500 | 0.398 | 0.325 | | | 25% | REAP | 0.529 | 0.456 | 0.421 | | | 50% | REAP | 0.471 | 0.339 | 0.263 | | Figure ˜ A5 plots non-agentic coding and MC accuracy versus compressed model size. Figure ˜ 6(a) depict the proportion of singleton clusters for HC-SMoE and M-SMoE. Figure ˜ 6(b) plots accuracy vs. maximum cluster sizes when the maximum cardinality of clusters is restricted. Figures ˜ A7 and A8 show the importance of using domain-specific calibration data, particularly at high compression ratios. Table ˜ A7 presents the complete $\tau^{2}$ -bench results across three domains (Retail, Airline, and Telecom) for the baseline model and REAP compression at 25% and 50% levels. The results show passˆk metrics for k=1, 2, and 3, demonstrating the impact of pruning on evaluating conversational agents, specifically designed to test their ability to collaborate with a user in real-world scenarios. <details> <summary>2510.13999v1/x17.png Details</summary>  ### Visual Description ## Technical Analysis of Model Performance Charts This image contains two scatter plots comparing the performance of various large language models (LLMs) and different pruning/merging methods against their total parameters. A single legend on the right is shared by both charts. ### Legend The legend defines the coding for methods (lines and colors) and models (markers). **Methods:** * **Baseline:** Solid black line (represented as black markers in the plots). * **Pruning Methods:** * **REAP (ours):** Blue dashed line with diamond markers. * **EAN:** Pink dashed line with 'x' markers. * **Frequency:** Green dashed line with diamond markers. * **Merging Methods:** * **HC-SMoE:** Yellow dashed line with square markers. * **M-SMoE:** Light blue dashed line with square markers. **Models:** * **ERNIE-4.5-21B-A3B:** Circle (•) * **Qwen3-30B-A3B:** Square (■) * **Mixtral-8x7B-Instruct-v0.1:** Downward-pointing triangle (▼) * **LLaMA-4-Scout-17B-16E-Instruct:** Star (★) * **GLM-4.5-Air:** Diamond (◆) * **Qwen3-Coder-480B-A35B-Instruct-FP8:** Plus sign (+) * **Kimi-K2-Instruct-W4A16:** Cross (x) --- ### Left Chart: Non-Agentic Code Accuracy vs. Total Parameters This chart plots "Non-Agentic Code Acc. (%)" on the y-axis against "Log Scaled Total Parameters (in billions)" on the x-axis. The x-axis is logarithmic, with major ticks at 10¹, 10², and 10³. The y-axis ranges from 0 to 60% with ticks every 10%. **Key Trends and Data Points:** * **General Trend:** For most methods, accuracy tends to increase as the number of parameters increases. The "REAP (ours)" method consistently outperforms other pruning and merging methods, often approaching or exceeding the baseline performance. * **ERNIE-4.5-21B-A3B (•):** Baseline is at ~21B parameters with ~55% accuracy. All methods show an upward trend from ~10B to ~21B parameters, with REAP reaching the highest accuracy (~58%). * **Qwen3-30B-A3B (■):** Baseline is at ~30B parameters with ~58% accuracy. Methods show an upward trend from ~20B to ~30B parameters. REAP reaches ~28% accuracy. * **Mixtral-8x7B-Instruct-v0.1 (▼):** Baseline is at ~47B parameters with ~32% accuracy. Methods show an upward trend from ~60B to ~80B parameters. REAP reaches ~59% accuracy. * **LLaMA-4-Scout-17B-16E-Instruct (★):** Baseline is at ~110B parameters with ~55% accuracy. Methods show an upward trend from ~60B to ~80B parameters. REAP reaches ~56% accuracy. * **GLM-4.5-Air (◆):** Baseline is at ~110B parameters with ~60% accuracy. Methods show an upward trend from ~250B to ~400B parameters. REAP reaches ~65% accuracy. Frequency pruning shows a very steep drop to ~1% at ~250B before recovering to ~54% at ~400B. * **Qwen3-Coder-480B-A35B-Instruct-FP8 (+):** Baseline is at ~500B parameters with ~66% accuracy. Methods show an upward trend from ~600B to ~800B parameters. REAP reaches ~66% accuracy. Frequency pruning starts very low at ~6% and rises to ~30%. * **Kimi-K2-Instruct-W4A16 (x):** Only the baseline is shown at ~1000B parameters with ~66% accuracy. --- ### Right Chart: MC Accuracy vs. Total Parameters This chart plots "MC Accuracy (%)" on the y-axis against "Log Scaled Total Parameters (in billions)" on the x-axis. The x-axis is logarithmic, with major ticks at 10¹, 10², and 10³. The y-axis ranges from 45% to 75% with ticks every 5%. **Key Trends and Data Points:** * **General Trend:** Similar to the left chart, accuracy generally increases with parameter count. The "REAP (ours)" method again shows strong performance, often being the top-performing method after the baseline. * **ERNIE-4.5-21B-A3B (•):** Baseline is at ~21B parameters with ~72% accuracy. All methods show an upward trend from ~10B to ~21B parameters, with REAP reaching ~68% accuracy. * **Qwen3-30B-A3B (■):** Baseline is at ~30B parameters with ~72% accuracy. Methods show an upward trend from ~20B to ~30B parameters. REAP reaches ~68% accuracy. * **Mixtral-8x7B-Instruct-v0.1 (▼):** Baseline is at ~47B parameters with ~74% accuracy. Methods show an upward trend from ~60B to ~80B parameters. REAP reaches ~72% accuracy. * **LLaMA-4-Scout-17B-16E-Instruct (★):** Baseline is at ~110B parameters with ~74% accuracy. Methods show an upward trend from ~60B to ~80B parameters. REAP reaches ~68% accuracy. * **GLM-4.5-Air (◆):** Baseline is at ~110B parameters with ~75% accuracy. Methods show an upward trend from ~250B to ~400B parameters. REAP reaches ~75% accuracy. * **Qwen3-Coder-480B-A35B-Instruct-FP8 (+):** Baseline is at ~500B parameters with ~75% accuracy. Methods show an upward trend from ~600B to ~800B parameters. REAP reaches ~77% accuracy, surpassing the baseline. * **Kimi-K2-Instruct-W4A16 (x):** Only the baseline is shown at ~1000B parameters with ~78% accuracy. </details> Figure A5: Coding and MC accuracy across all models vs. parameters. The benefits of REAP over other compression methods are evident at 50% compression. For large-scale SMoEs, REAP is near-lossless whereas the shortcomings of frequency-based pruning become apparent. <details> <summary>2510.13999v1/x18.png Details</summary>  ### Visual Description ## Technical Description of the Bar Chart This image is a grouped bar chart comparing the percentage of "Singleton clusters" for two different methods, "HC-SMoE" and "M-SMoE", across five different language models. The chart includes error bars for each data point. ### Chart Components * **Y-axis:** The vertical axis is labeled "Singleton clusters (%)" and ranges from 0 to 100, with major tick marks at 0, 20, 40, 60, 80, and 100. * **X-axis:** The horizontal axis represents five different language models, labeled as follows: * ERNIE-4.5-21B-A3B-PT * Qwen3-30B-A3B * Mixtral-8x7B-Instruct-v0.1 * Llama-4-Scout-17B-16E-Instruct * GLM-4.5-Air * **Legend:** A legend is located in the upper right area of the chart, indicating the color coding for the two methods: * **Blue:** HC-SMoE * **Red/Pink:** M-SMoE ### Data Analysis The chart presents the following data for each model, with estimated values and error ranges: 1. **ERNIE-4.5-21B-A3B-PT:** * **HC-SMoE (Blue):** The bar reaches approximately 97%. The error bar is very small, indicating a tight range around this value. * **M-SMoE (Red):** The bar reaches approximately 56%. The error bar extends from about 48% to 64%. 2. **Qwen3-30B-A3B:** * **HC-SMoE (Blue):** The bar reaches approximately 98%. The error bar is very small. * **M-SMoE (Red):** The bar reaches approximately 63%. The error bar extends from about 57% to 69%. 3. **Mixtral-8x7B-Instruct-v0.1:** * **HC-SMoE (Blue):** The bar reaches approximately 57%. The error bar extends from about 40% to 74%. * **M-SMoE (Red):** The bar reaches approximately 27%. The error bar extends from about 4% to 50%. 4. **Llama-4-Scout-17B-16E-Instruct:** * **HC-SMoE (Blue):** The bar reaches approximately 86%. The error bar extends from about 82% to 90%. * **M-SMoE (Red):** The bar reaches approximately 73%. The error bar extends from about 63% to 83%. 5. **GLM-4.5-Air:** * **HC-SMoE (Blue):** The bar reaches approximately 99%. The error bar is very small. * **M-SMoE (Red):** The bar reaches approximately 66%. The error bar extends from about 60% to 72%. ### Key Trends * Across all five models, the **HC-SMoE** method consistently results in a higher percentage of singleton clusters compared to the **M-SMoE** method. * The difference between the two methods is most pronounced for **Mixtral-8x7B-Instruct-v0.1** and **ERNIE-4.5-21B-A3B-PT**. * The **HC-SMoE** method generally shows smaller error bars, suggesting more consistent results, except for the **Mixtral-8x7B-Instruct-v0.1** model. * The **M-SMoE** method exhibits larger error bars across most models, particularly for **Mixtral-8x7B-Instruct-v0.1**. </details> (a) Singleton cluster proportion <details> <summary>2510.13999v1/x19.png Details</summary>  ### Visual Description Here is a detailed technical description of the bar chart: ### Chart Description This is a grouped bar chart comparing the "Accuracy (%)" of two categories, "Coding" and "MC", across different "Max cluster size" settings. ### Axes and Legend * **Y-axis:** The vertical axis is titled "Accuracy (%)" and has markers at 0.0, 0.1, 0.2, 0.3, and 0.4. * **X-axis:** The horizontal axis is titled "Max cluster size" and features six categorical labels: "None", "32", "16", "8", "4", and "2". * **Legend:** Located in the top right corner [0.8, 0.9], the legend indicates that blue bars represent "Coding" and red bars represent "MC". ### Data Points The chart displays the following approximate accuracy values for each "Max cluster size": | Max cluster size | Coding (Blue) | MC (Red) | | :--- | :--- | :--- | | **None** | ~0.41 | ~0.45 | | **32** | ~0.12 | ~0.45 | | **16** | ~0.11 | ~0.45 | | **8** | ~0.11 | ~0.45 | | **4** | ~0.21 | ~0.44 | | **2** | 0.0 (No bar) | ~0.36 | ### Trends * **Coding:** The accuracy for "Coding" is highest when the "Max cluster size" is "None" (~0.41). It drops significantly to around 0.11-0.12 for cluster sizes of 32, 16, and 8. There is a slight increase to ~0.21 at a cluster size of 4, and no data is presented for a cluster size of 2. * **MC:** The accuracy for "MC" remains relatively stable and high, around 0.44-0.45, for "Max cluster size" settings from "None" down to 4. It shows a noticeable decrease to approximately 0.36 at a cluster size of 2. * **Comparison:** For all "Max cluster size" settings where both data points are present, the "MC" accuracy is consistently higher than the "Coding" accuracy. </details> (b) Restricted cluster sizes Figure A6: (6(a)) Average proportion of singleton clusters vs. model for HC-SMoE and M-SMoE. We find that the clustering algorithms used by our baseline merging methods tend to generate a high proportion of singleton clusters containing just a single expert. In order to achieve the desired compression ratio, the large number of singletons conversely results in some clusters which contain many experts, in some cases $N/2+1$ experts for a layer with $N$ experts are grouped into a single cluster. (6(b)) Accuracy vs. maximum cluster size using M-SMoE to compress 50% of experts in Qwen3-30B. While MC accuracy remains stable up to a maximum cluster size of 4, generative coding capabilities are severely diminished by restricting the clustering algorithm. <details> <summary>2510.13999v1/x20.png Details</summary>  ### Visual Description Here is a detailed description of the information contained in the image. ### **Overview** The image is a multi-panel bar chart comparing the "Coding Accuracy (%)" of three different language models under various compression techniques and datasets. The chart is divided into three panels, each representing a different model. ### **Common Elements** * **Y-axis:** The vertical axis is labeled "Coding Accuracy (%)" and is common to all three panels. The scale ranges from 0 to 60, with major tick marks at every 10 units (0, 10, 20, 30, 40, 50, 60). * **Legend:** A legend is located on the far right of the image, defining the coding for compression levels and datasets. * **Compression:** * **0%:** Represented by a dashed grey horizontal line. * **50%:** Represented by a bar with diagonal hatching. * **25%:** Represented by a solid grey bar (Note: In the chart, this corresponds to the solid colored bars). * **Dataset:** * **c4:** Represented by the color blue. * **CodeAlpaca:** Represented by the color pink. * **X-axis Categories:** Each panel shares the same categorical x-axis labels, representing different compression methods: "REAP", "EAN", "Freq.", "HC-SMoE", and "M-SMoE". --- ### **Panel 1: ERNIE-4.5-21B-A3B-PT** This panel displays the results for the **ERNIE-4.5-21B-A3B-PT** model. * **0% Compression Baseline:** The dashed grey line indicates a baseline accuracy of approximately **55%**. **Data Table for ERNIE-4.5-21B-A3B-PT:** | Method | Dataset | Solid Bar (approx. 25% Comp.) | Hatched Bar (50% Comp.) | | :--- | :--- | :--- | :--- | | **REAP** | c4 (Blue) | ~13% | ~2% | | | CodeAlpaca (Pink) | ~50% | ~44% | | **EAN** | c4 (Blue) | ~14% | ~1% | | | CodeAlpaca (Pink) | ~49% | ~38% | | **Freq.** | c4 (Blue) | ~13% | ~1% | | | CodeAlpaca (Pink) | ~48% | ~37% | | **HC-SMoE**| c4 (Blue) | ~25% | ~1% | | | CodeAlpaca (Pink) | ~50% | ~28% | | **M-SMoE** | c4 (Blue) | ~4% | ~1% | | | CodeAlpaca (Pink) | ~46% | ~9% | **Key Trends for ERNIE:** * The CodeAlpaca dataset consistently yields significantly higher accuracy than the c4 dataset across all methods. * Performance drops as compression increases (0% > solid bar > hatched bar). * The HC-SMoE method shows the highest accuracy for the c4 dataset among the compressed models. --- ### **Panel 2: Qwen3-30B-A3B** This panel displays the results for the **Qwen3-30B-A3B** model. * **0% Compression Baseline:** The dashed grey line indicates a baseline accuracy of approximately **58%**. **Data Table for Qwen3-30B-A3B:** | Method | Dataset | Solid Bar (approx. 25% Comp.) | Hatched Bar (50% Comp.) | | :--- | :--- | :--- | :--- | | **REAP** | c4 (Blue) | ~48% | ~1% | | | CodeAlpaca (Pink) | ~57% | ~56% | | **EAN** | c4 (Blue) | ~1% | ~1% | | | CodeAlpaca (Pink) | ~57% | ~55% | | **Freq.** | c4 (Blue) | ~1% | ~1% | | | CodeAlpaca (Pink) | ~58% | ~47% | | **HC-SMoE**| c4 (Blue) | ~55% | ~47% | | | CodeAlpaca (Pink) | ~53% | ~38% | | **M-SMoE** | c4 (Blue) | ~1% | ~1% | | | CodeAlpaca (Pink) | ~56% | ~41% | **Key Trends for Qwen:** * For the CodeAlpaca dataset, performance is very close to the 0% baseline for most methods, even at 50% compression. * For the c4 dataset, the REAP and HC-SMoE methods maintain high accuracy at the solid bar level, but performance drops drastically for other methods and at 50% compression for REAP. * HC-SMoE shows strong performance for both datasets. --- ### **Panel 3: Mixtral-8x7B-Instruct-v0.1** This panel displays the results for the **Mixtral-8x7B-Instruct-v0.1** model. * **0% Compression Baseline:** The dashed grey line indicates a baseline accuracy of approximately **31%**. **Data Table for Mixtral-8x7B-Instruct-v0.1:** | Method | Dataset | Solid Bar (approx. 25% Comp.) | Hatched Bar (50% Comp.) | | :--- | :--- | :--- | :--- | | **REAP** | c4 (Blue) | ~24% | ~9% | | | CodeAlpaca (Pink) | ~27% | ~13% | | **EAN** | c4 (Blue) | ~24% | ~7% | | | CodeAlpaca (Pink) | ~26% | ~15% | | **Freq.** | c4 (Blue) | ~23% | ~9% | | | CodeAlpaca (Pink) | ~27% | ~13% | | **HC-SMoE**| c4 (Blue) | ~27% | ~10% | | | CodeAlpaca (Pink) | ~28% | ~12% | | **M-SMoE** | c4 (Blue) | ~18% | ~1% | | | CodeAlpaca (Pink) | ~19% | ~5% | **Key Trends for Mixtral:** * The overall accuracy is lower compared to the other two models. * The difference in performance between the c4 and CodeAlpaca datasets is less pronounced than in the other panels. * There is a significant drop in accuracy from the solid bar (approx. 25% compression) to the hatched bar (50% compression) across all methods and datasets. * HC-SMoE generally performs the best among the compressed methods. </details> Figure A7: Coding accuracy vs. calibration dataset. Using domain-specific calibration datasets substantially improves compressed model quality within the target domain. Fine-grained models such as Qwen3-30B and ERNIE suffers greater degradation, with several compression methods failing to produce any coherent output when calibrated on c4. <details> <summary>2510.13999v1/x21.png Details</summary>  ### Visual Description ## Technical Document Image Analysis ### Image Description The image is a grouped bar chart illustrating the "Mean Accuracy (%)" of various methods across four different tasks: "Coding", "Math", "Multiple Choice", and "Creative Writing". The chart compares the performance of these methods at two different compression ratios (50% and 25%) against a 0% compression baseline. ### Chart Details * **Y-axis:** Labeled "Mean Accuracy (%)" with markers at 0, 20, 40, 60, and 80. * **X-axis:** Categorized into four tasks: "Coding", "Math", "Multiple Choice", and "Creative Writing". * **Legend (Compression Ratio):** * **0%:** Represented by a dashed horizontal line indicating the baseline accuracy for each task. * **50%:** Represented by bars with diagonal hatching (///). * **25%:** Represented by solid bars. * **Legend (Method):** * **REAP (ours) (specific):** Light Blue * **REAP (ours) (general):** Dark Blue * **EAN (specific):** Light Red * **EAN (general):** Dark Red * **Frequency (specific):** Light Green * **Frequency (general):** Dark Green * **HC-SMoE (specific):** Light Yellow/Gold * **HC-SMoE (general):** Dark Yellow/Gold * **M-SMoE (specific):** Light Cyan * **M-SMoE (general):** Dark Cyan ### Data Extraction The following table summarizes the approximate Mean Accuracy (%) for each method, task, and compression ratio, based on the visual representation in the chart. | Task | Method | 0% Compression (Baseline) | 50% Compression (Hatched) | 25% Compression (Solid) | | :--- | :--- | :--- | :--- | :--- | | **Coding** | | **~58%** | | | | | REAP (ours) (specific) | | ~56% | ~54% | | | REAP (ours) (general) | | ~53% | ~51% | | | EAN (specific) | | ~57% | ~55% | | | EAN (general) | | ~32% | ~30% | | | Frequency (specific) | | ~57% | ~55% | | | Frequency (general) | | ~47% | ~45% | | | HC-SMoE (specific) | | ~53% | ~51% | | | HC-SMoE (general) | | ~41% | ~39% | | | M-SMoE (specific) | | ~57% | ~55% | | | M-SMoE (general) | | ~41% | ~39% | | **Math** | | **~89%** | | | | | REAP (ours) (specific) | | ~88% | ~86% | | | REAP (ours) (general) | | ~81% | ~79% | | | EAN (specific) | | ~88% | ~86% | | | EAN (general) | | ~86% | ~84% | | | Frequency (specific) | | ~88% | ~86% | | | Frequency (general) | | ~40% | ~38% | | | HC-SMoE (specific) | | ~85% | ~83% | | | HC-SMoE (general) | | ~73% | ~71% | | | M-SMoE (specific) | | ~89% | ~87% | | | M-SMoE (general) | | ~83% | ~81% | | **Multiple Choice** | | **~72%** | | | | | REAP (ours) (specific) | | ~69% | ~67% | | | REAP (ours) (general) | | ~59% | ~57% | | | EAN (specific) | | ~65% | ~63% | | | EAN (general) | | ~59% | ~57% | | | Frequency (specific) | | ~65% | ~63% | | | Frequency (general) | | ~56% | ~54% | | | HC-SMoE (specific) | | ~67% | ~65% | | | HC-SMoE (general) | | ~54% | ~52% | | | M-SMoE (specific) | | ~56% | ~54% | | | M-SMoE (general) | | ~45% | ~43% | | **Creative Writing** | | **~81%** | | | | | REAP (ours) (specific) | | ~80% | ~78% | | | REAP (ours) (general) | | ~72% | ~70% | | | EAN (specific) | | ~80% | ~78% | | | EAN (general) | | ~70% | ~68% | | | Frequency (specific) | | ~80% | ~78% | | | Frequency (general) | | ~68% | ~66% | | | HC-SMoE (specific) | | ~50% | ~48% | | | HC-SMoE (general) | | ~4% | ~2% | | | M-SMoE (specific) | | ~80% | ~78% | | | M-SMoE (general) | | ~72% | ~70% | ### Key Trends * **General vs. Specific:** Across all tasks and methods, the "specific" version of a method consistently outperforms its "general" counterpart at both 50% and 25% compression ratios. * **Compression Impact:** For all methods and tasks, accuracy decreases as the compression ratio increases from 50% to 25%. * **Baseline Comparison:** In most cases, the compressed models (both 50% and 25%) achieve lower accuracy than the 0% compression baseline. However, some "specific" methods at 50% compression come very close to the baseline, particularly in the "Math" and "Creative Writing" tasks. * **Method Performance:** "REAP (ours) (specific)" and "EAN (specific)" generally show strong performance across tasks, often being the top-performing methods at both compression ratios. "HC-SMoE (general)" shows a significant drop in performance, especially in the "Creative Writing" task. </details> Figure A8: Mean accuracy vs. task type for models calibrated with domain specific data versus general data. The “general” calibration data consists of the combination of evol-codealpaca-v1, WritingPrompts curated, and tulu-3-sft-personas-math and includes three times the total number of samples as the domain-specific calibration datasets. While the general data calibrated models perform reasonably well at 25% compression, domain-specific data is crucial for high-quality compressed SMoE accuracy at 50% compression.