# IG-Pruning: Input-Guided Block Pruning for Large Language Models
Abstract
With the growing computational demands of large language models (LLMs), efficient inference has become increasingly critical for practical deployment. Depth pruning has emerged as a promising approach for reducing the computational costs of large language models by removing transformer layers. However, existing methods typically rely on fixed block masks, which can lead to suboptimal performance across different tasks and inputs. In this paper, we propose IG-Pruning, a novel input-aware block-wise pruning method that dynamically selects layer masks at inference time. Our approach consists of two stages: (1) Discovering diverse mask candidates through semantic clustering and $L_{0}$ optimization, and (2) Implementing efficient dynamic pruning without the need for extensive training. Experimental results demonstrate that our method consistently outperforms state-of-the-art static depth pruning methods, making it particularly suitable for resource-constrained deployment scenarios. https://github.com/ictnlp/IG-Pruning
IG-Pruning: Input-Guided Block Pruning for Large Language Models
Kangyu Qiao 1,3, Shaolei Zhang 1,3, Yang Feng 1,2,3 Corresponding author: Yang Feng. 1 Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) 2 Key Laboratory of AI Safety, Chinese Academy of Sciences 3 University of Chinese Academy of Sciences, Beijing, China {qiaokangyu24s, zhangshaolei20z, fengyang}@ict.ac.cn
1 Introduction
Large Language Models (LLMs) Brown et al. (2020); AI@Meta (2024); QwenTeam (2025); Zhang et al. (2024b, 2023a) have demonstrated remarkable capabilities across a wide range of natural language processing tasks. However, their immense model size and computational demands present significant deployment challenges Wang et al. (2024); Zhou et al. (2024), particularly in resource-constrained environments and for latency-sensitive real-time inference scenarios. To address this, pruning techniques have become a crucial area of research Ma et al. (2023); Sun et al. (2023); Frantar and Alistarh (2023); Ashkboos et al. (2024); Fang et al. (2024); Ling et al. (2024); Zhang et al. (2023b); Gu et al. (2021), being highly favored due to their potential for reducing parameters for efficient inference.
As large LLMs continue to scale in size, researchers have identified significant redundancy within their layer structures. Studies from Liu et al. (2023); Men et al. (2024); Gromov et al. (2024) reveal that word embeddings in adjacent layers often change slightly due to residual connection, suggesting that selective layer removal may have minimal impact on performance. These findings have motivated increasing research interest in discovering effective depth pruning strategies for LLMs, which aim to reduce the number of transformer layers or blocks in the model architecture while maintaining performance. In recent years, depth pruning methods Song et al. (2024); Sieberling et al. (2024); Kim et al. (2024); Ling et al. (2024) have emerged as a promising approach for reducing LLM computational costs. Compared with fine-grained structured pruning methods (which remove the neurons or channels), depth pruning has demonstrated superior computational efficiency advantages in practical deployments Kim et al. (2024).
<details>
<summary>x1.png Details</summary>
### Visual Description
## Diagram: Model Performance and Masking
### Overview
The image presents a diagram comparing the performance of two models, likely with different masking strategies, on two sets of tasks: "PPL" (Perplexity) and "Other Tasks." The diagram includes bar charts representing performance on these tasks and visual representations of the masking strategies used (Mask1 and Mask2).
### Components/Axes
* **PPL Chart:**
* X-axis: Implied categories for two models.
* Y-axis: Perplexity score (lower is better). No explicit scale is provided.
* Horizontal dashed line: Represents a performance threshold or target.
* **Other Tasks Chart:**
* X-axis: Three task categories represented by icons: a cloud (likely representing natural language understanding or generation), a calculator (likely representing arithmetic or reasoning), and a lightbulb (likely representing knowledge or problem-solving).
* Y-axis: Performance metric (higher is better). No explicit scale is provided.
* **Masking Strategies:**
* Mask1: A sequence of rectangular blocks, some solid (purple) and some dashed.
* Mask2: A sequence of rectangular blocks, some solid (orange) and some dashed.
* Horizontal lines connect the blocks, indicating a sequence or flow.
### Detailed Analysis
**PPL Chart:**
* Two bars are shown. The left bar is purple, and the right bar is orange.
* The height of the purple bar is approximately equal to the height of the orange bar.
* Both bars reach the horizontal dashed line.
**Other Tasks Chart:**
* Three pairs of bars are shown, corresponding to the three task categories.
* For the cloud task: The purple bar is shorter than the orange bar. Purple bar height is ~ 0.4, orange bar height is ~0.6.
* For the calculator task: The purple bar is taller than the orange bar. Purple bar height is ~ 0.9, orange bar height is ~0.7.
* For the lightbulb task: The purple bar is shorter than the orange bar. Purple bar height is ~ 0.4, orange bar height is ~0.6.
**Masking Strategies:**
* **Mask1 (Purple):** Dashed-Solid-Solid-Dashed-Solid-Dashed
* **Mask2 (Orange):** Dashed-Solid-Dashed-Dashed-Solid-Solid
### Key Observations
* On the PPL task, both models perform similarly, reaching the target performance level.
* On the "Other Tasks," the models exhibit different performance profiles across the three task categories. Mask1 (purple) performs better on the calculator task, while Mask2 (orange) performs better on the cloud and lightbulb tasks.
* The masking strategies differ in the arrangement of solid and dashed blocks, suggesting different approaches to information processing or attention.
### Interpretation
The diagram suggests that the two models, employing different masking strategies, achieve comparable performance on the PPL task. However, their performance diverges on other tasks. Mask1, with its specific masking pattern, seems to excel at tasks represented by the calculator icon (potentially arithmetic or logical reasoning). Mask2, with its alternative masking pattern, appears to be more effective on tasks represented by the cloud and lightbulb icons (potentially natural language understanding and knowledge-based tasks).
The masking strategies likely influence how the models process information, leading to these performance differences. The solid blocks may represent active processing or attention, while the dashed blocks may represent masked or ignored information. The specific arrangement of these blocks could be optimized for different types of tasks.
The diagram highlights the importance of masking strategies in model performance and suggests that different strategies may be better suited for different tasks. Further investigation would be needed to understand the specific mechanisms by which these masking strategies influence performance.
</details>
Figure 1: Different Mask structure can lead to similar perplexity scores but exhibit significant performance variations across different downstream tasks.
However, a critical limitation of existing depth pruning methods is their reliance on a fixed layer pruning mask determined offline based on global layer importance metrics at a given sparsity level. This static approach is problematic because different fixed pruning masks, even at the same sparsity level, can exhibit significant performance variations across different downstream tasks. For instance, we observe that perplexity (PPL) is commonly used as a saliency metric for layer pruning Sieberling et al. (2024); Kim et al. (2024), but as illustrated in Figure 1, different mask structures can achieve similar perplexity scores while exhibiting substantially different performance across various downstream tasks. To overcome these limitations and enable adaptive computation pathways, researchers have explored various dynamic routing approaches Elhoushi et al. (2024); Fan et al. (2024); Del Corro et al. ; Schuster et al. (2022); Raposo et al. (2024); Tan et al. (2024); Wu et al. (2024). However, most existing methods perform dynamic routing at the token level, which introduces significant drawbacks: they lack comprehensive understanding of sentence-level semantics, potentially leading to globally inconsistent routing decisions. Furthermore, these approaches typically incur substantial computational overhead from frequent token-level routing calls and require extensive training of additional router networks alongside the original model parameters, making them computationally expensive and time-consuming to implement.
To address the challenges identified in existing works, we propose IG-Pruning, a novel block-wise pruning method that dynamically selects layer masks based on input characteristics at inference time. Our approach consists of two stages: (1) a semantic clustering-based mask discovery stage that identifies diverse, high-quality mask candidates while capturing global information through rapidly converging trainable masks, and (2) a lightweight inference-time routing mechanism that requires no additional training of the base model parameters, enabling efficient dynamic adaptation to varying inputs.
Extensive evaluations demonstrate that our approach consistently outperforms state-of-the-art static pruning methods across different sparsity levels and model architectures on various zero-shot tasks. For Llama-3-8B at 25% sparsity, IG-Pruning preserves 87.18% of dense model performance, surpassing the best baseline by 10.86 percentage points. Similarly, for Qwen-3-8B, IG-Pruning maintains 96.01% of dense model performance at 13.9% sparsity, compared to 90.37% for the best baseline.
Our method trains only mask parameters while keeping model weights frozen, enabling rapid adaptation with minimal computational overhead. During inference stage, it incurs negligible routing overhead by efficiently skipping unimportant layers; and these advancements provides a viable path toward deploying powerful LLMs in environments with limited computational resources.
<details>
<summary>x2.png Details</summary>
### Visual Description
## Diagram: Two-Stage Model Architecture
### Overview
The image presents a two-stage model architecture, likely for natural language processing. Stage 1 focuses on data calibration and initialization using K-means clustering, while Stage 2 involves routing, embedding, and attention mechanisms.
### Components/Axes
**Stage 1 (Left, Peach Background):**
* **Calibration Data:** A dashed-line box at the bottom-left.
* **Sentence Encoder:** A dashed-line box above "Calibration Data".
* **K-means:** Text label with an arrow pointing upwards.
* **C1, C2, C3, C4:** Four circles, representing clusters. C2 is slightly darker than the others.
* **Soft Mask Training:** A dashed-line box at the top-left.
**Stage 2 (Right, Light Yellow Background):**
* **Input:** A dashed-line box at the bottom.
* **Router:** A rounded rectangle containing several components.
* **Sentence Encoder:** A dashed-line box at the bottom of the Router.
* **Embedding Pool:** A collection of circles, some filled with orange and some gray.
* **Min Distance:** Text label with an arrow pointing upwards.
* **Select:** Text label with an arrow pointing upwards.
* A stack of blue rectangles at the top of the Router, some with orange highlights.
* **Prune:** Text label with an arrow pointing to a stack of two dashed-line boxes.
* **FFN, ATTN:** Alternating blocks of "FFN" (Feed Forward Network) and "ATTN" (Attention) modules.
### Detailed Analysis
**Stage 1:**
* Calibration Data is fed into a Sentence Encoder.
* The output of the Sentence Encoder is used for K-means clustering, resulting in four clusters (C1, C2, C3, C4).
* These clusters initialize the Soft Mask Training process.
**Stage 2:**
* Input is fed into a Sentence Encoder within the Router.
* The output of the Sentence Encoder goes into the Embedding Pool.
* "Min Distance" is calculated, and a selection is made.
* The selected embeddings are then processed through a series of blue rectangles.
* The "Prune" operation leads to a stack of two dashed-line boxes.
* The pruned output is then processed through alternating layers of FFN and ATTN.
### Key Observations
* The diagram illustrates a sequential flow of data and processing steps.
* K-means clustering in Stage 1 initializes the Soft Mask Training.
* The Router in Stage 2 appears to be a key component for embedding selection and pruning.
* The alternating FFN and ATTN layers suggest a transformer-based architecture.
### Interpretation
The diagram depicts a two-stage model designed for a specific NLP task. Stage 1 focuses on data preparation and initialization, likely to improve the efficiency or accuracy of the subsequent stages. The K-means clustering suggests that the model is designed to handle different types of input data, with each cluster representing a distinct category or feature. Stage 2 implements a more complex architecture, potentially involving attention mechanisms and pruning techniques to optimize performance. The "Router" component seems to play a crucial role in selecting and routing relevant information within the model. The overall architecture suggests a sophisticated approach to NLP, combining data-driven initialization with advanced neural network techniques.
</details>
Figure 2: Overview of our method. The approach consists of two stages: (1) Preparing mask candidates through input clustering and soft mask training; (2) Dynamic pruning that selects the appropriate mask for each input at inference time. This enables efficient computation by selectively skipping layers based on input characteristics while maintaining model performance.
2 Related Work
Most static depth pruning approaches focus on calculating saliency scores for each transformer block, and removing layers according to these scores. Commonly used saliency metrics include cosine similarity Song et al. (2024); Men et al. (2024), magnitude, second-order derivatives Kim et al. (2024), and perplexity Sieberling et al. (2024). These works calculate layer importance as if they are independent of others, which ignores the coupling connections between layers. As discovered in Fan et al. (2024), contiguous middle layers often exhibit similar saliency scores, which inspired Chen et al. (2024) to use small FFN or transformer blocks to replace contiguous layers. EvoPress Sieberling et al. (2024) found that lower per-layer error does not necessarily lead to better performance, and proposed an evolutionary search algorithm to generate offspring from parent masks, then select better candidates with lower perplexity or KL divergence. Rather than directly removing layers, LaCO Yang et al. (2024) collapses consecutive redundant model layers via layer averaging. MKA Liu et al. (2024a) transforms layer activations into low-dimensional manifolds using diffusion kernel algorithms and evaluates saliency using the NPIB metric.
Beyond one-shot pruning approaches, dynamically skipping unimportant layers during inference has also emerged as a promising research direction. Early approaches include early skipping Del Corro et al. ; Zhu et al. (2024), early exit Elhoushi et al. (2024), and periodic skipping Liu et al. (2024b). However, these methods typically require routers for each layer and demand elaborate training of original weights to recover performance. Dynamic skipping has also been adopted in long-context and multimodal models. Adaskip He et al. (2024) focused on adaptive layer skipping for long-context models, accelerating both prefilling and decoding phases. RoE Wu et al. (2024) employs token-wise routing for multimodal LLMs and trains low-rank adapters to replace the skipped layers.
3 Method
As illustrated in Figure 2, our framework consists of two main stages: (1) Mask candidate discovery and (2) Dynamic routing. In the first stage, we cluster the semantic space of inputs and train cluster-specific masks using hard concrete distributions, resulting in diverse yet high-quality mask candidates that each specialize in handling different input patterns. During the second stage, at inference time, we employ a lightweight routing mechanism that maps each input to its most semantically similar cluster and applies the corresponding pre-trained mask, enabling efficient dynamic adaptation without requiring additional training of router networks or base model parameters.
3.1 Stage 1: Discovering Mask Candidates
In the first stage, we aim to discover a set of effective mask candidates for dynamic routing. Unlike existing routing methods that typically employ per-layer router networks to make skip decisions, we propose a global routing strategy that dynamically selects routing paths from a carefully curated candidate mask set.
We design our mask candidate discovery process to satisfy two key requirements: Quality: Masks must maintain strong general language generation capabilities. Diversity: The candidate set must provide sufficient variety to handle different input patterns effectively.
To meet these requirements, we leverage hard concrete distribution to model transformer block masks to capture global routing information, and apply $L_{0}$ optimization with cluster-specific calibration data, generating masks that cover diverse computational needs.
Input Clustering.
First, an encoder is used to encode each sentence $x_{i}$ in the calibration dataset into a fixed-dimensional embedding vector $e_{i}$ :
$$
e_{i}=\text{Encoder}(x_{i}) \tag{1}
$$
where $x_{i}$ represents the $i$ -th input, and $e_{i}∈\mathbb{R}^{d}$ , with $d$ being the dimension of the embedding vector. Next, the K-means algorithm is applied to cluster all embedding vectors $e_{1},e_{2},...,e_{M}$ , where $M$ is the size of the calibration set. The K-means algorithm aims to find $N$ clusters $S=\{S_{1},S_{2},...,S_{N}\}$ that minimize the within-cluster sum of squares:
$$
\arg\min_{S}\sum_{k=1}^{N}\sum_{e_{i}\in S_{k}}\|e_{i}-\mu_{k}\|^{2} \tag{2}
$$
where $\mu_{k}$ is the centroid of cluster $S_{k}$ . This results in $N$ cluster centers, each representing a class of semantically similar input sentences.
Mask Training.
Hard concrete distribution Louizos et al. (2018); Xia et al. (2022, 2024) has been widely adopted in structured pruning. Following prior work, we incorporate hard concrete distribution to model transformer block masks, and use $L_{0}$ optimization to generate layer masks, enabling joint learning of all layer masks while incorporating global information.
For each cluster $S_{k}$ , we train a dedicated layer mask $z^{(k)}∈\mathbb{R}^{B}$ using hard concrete distribution and Lagrangian sparsity, where $B$ is the total number of blocks in the model (for block-wise pruning, $B=2L$ where $L$ is the number of transformer layers, representing both attention and FFN blocks separately). Specifically, the masks $z^{(k)}$ are modeled as follows:
First, for each block $i$ in the model, sample $u^{(k)}_{i}$ from a uniform distribution:
$$
u^{(k)}_{i}\sim\text{Uniform}(0,1),\quad i\in\{1,2,\ldots,B\} \tag{3}
$$
Then, compute the soft mask value $s^{(k)}_{i}$ for each block using the sigmoid function:
$$
s^{(k)}_{i}=\sigma\left(\frac{1}{\beta}\log{\frac{u^{(k)}_{i}}{1-u^{(k)}_{i}}}+\log\alpha^{(k)}_{i}\right) \tag{4}
$$
Stretch the soft mask values to a specific interval $[l,r]$ :
$$
\tilde{s}^{(k)}_{i}=s^{(k)}_{i}\times(r-l)+l \tag{5}
$$
Finally, obtain the hardened mask $z^{(k)}_{i}$ for each block by clipping:
$$
z^{(k)}_{i}=\min(1,\max(0,\tilde{s}^{(k)}_{i})) \tag{6}
$$
The complete mask vector for cluster $k$ is then $z^{(k)}=[z^{(k)}_{1},z^{(k)}_{2},...,z^{(k)}_{B}]$ , where each element corresponds to a specific transformer block in the model. During training, these mask values are soft (continuous values between 0 and 1), functioning as scaling parameters. During inference, they are binarized to either 0 (block skipped) or 1 (block executed).
Here, $\sigma$ denotes the sigmoid function. The temperature $\beta$ is fixed hyperparameter, and $l<0,r>0$ are two constants that stretch the sigmoid function output. $\alpha^{(k)}_{i}$ are the main learnable parameters for i-th block mask value in cluster $k$ .
We enforce a target sparsity via a Lagrangian term. Let $s_{\text{target}}$ be the target sparsity and $t^{(k)}$ be the current sparsity of mask $z^{(k)}$ (computed as the proportion of zeroes in the mask), the Lagrangian penalty term $L_{s}^{(k)}$ is:
$$
L_{s}^{(k)}=\lambda_{1}^{(k)}(t^{(k)}-s_{\text{target}})+\lambda_{2}^{(k)}(t^{(k)}-s_{\text{target}})^{2} \tag{7}
$$
For the $k$ -th cluster, the optimization objective for its mask parameters $\log\alpha^{(k)}$ is to minimize:
$$
L_{\text{total}}^{(k)}=\sum_{x_{j}\in S_{k}}L_{\text{LM}}(x_{j};W\odot z^{(k)})+L_{s}^{(k)} \tag{8}
$$
where $L_{\text{LM}}$ is the language modeling loss and $W$ represents the model weights.
Routing Decision.
To implement dynamic routing decisions, we maintain an embedding pool for each semantic cluster to represent the cluster’s features. These embeddings $c_{k}$ are initialized using the cluster centers $\mu_{k}$ . During inference, for each input sequence, we first extract its embedding representation $e_{x}$ through the encoder, then calculate the Euclidean distance between this embedding and each cluster embedding $c_{k}$ . Based on the calculated distances, we select the most similar cluster as the best match for that input:
$$
k^{*}=\arg\min_{k}||e_{x}-c_{k}||_{2}^{2},k\in\{1,2,\ldots,N\} \tag{9}
$$
After determining the best matching cluster, we directly adopt the trained mask corresponding to that cluster as the final execution mask for input $x$ :
$$
M^{x}=z^{(k^{*})} \tag{10}
$$
where $z^{(k^{*})}$ is the binary mask vector associated with cluster $k^{*}$ , containing all block-level mask values.
Dynamic Routing for FFN and Attention Blocks.
Our dynamic routing approach employs different strategies for Feed-Forward layers and Attention layers. During training, the layer mask values are soft, functioning as scaling parameters that directly multiply with the outputs of FFN and Attention components. This enables gradient-based optimization through backpropagation. During inference, we use hard binary masks containing only 0 and 1, where FFN layers are completely skipped when the corresponding mask value is 0. For Attention layers, the approach is more nuanced due to the necessity of maintaining key-value caches for autoregressive generation. When an Attention layer is marked for skipping, we still compute the key and value projections to maintain the KV cache, but we bypass the computationally expensive scaled dot-product operation between queries and keys. Specifically, for a transformer layer $i$ with mask value $M_{i}^{x}=0$ , the FFN computation $\text{FFN}(x_{i})$ is entirely skipped, while for Attention, we compute $K=W_{K}x_{i}$ and $V=W_{V}x_{i}$ for the cache but skip $\text{Attention}(Q,K,V)=\text{softmax}(QK^{T}/\sqrt{d})V$ . This selective computation strategy preserves the model’s autoregressive capabilities while reducing computational overhead.
4 Experiment
| Model | Sparsity | Method | OBQA | WG | HS | PIQA | ARC-E | ARC-C | Average | Percentage |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Llama-3-8B | 0% | Dense | 44.6 | 73.24 | 79.16 | 80.79 | 77.82 | 53.24 | 68.14 | 100% |
| 12.5% | SLEB | 38.6 | 69.45 | 70.71 | 77.63 | 70.28 | 43.00 | 61.61 | 90.42% | |
| ShortenedLlama | 39.2 | 61.56 | 66.84 | 76.33 | 67.63 | 38.57 | 58.36 | 85.64% | | |
| EvoPress | 41.2 | 70.17 | 72.03 | 77.75 | 71.00 | 43.69 | 62.64 | 91.93% | | |
| IG-Pruning | 43.6 | 72.93 | 77.26 | 79.38 | 77.06 | 51.62 | 66.98 | 98.29% | | |
| 25% | SLEB | 33.8 | 53.90 | 57.96 | 72.25 | 57.32 | 31.56 | 51.13 | 75.04% | |
| EvoPress | 32.8 | 57.93 | 58.16 | 71.06 | 58.38 | 33.70 | 52.01 | 76.32% | | |
| ShortenedLlama | 33.6 | 53.91 | 57.98 | 72.31 | 57.15 | 31.74 | 51.12 | 75.01% | | |
| IG-Pruning | 40.0 | 68.98 | 67.53 | 76.12 | 63.43 | 40.36 | 59.40 | 87.18% | | |
| 37.5% | SLEB | 28.4 | 52.24 | 46.46 | 65.77 | 46.96 | 28.41 | 44.71 | 65.61% | |
| EvoPress | 28.2 | 51.22 | 45.58 | 65.18 | 48.15 | 28.50 | 44.47 | 65.26% | | |
| ShortenedLlama | 28.6 | 52.41 | 45.90 | 64.69 | 42.68 | 27.47 | 43.63 | 64.02% | | |
| IG-Pruning | 31.8 | 58.01 | 49.63 | 65.94 | 48.44 | 30.38 | 47.37 | 69.51% | | |
| Qwen-3-8B | 0% | Dense | 41.8 | 67.96 | 74.93 | 77.48 | 80.77 | 56.40 | 66.56 | 100% |
| 13.9% | SLEB | 37.4 | 60.85 | 62.45 | 77.52 | 74.45 | 47.09 | 59.96 | 90.09% | |
| ShortenedLlama | 37.0 | 59.27 | 61.82 | 75.14 | 71.00 | 45.14 | 58.23 | 87.49% | | |
| EvoPress | 39.0 | 61.96 | 67.76 | 75.57 | 70.33 | 46.25 | 60.15 | 90.37% | | |
| IG-Pruning | 39.8 | 65.82 | 69.44 | 77.09 | 77.35 | 53.92 | 63.90 | 96.01% | | |
| 25% | SLEB | 36.6 | 56.35 | 53.95 | 72.47 | 65.36 | 37.20 | 53.66 | 80.62% | |
| EvoPress | 37.0 | 58.08 | 57.18 | 71.43 | 62.28 | 38.65 | 54.10 | 81.29% | | |
| ShortenedLlama | 35.6 | 53.99 | 52.20 | 70.84 | 64.69 | 36.43 | 52.29 | 78.56% | | |
| IG-Pruning | 35.6 | 60.46 | 61.65 | 73.39 | 68.94 | 44.80 | 57.47 | 86.35% | | |
| 36.1% | SLEB | 29.6 | 52.40 | 44.02 | 65.77 | 51.68 | 31.39 | 45.81 | 68.82% | |
| EvoPress | 31.6 | 52.17 | 45.29 | 62.95 | 51.09 | 29.18 | 45.38 | 68.18% | | |
| ShortenedLlama | 28.2 | 50.91 | 37.08 | 61.75 | 46.13 | 25.43 | 41.58 | 62.48% | | |
| IG-Pruning | 32.6 | 53.43 | 49.17 | 65.83 | 54.21 | 32.17 | 47.90 | 71.96% | | |
Table 1: Zero-shot evaluation results on Llama-3-8B and Qwen-3-8B across multiple sparsity levels.
4.1 Experimental Setup
Datasets and Evaluation Metrics.
Following prior work, we use lm-evaluation-harness Gao et al. (2023) to evaluate our method on six widely-used zero-shot tasks: OpenBookQA Mihaylov et al. (2018), which tests elementary-level science reasoning requiring the combination of facts with commonsense knowledge; Winogrande Sakaguchi et al. (2021), a large-scale adversarial dataset for testing pronoun disambiguation through commonsense reasoning; HellaSwag Zellers et al. (2019), which challenges models to select plausible scenario completions through commonsense inference; PIQA Bisk et al. (2020), focused on physical commonsense knowledge; and the ARC dataset Clark et al. (2018), divided into ARC-Easy and ARC-Challenge subsets for testing scientific reasoning at different difficulty levels. Llama-3-8B AI@Meta (2024) and Qwen-3-8B QwenTeam (2025) are used as our base models, and we use all-MiniLM-L6-v2 from sentence transformer Reimers and Gurevych (2019) as sentence encoder. For calibration data for clustering and layer mask training, we use fineweb-edu Lozhkov et al. (2024), which contains high quality synthetic data used for LLM pretraining.
Baselines and Setups.
To evaluate our dynamic block pruning approach against static methods, we select three representative block pruning techniques for comparison:
- SLEB Song et al. (2024): A method that iteratively eliminates redundant transformer blocks based on cosine similarity between adjacent layers.
- ShortenedLlama Kim et al. (2024): An approach that uses magnitude, second-order derivatives, or perplexity to measure block-level importance. After identifying unimportant blocks, this method removes them in a single pass.
- EvoPress Sieberling et al. (2024): A technique leveraging evolutionary algorithms to search for optimal pruning masks with improved perplexity or KL divergence. Starting with a random initial configuration, in each generation it mutates the compression levels of selected layers and retains the best candidates according to a fitness function. This approach yields better results but incurs higher computational costs.
For all baseline methods, we perform one-shot pruning that identifies and eliminates redundant transformer blocks without retraining, and we use wikitext2 Merity et al. (2016) as calibration set for baselines.
4.2 Main Results
IG-Pruning consistently outperforms all baseline methods across all evaluated sparsity configurations for both Llama-3-8B and Qwen-3-8B models. In this paper, the sparsity level is defined as the ratio of the number of skipped blocks to the total number of blocks in the model. For Llama-3-8B at 12.5% sparsity, IG-Pruning maintains 98.29% of the dense model performance, surpassing the best baseline (EvoPress) by 6.36 percentage points. This advantage becomes even more significant at 25% sparsity, where IG-Pruning achieves 87.18% of dense performance compared to the best baseline at 76.32%, representing a 10.86 percentage point improvement. Similarly, for Qwen-3-8B, IG-Pruning preserves 96.01% of dense model performance at 13.9% sparsity, compared to 90.37% for the best baseline. These consistent improvements across different model architectures demonstrate the inherent advantage of our dynamic routing strategy over static pruning methods.
4.3 Analysis
Mask Training Efficiency.
In Stage 1 of our approach, model parameters remain frozen while only layer mask parameters undergo optimization. We set a higher learning rate for $L_{0}$ module, enabling rapid mask convergence without extensive training periods. For our experiments, we sample 1,000 examples from each cluster for training, utilizing 4 NVIDIA H800 GPUs. Hyperparameters can be found in Appendix 5. For configurations with sparsity levels below 25% across 16 clusters, all masks can be trained in approximately 15 minutes. Higher sparsity (37%) requires around one hour of training time for mask convergence. Our method requires training, but it only trains the block mask parameters, while the parameters in the original models are frozen. Therefore, it doesn’t require excessive memory, which has been tested successfully on a single RTX 3090 for 8B model.
Block-level vs. Layer-level Pruning.
To investigate the impact of pruning granularity on model performance, we conducted comprehensive experiments comparing block-level and layer-level pruning across different sparsity configurations. As shown in Figure 3, block-level pruning consistently outperforms layer-level pruning across all tasks, with performance advantages that vary based on sparsity levels. The gap between these approaches is most significant at sparsity levels around 20%, where block pruning demonstrates substantially better performance. This suggests that independently pruning Attention and FFN components provides the model with greater flexibility to maintain critical capabilities while reducing computational costs.
<details>
<summary>x3.png Details</summary>
### Visual Description
## Chart Type: Multiple Line Graphs
### Overview
The image presents six line graphs arranged in a 2x3 grid. Each graph displays the relationship between "Sparsity" (x-axis) and "Accuracy (%)" (y-axis) for two different methods: "Block" and "Layer". The graphs are titled OPENBOOKQA, WINOGRANDE, HELLASWAG, PIQA, ARC-EASY, and ARC-CHALLENGE.
### Components/Axes
* **X-axis (Horizontal):** "Sparsity" ranging from 0% to 40% in increments of 8% (0%, 8%, 16%, 24%, 32%, 40%).
* **Y-axis (Vertical):** "Accuracy (%)". The range varies for each graph, but generally spans from approximately 25% to 80%.
* **Titles:** Each graph has a title indicating the dataset or task: OPENBOOKQA, WINOGRANDE, HELLASWAG, PIQA, ARC-EASY, ARC-CHALLENGE.
* **Legend:** Each graph includes a legend in the top-right corner:
* Blue line with circular markers: "Block"
* Orange dashed line with square markers: "Layer"
### Detailed Analysis
**1. OPENBOOKQA**
* **Y-axis:** Ranges from 28% to 44%.
* **Block (Blue):** Starts at approximately 44%, remains relatively stable until 8% sparsity, then gradually decreases to approximately 31% at 40% sparsity.
* (0%, 44%), (8%, 44%), (16%, 43.5%), (24%, 40%), (32%, 35%), (40%, 31%)
* **Layer (Orange):** Starts at approximately 44%, decreases sharply until 16% sparsity, fluctuates between 33% and 37% until 32% sparsity, then decreases to approximately 28% at 40% sparsity.
* (0%, 44%), (8%, 39%), (16%, 33%), (24%, 37%), (32%, 33%), (40%, 28%)
**2. WINOGRANDE**
* **Y-axis:** Ranges from 50% to 75%.
* **Block (Blue):** Starts at approximately 73%, remains relatively stable until 24% sparsity, then decreases to approximately 52% at 40% sparsity.
* (0%, 73%), (8%, 73%), (16%, 74%), (24%, 73%), (32%, 60%), (40%, 52%)
* **Layer (Orange):** Starts at approximately 72%, decreases until 24% sparsity, remains relatively stable until 32% sparsity, then decreases to approximately 52% at 40% sparsity.
* (0%, 72%), (8%, 68%), (16%, 63%), (24%, 57%), (32%, 58%), (40%, 52%)
**3. HELLASWAG**
* **Y-axis:** Ranges from 40% to 80%.
* **Block (Blue):** Starts at approximately 80%, gradually decreases to approximately 42% at 40% sparsity.
* (0%, 80%), (8%, 79%), (16%, 78%), (24%, 73%), (32%, 60%), (40%, 42%)
* **Layer (Orange):** Starts at approximately 79%, decreases to approximately 38% at 40% sparsity.
* (0%, 79%), (8%, 74%), (16%, 66%), (24%, 62%), (32%, 50%), (40%, 38%)
**4. PIQA**
* **Y-axis:** Ranges from 60% to 82%.
* **Block (Blue):** Starts at approximately 81%, gradually decreases to approximately 63% at 40% sparsity.
* (0%, 81%), (8%, 81%), (16%, 79%), (24%, 77%), (32%, 72%), (40%, 63%)
* **Layer (Orange):** Starts at approximately 80%, decreases to approximately 61% at 40% sparsity.
* (0%, 80%), (8%, 77%), (16%, 75%), (24%, 73%), (32%, 68%), (40%, 61%)
**5. ARC-EASY**
* **Y-axis:** Ranges from 40% to 80%.
* **Block (Blue):** Starts at approximately 78%, gradually decreases to approximately 40% at 40% sparsity.
* (0%, 78%), (8%, 78%), (16%, 76%), (24%, 72%), (32%, 53%), (40%, 40%)
* **Layer (Orange):** Starts at approximately 77%, decreases to approximately 36% at 40% sparsity.
* (0%, 77%), (8%, 69%), (16%, 64%), (24%, 57%), (32%, 50%), (40%, 36%)
**6. ARC-CHALLENGE**
* **Y-axis:** Ranges from 24% to 56%.
* **Block (Blue):** Starts at approximately 54%, gradually decreases to approximately 26% at 40% sparsity.
* (0%, 54%), (8%, 55%), (16%, 53%), (24%, 50%), (32%, 40%), (40%, 26%)
* **Layer (Orange):** Starts at approximately 54%, decreases to approximately 25% at 40% sparsity.
* (0%, 54%), (8%, 48%), (16%, 40%), (24%, 36%), (32%, 30%), (40%, 25%)
### Key Observations
* In all six graphs, both "Block" and "Layer" accuracy generally decreases as sparsity increases.
* The "Block" method tends to maintain a higher accuracy than the "Layer" method, especially at higher sparsity levels.
* The rate of accuracy decrease varies across different datasets/tasks. OPENBOOKQA and WINOGRANDE show a more stable accuracy for the "Block" method at lower sparsity levels compared to the other datasets.
### Interpretation
The graphs demonstrate the impact of sparsity on the accuracy of two different methods ("Block" and "Layer") across various datasets/tasks. The consistent downward trend in accuracy with increasing sparsity suggests that higher sparsity levels negatively affect model performance. The "Block" method generally outperforms the "Layer" method, indicating it is more robust to sparsity. The specific performance and sensitivity to sparsity vary depending on the dataset/task, highlighting the importance of considering the nature of the problem when applying sparsity techniques. The initial stability of the "Block" method in OPENBOOKQA and WINOGRANDE might suggest that these datasets are less sensitive to sparsity at lower levels, or that the "Block" method is particularly well-suited for these tasks in the presence of some sparsity.
</details>
Figure 3: Results on average zero-shot task performance of Llama-3-8B, with block and layer pruning.
<details>
<summary>x4.png Details</summary>
### Visual Description
## Heatmap: MLP and Attention Masks Across Clusters for Llama-3-8B and Qwen-3-8B
### Overview
The image presents four heatmaps, arranged in a 2x2 grid. The top row displays "MLP Masks Across Clusters" for two different models, while the bottom row shows "Attention Masks Across Clusters" for the same models. The left column corresponds to "Llama-3-8B" and the right column to "Qwen-3-8B". The heatmaps visualize the activation patterns across different layers and clusters within each model. The x-axis represents the "Layer" number, and the y-axis represents the "Cluster" number. Each cell in the heatmap is colored either dark blue or light beige, indicating the state of the mask (presumably active or inactive).
### Components/Axes
* **Titles:**
* Top-Left: "MLP Masks Across Clusters"
* Top-Right: "MLP Masks Across Clusters"
* Bottom-Left: "Attention Masks Across Clusters"
* Bottom-Right: "Attention Masks Across Clusters"
* **X-Axis (Layer):**
* Top-Left: 0 to 31
* Top-Right: 0 to 35
* Bottom-Left: 0 to 31
* Bottom-Right: 0 to 35
* **Y-Axis (Cluster):**
* All plots: 0 to 15
* **Model Names:**
* Bottom-Left: "Llama-3-8B"
* Bottom-Right: "Qwen-3-8B"
* **Colors:**
* Dark Blue: Represents one state (likely inactive)
* Light Beige: Represents another state (likely active)
### Detailed Analysis
**1. MLP Masks Across Clusters (Llama-3-8B - Top-Left)**
* The heatmap shows a distinct vertical band of light beige around Layer 8.
* Most of the heatmap is dark blue, indicating a generally inactive state.
* There are a few isolated light beige cells scattered outside the main band.
* Clusters 0-15 are represented.
**2. MLP Masks Across Clusters (Qwen-3-8B - Top-Right)**
* The heatmap shows a concentration of light beige cells between Layers 15 and 20.
* There are also some light beige cells towards the right side of the plot, between layers 30 and 35.
* Clusters 0-15 are represented.
**3. Attention Masks Across Clusters (Llama-3-8B - Bottom-Left)**
* The heatmap shows several vertical bands of light beige, particularly around Layers 1, 7, 10, and 24.
* There are also scattered light beige cells throughout the heatmap.
* Clusters 0-15 are represented.
**4. Attention Masks Across Clusters (Qwen-3-8B - Bottom-Right)**
* The heatmap shows a more distributed pattern of light beige cells compared to the other plots.
* There are concentrations of light beige cells between Layers 0-5, 15-20, and 30-35.
* Clusters 0-15 are represented.
### Key Observations
* The MLP masks for Llama-3-8B are highly concentrated around a single layer (Layer 8), while Qwen-3-8B's MLP masks are more spread out.
* The attention masks for both models show more distributed patterns compared to the MLP masks.
* Qwen-3-8B's attention masks appear to be more active overall than Llama-3-8B's.
### Interpretation
The heatmaps visualize the activation patterns of MLP and attention masks across different layers and clusters in the Llama-3-8B and Qwen-3-8B models. The patterns suggest differences in how these models process information. The concentrated MLP mask in Llama-3-8B might indicate a specific layer that is crucial for processing, while the more distributed MLP mask in Qwen-3-8B could suggest a more distributed processing approach. The attention masks, which are generally more active, likely play a role in focusing on relevant parts of the input sequence. The differences in these patterns could contribute to the different performance characteristics of the two models. The beige color likely represents an "active" state, while the dark blue represents an "inactive" state. The data suggests that the Qwen-3-8B model utilizes more layers for attention than the Llama-3-8B model.
</details>
Figure 4: Block mask visualization of Llama-3-8B(left) and Qwen-3-8B(right) with 16 clusters and 25% sparsity. Upper part is FFN Block and the lower part is Attention Block. The color indicates the mask value, with 1 being blue and 0 being yellow.
Interestingly, the performance differential diminishes as sparsity decreases. At sparsity levels higher than 40%, the differences become minimal, and in specific tasks such as Winogrande, layer-level pruning occasionally outperforms block-level pruning. To better understand the results, we analyze the layer masks. Visualization in Figure 4 reveals that Llama attention blocks are more likely to be pruned compared to FFN blocks, especially in middle layers, aligning with previous observations about layer representation similarity in Men et al. (2024). This phenomenon also exists in the Qwen-3 model, but shows a more balanced distribution between attention and FFN blocks. Additionally, attention masks are more separate for Qwen, with no long ranges of consecutive blocks being masked. We analyzed the mask distribution at various sparsity levels and found this phenomenon was commonly observed. This suggests that, in higher sparsity settings, retaining the FFN blocks is more beneficial for model performance, as they are more likely to contain important information. For higher sparsity levels, more FFN blocks are pruned, leading to similar performance between block-level and layer-level pruning.
Computational Efficiency Analysis.
To quantify efficiency improvements, we measured FLOPs (floating point operations) for Llama-3-8B with different sparsity settings, as shown in Table 2. Our analysis reveals that block-wise pruning provides significant computational savings while maintaining model performance. At 25% sparsity, our approach reduces the computational cost to 89.8% of the dense model, representing a reasonable trade-off between efficiency and effectiveness. As sparsity increases to 37.5%, computational requirements drop to 75.8% of the original model.
| 0% 3.12% 6.25% | 32.94T 32.66T 32.39T | 100.0% 99.1% 98.3% | 21.88% 25.00% 28.12% | 31.01T 29.57T 28.71T | 94.1% 89.8% 87.2% |
| --- | --- | --- | --- | --- | --- |
| 9.38% | 32.11T | 97.5% | 31.25% | 27.27T | 82.8% |
| 12.50% | 31.84T | 96.7% | 34.38% | 26.41T | 80.2% |
| 15.62% | 31.56T | 95.8% | 37.50% | 24.97T | 75.8% |
| 18.75% | 31.29T | 95.0% | 40.62% | 24.69T | 74.9% |
Table 2: Computational efficiency at different sparsity for block-wise pruning. The FLOPs values represent the computational cost, while the percentage shows the proportion relative to the dense model.
4.3.1 Analyze clustering effectiveness
Number of Clusters.
To investigate how the number of clusters affects model performance, we conducted experiments with varying cluster counts (N = 4, 8, 16) at different sparsity levels, as shown in Figure 5. The results demonstrate a clear trend: as the number of clusters increases, overall performance improves consistently across all pruning configurations. At lower sparsity, models with 16 clusters achieve an average performance of 66.98%, compared to 61.05% with 8 clusters and 63.82% with 4 clusters. This advantage becomes even more pronounced at higher sparsity levels. With sparsity of 37.5%, the 16-cluster configuration outperforms the 4-cluster variant by 10.64 percentage points. This pattern confirms that a higher number of clusters enables more specialized mask combinations tailored to different input types. With more clusters, the model can develop a more diverse set of computational paths, each optimized for specific semantic patterns in the input data. The performance improvements with increased cluster count provide strong evidence supporting our hypothesis that dynamic routing significantly benefits model effectiveness by enabling adaptive computation. Rather than forcing all inputs through a single pruned structure, our approach leverages the complementary strengths of mask combinations, explained why our dynamic pruning strategy consistently outperforms static pruning methods.
<details>
<summary>x5.png Details</summary>
### Visual Description
## Bar Chart: Average Accuracy vs. Sparsity for Different Number of Clusters
### Overview
The image is a bar chart comparing the average accuracy of a model at different sparsity levels (12.5%, 25.0%, and 37.5%) for varying numbers of clusters (4, 8, and 16). The chart displays three bars for each sparsity level, each representing a different number of clusters.
### Components/Axes
* **X-axis:** Sparsity, with values 12.5%, 25.0%, and 37.5%.
* **Y-axis:** Average Accuracy, ranging from 0 to 70.
* **Legend (Top-Right):**
* Pink: 4 Clusters
* Light Blue: 8 Clusters
* Peach: 16 Clusters
### Detailed Analysis
The chart presents the average accuracy for different combinations of sparsity and number of clusters.
* **Sparsity 12.5%:**
* 4 Clusters (Pink): Accuracy ~ 63
* 8 Clusters (Light Blue): Accuracy ~ 61
* 16 Clusters (Peach): Accuracy ~ 66
* **Sparsity 25.0%:**
* 4 Clusters (Pink): Accuracy ~ 49
* 8 Clusters (Light Blue): Accuracy ~ 45
* 16 Clusters (Peach): Accuracy ~ 59
* **Sparsity 37.5%:**
* 4 Clusters (Pink): Accuracy ~ 37
* 8 Clusters (Light Blue): Accuracy ~ 41
* 16 Clusters (Peach): Accuracy ~ 47
### Key Observations
* For all cluster sizes, the average accuracy decreases as sparsity increases.
* At each sparsity level, 16 clusters generally yield the highest average accuracy, followed by 4 clusters, and then 8 clusters.
* The difference in average accuracy between different numbers of clusters is more pronounced at lower sparsity levels (12.5% and 25.0%) compared to 37.5%.
### Interpretation
The data suggests that increasing sparsity negatively impacts the average accuracy of the model, regardless of the number of clusters. Using 16 clusters generally results in higher accuracy compared to 4 or 8 clusters, especially when sparsity is lower. The relationship between sparsity and accuracy is likely due to the removal of data points, which reduces the information available for the model to learn from. The varying performance of different cluster sizes may be related to how well the data structure aligns with the clustering algorithm at different sparsity levels.
</details>
Figure 5: Impact of cluster number on performance across evaluation tasks. Results on average zero-shot task performance on Llama-3-8B, with cluster N=4, 8, and 16.
Calibration Data Quality.
The quality of calibration data proves critical for effective mask training, as demonstrated in our ablation studies (Table 3). We found that using high-quality, diverse pretraining data from fineweb-edu Lozhkov et al. (2024) yields the best results, achieving an average score of 59.40. In contrast, using wikitext2, the calibration dataset for baseline models, leads to a significant performance degradation, with the average score dropping to 55.85. Also, instruction dataset in Gou et al. (2023), achieved a competitive score of 58.20 but was still lower than fineweb-edu. Our experiments demonstrate that clustering semantically-rich texts creates more meaningfully differentiated clusters, enabling the discovery of truly specialized computational paths. This finding highlights the importance of data diversity and representational richness in training effective dynamic routing mechanisms.
| Instruction Wikitext2 Fineweb-edu | 36.4 39.0 40.0 | 68.27 63.06 68.98 | 68.14 64.12 67.53 | 73.06 73.07 76.12 | 63.38 60.19 63.43 | 39.93 35.67 40.36 | 58.20 55.85 59.40 |
| --- | --- | --- | --- | --- | --- | --- | --- |
Table 3: Ablation results on Llama-3-8B with 25% sparsity across different datasets. Comparing with fineweb-edu, instruction set show minor difference, while wikitext cause average score degradation.
To verify that the observed performance enhancement is attributable to our proposed method rather than the calibration data, we benchmarked the SLEB baseline on both the wikitext2 and fineweb-edu datasets. As detailed in Table 4, the baseline’s performance did not improve when using fineweb-edu. Crucially, our method continues to outperform the baseline even when using wikitext2. This evidence indicates that the performance gains originate from our method’s dynamic architecture and its ability to leverage high-quality data, rather than from an unfair data advantage.
| SLEB SLEB IG-Pruning | Wikitext2 Fineweb-edu Wikitext2 | 33.8 33.0 39.0 | 53.95 52.56 63.06 | 57.96 57.19 64.12 | 72.25 72.79 73.07 | 57.32 56.60 60.19 | 31.56 32.84 35.67 | 51.13 50.83 55.85 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| IG-Pruning | Fineweb-edu | 40.0 | 68.98 | 67.53 | 76.12 | 63.43 | 40.36 | 59.40 |
Table 4: Comparison with baseline models on different datasets. Our method outperforms the baseline (SLEB) regardless of the dataset used.
5 Conclusion
We introduced IG-Pruning, a novel approach for efficient LLM inference through input-adaptive dynamic block pruning. Our method addresses critical limitations of static pruning, and demonstrates that IG-Pruning consistently outperforms state-of-the-art static pruning methods across various configurations and model architectures. Our approach offers four key advantages: (1) improved accuracy through input-adaptive computation that tailors pruning decisions to specific input characteristics, (2) efficient training that keeps model weights frozen while only optimizing lightweight mask parameters, (3) minimal inference overhead via a simple yet effective semantic-based routing mechanism, and (4) flexible block-level pruning granularity that allows independent treatment of attention and FFN components. The success of IG-Pruning highlights the importance of input-adaptive computation in efficient LLM deployment and represents a promising direction for developing high-performing LLMs for resource-constrained environments.
Limitations
The performance heavily depends on clustering quality, potentially diminishing if semantic clusters aren’t effectively differentiated. Moreover, the result is sensitive to calibration data quality, as instruction datasets led to performance degradation compared to diverse pretraining data. Also, our evaluation focused primarily on specific zero-shot tasks, leaving generalization to other task types or domain-specific applications less thoroughly validated. Additionally, the method introduces sensitivity to multiple hyperparameters, including $L_{0}$ regularization, lagrangian parameters, and cluster numbers. Finally, our work does not investigate the impact of block pruning on model factuality. Removing computational blocks risks eliminating components that are critical for factual recall, which may increase the model’s propensity for hallucination. A promising direction for future work would be to combine our dynamic pruning strategy with hallucination mitigation techniques. For instance, integrating methods like TruthX Zhang et al. (2024a), which enhances truthfulness by editing internal model representations, or Truth-Aware Context Selection Yu et al. (2024), which filters untruthful information from the input context. Such an approach could lead to models that are not only efficient but also more robust and factually reliable.
Acknowledgements
We thank all the anonymous reviewers for their insightful and valuable comments on this paper. This work was supported by the grant from the National Natural Science Foundation of China (No. 62376260).
References
- AI@Meta (2024) AI@Meta. 2024. Llama 3 model card.
- Ashkboos et al. (2024) Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15024.
- Bisk et al. (2020) Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, and 1 others. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439.
- Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
- Chen et al. (2024) Xiaodong Chen, Yuxuan Hu, Jing Zhang, Yanling Wang, Cuiping Li, and Hong Chen. 2024. Streamlining redundant layers to compress large language models. arXiv preprint arXiv:2403.19135.
- Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.
- (7) Luciano Del Corro, Allison Del Giorno, Sahaj Agarwal, Bin Yu, Ahmed Hassan Awadallah, and Subhabrata Mukherjee. Skipdecode: Autoregressive skip decoding with batching and caching for efficient llm inference.
- Elhoushi et al. (2024) Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, and 1 others. 2024. Layerskip: Enabling early exit inference and self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12622–12642.
- Fan et al. (2024) Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng, Peng Han, Shuo Shang, Aixin Sun, Yequan Wang, and Zhongyuan Wang. 2024. Not all layers of llms are necessary during inference. arXiv preprint arXiv:2403.02181.
- Fang et al. (2024) Gongfan Fang, Hongxu Yin, Saurav Muralidharan, Greg Heinrich, Jeff Pool, Jan Kautz, Pavlo Molchanov, and Xinchao Wang. 2024. Maskllm: Learnable semi-structured sparsity for large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
- Frantar and Alistarh (2023) Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323–10337. PMLR.
- Gao et al. (2023) Leo Gao, Jonathan Tow, Baber Abbasi, S Biderman, S Black, A DiPofi, C Foster, L Golding, J Hsu, A Le Noac’h, and 1 others. 2023. A framework for few-shot language model evaluation, 12 2023. URL https://zenodo. org/records/10256836, 7.
- Gou et al. (2023) Yunhao Gou, Zhili Liu, Kai Chen, Lanqing Hong, Hang Xu, Aoxue Li, Dit-Yan Yeung, James T Kwok, and Yu Zhang. 2023. Mixture of cluster-conditional lora experts for vision-language instruction tuning. arXiv preprint arXiv:2312.12379.
- Gromov et al. (2024) Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Dan Roberts. 2024. The unreasonable ineffectiveness of the deeper layers. In NeurIPS 2024 Workshop on Scientific Methods for Understanding Deep Learning.
- Gu et al. (2021) Shuhao Gu, Yang Feng, and Wanying Xie. 2021. Pruning-then-expanding model for domain adaptation of neural machine translation. arXiv preprint arXiv:2103.13678.
- He et al. (2024) Zhuomin He, Yizhen Yao, Pengfei Zuo, Bin Gao, Qinya Li, Zhenzhe Zheng, and Fan Wu. 2024. AdaSkip: Adaptive sublayer skipping for accelerating long-context LLM inference. 39(22):24050–24058.
- Kim et al. (2024) Bo-Kyeong Kim, Geon-min Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Kyu Song. 2024. Shortened llama: A simple depth pruning for large language models. CoRR.
- Ling et al. (2024) Gui Ling, Ziyang Wang, Qingwen Liu, and 1 others. 2024. Slimgpt: Layer-wise structured pruning for large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
- Liu et al. (2024a) Deyuan Liu, Zhanyue Qin, Hairu Wang, Zhao Yang, Zecheng Wang, Fangying Rong, Qingbin Liu, Yanchao Hao, Bo Li, Xi Chen, and 1 others. 2024a. Pruning via merging: Compressing llms via manifold alignment based layer merging. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17817–17829.
- Liu et al. (2024b) Yijin Liu, Fandong Meng, and Jie Zhou. 2024b. Accelerating inference in large language models with a unified layer skipping strategy. arXiv preprint arXiv:2404.06954.
- Liu et al. (2023) Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, and 1 others. 2023. Deja vu: Contextual sparsity for efficient llms at inference time. In International Conference on Machine Learning, pages 22137–22176. PMLR.
- Louizos et al. (2018) Christos Louizos, Max Welling, and Diederik P Kingma. 2018. Learning sparse neural networks through l_0 regularization. In International Conference on Learning Representations.
- Lozhkov et al. (2024) Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. 2024. Fineweb-edu: the finest collection of educational content.
- Ma et al. (2023) Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702–21720.
- Men et al. (2024) Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. 2024. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853.
- Merity et al. (2016) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. Preprint, arXiv:1609.07843.
- Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391.
- QwenTeam (2025) QwenTeam. 2025. Qwen3.
- Raposo et al. (2024) David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. 2024. Mixture-of-depths: Dynamically allocating compute in transformer-based language models. arXiv preprint arXiv:2404.02258.
- Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992.
- Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106.
- Schuster et al. (2022) Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. 2022. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 35:17456–17472.
- Sieberling et al. (2024) Oliver Sieberling, Denis Kuznedelev, Eldar Kurtic, and Dan Alistarh. 2024. Evopress: Towards optimal dynamic model compression via evolutionary search. arXiv preprint arXiv:2410.14649.
- Song et al. (2024) Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, and Jae-Joon Kim. 2024. Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks. In International Conference on Machine Learning, pages 46136–46155. PMLR.
- Sun et al. (2023) Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. 2023. A Simple and Effective Pruning Approach for Large Language Models. In ICML.
- Tan et al. (2024) Zhen Tan, Daize Dong, Xinyu Zhao, Jie Peng, Yu Cheng, and Tianlong Chen. 2024. Dlo: Dynamic layer operation for efficient vertical scaling of llms. CoRR.
- Wang et al. (2024) Wenxiao Wang, Wei Chen, Yicong Luo, Yongliu Long, Zhengkai Lin, Liye Zhang, Binbin Lin, Deng Cai, and Xiaofei He. 2024. Model compression and efficient inference for large language models: A survey. arXiv preprint arXiv:2402.09748.
- Wu et al. (2024) Qiong Wu, Zhaoxi Ke, Yiyi Zhou, Xiaoshuai Sun, and Rongrong Ji. 2024. Routing experts: Learning to route dynamic experts in existing multi-modal large language models. In The Thirteenth International Conference on Learning Representations.
- Xia et al. (2024) Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2024. Sheared llama: Accelerating language model pre-training via structured pruning. In 12th International Conference on Learning Representations, ICLR 2024.
- Xia et al. (2022) Mengzhou Xia, Zexuan Zhong, and Danqi Chen. 2022. Structured pruning learns compact and accurate models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1513–1528.
- Yang et al. (2024) Yifei Yang, Zouying Cao, and Hai Zhao. 2024. Laco: Large language model pruning via layer collapse. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6401–6417.
- Yu et al. (2024) Tian Yu, Shaolei Zhang, and Yang Feng. 2024. Truth-aware context selection: Mitigating hallucinations of large language models being misled by untruthful contexts. In Findings of the Association for Computational Linguistics ACL 2024, pages 10862–10884.
- Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800.
- Zhang et al. (2023a) Shaolei Zhang, Qingkai Fang, Zhuocheng Zhang, Zhengrui Ma, Yan Zhou, Langlin Huang, Mengyu Bu, Shangtong Gui, Yunji Chen, Xilin Chen, and 1 others. 2023a. Bayling: Bridging cross-lingual alignment and instruction following through interactive translation for large language models. arXiv preprint arXiv:2306.10968.
- Zhang et al. (2024a) Shaolei Zhang, Tian Yu, and Yang Feng. 2024a. Truthx: Alleviating hallucinations by editing large language models in truthful space. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8908–8949.
- Zhang et al. (2024b) Shaolei Zhang, Kehao Zhang, Qingkai Fang, Shoutao Guo, Yan Zhou, Xiaodong Liu, and Yang Feng. 2024b. Bayling 2: A multilingual large language model with efficient language alignment. arXiv preprint arXiv:2411.16300.
- Zhang et al. (2023b) Yuxin Zhang, Lirui Zhao, Mingbao Lin, Sun Yunyun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. 2023b. Dynamic sparse no training: Training-free fine-tuning for sparse llms. In The Twelfth International Conference on Learning Representations.
- Zhou et al. (2024) Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, and 1 others. 2024. A survey on efficient inference for large language models. CoRR.
- Zhu et al. (2024) Yunqi Zhu, Xuebing Yang, Yuanyuan Wu, and Wensheng Zhang. 2024. Hierarchical skip decoding for efficient autoregressive text generation. CoRR.
Appendix A Hyperparameter
The hyperparamters we use in our experiments are listed in Table 5.
| $L_{0}$ Module Learning Rate | 0.1 |
| --- | --- |
| Lagrangian Learning Rate | 0.1 |
| $\epsilon$ | 1e-6 |
| $1/\beta$ | 2/3 |
| $l$ | -0.1 |
| $r$ | 1.1 |
| Number of Clusters | 16, 8, 4 |
| Calibration Data Size for each cluster | 1000 |
| Clustering Stage Sequence Length | 4096 |
| Mask Training Sequence Length | 512 |
Table 5: Hyperparameters used in our experiments.
Appendix B More results on various models
To further validate the generalizability and robustness of our approach, we conducted additional experiments on a wider range of models, including Llama-3.2-3B (Table 6), Llama-3.2-1B (Table 7), and Qwen-3-4B (Table 8). Across all tested models and architectures, the input-adaptive nature of IG-Pruning allows it to retain significantly more of the original model’s performance compared to baselines, especially at moderate sparsity levels. As sparsity becomes extremely high, the performance of both methods naturally converges. These comprehensive results validate that our dynamic approach is a consistently superior and more robust solution for model pruning.
| Model Llama-3.2-3B 14% (4/28) | Sparsity 0% (0/28) SLEB | Method Dense 35.80 | OpenBookQA 43.20 58.45 | Winogrande 69.38 58.20 | Hellaswag 73.73 73.12 | PIQA 77.27 57.02 | ARC-E 71.84 33.70 | ARC-C 45.99 52.71 | Average 63.55 82.94% | Percentage(%) 100% |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| IG-Pruning | 41.40 | 66.45 | 68.20 | 75.95 | 68.13 | 43.34 | 60.58 | 95.32% | | |
| 25% (7/28) | SLEB | 25.00 | 53.82 | 46.67 | 68.28 | 50.96 | 29.01 | 46.79 | 73.63% | |
| IG-Pruning | 36.40 | 57.76 | 60.14 | 71.87 | 54.88 | 33.19 | 52.36 | 82.40% | | |
| 39% (11/28) | SLEB | 26.80 | 51.06 | 37.26 | 61.58 | 40.02 | 24.65 | 40.23 | 63.30% | |
| IG-Pruning | 28.00 | 49.83 | 38.52 | 61.53 | 38.17 | 24.40 | 40.07 | 63.05% | | |
Table 6: Results on Llama-3.2-3B.
| Model Llama-3.2-1B 12.5% (2/16) | Sparsity 0% (0/16) SLEB | Method Dense 30.60 | OpenBookQA 37.40 55.16 | Winogrande 60.36 48.74 | Hellaswag 63.64 68.55 | PIQA 74.43 48.48 | ARC-E 60.27 28.41 | ARC-C 36.26 46.66 | Average 55.38 84.24% | Percentage(%) 100% |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| IG-Pruning | 35.00 | 60.45 | 59.65 | 72.79 | 57.32 | 33.87 | 53.18 | 96.02% | | |
| 25% (4/16) | SLEB | 27.80 | 51.63 | 37.50 | 63.11 | 40.19 | 23.72 | 40.65 | 73.40% | |
| IG-Pruning | 27.00 | 54.78 | 40.30 | 62.08 | 40.24 | 27.22 | 41.94 | 75.72% | | |
| 37.5% (6/16) | SLEB | 27.00 | 49.88 | 29.90 | 56.03 | 30.93 | 22.01 | 35.96 | 64.93% | |
| IG-Pruning | 24.40 | 50.98 | 30.90 | 56.31 | 30.72 | 25.08 | 36.40 | 65.72% | | |
Table 7: Results on Llama-3.2-1B.
| Model Qwen-3-4B 14% (5/36) | Sparsity 0% (0/36) SLEB | Method Dense 35.40 | OpenBookQA 40.40 56.19 | Winogrande 65.82 57.36 | Hellaswag 68.42 72.85 | PIQA 75.13 65.78 | ARC-E 53.75 39.84 | ARC-C 53.75 54.57 | Average 59.55 91.64% | Percentage(%) 100% |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| IG-Pruning | 37.60 | 62.58 | 59.76 | 73.55 | 68.35 | 44.62 | 57.74 | 96.97% | | |
| 25% (9/36) | SLEB | 32.20 | 53.03 | 46.94 | 67.46 | 58.37 | 31.22 | 48.20 | 80.95% | |
| IG-Pruning | 35.80 | 56.43 | 53.78 | 69.85 | 60.01 | 39.07 | 52.49 | 88.15% | | |
| 36% (13/36) | SLEB | 29.80 | 53.43 | 39.54 | 62.67 | 47.01 | 26.79 | 43.21 | 72.56% | |
| IG-Pruning | 30.60 | 54.69 | 42.74 | 63.65 | 47.26 | 28.66 | 44.60 | 74.90% | | |
Table 8: Results on Qwen-3-4B.