# A Survey on Transformer Compression
**Authors**: Yehui Tang, Yunhe Wang, Jianyuan Guo, Zhijun Tu, Kai Han, Hailin Hu, and Dacheng Tao
> Yehui Tang, Jianyuan Guo, Zhijun Tu, Kai Han, Hailin Hu, and Yunhe Wang are with Huawei Noah’s Ark Lab. E-mail: Dacheng Tao is with the School of Computer Science, in the Faculty of Engineering, at The University of Sydney, 6 Cleveland St, Darlington, NSW 2008, Australia. E-mail: Corresponding to Yunhe Wang and Dacheng Tao.
## Abstract
Transformer plays a vital role in the realms of natural language processing (NLP) and computer vision (CV), specially for constructing large language models (LLM) and large vision models (LVM). Model compression methods reduce the memory and computational cost of Transformer, which is a necessary step to implement large language/vision models on practical devices. Given the unique architecture of Transformer, featuring alternative attention and feedforward neural network (FFN) modules, specific compression techniques are usually required. The efficiency of these compression methods is also paramount, as retraining large models on the entire training dataset is usually impractical. This survey provides a comprehensive review of recent compression methods, with a specific focus on their application to Transformer-based models. The compression methods are primarily categorized into pruning, quantization, knowledge distillation, and efficient architecture design (Mamba, RetNet, RWKV, etc). In each category, we discuss compression methods for both language and vision tasks, highlighting common underlying principles. Finally, we delve into the relation between various compression methods, and discuss further directions in this domain.
Index Terms: Model Compression, Transformer, Large Language Model, Large Vision Model, LLM
## 1 Introduction
Deep neural networks have become indispensable in numerous artificial intelligence applications, with architectures encompassing diverse formulations, such as multilayer perceptron (MLP), convolutional neural network (CNN), recurrent neural network (RNN), long short-term memory (LSTM), Transformers, etc. In recent times, Transformer-based models have emerged as the prevailing choice across various domains, including both natural language processing (NLP) and computer vision (CV) domains. Considering their strong scalability, most of the large models with over billions of parameters are based on the Transformer architecture, which are considered as foundational elements for artificial general intelligence (AGI) [1, 2, 3, 4, 5, 6].
While large models have demonstrated significant capabilities, their exceptionally vast sizes pose challenges for practical development. For instance, the GPT-3 model has 175 billion parameters and demands about 350GB memory model storage (float16). The sheer volume of parameters and the associated computational expenses necessitate devices with exceedingly high memory and computational capabilities. Directly deploying such models will incur substantial resource costs and contributes significantly to carbon dioxide emissions. Moreover, on edge devices like mobile phones, the development of these models becomes impractical due to the limited storage and computing resources of such devices.
<details>
<summary>x1.png Details</summary>
### Visual Description
## Composite Diagram: Evolution and Deployment of Transformer-Based AI Models
### Overview
The image is a composite technical diagram illustrating the growth of Transformer-based research, the evolution of AI model architectures leading to large Transformer-based models, and the subsequent pipeline for compressing and deploying these models. It consists of three main sections arranged horizontally from left to right: a bar chart, a flow diagram of model evolution, and a deployment pipeline.
### Components/Axes
**1. Left Section: Bar Chart**
* **Title:** "Number of publications based on Transformers"
* **X-axis:** Labeled "Year". Categories: 2017, 2018, 2019, 2020, 2021, 2022, 2023.
* **Y-axis:** Labeled "# Publications". Scale markers: 0, 10000, 20000, 30000, 40000, 50000.
* **Data Series:** A single series of blue vertical bars representing the publication count for each year.
**2. Center Section: Model Evolution Flow Diagram**
* **Left Cluster (Pre-Transformer Architectures):** Six light gray ovals containing text:
* Top: "RNN"
* Upper Middle: "LSTM"
* Middle: "NLP" (connected via arrow to central oval)
* Lower Middle: "CV" (connected via arrow to central oval)
* Bottom: "MLP"
* Bottom Right: "CNN"
* **Central Oval (Target Models):** A large, light gray oval containing the text: "ChatGPT, LLaMA, BERT, CLIP, ViT..." The ellipsis (...) indicates this is a non-exhaustive list.
* **Label Below Central Oval:** "Large Transformer-Based Models"
* **Arrows:** Two black arrows originate from the "NLP" and "CV" ovals, pointing directly to the central "Large Transformer-Based Models" oval.
**3. Right Section: Compression and Implementation Pipeline**
* **Compression Techniques:** Four horizontal black arrows point from left to right. Above each arrow is a label:
* "Pruning"
* "Quantization"
* "Knowledge Distillation"
* "Efficient Architecture"
* **Ellipsis:** Below the fourth arrow, three vertical dots ("...") indicate additional, unspecified compression techniques.
* **Label Below Arrows:** "Compression"
* **Implementation Icons:** On the far right, three black line-art icons represent deployment targets:
* Top: A cloud with network connections (representing cloud/server deployment).
* Middle: A smartphone (representing mobile deployment).
* Bottom: A desktop computer and server tower (representing edge/desktop deployment).
* **Label Below Icons:** "Implementation"
### Detailed Analysis
**Bar Chart Data (Approximate Values):**
* **Trend Verification:** The data series shows a clear, steep upward trend from 2017 to 2022, followed by a slight decline in 2023.
* **2017:** ~0 publications (bar is barely visible).
* **2018:** ~1,000 publications.
* **2019:** ~5,000 publications.
* **2020:** ~11,000 publications.
* **2021:** ~21,000 publications.
* **2022:** ~42,000 publications (peak).
* **2023:** ~37,000 publications.
**Flow Diagram Relationships:**
* The diagram positions "NLP" (Natural Language Processing) and "CV" (Computer Vision) as the primary application domains driving the development of large Transformer-based models, as indicated by the direct arrows.
* Other architectures like RNN, LSTM, MLP, and CNN are shown as related or predecessor technologies in the same conceptual space but without direct arrows to the central models.
**Compression Pipeline:**
* The four listed techniques (Pruning, Quantization, Knowledge Distillation, Efficient Architecture) are presented as parallel or sequential methods for model compression.
* The flow is linear: Large Transformer-Based Models → Compression Techniques → Implementation on various hardware (cloud, mobile, desktop).
### Key Observations
1. **Exponential Growth:** The publication chart demonstrates explosive, near-exponential growth in Transformer research from 2018 to 2022.
2. **Peak and Plateau:** The year 2022 represents the peak of publication volume, with a noticeable but not drastic decrease in 2023, suggesting a potential maturation or consolidation phase in the field.
3. **Central Role of Transformers:** The diagram's structure places "Large Transformer-Based Models" as the central, convergent outcome of prior work in NLP and CV.
4. **Deployment Focus:** The rightmost section highlights that a major current challenge in the field is not just creating large models, but compressing them for practical "Implementation" across diverse computing environments.
### Interpretation
This diagram tells a concise story of a technological paradigm shift. The bar chart provides quantitative evidence of a research revolution centered on Transformers, peaking around 2022. The flow diagram contextualizes this revolution, showing how it synthesized advances from both language (NLP) and vision (CV) domains, moving beyond older architectures like RNNs and CNNs.
The most critical insight lies in the pipeline's conclusion: the creation of massive models (like ChatGPT, LLaMA) is not the end goal. The diagram argues that the current frontier is **compression**—making these powerful but resource-intensive models efficient enough for real-world **implementation** on everyday devices (phones, desktops) and cloud infrastructure. The ellipsis under "Compression" acknowledges this is an active, open area of research with many emerging techniques. The overall message is one of rapid theoretical advancement now grappling with the practical engineering challenges of deployment.
</details>
Figure 1: Transformer-based models have emerged as the predominant architectures in both natural language processing (NLP) and computer vision (CV) domains, resulting in a surge in publications. As these models tend to possess substantial dimensions, it becomes imperative to compress their parameters and streamline computational redundancies. This compression is essential for facilitating efficient implementation on practical platforms, ensuring the feasibility of deploying Transformer models in real-world applications.
Model compression is an effective strategy for mitigating the development costs associated with Transformer models. This approach, grounded in the principle of reducing redundancy, encompasses various categories, including pruning, quantization, knowledge distillation, efficient architecture design, etc. Network pruning directly removes redundant components, such as blocks, attention heads, FFN layers, and individual parameters. Diverse sub-models can be derived by employing different pruning granularity and pruning criteria. Quantization reduces the development cost by representing model weights and intermediate features with lower bits. For example, when quantizing a full-precision model (float32) into 8-bit integers, the memory cost can be reduced by a factor of four. According the computational process, it can be divided into post-training quantization(PTQ) or quantization-aware training (QAT), in which the former incurs only limited training costs and is more efficient for large models. Knowledge distillation serves as a training strategy, and transfers knowledge from a large model (teacher) to a smaller model (student). The student mimics the behavior of the teacher by emulating the model’s output and intermediate features. Notably, for advanced models like GPT-4, accessible only through APIs, their generated instructions and explanations can also guide the learning of the student model [7, 8].In addition to obtaining models from predefined large models, some methods yield efficient architectures by directly reducing the computational complexity of attention modules or FFN modules. Combining different methods enables extreme compression. For instance, Han et al. [9] combined network pruning, quantization, and Huffman coding to achieve an impressive 49 $\times$ compression rate on a conventional VGGNet [10].
Regarding Transformer models, their compression strategies exhibit distinct characteristics. Unlike other architectures such as CNN or RNN, the Transformer features a unique design with alternative attention and FFN modules. The former captures the global information by calculating the attention map over different tokens while the latter extracts information from each token respectively. This specific architecture can enable a tailored compression strategy for optimal compression rates. What’s more, the efficiency of compression method becomes especially important for such large models. Due to the high computational cost of large model, it is usually unaffordable to retrain the whole model on the original training set. Some training-efficient methods like post-training compression are preferable.
In this survey, we aim to comprehensively investigate how to compress these Transformer models (Figure 1), and categorize the methods by quantization, knowledge distillation, pruning, efficient architecture design, etc. In each category, we investigate the compression methods for NLP and CV domains, respectively. Table I summarizes the main compression categories and lists representative method suitable for large Transformer models. Though NLP and CV are usually treated as very different domains, we observe that their models compression methods actually share the similar principles. Finally, we discuss the relationship between different compression methods and outline some future research directions.
The rest of the paper is organized as follows. Section 2 introduces the fundamental concept of Transformers. Following this, Section 3 provides an in-depth discussion on compression methods that preserve the architecture, encompassing quantization and knowledge distillation—techniques that maintain the model’s architecture. Section 4 delves further into architecture-preserving compression, including pruning and efficient architecture design. Additional Transformer compression methods are explored in Section 5. Finally, Section 6 draws conclusions on the compression methods and discusses future research directions.
## 2 Concept of Transformer
The Transformer architecture is firstly proposed to tackle tasks like machine translation [11]. A standard Transformer architecture contains main blocks, multi-head attention (MHA) and feed-forward networks (FFN). The attention is formulated as
$$
\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d}})V, \tag{1}
$$
where $Q$ , $K$ , $V$ are query, key and value matrix, respectively. $d$ is the feature’s dimension. The multi-head attention jointly extracts information from diverse subspaces, which is the concatenation of different heads,
$$
\displaystyle\mathrm{MultiHead}(Q,K,V)=\mathrm{Concat}(\mathrm{head_{1}},...,
\mathrm{head_{h}})W^{O}, \displaystyle\text{where}~{}\mathrm{head_{i}}=\mathrm{Attention}(QW^{Q}_{i},KW
^{K}_{i},VW^{V}_{i}). \tag{2}
$$
$W^{Q}$ , $W^{K}$ , $W^{V}$ , $W^{O}$ are the corresponding parameter matrices. The FFN module transforms features from each token independently. It is usually constructed by stacking two FC layers with activation functions,
$$
\mathrm{FFN}(x)=\phi(xW_{1}+b_{1})W_{2}+b_{2}, \tag{3}
$$
where $x$ is input feature, and $\phi$ is activation function (e.g., GELU). $W_{1}$ , $W_{2}$ , $b_{1}$ , $b_{2}$ are the weight and bias parameters in FC layers. The MHA and FFN module are stacked alternatively to construct the whole model.
The Transformer architecture has strong scalability and so can be used to construct extremely large models with several billion or trillion parameters. It supports most of the predominant large models in NLP, CV and multiple modality domains. For example, the well-known large language models (e.g., GPT-series [4, 2], LLaMA [1],Pangu [5, 6]) are its decoder-only variants. By simply splitting an image into multiple patches, it can be used to tackle vision tasks [12, 13, 14]. The multiple model like CLIP [15], BLIP [16], LLaVA [17] also use Transformer as the backbones.
TABLE I: Representative compression method for Transformer models.
| Category | Sub-category | Method | Highlights | Publication |
| --- | --- | --- | --- | --- |
| Quantization | NLP | SmoothQuant [18] | Training-free, smooth outliers, equivalent transformation | ICML 2023 |
| OmniQuant [19] | Weight clipping, learnable transformation, block-wise | Arxiv 2023 | | |
| QLoRA [20] | Parameter-efficient fine-tuning, memory management | Arxiv 2023 | | |
| CV | PTQ-ViT [21] | Self-attention preservation, mixed-precision | NeurIPS 2021 | |
| FQ-ViT [22] | Fully-quantized, log2 quantization, power-of-two factor | IJCAI 2022 | | |
| OFQ [23] | Confidence-guided annealing, query-key reparameterization | ICML 2023 | | |
| Knowledge Distillation | NLP | DistilBERT [24] | Small version of BERT, trained with logits of the teacher | NeurIPS 2019 |
| MiniLM [25] | Mimicking attention distribution and value-relation of teacher | NeurIPS 2020 | | |
| Lion [7] | Adversarial distillation: imitation, discrimination, generation | EMNLP 2023 | | |
| CV | DeiT [26] | Hard labels, novel distillation token in ViTs | ICML 2021 | |
| TinyViT [27] | Large-scale pretraining data, encoded data augmentation | ECCV 2022 | | |
| ManifoldKD [28] | Patch-level, batch-level manifold information | NeurIPS 2022 | | |
| Pruning | NLP | LLM Pruner [29] | Structured, coupled-module identification | NeurIPS 2023 |
| Sheared LLaMA [30] | Structured, pre-defined target, dynamic data loading | NeurIPS 2023 | | |
| Dynamic Context Pruning [31] | Sigmoid-based context selection, KV-cache aware | NeurIPS 2023 | | |
| CV | ViT-Slim [32] | Structured, single-shot architecture search | CVPR 2022 | |
| Patch Sliming [33] | Top-down unimportant patch removing | CVPR 2022 | | |
| X-pruner [34] | Structured, class-aware, layer-wise fully differentiable pruning | CVPR 2023 | | |
| Efficient Architecture | NLP | PaLM [35] | SwiGLU activation in FFN, densely activated | JMLR 2023 |
| RetNet [36] | Training parallelism, low-cost inference, parallel, recurrent | Arxiv 2023 | | |
| Reformer [37] | Efficient attention, locality-sensitive hashing | ICLR 2020 | | |
| CV | Swin [38] | Hierarchical structures, shifted local window attention | ICCV 2021 | |
| MetaFormer [39] | Non-parametric pooling as basic token mixing | CVPR 2022 | | |
| MLP-Mixer [40] | Architecture based exclusively on multi-layer perceptrons | NeurIPS 2021 | | |
## 3 Architecture Preserved Compression
### 3.1 Quantization
#### 3.1.1 Overview of Quantization
Quantization is a crucial step for deploying Transformers on various devices, especially GPUs and NPUs, that have specialized circuits for low-precision arithmetic. During the quantization process as shown in Equation 4, a floating-point tensor $x$ is converted to the integer one $x_{int}$ with corresponding quantization parameters (scale factor $s$ and zero point $z$ ), then the integer tensor $x_{int}$ could be quantized back to floating-point $x_{quant}$ but causes some precision error compared with the original $x$ ,
$$
\displaystyle x_{int} \displaystyle=\textrm{Clamp}(\lfloor x/s\rceil+z,0,2^{b}-1), \displaystyle x_{quant} \displaystyle=s(x_{int}-z), \tag{4}
$$
where $b$ denotes the bit-width, $\lfloor\cdot\rceil$ represents the rounding function and ‘Clamp’ clips the values that exceed the given range. For the matrix multiplication, the weight w adopts symmetrical quantization with zero point $z_{w}=0$ , and the input embedding tensor $e$ is quantized with unsymmetrical quantization, as shown in Equation 5:
$$
\displaystyle y \displaystyle=\textrm{MatMul}(e,\textrm{w})\approx\textrm{MatMul}(e_{quant},
\textrm{w}_{quant}) \displaystyle=\textrm{MatMul}(s_{e}(e_{int}-z_{e}),s_{w}\textrm{w}_{int}) \displaystyle=s_{e}s_{w}\textrm{MatMul}(e_{int},\textrm{w}_{int})+C, \tag{5}
$$
where $s_{w},s_{e},z_{e}$ are quantization parameters of weights and input embedding, and $e_{int}$ and $\textrm{w}_{int}$ are the integer input and weights, which are calculated by Equation 4. $C$ could be pre-computed with $s_{e},z_{e},s_{e}$ and $\textrm{w}_{int}$ . Thus the floating-point multiplication could be accelerated with efficient integer multiplication in the inference. To minimize the performance degradation of quantized models, different optimization methods have been proposed and they can be divided into two categories: (1) Post-training quantization (PTQ) [21, 41, 22, 42, 43, 44, 45] , mainly focuses on optimizing the quantization parameters of weights and activations with a few unlabeled calibration data, and some of the latest methods also explore adaptive rounding for weight quantization. (2) Quantization-aware training (QAT) [46, 47, 48, 49, 50, 51, 23, 52, 53, 54, 55, 56], inserts the quantization nodes into networks and conducts training with complete training data, where all the weights and quantization parameters are optimized together. In this section, we systematically introduce the research of model quantization on Transformer-based vision models and large language models, as shown in Figure 2.
<details>
<summary>x2.png Details</summary>
### Visual Description
\n
## Diagram: Transformer Model Quantization Framework
### Overview
The image is a technical diagram illustrating a framework for quantizing Transformer-based models, split into two distinct sections. The top section is a high-level flowchart showing the quantization process for two major application domains: Computer Vision and Natural Language Processing. The bottom section is a detailed architectural diagram of a quantized Transformer block, showing the specific sequence of operations and data type conversions (FP32 to INT8 and back).
### Components/Axes
The diagram contains no traditional chart axes or legends. Its components are labeled blocks, arrows indicating data flow, and textual annotations.
**Top Flowchart Components:**
1. **Root Block:** "Transformer" (blue rectangle, left-center).
2. **Domain Branches:**
* "Computer Vision" (light green rectangle, top-left).
* "Natural Language Processing" (pink rectangle, bottom-left).
3. **Data Blocks (for each domain):**
* "Calibration images" / "Calibration sentences" (light purple/peach rectangles).
* "Training images" / "Training sentences" (light purple/peach rectangles).
4. **Quantization Method Labels:** "PTQ" (Post-Training Quantization) and "QAT" (Quantization-Aware Training) are written on the arrows leading to the output blocks.
5. **Output Blocks:**
* "Quantized ViTs" (Vision Transformers) (yellow rectangle, top-right).
* "Quantized PLMs/LLMs" (Pre-trained Language Models/Large Language Models) (yellow rectangle, bottom-right).
6. **Bullet Point Lists (to the right of each output block):**
* For **Quantized ViTs**:
* Self-Attention Rank Consistency
* Post-SoftMax/GELU/LayerNorm Rectification
* Weight Oscillation / Data Variations
* For **Quantized PLMs/LLMs**:
* Outliers Clipping/Scale
* Module-wise Reconstruction
* Parameter-Efficient Fine-Tune
**Bottom Architecture Diagram Components (Left to Right Flow):**
The diagram shows a sequential pipeline with residual connections. Each major operation is in a rounded rectangle. Data types (FP32, INT8) are labeled on the connecting arrows.
1. **Input:** "FP32" data enters.
2. **First Sub-block (Attention):**
* `LayerNorm` (FP32 -> FP32)
* `Quantize` (FP32 -> INT8)
* `Query-Key-Value proj` (INT8 -> Q/K/V in INT8)
* `Self attention` & `SoftMax` (INT8 -> FP32)
* `Quantize` (FP32 -> INT8)
* `out proj` (INT8 -> INT32)
* `Dequantize` (INT32 -> FP32)
* **Residual Connection:** Adds the original FP32 input to the output of `Dequantize`.
3. **Second Sub-block (Feed-Forward):**
* `LayerNorm` (FP32 -> FP32)
* `Quantize` (FP32 -> INT8)
* `FC1` (Fully Connected 1) (INT8 -> INT32)
* `Dequantize` (INT32 -> FP32)
* `GELU` (activation function) (FP32 -> FP32)
* `Quantize` (FP32 -> INT8)
* `FC2` (Fully Connected 2) (INT8 -> FP32)
* **Residual Connection:** Adds the output from the first sub-block's residual to the output of `FC2`.
4. **Output:** "FP32" data exits.
### Detailed Analysis
**Flow and Relationships:**
The top flowchart establishes that the quantization framework is domain-aware. It takes a generic "Transformer" and applies specialized techniques depending on whether it's for vision (ViTs) or language (PLMs/LLMs). The process uses both calibration data (for PTQ) and training data (for QAT).
The bottom diagram details the *how* of quantization within a single Transformer block. It reveals a pattern of quantizing activations (to INT8) before compute-heavy operations (like projections, attention, FC layers) and dequantizing the results (to FP32) for operations that require higher precision (like SoftMax, GELU, LayerNorm, and residual additions). The data type transitions are meticulously labeled: FP32 -> INT8 (Quantize), INT8 -> INT32 (during computation), INT32 -> FP32 (Dequantize).
**Spatial Grounding:**
* The "Computer Vision" branch is positioned above the "Natural Language Processing" branch.
* The bullet-point techniques are aligned to the right of their respective "Quantized" output blocks.
* The detailed architecture diagram spans the entire bottom half of the image, flowing strictly left-to-right. The two residual connections are shown as lines arching over the main computational blocks.
### Key Observations
1. **Domain-Specific Strategies:** The framework explicitly lists different core challenges/techniques for quantizing vision models (e.g., "Self-Attention Rank Consistency") versus language models (e.g., "Outliers Clipping/Scale").
2. **Hybrid Precision Pipeline:** The architecture does not quantize everything to INT8 uniformly. It uses a mix of FP32, INT8, and INT32, strategically placing quantization and dequantization steps around specific operations to balance efficiency and accuracy.
3. **Focus on Activation Quantization:** The detailed diagram primarily shows the quantization of *activations* (the data flowing between layers), as indicated by the `Quantize`/`Dequantize` blocks on the main path. Weight quantization is implied but not explicitly diagrammed in this view.
4. **Preservation of Residual Paths:** The residual connections (the `⊕` symbols) operate in FP32, suggesting that the high-precision addition is critical for maintaining model performance.
### Interpretation
This diagram serves as a technical blueprint for efficiently deploying Transformer models. It communicates two key messages:
1. **Strategic Abstraction:** Quantization is not a one-size-fits-all process. The top chart argues that effective quantization requires understanding the unique statistical properties and failure modes of the model's application domain (vision vs. language). The listed bullet points are likely the titles of specific methods or research papers addressing these domain-specific issues.
2. **Operational Precision:** The bottom diagram reveals the engineering precision required. It shows that quantization is a careful dance of data type conversion. The goal is to confine lower-precision (INT8) math to the most computationally expensive parts (matrix multiplications in projections and FC layers) while maintaining higher precision (FP32) for sensitive non-linear functions (SoftMax, GELU) and normalization. This hybrid approach aims to achieve the speed and memory benefits of integer arithmetic without catastrophic loss of model accuracy.
In essence, the image bridges high-level methodology (domain-aware quantization) with low-level implementation (mixed-precision tensor operations), providing a comprehensive view of the challenges and solutions in making massive Transformer models more efficient.
</details>
Figure 2: The overview of quantization for Transformers. The top demonstrates the different problems that are addressed in existing works for computer vision and natural language processing, and the bottom shows a normal INT8 inference process of a standard Transformer block.
TABLE II: Comparison of different PTQ and QAT methods for transformer-based vision models. W/A denotes the bit-width of weight and activation and the results show the top-1 accuracy on ImageNet-1k validation set. * represents for mixed-precision.
| Method of PTQ | W/A (bit) | ViT-B | DeiT-S | DeiT-B | Swin-S | Swin-B |
| --- | --- | --- | --- | --- | --- | --- |
| Full Precision | 32/32 | 84.54 | 79.85 | 81.80 | 83.23 | 85.27 |
| PTQ-ViT [21] | 8*/8* | 76.98 | 78.09 | 81.29 | - | - |
| PTQ4ViT [41] | 8/8 | 84.54 | 79.47 | 81.48 | 83.10 | 85.14 |
| FQ-ViT [22] | 8/8 | 83.31 | 79.17 | 81.20 | 82.71 | 82.97 |
| APQ-ViT [42] | 8/8 | 84.26 | 79.78 | 81.72 | 83.16 | 85.16 |
| NoiseQuant [44] | 8/8 | 84.22 | 79.51 | 81.45 | 83.13 | 85.20 |
| PTQ-ViT [21] | 6*/6* | 75.26 | 75.10 | 77.47 | - | - |
| PTQ4ViT [41] | 6/6 | 81.65 | 76.28 | 80.25 | 82.38 | 84.01 |
| APQ-ViT [42] | 6/6 | 82.21 | 77.76 | 80.42 | 84.18 | 85.60 |
| NoiseQuant [44] | 6/6 | 82.32 | 77.43 | 80.70 | 82.86 | 84.68 |
| RepQ-ViT [45] | 6/6 | 83.62 | 78.90 | 81.27 | 82.79 | 84.57 |
| APQ-ViT [42] | 4/8 | 72.63 | 77.14 | 79.55 | 80.56 | 81.94 |
| PTQ-ViT [21] | 4*/4* | - | - | 75.94 | - | - |
| APQ-ViT [42] | 4/4 | 41.41 | 43.55 | 67.48 | 83.16 | 85.16 |
| RepQ-ViT [45] | 4/4 | 68.48 | 69.03 | 75.61 | 79.45 | 78.32 |
| Method of QAT | W/A (bit) | DeiT-T | DeiT-S | DeiT-B | Swin-T | Swin-S |
| Full Precision | 32/32 | 72.20 | 79.85 | 81.80 | 81.20 | 83.23 |
| I-ViT [50] | 8/8 | 72.24 | 80.12 | 81.74 | 81.50 | 83.01 |
| Q-ViT [48] | 4/4 | 72.79 | 80.11 | - | 80.59 | - |
| AFQ-ViT [48] | 4/4 | - | 80.90 | 83.00 | 82.50 | 84.40 |
| Quantformer [46] | 4/4 | 69.90 | 78.20 | 79.70 | 78.30 | 81.00 |
| OFQ [23] | 4/4 | 75.46 | 81.10 | - | 81.88 | - |
| VVTQ [56] | 4/4 | 74.71 | - | - | 82.42 | - |
| Q-ViT [48] | 3/3 | 69.62 | 78.08 | - | 79.45 | - |
| AFQ-ViT [48] | 3/3 | - | 79.00 | 81.00 | 80.90 | 82.70 |
| Quantformer [46] | 3/3 | 65.20 | 75.40 | 78.30 | 77.40 | 79.20 |
| OFQ [23] | 3/3 | 72.72 | 79.57 | - | 81.09 | - |
| AFQ-ViT [48] | 2/2 | - | 72.10 | 74.20 | 74.70 | 76.90 |
| Quantformer [46] | 2/2 | 60.70 | 65.20 | 73.80 | 74.20 | 76.60 |
| OFQ [23] | 2/2 | 64.33 | 75.72 | - | 78.52 | - |
(a) ViT-B_16
(b) ViT-B_16-224
(c) ViT-L_16
(d) ViT-L_16-224
(e) OPT-13B
(f) OPT-30B
(g) OPT-66B
<details>
<summary>x3.png Details</summary>
### Visual Description
\n
## Bar Chart: Latency vs. Batch Size for FP16 and INT8 Precision
### Overview
The image is a grouped bar chart comparing the inference latency (in milliseconds) of two numerical precision formats, FP16 (16-bit floating point) and INT8 (8-bit integer), across four different batch sizes. The chart demonstrates how latency scales with increasing batch size for each format and highlights the performance difference between them.
### Components/Axes
* **Chart Type:** Grouped vertical bar chart.
* **X-Axis:** Labeled **"Batch Size"**. It has four discrete categories: **1, 8, 16, and 32**.
* **Y-Axis:** Labeled **"Latency(ms)"**. The scale is linear, ranging from **0.0 to 50.0**, with major tick marks at intervals of 12.5 (0.0, 12.5, 25.0, 37.5, 50.0).
* **Legend:** Located in the **top-left corner** of the chart area.
* A light gray rectangle corresponds to **"FP16"**.
* A dark red (maroon) rectangle corresponds to **"INT8"**.
* **Data Labels:** The exact latency value is printed above each bar.
### Detailed Analysis
The chart presents the following precise data points, confirmed by matching bar color to the legend:
| Batch Size | FP16 Latency (ms) | INT8 Latency (ms) |
| :--- | :--- | :--- |
| **1** | 2.24 | 2.26 |
| **8** | 11.14 | 7.93 |
| **16** | 21.5 | 14.66 |
| **32** | 43.81 | 29.07 |
**Trend Verification:**
* **FP16 Series (Light Gray Bars):** The latency shows a clear, near-linear upward trend as batch size increases. The line formed by the tops of the gray bars slopes steeply upward from left to right.
* **INT8 Series (Dark Red Bars):** The latency also increases with batch size, but the slope is less steep than for FP16. The upward trend is consistent.
### Key Observations
1. **Performance Crossover:** At the smallest batch size of 1, the latencies are virtually identical (2.24 ms vs. 2.26 ms), with INT8 being marginally slower. This is the only point where INT8 does not show an advantage.
2. **Growing Advantage with Batch Size:** For all batch sizes greater than 1, INT8 demonstrates significantly lower latency than FP16. The absolute and relative performance gap widens as the batch size increases.
3. **Scaling Behavior:** Both precision formats exhibit latency that scales roughly linearly with batch size. However, the scaling factor (the slope) is lower for INT8.
4. **Magnitude of Difference:** At the largest measured batch size (32), INT8 latency (29.07 ms) is approximately **33.6% lower** than FP16 latency (43.81 ms).
### Interpretation
This chart provides a clear performance comparison relevant to machine learning inference optimization. The data suggests that:
* **INT8 quantization offers a substantial latency benefit** over FP16 for batched inference workloads. This benefit becomes more pronounced as the batch size increases, making INT8 particularly advantageous for high-throughput scenarios.
* The minimal difference at batch size 1 indicates that the overhead of processing in INT8 format is negligible compared to the computational savings it provides. The primary advantage stems from more efficient computation and memory bandwidth usage with 8-bit integers versus 16-bit floats.
* The consistent linear scaling for both formats implies predictable performance characteristics, which is crucial for system design and capacity planning. The lower slope for INT8 means it can handle larger batches with a smaller relative penalty in latency.
* **Practical Implication:** For applications where maximizing throughput (processed items per second) is critical, using INT8 precision is strongly favored, especially when operating with batch sizes of 8 or more. The choice between FP16 and INT8 at very low batch sizes (like 1) may depend on other factors such as model accuracy requirements, as the latency difference is negligible.
</details>
<details>
<summary>x4.png Details</summary>
### Visual Description
## Grouped Bar Chart: Latency vs. Batch Size for FP16 and INT8 Precision
### Overview
The image displays a grouped bar chart comparing the inference latency (in milliseconds) of two numerical precision formats, FP16 (16-bit floating point) and INT8 (8-bit integer), across four different batch sizes. The chart demonstrates how latency scales with increasing batch size for each precision type.
### Components/Axes
* **Chart Type:** Grouped (clustered) bar chart.
* **X-Axis (Horizontal):**
* **Label:** "Batch Size"
* **Categories/Markers:** 1, 8, 16, 32. These represent the number of input samples processed in a single inference pass.
* **Y-Axis (Vertical):**
* **Label:** "Latency(ms)"
* **Scale:** Linear scale from 0.0 to 10.0, with major tick marks at 0.0, 2.5, 5.0, 7.5, and 10.0.
* **Legend:**
* **Position:** Top-left corner of the chart area.
* **Items:**
* A light gray rectangle labeled "FP16".
* A dark red (maroon) rectangle labeled "INT8".
* **Data Labels:** Numerical values are printed directly above each bar, indicating the precise latency measurement.
### Detailed Analysis
The chart presents the following data points, grouped by batch size:
| Batch Size | FP16 Latency (ms) | INT8 Latency (ms) |
| :--- | :--- | :--- |
| **1** | 1.53 | 1.52 |
| **8** | 3.03 | 2.38 |
| **16** | 5.3 | 3.74 |
| **32** | 10.04 | 6.43 |
**Trend Verification:**
* **FP16 Series (Light Gray Bars):** The latency shows a clear, steep upward trend as batch size increases. The increase appears to be more than linear, accelerating notably between batch sizes 16 and 32.
* **INT8 Series (Dark Red Bars):** The latency also increases with batch size, but the slope is consistently less steep than that of the FP16 series. The growth appears more gradual and controlled.
### Key Observations
1. **Performance Crossover at Low Batch Size:** At a batch size of 1, the latencies are nearly identical (1.53 ms vs. 1.52 ms), with INT8 being marginally faster.
2. **Diverging Performance Gap:** As batch size increases, the performance gap between INT8 and FP16 widens significantly. The advantage of INT8 becomes more pronounced with larger batches.
3. **Maximum Observed Difference:** The largest absolute difference occurs at batch size 32, where INT8 (6.43 ms) is approximately 3.61 ms faster than FP16 (10.04 ms), representing a ~36% reduction in latency.
4. **Scaling Behavior:** FP16 latency scales poorly with batch size, nearly doubling from batch 16 (5.3 ms) to batch 32 (10.04 ms). INT8 latency scales more favorably, increasing by a smaller factor over the same range (3.74 ms to 6.43 ms).
### Interpretation
This chart provides a clear performance comparison relevant to machine learning model deployment and optimization. The data suggests that **using INT8 quantization offers a significant latency advantage over FP16, and this advantage becomes increasingly valuable as the workload (batch size) grows.**
* **Why it matters:** In production environments where throughput (processing many samples quickly) is critical, using INT8 precision can lead to substantially faster inference times, especially for services that handle large batches of requests. This can translate to better user experience, lower computational costs, and higher system efficiency.
* **Underlying reason:** The trend is consistent with the expected benefits of quantization. INT8 operations typically require less memory bandwidth and computational resources than FP16 operations. The widening gap suggests that the overhead of managing larger batches exacerbates the inherent efficiency differences between the two numerical formats.
* **Practical implication:** For applications where batch size is large (e.g., offline processing, high-throughput servers), adopting INT8 quantization is strongly indicated. For very small batch sizes (e.g., real-time, single-request serving), the performance benefit may be negligible, and other factors like model accuracy post-quantization would become the primary consideration.
</details>
<details>
<summary>x5.png Details</summary>
### Visual Description
## Bar Chart: Latency vs. Batch Size for FP16 and INT8 Precision
### Overview
The image is a grouped bar chart comparing the inference latency (in milliseconds) of two numerical precision formats, FP16 (16-bit floating point) and INT8 (8-bit integer), across four different batch sizes. The chart demonstrates how latency scales with increasing batch size for each precision type.
### Components/Axes
* **Chart Type:** Grouped vertical bar chart.
* **X-Axis:** Labeled **"Batch Size"**. It has four discrete categories: **1, 8, 16, and 32**.
* **Y-Axis:** Labeled **"Latency(ms)"**. The scale is linear, ranging from **0.0 to 130.0**, with major tick marks at intervals of 32.5 (0.0, 32.5, 65.0, 97.5, 130.0).
* **Legend:** Positioned in the **top-left corner** of the chart area. It contains two entries:
* A light gray rectangle labeled **"FP16"**.
* A dark red (maroon) rectangle labeled **"INT8"**.
* **Data Series:** Two series of bars, grouped by batch size.
* **FP16 Series:** Represented by light gray bars.
* **INT8 Series:** Represented by dark red bars.
### Detailed Analysis
The precise latency values are annotated directly above each bar.
| Batch Size | FP16 Latency (ms) | INT8 Latency (ms) |
| :--- | :--- | :--- |
| **1** | 5.36 | 4.77 |
| **8** | 30.95 | 20.25 |
| **16** | 60.99 | 39.43 |
| **32** | 124.85 | 80.05 |
**Trend Verification:**
* **FP16 Trend (Gray Bars):** The latency shows a strong, near-linear upward trend as batch size increases. The slope is steep, indicating significant latency cost for larger batches.
* **INT8 Trend (Red Bars):** The latency also increases with batch size, but the slope is noticeably less steep than that of FP16. The growth appears more controlled.
### Key Observations
1. **Consistent Performance Advantage:** For every batch size shown, INT8 (dark red) exhibits lower latency than FP16 (light gray).
2. **Diverging Performance Gap:** The absolute difference in latency between FP16 and INT8 grows substantially as the batch size increases.
* At Batch Size 1: Difference = 0.59 ms (FP16 is ~12% slower).
* At Batch Size 32: Difference = 44.80 ms (FP16 is ~56% slower).
3. **Scaling Behavior:** Both precision formats show increased latency with larger batch sizes, which is expected due to increased computational load. However, INT8 demonstrates superior scalability.
### Interpretation
This chart provides a clear performance comparison relevant to machine learning model inference optimization. The data suggests that using INT8 quantization offers a significant latency benefit over FP16, and this advantage becomes dramatically more pronounced at larger batch sizes.
The underlying reason is likely that INT8 operations are more computationally efficient (requiring less memory bandwidth and enabling more parallel operations) on the target hardware. The widening gap indicates that the efficiency gains of INT8 are not just a fixed offset but compound as the workload scales. For a system designer, this implies that choosing INT8 over FP16 is particularly crucial for high-throughput scenarios where large batch sizes are used to maximize hardware utilization. The chart makes a strong case for quantization as a key technique for reducing inference latency, especially under heavy load.
</details>
<details>
<summary>x6.png Details</summary>
### Visual Description
## Bar Chart: Latency vs. Batch Size for FP16 and INT8 Precision
### Overview
The image is a grouped bar chart comparing the inference latency (in milliseconds) of two numerical precision formats, FP16 (16-bit floating point) and INT8 (8-bit integer), across four different batch sizes. The chart visually demonstrates the performance advantage of INT8 quantization over FP16, particularly as the workload (batch size) increases.
### Components/Axes
* **Chart Type:** Grouped vertical bar chart.
* **X-Axis:** Labeled **"Batch Size"**. It has four discrete categories: **1, 8, 16, and 32**.
* **Y-Axis:** Labeled **"Latency(ms)"**. The scale is linear, ranging from **0.0 to 30.0** with major tick marks at intervals of 7.5 (0.0, 7.5, 15.0, 22.5, 30.0).
* **Legend:** Positioned in the **top-left corner** of the chart area. It contains two entries:
* A light gray rectangle labeled **"FP16"**.
* A dark red (maroon) rectangle labeled **"INT8"**.
* **Data Series:** Two series of bars, grouped by batch size.
* **FP16 Series:** Light gray bars.
* **INT8 Series:** Dark red bars.
### Detailed Analysis
The exact latency values are annotated above each bar.
**Batch Size = 1:**
* FP16 (light gray bar): **2.97 ms**
* INT8 (dark red bar): **2.91 ms**
* *Trend Check:* Both bars are very short and nearly equal in height, indicating minimal latency difference at the smallest batch size.
**Batch Size = 8:**
* FP16 (light gray bar): **8.09 ms**
* INT8 (dark red bar): **5.44 ms**
* *Trend Check:* Both bars are taller than at batch size 1. The FP16 bar is now noticeably taller than the INT8 bar.
**Batch Size = 16:**
* FP16 (light gray bar): **15.03 ms**
* INT8 (dark red bar): **9.23 ms**
* *Trend Check:* Latency continues to increase for both. The height difference between the FP16 and INT8 bars has grown larger.
**Batch Size = 32:**
* FP16 (light gray bar): **29.66 ms**
* INT8 (dark red bar): **17.28 ms**
* *Trend Check:* This is the tallest pair of bars. The FP16 bar is significantly taller than the INT8 bar, showing the largest absolute difference in latency.
### Key Observations
1. **Consistent Performance Advantage:** For every batch size tested, the INT8 (dark red) bar is shorter than the corresponding FP16 (light gray) bar, indicating lower latency.
2. **Diverging Trend:** The performance gap between FP16 and INT8 widens as the batch size increases. At batch size 1, the difference is negligible (0.06 ms). At batch size 32, the difference is substantial (12.38 ms).
3. **Scaling Behavior:** Latency for both precision formats increases with batch size. However, the rate of increase (slope) is steeper for FP16 than for INT8.
4. **Data Integrity:** All values are explicitly labeled, removing ambiguity in reading the bar heights against the y-axis scale.
### Interpretation
This chart provides clear empirical evidence for the inference performance benefits of using INT8 quantization over FP16 precision in a computational workload (likely a neural network inference task).
* **What the data suggests:** INT8 quantization reduces computational latency. This benefit becomes more pronounced under heavier loads (larger batch sizes), suggesting that INT8 operations scale more efficiently. The near-equal latency at batch size 1 indicates that the overhead of quantization/dequantization or other fixed costs may be minimal compared to the compute savings.
* **How elements relate:** The x-axis (Batch Size) represents the independent variable—the workload intensity. The y-axis (Latency) is the dependent variable—the performance cost. The two bar colors (FP16 vs. INT8) represent the independent variable being tested—the precision format. The chart's structure directly compares the dependent variable across the two conditions for each level of workload.
* **Notable implications:** For system designers and ML engineers, this data argues for adopting INT8 quantization to improve throughput and reduce response times in production inference systems, especially for services that process requests in batches. The significant saving at batch size 32 (a ~42% reduction in latency) could translate to major cost savings or increased capacity in a deployed service. The chart effectively communicates that the performance advantage of INT8 is not static but scales favorably with demand.
</details>
<details>
<summary>x7.png Details</summary>
### Visual Description
## Bar Chart: Latency Comparison: FP16 vs. w8a8 by Batch Size
### Overview
This is a grouped bar chart comparing the inference latency (in milliseconds) of two different numerical precision formats, FP16 and w8a8, across four increasing batch sizes. The chart demonstrates how latency scales with batch size for each format.
### Components/Axes
* **Chart Type:** Grouped vertical bar chart.
* **X-Axis (Horizontal):** Labeled "Batch Size". It has four discrete categories: `128`, `256`, `512`, and `1024`.
* **Y-Axis (Vertical):** Labeled "Latency(ms)". The scale runs from 0 to 200, with major tick marks at 0, 50, 100, 150, and 200.
* **Legend:** Located in the top-left quadrant of the chart area. It contains two entries:
* A light gray rectangle labeled "FP16".
* A dark red (maroon) rectangle labeled "w8a8".
* **Data Labels:** The exact latency value is printed above each bar.
### Detailed Analysis
The chart presents the following data points, confirmed by matching bar color to the legend and reading the labels:
**Batch Size = 128**
* **FP16 (Light Gray Bar):** 28 ms
* **w8a8 (Dark Red Bar):** 22 ms
**Batch Size = 256**
* **FP16 (Light Gray Bar):** 44 ms
* **w8a8 (Dark Red Bar):** 33 ms
**Batch Size = 512**
* **FP16 (Light Gray Bar):** 87 ms
* **w8a8 (Dark Red Bar):** 63 ms
**Batch Size = 1024**
* **FP16 (Light Gray Bar):** 181 ms
* **w8a8 (Dark Red Bar):** 125 ms
**Visual Trend Verification:**
* **FP16 Series:** The light gray bars show a clear, accelerating upward trend. The increase from 128 to 256 is +16 ms, from 256 to 512 is +43 ms, and from 512 to 1024 is +94 ms. The slope steepens significantly at larger batch sizes.
* **w8a8 Series:** The dark red bars also show a consistent upward trend, but the rate of increase is more linear and less steep than FP16. The increases are +11 ms, +30 ms, and +62 ms for the same intervals.
### Key Observations
1. **Consistent Performance Advantage:** For every batch size shown, the w8a8 format exhibits lower latency than the FP16 format.
2. **Diverging Performance Gap:** The absolute difference in latency between FP16 and w8a8 grows as the batch size increases.
* At batch size 128, the difference is 6 ms.
* At batch size 1024, the difference is 56 ms.
3. **Scaling Behavior:** Both formats show increased latency with larger batch sizes, but FP16's latency scales more poorly (non-linearly) compared to the more moderate scaling of w8a8.
### Interpretation
The data suggests that the **w8a8 precision format offers superior latency performance and better scalability** for inference workloads compared to FP16, particularly as the computational load (batch size) increases. This is a critical insight for optimizing machine learning systems where throughput and response time are key.
The widening gap indicates that the efficiency benefits of w8a8 become more pronounced under heavier load. This could be due to factors like reduced memory bandwidth requirements, better cache utilization, or more efficient arithmetic operations inherent to the w8a8 format. For a system designer, this chart provides a clear quantitative argument for adopting w8a8 (or similar quantization schemes) to handle larger batches without incurring the same latency penalty as FP16. The chart effectively communicates that the choice of numerical precision is not just about accuracy, but is a fundamental lever for performance engineering.
</details>
<details>
<summary>x8.png Details</summary>
### Visual Description
\n
## Bar Chart: Latency vs. Batch Size for FP16 and w8a8 Precision
### Overview
This is a grouped bar chart comparing the inference latency (in milliseconds) of two different numerical precision formats—FP16 (16-bit floating point) and w8a8 (8-bit weights and 8-bit activations)—across four increasing batch sizes. The chart demonstrates the performance advantage of the w8a8 quantization format over the FP16 baseline.
### Components/Axes
* **Chart Type:** Grouped Bar Chart.
* **Y-Axis:** Labeled **"Latency(ms)"**. The scale runs from 0 to 400 with major tick marks at 0, 100, 200, 300, and 400.
* **X-Axis:** Labeled **"Batch Size"**. It displays four discrete categories: **128**, **256**, **512**, and **1024**.
* **Legend:** Positioned in the **top-left corner** of the chart area.
* A light gray rectangle corresponds to the label **"FP16"**.
* A dark red (maroon) rectangle corresponds to the label **"w8a8"**.
* **Data Series:** Two series are plotted for each batch size category.
* **Series 1 (FP16):** Light gray bars, positioned on the left within each group.
* **Series 2 (w8a8):** Dark red bars, positioned on the right within each group.
* **Data Labels:** The exact latency value in milliseconds is printed directly above each bar.
### Detailed Analysis
The chart presents the following precise data points:
| Batch Size | FP16 Latency (ms) | w8a8 Latency (ms) |
| :--- | :--- | :--- |
| **128** | 59 | 43 |
| **256** | 98 | 65 |
| **512** | 186 | 119 |
| **1024** | 380 | 249 |
**Trend Verification:**
* **FP16 Series (Light Gray):** The bar heights show a clear, steep upward trend. Latency increases significantly with each doubling of the batch size, from 59 ms at 128 to 380 ms at 1024.
* **w8a8 Series (Dark Red):** The bar heights also show a consistent upward trend, but the slope is less steep than the FP16 series. Latency increases from 43 ms at 128 to 249 ms at 1024.
### Key Observations
1. **Consistent Performance Advantage:** For every batch size, the w8a8 format exhibits lower latency than the FP16 format.
2. **Widening Performance Gap:** The absolute difference in latency between FP16 and w8a8 grows as the batch size increases.
* At batch size 128, the difference is 16 ms.
* At batch size 1024, the difference is 131 ms.
3. **Scaling Behavior:** Both precision formats show latency that scales roughly linearly with batch size within this range, but the scaling factor (the slope) is lower for w8a8.
### Interpretation
The data demonstrates the efficacy of the **w8a8 quantization technique** in reducing computational latency for inference tasks compared to the standard **FP16** precision. The key insight is that the performance benefit of w8a8 becomes **more pronounced at larger batch sizes**.
This suggests that w8a8 is particularly advantageous for high-throughput scenarios where processing large batches of data simultaneously is common (e.g., in data centers or batch processing pipelines). The reduced latency at scale implies potential for higher system throughput, lower operational costs, or the ability to handle more concurrent requests within a given time frame. The chart provides a clear, quantitative argument for adopting w8a8 precision in performance-sensitive deployment environments.
</details>
<details>
<summary>x9.png Details</summary>
### Visual Description
\n
## Bar Chart: Latency vs. Batch Size for FP16 and w8a8 Precision
### Overview
This is a grouped bar chart comparing the inference latency (in milliseconds) of two different numerical precision formats—FP16 (16-bit floating point) and w8a8 (8-bit weights and 8-bit activations)—across four different batch sizes. The chart visually demonstrates how latency scales with increasing batch size for each format.
### Components/Axes
* **Chart Type:** Grouped vertical bar chart.
* **Y-Axis (Vertical):**
* **Label:** `Latency(ms)`
* **Scale:** Linear scale from 0 to 500 ms.
* **Major Tick Marks:** 0, 125, 250, 375, 500.
* **X-Axis (Horizontal):**
* **Label:** `Batch Size`
* **Categories:** 128, 256, 512, 1024.
* **Legend:**
* **Position:** Top-left corner of the chart area.
* **Items:**
* Light gray square: `FP16`
* Dark red square: `w8a8`
* **Data Labels:** Numerical latency values are printed directly above each bar.
### Detailed Analysis
The chart presents paired bars for each batch size. The left bar (light gray) represents FP16 latency, and the right bar (dark red) represents w8a8 latency.
**Data Series & Values:**
| Batch Size | FP16 Latency (ms) | w8a8 Latency (ms) |
| :--- | :--- | :--- |
| **128** | 79 | 75 |
| **256** | 122 | 131 |
| **512** | 236 | 229 |
| **1024** | 489 | 490 |
**Trend Verification:**
* **FP16 (Light Gray Bars):** The series shows a clear, steep upward trend. Latency increases monotonically and significantly with batch size, from 79 ms at batch size 128 to 489 ms at batch size 1024.
* **w8a8 (Dark Red Bars):** This series also shows a strong upward trend. Latency increases from 75 ms at batch size 128 to 490 ms at batch size 1024.
### Key Observations
1. **Dominant Trend:** For both precision formats, latency increases approximately linearly with batch size. The jump from batch size 512 to 1024 results in roughly a doubling of latency for both.
2. **Performance Crossover:** The relative performance of the two formats changes with batch size.
* At **batch size 128**, w8a8 (75 ms) is slightly faster than FP16 (79 ms).
* At **batch size 256**, the relationship inverts, with FP16 (122 ms) becoming faster than w8a8 (131 ms).
* At **batch size 512**, w8a8 (229 ms) regains a slight lead over FP16 (236 ms).
* At **batch size 1024**, the latencies are nearly identical (489 ms vs. 490 ms), with FP16 having a negligible advantage.
3. **Magnitude of Difference:** The absolute difference in latency between the two formats is relatively small across all tested batch sizes, ranging from 4 ms to 9 ms.
### Interpretation
This chart provides a performance comparison relevant to machine learning model deployment, where choosing between higher precision (FP16) and lower precision (w8a8 quantization) involves trade-offs between accuracy, speed, and memory usage.
* **What the data suggests:** The data indicates that for this specific model and hardware configuration, the latency benefit of using w8a8 quantization over FP16 is minimal and inconsistent. The performance is highly dependent on the batch size. There is no clear, consistent "winner" in terms of raw inference speed across the tested range.
* **Relationship between elements:** The primary relationship shown is the direct, positive correlation between batch size and latency for both precision types. The secondary relationship is the comparison between the two types, which shows their performance is tightly coupled and crosses over multiple times.
* **Notable anomalies/outliers:** The most notable pattern is the inconsistency in which format is faster. The inversion at batch size 256 (where FP16 outperforms w8a8) is particularly interesting, as one might expect quantization to consistently reduce latency. This could point to specific hardware utilization characteristics, kernel optimization differences, or overheads associated with the quantization/dequantization process that become more or less prominent at different batch sizes.
* **Conclusion:** The choice between FP16 and w8a8 for this workload cannot be made on latency grounds alone based on this data. The decision would likely hinge on other factors such as model accuracy after quantization, memory footprint reduction (which w8a8 would significantly improve), and the typical batch sizes used in the production environment. The chart demonstrates that latency scaling is predictable, but the relative advantage of quantization is not guaranteed.
</details>
<details>
<summary>x10.png Details</summary>
### Visual Description
## Bar Chart: Latency vs. Batch Size for FP16 and w8a8 Precision Formats
### Overview
This is a grouped bar chart comparing the inference latency (in milliseconds) of two different numerical precision formats—FP16 (16-bit floating point) and w8a8 (likely 8-bit weights and 8-bit activations)—across four different batch sizes. The chart demonstrates that latency increases with batch size for both formats, but the w8a8 format consistently exhibits lower latency than FP16.
### Components/Axes
* **Chart Type:** Grouped vertical bar chart.
* **X-Axis:** Labeled **"Batch Size"**. It has four discrete categories: `128`, `256`, `512`, and `1024`.
* **Y-Axis:** Labeled **"Latency(ms)"**. The scale is linear, with major tick marks at 0, 225, 450, 675, and 900.
* **Legend:** Located in the top-left quadrant of the chart area.
* A light gray rectangle corresponds to the label **"FP16"**.
* A dark red (maroon) rectangle corresponds to the label **"w8a8"**.
* **Data Labels:** The exact latency value is printed directly above each bar.
### Detailed Analysis
The chart presents the following precise data points, confirmed by matching bar color to legend and reading the labels:
| Batch Size | FP16 Latency (ms) | w8a8 Latency (ms) |
| :--- | :--- | :--- |
| **128** | 139 | 122 |
| **256** | 228 | 194 |
| **512** | 432 | 366 |
| **1024** | 848 | 720 |
**Trend Verification:**
* **FP16 Series (Light Gray Bars):** The line formed by the tops of the light gray bars slopes sharply upward from left to right. Latency increases monotonically and significantly with batch size, growing from 139 ms at batch size 128 to 848 ms at batch size 1024.
* **w8a8 Series (Dark Red Bars):** The line formed by the tops of the dark red bars also slopes upward from left to right, but remains consistently below the FP16 series. Latency increases from 122 ms at batch size 128 to 720 ms at batch size 1024.
### Key Observations
1. **Consistent Performance Advantage:** For every batch size tested, the w8a8 format has lower latency than the FP16 format.
2. **Widening Performance Gap:** The absolute difference in latency between FP16 and w8a8 grows as the batch size increases.
* At batch size 128, the difference is 17 ms.
* At batch size 1024, the difference is 128 ms.
3. **Non-Linear Scaling:** The increase in latency is not linear with the batch size. Doubling the batch size from 512 to 1024 results in a near-doubling of latency for both formats (FP16: 432ms → 848ms; w8a8: 366ms → 720ms), suggesting a roughly linear relationship between batch size and latency within this range.
### Interpretation
This chart provides empirical evidence for the performance benefits of using lower-precision arithmetic (w8a8) in computational workloads, likely machine learning model inference, compared to higher-precision arithmetic (FP16).
* **What the data suggests:** The w8a8 format offers a consistent and scalable reduction in processing time (latency) compared to FP16. The benefit becomes more pronounced at larger batch sizes, which are common in high-throughput serving scenarios.
* **How elements relate:** The x-axis (Batch Size) is the independent variable, representing the workload. The y-axis (Latency) is the dependent variable, measuring performance. The two series (FP16, w8a8) represent different system configurations. The chart directly visualizes the trade-off: adopting w8a8 quantization reduces latency for a given workload.
* **Notable implications:** The data implies that for applications where inference speed is critical, especially at scale (large batch sizes), implementing w8a8 quantization could yield significant performance improvements over FP16. The near-linear scaling suggests predictable performance characteristics, which is valuable for capacity planning. The chart does not show accuracy or quality metrics, so the interpretation is strictly about computational speed.
</details>
(a) ViT-B_16
(b) ViT-B_16-224
(c) ViT-L_16
(d) ViT-L_16-224
(e) OPT-13B
(f) OPT-30B
(g) OPT-66B
(h) OPT-175B
Figure 3: Inference latency of the ViT and OPT using FasterTransformer on NVIDIA A100-80GB GPUs. The data of OPT is from [18].
#### 3.1.2 Quantization for Transformer-Based Large Language Models
Before 2023, the study of quantization [57, 58, 59, 60, 61, 62, 63, 64, 65, 66] on Transformer-based NLP focused almost entirely on the BERT architecture. With the popularity of pretrained large language models [67, 1, 2], researchers [68, 69, 70] stated to explore how to quantize the Transformer with billions of parameters and explores more efficient quantization schemes with limited data and computational overhead.
Post-training quantization. Based on the analysis of outliers for quantized Transformers, Outlier Suppression [71] proposes to migrate the gamma of LayerNorm to the next module for that the gamma amplifies the outliers in the output and causes large quantization error, and then clips the tokens in a coarse-to-fine procedure. MREM [59] focuses on reducing the computational cost of quantization, and minimizes the output quantization error for all modules in parallel manner. Zeroquant [70] finds that the performance degradation is due to the different dynamic range of tokens and weights, proposes to adopt group-wise quantization for weights and token-wise quantization for activations, and utilizes knowledge distillation layer by layer. For large Transformers with billions of parameters, the outlier is still the main reason for large accuracy degradation for quantized models. To address this, LLM.int8() [68] represents the activations and outliers of weights with 16-bit and performs 8-bit vector-wise quantization for weight tensor, but the acceleration is limited and even worsened due to the irregular quantization granularity [18]. GPTQ [72] also only quantizes weight parameters as LLM.int8(), but adopts unified quantization strategy. GPTQ quantizes the weights based on approximate second-order information, thereby getting much more accurate quantized weight in a few hours. AWQ [73] proposes to search for the optimal per-channel scale factors by observing the distribution of activations instead of weights, allowing the LLMs retain the capabilities for different domains and modalities. Outlier Suppression+ [74] explores more accurate outliers suppression schemes with channel-wise shifting and scale, thereby helping align the range of different activation channels and scale down the outliers. The shifting and scale factors could also be merged with other weight parameters. Similarly, SmoothQuant [18] and QLLM [75] propose the mathematically equivalent per-channel scaling transformation that migrates the quantization difficulty from activations to weights. To further improve the performance of quantized LLMs, QLLM also learns low-rank parameters by minimizing the reconstruction error of the outputs between floating-point and quantized LLMs with limited calibration data. Similarly, RPTQ [76] utilizes a reorder-based scheme that rearranges the channels of activations with a similar range and quantizes them in cluster, and then migrates the scale into LayerNorm and weights of linear layers without extra computational overhead in the inference.Based on scale migration, OmniQuant [19] further proposes a learnable PTQ method module by module, where the weight clipping parameters and transformation scale of activations are optimized with gradient descent. To minimize error accumulation in adjacent blocks, CBQ [77] presents a cross-block reconstruction framework, simultaneously learn the rounding matrices of weight and step sizes of weights and activations. The rounding matrices are learned with LoRA technique, which does not bring much extra cost for PTQ. Like GPTQ, SqueezeLLM [78] also focuses on quantizing weights for the memory bandwidth and proposes to search for the optimal bit based on second-order information. Also, SqueezeLLM does not suppress the outliers and sensitive weight values but stores them in an efficient sparse format for more accurate quantized LLMs. Unlike the previous methods, SignRound [79] proposes to optimize quantized LLMs from the perspective of adaptive rounding. SignRound designs block-wise tuning using signed gradient descent to learn the weight rounding, thereby greatly helping the output reconstruction of each block.
TABLE III: Perplexity (PPL) comparison of different quantization methods on WikiText2. W/E denotes weights and embedding, respectively. * represents adopting different methods to get the perplexity, please refer to the corresponding papers.
| Method | PTQ | Bits | LLaMA | | | |
| --- | --- | --- | --- | --- | --- | --- |
| (W-E) | 7B | 13B | 30B | 65B | | |
| FP16 | $\times$ | 16/16 | 5.68 | 5.09 | 4.10 | 3.56 |
| SmoothQuant* [18] | ✓ | 8/8 | 11.56 | 10.08 | 7.56 | 6.20 |
| LLM-QAT* [80] | $\times$ | 8/8 | 10.30 | 9.50 | 7.10 | - |
| OS+ [74] | ✓ | 6/6 | 5.76 | 5.22 | 4.30 | 3.65 |
| OmniQuant [19] | ✓ | 6/6 | 5.96 | 5.28 | 4.38 | 3.75 |
| QLLM [75] | ✓ | 6/6 | 5.89 | 5.28 | 4.30 | 3.73 |
| SqueezeLLM [78] | ✓ | 4/16 | 5.79 | 5.18 | 4.22 | 3.76 |
| SignRound [79] | ✓ | 4/16 | 6.12 | 5.32 | 4.52 | 3.90 |
| OmniQuant [19] | ✓ | 4/16 | 5.86 | 5.21 | 4.25 | 3.71 |
| LLM-QAT* [80] | $\times$ | 4/8 | 10.90 | 10.00 | 7.50 | - |
| OS+ [74] | ✓ | 4/4- | 14.17 | 18.95 | 22.61 | 9.33 |
| OmniQuant [19] | ✓ | 4/4 | 11.26 | 10.87 | 10.33 | 9.17 |
| QLLM [75] | ✓ | 4/4 | 9.65 | 8.41 | 8.37 | 6.87 |
| PEQA [81] | $\times$ | 4/4 | 5.84 | 5.30 | 4.36 | 4.02 |
| SqueezeLLM [78] | ✓ | 3/16 | 6.32 | 5.60 | 4.66 | 4.05 |
| OmniQuant [19] | ✓ | 3/16 | 6.49 | 5.68 | 4.74 | 4.04 |
| PEQA [81] | $\times$ | 3/3 | 6.19 | 5.54 | 4.58 | 4.27 |
| OmniQuant [19] | ✓ | 2/16 | 15.47 | 13.21 | 8.71 | 7.58 |
Quantization-aware training. Q-BERT [57] is the early work that conducts quantization-aware training on Transformer-based architecture for natural language processing. Inspired by HAWQ [82], Q-BERT searches mixed-quantization settings based on the second order Hessian information and adopts group-wise quantization to partition parameter matrix into multiple groups to reduce accuracy degradation. I-BERT [62] not only quantizes the linear and self-attention layer, but also designs an integer-only inference scheme for the nonlinear operations (GELU, SoftMax and LayerNorm) in Transformers, which also inspires the FQ-ViT and I-ViT for the quantization of vision Transformer. Specifically, I-BERT proposes to use the second-order polynomial approximate for GELU and exponential function of SoftMax, and calculate the standard deviation of LayerNorm based on Newton’s Method. In this way, I-BERT can achieve faster inference compared to the baseline and normal quantization methods. To compensate for the disadvantages of the hand-crafted heuristics in Q-BERT, AQ-BERT [66] proposes an automatic mixed-precision quantization scheme to learn the bit-width and parameters of each layer simultaneously, which is inspired from differentiable network architecture search. AQ-BERT achieves better results than Q-BERT and more suitable for resource-limited devices. Besides the QAT schemes for BERT, there are only a few papers that conducts quantization-aware training on large language models for the huge training overhead. PEQA [81] and QLoRA [20] explore parameter-efficient fine-tuning techniques to train the quantized LLMs, with the former only optimizing the scale factors while freezing the quantized weights, and the latter adopting 4-bit NormalFloat, double quantization, and paged optimizers for normally distributed weights and reduce memory footprint during training. LLM-QAT [80] proposes a data-free quantization-aware training method, where the training data is generated from the original pretrained large language model. LLM-QAT not only quantizes all the linear layers and self-attention, but also quantizes the KV cache with cross-entropy based logits distillation.
#### 3.1.3 Quantization for Transformer-Based Vision Models
Post-training quantization. PTQ-ViT [21] firstly explores post-training quantization on vision Transformers, using the nuclear norm of the attention map and output feature in the Transformer layer to determine the bit-width of each layer. To get more accurate quantized Transformers, PTQ-ViT further proposes a rank loss to keep the relative order of the self-attention of quantized Transformers. PTQ4ViT [41] observers that the distributions of post-softmax and post-GELU activations are hard to quantize with conventional methods, and so introduces a twin uniform quantizer and use a Hessian guild metric to find the optimal scale parameters. To construct a full-quantized vision Transformer, FQ-ViT [22] not only quantizes all the linear and self-attention layers, but also quantizes LayerNorm and SoftMax with the power-of-two factor and Log2 quantizer. APQ-ViT [42] explores block-wise optimization scheme to determine the optimal quantizer for extremely low-bit PTQ, and utilizes an asymmetric linear quantization to quantize the attention map for maintaining the Matthew-effect of Softmax. Based on the observation that the normal uniform quantizer can not effectively handle the heavy-tailed distribution of vision Transformer activations, NoisyQuant [44] proposes to enhance the quantizer by adding a fixed uniform noise following the theoretical results, refining the quantized distribution to reduce quantization error with minimal computation cost. For the post-LayerNorm and post-SoftMax activations that have extreme distributions, RepQ-ViT [45] applies channel-wise quantization to deal with the severe inter-channel variation of the former and utilize log $\sqrt{2}$ quantizer to compress the power-law features of the later one. Before the inference, RepQ-ViT reparameterizes the scale factors to the layer-wise and log2 quantizer with mere computations.
<details>
<summary>x11.png Details</summary>
### Visual Description
## Diagram: Knowledge Distillation for Large Transformer-Based Models
### Overview
The image is a hierarchical flowchart or taxonomy diagram illustrating the categorization of knowledge distillation techniques for large transformer-based models. It organizes methods into two primary domains—Computer Vision and Natural Language Processing—and further subdivides them by the type of distillation signal or approach used, providing specific model examples for each category.
### Components/Axes
The diagram is structured as a top-down tree with the following components:
1. **Root Node (Top Center):** A blue rectangular box containing the main title: "Knowledge Distillation for Large Transformer-Based Models".
2. **Primary Branches (Second Level):** Two arrows descend from the root node to two main category boxes:
* **Left Branch:** A green box labeled "Computer Vision".
* **Right Branch:** A pink box labeled "Natural Language Processing".
3. **Secondary Branches (Third Level):** Arrows descend from each primary category to sub-category boxes.
* **Under "Computer Vision":** Three boxes.
* Left: A purple box labeled "Logits-based".
* Center: A yellow box labeled "Hint-based".
* Right: A grey box labeled "Others".
* **Under "Natural Language Processing":** Four boxes.
* Left: A purple box labeled "Logits-based".
* Center-Left: A yellow box labeled "Hint-based".
* Center-Right: A peach box labeled "API-based".
* Right: A grey box labeled "Others".
4. **Descriptive Text & Examples (Fourth Level):** Below each third-level sub-category box, there is a text block consisting of a brief description followed by a bulleted list of example models or methods.
### Detailed Analysis
The diagram's content is fully transcribed below, organized by the primary domain.
**Computer Vision Branch:**
* **Sub-category: Logits-based**
* Description: "Output logits"
* Examples:
* DeiT
* TinyViT
* **Sub-category: Hint-based**
* Description: "Intermediate features"
* Examples:
* ViTKD
* ManifoldKD
* **Sub-category: Others**
* Description: "Other contexts"
* Examples:
* GPT4Image
* BLIP
**Natural Language Processing Branch:**
* **Sub-category: Logits-based**
* Description: "Output logits"
* Examples:
* DistilBERT
* MINILLM
* **Sub-category: Hint-based**
* Description: "Intermediate features"
* Examples:
* MobileBERT
* TinyBERT
* **Sub-category: API-based**
* Description: "Generated contexts"
* Examples:
* PaD
* Lion
* **Sub-category: Others**
* Description: "Parameters"
* Examples:
* Bert-of-theseus
* ProKT
### Key Observations
1. **Structural Symmetry and Divergence:** The diagram shows a parallel structure for the "Logits-based" and "Hint-based" categories across both CV and NLP domains, using identical labels and color coding (purple and yellow, respectively). This suggests these are fundamental, cross-domain distillation paradigms.
2. **Domain-Specific Categories:** The NLP branch includes an additional, distinct category—"API-based" (peach box)—which is not present under Computer Vision. This indicates a methodological approach specific to or more prevalent in language model distillation.
3. **Consistent Categorization Logic:** The "Others" category (grey box) exists in both domains but contains different descriptive text and examples, reflecting domain-specific alternative techniques. In CV, it refers to "Other contexts," while in NLP, it refers to "Parameters."
4. **Model Examples:** The listed examples are well-known models or frameworks within their respective fields (e.g., DistilBERT, TinyBERT for NLP; DeiT, ViTKD for CV), serving as concrete instantiations of each distillation approach.
### Interpretation
This diagram serves as a **conceptual taxonomy** for the field of knowledge distillation applied to transformer models. It does not present empirical data or performance metrics but rather organizes the landscape of methodological approaches.
* **What it demonstrates:** The primary insight is that knowledge distillation is not a monolithic technique. The field has evolved specialized strategies based on *what* knowledge is transferred (logits, intermediate features, generated contexts, parameters) and the *context* in which it is applied (vision vs. language tasks).
* **Relationship between elements:** The hierarchy explicitly shows that the choice of distillation method is a secondary consideration, dependent first on the primary application domain (CV or NLP). The color-coding reinforces that similar methodological families (Logits-based, Hint-based) exist across domains, while unique categories (API-based) highlight domain-specific innovations.
* **Notable patterns:** The presence of "API-based" distillation only under NLP is significant. It likely refers to methods where a large "teacher" model acts as an API to generate training contexts or data for a smaller "student" model, a paradigm more naturally suited to generative language tasks. The "Others" category in NLP focusing on "Parameters" (e.g., Bert-of-theseus, which replaces layers) points to architectural manipulation as a distinct form of knowledge transfer.
* **Purpose and utility:** This diagram is a foundational reference for researchers or engineers. It helps answer: "Given my domain (vision or language), what are the high-level families of distillation techniques available, and what are some representative models I should look at?" It provides a structured entry point into the literature, mapping broad strategies to specific, citable works.
</details>
Figure 4: The taxonomy of knowledge distillation used for large Transformer-based models.
Quantization-aware training. When compressing vision Transformers to extremely low-bit precision, PTQ can not optimize the large quantization error with limited calibration images and suffers from significant performance reduction. As such, QAT is urgently required for more accurate low-bit vision Transformers. Q-ViT [48] finds that MSA and GELU are highly sensitive to quantization, and so proposes a fully differentiable quantization method that adopts head-wise bit-width and switch scale during the quantization searching process. Quantformer [46] takes the self-attention rank into consideration, and proposes to maintain the consistency between quantized and full-precision vision Transformers. In addition, Quantformer presents the group-wise strategy to quantize feature of patches in different dimensions, where each group adopts different quantization parameters and the extra computation cost is negligible. Based on the observation that sever performance degradation suffers from the quantized attention map, AFQ-ViT [47] designs an information rectification module and a distribution guided distillation during quantization training. The former helps recover the distribution of attention maps with information entropy maximization in the inference, and the latter reduces the distribution variation with attention similarity loss in the backward. Similar with FQ-ViT [22], I-ViT [50] also explores the integer-only quantization scheme for ViTs. I-ViT designs Shiftmax, ShiftGELU and I-LayerNorm to replace the vanilla modules with bit-wise shift and integer matrix operations in the inference, achieving $3.72-4.11\times$ acceleration compared to the floating-point model. OFQ [23] finds that weight oscillation causes unstable quantization-aware training and leads to sub-optimal results, and the oscillation comes from the learnable scale factor and quantized query and key in self-attention. To address that, OFQ proposes statistical weight quantization to improve quantization robustness, freezing the weights with high confidence and calming the oscillating weights with confidence-guided annealing. For the query and key in the self-attention, OFQ presents query-key reparameterization to decouple the negative mutual-influence between quantized query and key weights oscillation. Similarly, VVTQ [56] analyzes ViT quantization from the perspective of variation, which indicates that the data variance within mini-batch is much harmful to quantization and slow down the training convergence. To reduce the impact of variations in the quantization-aware training, VVTQ proposes a multi-crop knowledge distillation-based quantization methodology, and introduces module-dependent quantization and oscillation-aware regularization to enhance the optimization process.
We summarize the results of these PTQ and QAT methods in Table II. Most schemes conduct PTQ on 8-bit and 6-bit, and there is server degradation when quantized to 4-bit. With the complete training data, QAT could push the ViT quantization to 4-bit and less. Some methods even show better results than floating point models, which prove that quantization-aware training could tap into the true accuracy of quantized models efficiently. Figure 3 shows the latency results of ViT models using FasterTransformer https://github.com/NVIDIA/FasterTransformer.
#### 3.1.4 Discussion
With the presence of extreme distributions and outliers, quantization for Transformers is much more difficult than that for convolutional neural networks. To recover the performance of quantized models, various methods are proposed to address the quantization-unfriendly components in Transformers. Specifically, for vision tasks, the existing methods optimizes the quantized Transformers in three ways: retaining the self-attention rank, rectifying the distribution of extreme distribution, and addressing the weight oscillation and data variation in quantization-aware training, as shown in Figure 2. And for natural language processing task, most schemes aims to process the outlier of weights and activations. What’s more, the training overhead is usually unacceptable for large language models with billions of parameters, and therefore quantizing module by module and conducting parameter-efficient fine-tune are much popular. However, when compressing to extremely low bit-width, the quantized Transformers suffer significant performance degradation, and perform far worse than the floating-point models. As such, how to build a more accurate and low-bit Transformers is still a difficult problem to be solved.
### 3.2 Knowledge Distillation
In this section, we will introduce knowledge distillation frameworks used in compressing Transformer based foundation models, including both language and vision models.
#### 3.2.1 Overview of Knowledge Distillation
Knowledge distillation (KD) aims to train student networks by compressing [83, 84, 85] or transferring [86] knowledge from teacher networks. In this paper, our main focus lies on distillation methods proposed to achieve a compact student model while preserving satisfactory performance compared to a cumbersome teacher model. The student models typically have narrower and shallower architectures, making them more suitable for deployment on resource-limited systems.
We will first discuss two different kinds of knowledge distillation in the following: logits-based methods [86, 87, 88, 89] which convey knowledge on the logits level and hint-based methods [90] which convey knowledge through intermediate features. To illustrate the objective of logits-based KD in classification tasks, we denote the logits output of the teacher and student network as $z^{\bf s},z^{\bf t}\in\mathbb{R}^{C}$ , where $C$ represents the number of classes. Neural networks often generate class probabilities by applying a softmax function to the logits, converting them into probabilities $p^{\bf s},p^{\bf t}$ as follows:
$$
p_{i}^{\bf s}=\frac{\exp(z_{i}^{\bf s})}{\sum_{j=1}^{C}\exp(z_{j}^{\bf s})},
\quad p_{i}^{\bf t}=\frac{\exp(z_{i}^{\bf t})}{\sum_{j=1}^{C}\exp(z_{i}^{\bf t
})}, \tag{6}
$$
With above probabilities, the logits-based KD minimizes the KL divergence between the probabilities of the teacher and student model as:
$$
L_{logits}=KL(p^{\bf t}||p^{\bf s})=\sum_{j=1}^{C}p_{j}^{\bf t}log(\frac{p_{j}
^{\bf t}}{p_{j}^{\bf s}}), \tag{7}
$$
As for hint-based KD, given the intermediate features of the teacher and student network, $F^{\bf s},F^{\bf t}\in\mathbb{R}^{H\times W\times C}$ , the corresponding loss is formulated as:
$$
L_{hint}=\mathcal{H}(F^{\bf s},F^{\bf t})=||F^{\bf t}-\phi(F^{\bf s})||^{2}, \tag{8}
$$
where $\phi$ is a function used to ensure that the student features have the same shape as the teacher features. $\mathcal{H}$ represents the chosen metric function, here we provide an example using mean squared error.
In addition to the two mainstream KD methods mentioned above, we will also discuss an API-based method, where only the teacher’s generated outputs are accessible in today’s large language models (LLMs).
#### 3.2.2 KD for Transformer-Based Large Language Models
Logits-based KD. Tang et al. [88] employs knowledge distillation to compress BERT [91], a large language model, into a much lighter bidirectional long short-term memory network (BiLSTM) [92] for natural language processing (NLP) tasks. The distillation process aims to minimize the mean squared error (MSE) loss between the student’s logits and the teacher’s logits. Following distillation, the shallow BiLSTM-based model achieves comparable results to ELMo [93], but with approximately 99% fewer parameters and a 15 $\times$ faster inference speed. Similarly, DistillBERT [24] initializes the shallower student with teacher’s parameters, and minimized the soft target probabilities between the teacher and the student, also known as word-level KD. SeqKD [94] fine-tunes the student model on the sequence-level teacher-generated data. MixKD [95] extends the idea of encouraging the student to mimic the teacher’s logits to the linear interpolation of example pairs [96]. Theoretical analysis has shown that, under reasonable conditions, MixKD can effectively reduce the gap between generalization error and empirical error. Turc et al. [97] demonstrates the ongoing significance of pre-training, even when employing smaller architectures. They introduce a methodology called Pre-trained Distillation (PD), which begins with pre-trained compact student models. Subsequently, it explores the transfer of task knowledge from large fine-tuned models using conventional logits-based KD. In contrast, MINILLM [98] highlights a limitation of conventional logits-based KD that minimize forward Kullback-Leibler divergence (KLD) in free-run generation. They find that such approaches can lead the student model to overestimate the low-probability regions of the teacher’s distribution. To address this issue, MINILLM proposes replacing the forward KLD objective (as shown in Eq. 7) with reverse KLD, i.e., $reverse~{}\mathrm{KLD}:=KL(p^{\bf s}||p^{\bf t})$ , which is better suited for knowledge distillation on generative large language models (LLMs). GKD [99] identifies a challenge in distilling knowledge into auto-regressive student models, which is the train-test distribution mismatch. Specifically, partial sequences encountered by the student during generation phase can be significantly different from the ones observed during the training phase. To address this issue, GKD trains the student on its self-generated output sequences by leveraging feedback (logits) from the teacher on such sequences. It also facilitates the integration of distillation with RL fine-tuning of LLMs. Huang et al. [100] further combines the hint-based KD with multitask in-context learning.
Hint-based KD. Sun et al. [101] proposes patient knowledge distillation to help shallow BERT student learn from multiple intermediate features [102, 103, 104, 25] of the deep BERT teacher. Li et al. [105] utilizes hints extracted from both intermediate hidden states and attention distributions to enhance the training of Non-AutoRegressive Translation (NART) models. The distilled NART models attain performance similar to a powerful LSTM-based AutoRegressive Translation (ART) baseline in various machine translation tasks. Notably, the distilled NART models exhibit a speed improvement of 17 $\times$ over their ART counterparts. Similarly, Mukherjee et al. [106] employs MBERT [107] as a teacher model to guide the training of smaller BiLSTM models, resulting in a significant 51 $\times$ speed improvement for batch inference. MobileBERT [103] implements two objective functions to distill knowledge from a BERT teacher incorporated with inverted bottleneck, including attention distributions and hidden states, to a slimmed-down version of BERT as the student model. In addition to these objectives, TinyBERT [104] extends the distillation process by also transferring knowledge from the teacher’s logits to the student model. Wang et al. [25] proposes MINILM to train the student networks to mimic both the attention distributions and the scaled dot-product between values of teacher models. It retains more than 99% accuracy on General Language Understanding Evaluation (GLUE) benchmark tasks using 50% of the Transformer parameters and computations of the teacher model.
API-based KD. Today’s language models have reached unprecedented scale [4, 108, 109] in terms of computation and model parameters, as evident from the significant advancements made in large language models (LLMs) like GPT-4 [2]. However, the power of such models is only accessible through their APIs, which limits the ability to utilize conventional KD methods that rely on intermediate features or logits of the teacher model. In this section, we will discuss API-based KD, also known as black-box KD, where only the teacher’s final generations are accessible for the purpose of distillation.
Several recent studies [110, 111, 112, 113, 114, 7, 115, 8, 116] have showed the promising results in fine-tuning small models on the outputs generated by LLMs. Specifically, they first prompt a very large teacher model, such as GPT-3 with 175B parameters, to solve complex questions via chain-of-thought (CoT) reasoning [117]. Then, they use the generated instructions [7, 115], explanations [8] or rationales [110] to fine-tune a much smaller student. Furthermore, Zhu et al. [118] proposes PaD, which utilizes program-aided reasoning, such as an additional Python interpreter, to help student models overcome faulty steps in the CoT reasoning with automated error checking. Shridhar et al. [119] introduces Socratic CoT that trains a combination of two small distilled models: a problem decomposer and a subproblem solver. These models work together to decompose and solve complex problems effectively. Fu et al. [120] demonstrates that specializing the small student model’s abilities for the specific reasoning task, by transferring the knowledge in larger teacher model’s generic directions, yields promising results. Jiang et al. [7] identifies a shortcoming in the aforementioned approaches, as they lack the ability to incorporate feedback for identifying challenging instructions where the student model’s performance falls short. To address this limitation, they propose a three-stage adversarial loop of distillation that incorporates feedback and addresses these challenging instructions.
Other KDs. Xu et al. [121] compresses the BERT model by progressive module replacing. The original model is first divided into multiple modules, and more compact substitutes are created for these modules. Then, they randomly substitute the original modules with their corresponding substitutes and train the compact modules without any additional loss functions.
Jha et al. [122] discovers that fitting both the student and teacher models, each with numerous parameters, into GPU memory with traditional knowledge distillation is impractical. To overcome this limitation, they propose a teacher-free task-agnostic distillation method which uses a truncated version (student) of the large model (teacher) for initialization [123] and then continuing the pre-training of student, without employing any distillation loss.
TABLE IV: Comparison with previous Transformer-based language model distillation approaches. The GLUE score is averaged on 8 tasks, i.e., SQuAD2, MNLI-m, SST-2, QNLI, CoLA, RTE, MRPC, QQP.
| Models | Distillation | Teacher | $\#$ Layer | $\#$ Params | Speed Up | GLUE |
| --- | --- | --- | --- | --- | --- | --- |
| BERT ${}_{BASE}$ [91] | - | - | 12 | 110M | $\times$ 1 | 79.6 ${}^{\dagger}$ (81.5 ${}^{\ddagger}$ ) |
| BERT ${}_{LARGE}$ [91] | - | - | 24 | 340M | - | 81.9 ${}^{\dagger}$ |
| DistilBERT [24] | logits | BERT ${}_{BASE}$ | 4 / 6 | 52M / 66M | $\times$ 3.0 / $\times$ 2.0 | 71.2 / 76.2 (75.2 ${}^{\ddagger}$ ) |
| TinyBERT [104] | hint + logits | BERT ${}_{BASE}$ | 4 / 6 | 15M / 66M | $\times$ 9.4 / $\times$ 2.0 | 76.5 / 79.0 (79.1 ${}^{\ddagger}$ ) |
| BERT-PKD [101] | hint + logits | BERT ${}_{BASE}$ | 3 / 6 | 46M / 66M | $\times$ 3.7 / $\times$ 1.6 | 76.0 ${}^{\dagger}$ / 80.6 ${}^{\dagger}$ |
| MobileBERT [103] | hint | BERT ${}_{LARGE}$ | 24 | 25M | $\times$ 4.0 | 79.7 ${}^{\dagger}$ |
| PD [97] | logits | BERT ${}_{BASE}$ | 6 | 66M | $\times$ 2.0 | 81.2 ${}^{\dagger}$ |
| MINILM [98] | hint | BERT ${}_{BASE}$ | 6 | 66M | $\times$ 2.0 | 80.4 ${}^{\ddagger}$ |
- ${}^{\dagger}$ The corresponding data is from [124].
- ${}^{\ddagger}$ The corresponding data is from [98]. They reported F1 for SQuAD 2.0, and accuracy for other datasets.
#### 3.2.3 KD for Transformer-Based Vision Models
Logits-based KD. While conventional KD in vision tasks typically involves using soft labels [86, 125, 126] generated by a powerful teacher network, Touvron et al. [127] makes a noteworthy discovery. Their experiments on the ImageNet benchmark reveal that employing hard labels, which are essentially the maximum of the score values predicted by the teacher model, leads to superior results. Additionally, they introduce a novel distillation token into vision Transformers (ViTs) [128]. TinyViT [27] observes that smaller ViTs can benefit from larger teacher using massive pre-training data, for example, pre-training (distilling) student on ImageNet-21k while fine-tuning student on ImageNet-1k. And they propose TinyViT to save computational memory by storing data augmentation information and logits in advance for large teacher models. Ren et al. [129] argues that student accuracy is primarily influenced by the inductive bias of the teacher models rather than their accuracy. To address this, they introduce a cross-inductive bias distillation approach, which involves distilling the student model with multiple teachers exhibiting distinct architectural inductive biases. To enhance the utilization of the inductive bias in CNNs and offer more image-friendly guidance, CSKD [130] removes the pooling operation after the final feature map and treats the features at each position as individual samples. Subsequently, these position-based features are fed into the classifier, leading to the generation of dense predictions for each corresponding position. The hard labels derived from the CNN’s dense predictions are used as target labels, and cross-entropy serves as the loss function for spatial knowledge transfer.
Hint-based KD. Chen et al. [131] introduces a two-stage DearKD framework. In the initial stage, knowledge distillation is employed from the intermediate layers of the CNN to help Transformer-based student capture the inductive biases. Subsequently, the second stage trains the student without distillation. Hao et al. [28] proposes a fine-grained manifold distillation method to fully utilize the patch-level and batch-level information inside Transformer-based architectures. Specifically, they manually select intermediate layers from teacher-student pair and distilled the student via decoupled relation maps. ViTKD et al. [132] delves into the characteristics of feature maps in ViTs and formulated three practical guidelines for hint-based distillation in ViTs, namely: (i) generating hints is more effective than mimicking in deeper layers, (ii) distillation on shallower layers remains suitable for ViTs distilled through mimicking, and (iii) features from the feed-forward network (FFN) are better than those from the multi-head attention (MHA) for the distillation process.
#### 3.2.4 Discussion
Transformer-based language models have shown strong performance on several NLP tasks including text classification, machine translation, speech recognition, question-answering, dialogue systems, etc. Knowledge distillation is a common and practical technique used to reduce the high computational resource demand of LLMs. Different models, such as T5 [133], BERT [91], LLAMA [1], GPT-3 [4], and GPT-4 [2], exhibit varying performance on different NLP downstream tasks. Therefore, selecting appropriate teachers and methods for model compression is crucial. For example, when compressing the BERT series, we compare various distillation methods in Table IV. In contrast to using only logits, hint-based KD methods can transmit richer intermediate layer information to the student, making the learning process easier for the student and yielding better results. However, performing layer-to-layer knowledge distillation sometimes requires carefully designed layer mappings between the teacher and student models. In certain domains where generative models excel, such as reasoning and language understanding, the effectiveness of strong teacher models can only be accessed through their APIs. The exploration of guiding LLMs to generate better outputs for distillation is still in its early stages.
One of the strengths of ViT models is their ability to scale to high parametric complexity, but this requires substantial computational resources and comes at a high cost. KD can be used to transfer knowledge into more compact student models, but there are still challenges in the context of vision that require further research. The first challenge is related to training costs. Both logits-based and hint-based KD methods require large GPU memory for the distillation process. Vision tasks differ from NLP as the input for vision is in the form of images, which are significantly larger in size compared to the limited sequence length used in NLP tasks. During distillation, even though the teacher network doesn’t require backward propagation, each forward pass consumes a significant amount of GPU memory due to the activations of intermediate features. Therefore, finding ways to reduce training costs and shorten training times is an area that needs exploration. Additionally, choosing the most suitable teacher model for Transformer-based students is an open question. For instance, in the context of ViTs, determining whether a CNN or a Transformer teacher is more effective for the student model is a potential research avenue. Moreover, as proposed by Hao et al. [126], there is a necessity for designing and evaluating KD approaches within practical scenarios, moving away from the limitations of small-scale datasets. This ensures that KD methods are applicable and effective in real-world, large-scale settings.
## 4 Architecture Adaptive Compression
### 4.1 Pruning
#### 4.1.1 Overview of Pruning
Neural network pruning has long been recognized as an effective method for making models more compact and speeding up model inference. The taxonomy of pruning methods can be quite complicated, including the sequential order of pruning and model training, structure specification, and the way to determine the pruned parameters [134].This study, however, limits the scope of the source model as a pre-trained large Transformer for natural language processing [135, 4] and for visual recognition [12, 26, 136], raising several specific categories of techniques to be addressed (Fig. 5).
As pre-training accounts for most of the model performance for downstream tasks, pruning after training (PAT) [137] becomes a major choice. From a high-level perspective, the whole process consists of pre-training, pruning, and retraining performance recovery. However, the multi-task nature of pre-training and huge computational in the retraining phase also lead to critical issues to be addressed for pruning large Transformer models, as detailed later.
The methods available for pruning are generally split into two categories according to the structure specification of pruning: unstructured pruning and structured pruning. The former conducts pruning at the finest-grained level [138, 139], i.e., weight-wise pruning, which follows the optimization problem below,
$$
\mathop{\min}_{\mathbf{\theta}}L(\mathbf{\theta};D),s.t.\|\mathbf{\theta}\|_{0
}\leq k \tag{9}
$$
where $L$ is the general loss function on the dataset $D$ , $\mathbf{\theta}$ denotes the model parameter, and $k$ denotes the targeted non-zero weight number.
Although unstructured pruning generally leads to a reduction in the parameter size or memory usage, it cannot guarantee latency speedups because the resulting model shape tends to be irregular, and such speedups requires specific hardware design. In contrast, specifying the eliminated structure as an entire layer, head, or other network unit results in structural pruning [140], which can generally shorten the latency on standard hardware. When the specification is a pattern suitable for speeding-up (e.g., a specific ratio of non-zero mask), the pruning is called semi-structural pruning [141].
In addition to network parameters, reducing the input size is also an appealing direction to explore. With the emphasis on removing redundant information between tokens, such reduction also achieves lower computation FLOPs with unpruned network parameters, as discovered both in computer vision [33] and language domain [142] Transformers, .
<details>
<summary>x12.png Details</summary>
### Visual Description
## Hierarchical Diagram: Pruning Method for Large Transformer-Based Model
### Overview
The image displays a hierarchical taxonomy chart titled "Pruning Method for Large Transformer-Based Model." It organizes pruning techniques into two primary application domains, which are further subdivided into categories and specific methods. The diagram uses a top-down tree structure with color-coded boxes to denote different levels of the hierarchy.
### Components/Axes
The diagram is structured as a tree with the following levels and elements:
1. **Root Node (Top Center):** A blue rectangular box containing the main title: **"Pruning Method for Large Transformer-Based Model"**.
2. **First-Level Branches:** The root splits into two main domain categories:
* **Left Branch (Green Box):** **"Computer Vision"**
* **Right Branch (Pink Box):** **"Natural Language Processing"**
3. **Second-Level Branches (Sub-categories):** Each domain branches into specific pruning categories.
* Under **Computer Vision**:
* **Left (Purple Box):** **"Unstructured"**
* **Center (Yellow Box):** **"Structured"**
* **Right (Grey Box):** **"Other"**
* Under **Natural Language Processing**:
* **Left (Purple Box):** **"Unstructured"**
* **Center-Left (Yellow Box):** **"Structured"**
* **Center-Right (Light Orange Box):** **"Context"**
* **Right (Grey Box):** **"Other"**
4. **Third-Level Details (Methods/Examples):** Below each sub-category box, a brief description and a bulleted list of specific methods or examples are provided in plain text.
### Detailed Analysis
The complete textual content and structure are transcribed below:
**Root:**
* **Pruning Method for Large Transformer-Based Model**
**Branch 1: Computer Vision**
* **Sub-category 1: Unstructured**
* Description: Feature dimensions
* Methods:
* DynamicViT
* PatchSliming
* **Sub-category 2: Structured**
* Description: Network Modules
* Methods:
* ViT-Slim
* X-pruner
* **Sub-category 3: Other**
* Description: Parameter Redistribution
* Methods:
* NViT
**Branch 2: Natural Language Processing**
* **Sub-category 1: Unstructured**
* Description: Individual Parameter
* Methods:
* SparseGPT
* **Sub-category 2: Structured**
* Description: Network Modules
* Methods:
* Sheared LLama
* LLM Pruner
* **Sub-category 3: Context**
* Description: Input Text
* Methods:
* Dynamic Context Pruning
* **Sub-category 4: Other**
* Description: Parameter Efficient
* Methods:
* LoRAPrune
### Key Observations
1. **Symmetry and Divergence:** The taxonomy is symmetric at the first level (two domains) and partially symmetric at the second level (both have "Unstructured," "Structured," and "Other"). However, the NLP branch includes an additional, unique category: **"Context"**.
2. **Color Coding:** A consistent color scheme is used for parallel categories across domains:
* Purple for "Unstructured"
* Yellow for "Structured"
* Grey for "Other"
* Light Orange is used exclusively for the NLP-specific "Context" category.
3. **Method Specificity:** The listed methods (e.g., DynamicViT, SparseGPT, Sheared LLama) are specific, named techniques, indicating this is a practical taxonomy of known algorithms rather than purely theoretical categories.
4. **Descriptive Labels:** Each sub-category is accompanied by a short descriptor (e.g., "Feature dimensions," "Network Modules," "Input Text") that clarifies the *target* or *mechanism* of the pruning approach within that category.
### Interpretation
This diagram serves as a conceptual map for navigating the landscape of model compression techniques for large transformers. It demonstrates that pruning strategies are not monolithic but are specialized based on the model's primary domain (vision vs. language) and the technical approach (unstructured, structured, etc.).
The inclusion of a "Context" category solely under NLP is a significant insight. It highlights a pruning paradigm unique to language models, where the sparsity is induced dynamically based on the input text itself, rather than being a static property of the model's weights or architecture. This contrasts with Computer Vision pruning, which appears more focused on spatial dimensions (patches) and network modules.
The taxonomy suggests that while the core goal—reducing model size and computation—is shared, the implementation details and research frontiers diverge between CV and NLP. The "Other" categories in both domains indicate there are emerging or hybrid techniques that don't fit neatly into the primary unstructured/structured dichotomy, such as parameter redistribution (NViT) or parameter-efficient methods (LoRAPrune). Overall, the chart provides a structured framework for understanding, comparing, and selecting pruning methods based on the application domain and desired pruning granularity.
</details>
Figure 5: The taxonomy of pruning methods used for Transformer models.
#### 4.1.2 Pruning for Transformer-Based Large Language Models
Pruning granularity. In response to the considerable parameter increase in the LLM era, there have been early attempts of pruning arise with the focus on unstructured and semi-structured pruning [143, 144, 145]. SparesGPT, for instance, conducts unstructured and semi-structured (mainly 2:4) pruning starting from OPT-175B [146] and BLOOM-176B [147] and achieve 50-60% sparsity with moderate perplexity increase and performance decrease on downstream datasets. This work demonstrates the feasibility of low-resource pruning in $>$ 100 B models with moderate sparsity.
Another line of work mainly focuses on the structured pruning method to achieve a more significant speed-up. Language structural pruning has been conducted at multiple granularities. For instance, Michel et al. [148] discuss the redundancy of attention head, Fan et al. [149] propose a task-specific extraction of pre-trained sub-networks, Santacroce et al. [150] focus on FFN layers, and Block Pruning [151] is introduced to prune MHA and FFN parameters separately. In comparison, CoFi [152] proposes a more comprehensive coarse- and fine-grained pruning method that learns to generate masks at different granularity, i.e., FFN layers, FFN intermediate dimensions, MHA layers, Attention heads, and hidden dimensions to encourage the optimization in the high sparsity domain. In this way, the pruning of a specific parameter can be determined at different granularity in an end-to-end learnable manner. When pruning the general domain pre-trained BERT model to typical downstream datasets, CoFi achieves more than $10\times$ speed-ups with a limited accuracy gap. Later, this method was extended to a larger decoder-based LLM with a 7B parameter size [30]. With a pre-defined model shape, the pruned model shows better instruction tuning performance than models of similar size but trained from scratch. Also, specifying the model configuration with uniform size avoids irregularities in model shape and further increases the inference throughput. In contrast to pre-defining the model shape, Ma et al. [29] propose first to identify groups of coupled structures considering the interdependency of parameters in FFN, MHA, and channel-wise groups and conduct the pruning process by group. Experiments at a relatively low compression rate (20%) demonstrate the varying effectiveness of these grouping strategies.
Pruning criteria. To determine the structure to be pruned, various metrics have been explored in the context of LLM pruning. For instance, Frantar et al. [143] follows [153] to find an optimal mask that can be used for weight reconstruction and uses Hessian Matrix (i.e., the second order indicator of loss change concerning the change of each parameter) to indicate the parameter importance, which is experimentally shown better than a magnitude criteria [154] and much faster than previous method AdaPrune [155]. In comparison, as noted by LLM Pruner [29], the first-order term should also be considered due to the shift of data used in the pruning process compared to the original language model training. Also, it proposes considering the weight importance at different levels and explores different ways of combining the importance information. Besides, Sun et al. [144] also explores the combination of magnitude and norm of the input activations as the pruning criteria, which lowers the computational cost used for Hessian computation.
Learning to prune. In addition to determining the importance of parameters using, one can also perform extra training, commonly with regularization on sparsity [156, 157]. In particular, the success of LLM as a multi-task language processor promotes the development of a pruning method that preserves this multi-task nature, as opposed to the task-specific trails that are more feasible in the traditional pretrain-finetune paradigm [152]. In response to this motivation, recent works such as SIMPLE [158], LLM Pruner [29] and Sheared LLama [30] incorporate the causal language modeling loss in the pruning objective. However, this “general pruning” strategy raises another issue of data selection in the pruning process, as the domain gap between pre-training and pruning may hinder the importance of estimation/performance recovery. As detailed in [30], a dynamic batch loading strategy is applied to adaptively input the data batch from a proper domain to balance the model performance as a multi-task learner.
Computation cost for parameter pruning. Considering the huge numbers of parameters, post-pruning retraining and parameter importance estimation may both generate a significant computational burden for large language models. In an early attempt, Frantar et al. [143] consider a one-shot strategy, i.e., only limited data are used to calibrate the Hessian Matrix in importance estimation, and no retraining was employed. From another perspective, LoRAPrune [159] shows that introducing parameter-efficient tuning such as Low-Rank Adaption can also decrease the computation cost in retraining while compensating for the performance degradation.
Context and token pruning. Different from the case in the computer vision domain, language sequence can be very long, and long-range reference or induction also calls for the expansion of input tokens (i.e., context) to prompt better language understanding. Also, since the inference cost of LLM is in quadratic complexity with respect to sequence length, this expansion greatly hinders the efficient inference of language models. To tackle these problems, methods have been proposed since BERT era to prune redundancy in context or context attention calculation. For context pruning, Kim et al. [160] propose to prune tokens across the Transformer layers progressively and improves the throughput by several folds on GLUE benchmark with less than 1% accuracy loss. Among the second line of research, various sparse attention techniques have been proposed, including local attention restricted to nearby tokens [161, 162, 163], introducing global tokens [164, 162], content-based token grouping [37, 163], and attention back tracking [165]. With the advance of the modeling capacity of LLM, the context input for LLM is also growing readily. To provide a more adaptive attention selection, Anagnostidis et al. [31] proposes a sparse sigmoid-based selection scheme. Also, erasing tokens from the key-value cache makes this method more hardware-friendly in a decoder-based language model. While the experiments are mainly conducted on modest size GPT-2 model [3], the empirical results that up to 80% of the context can be successfully pruned with negligible degradation in perplexity indicates an appealing potential to explore along this direction.
#### 4.1.3 Pruning for Transformer-Based Vision Model
TABLE V: comparison of representative pruning methods on vision Transformer DeiT-Base. The Speed-up ratio is calculated using the latency or throughput reported in the original paper. Top-1 Acc. is measured on ImageNet 1K. “/” indicates the keep ratio of dynamic tokens, and “-” denotes the pruning ratio of parameters.
| Models | Pruning Type | $\#$ Params | FLOPs | Speed Up | Top-1 Acc. (%) |
| --- | --- | --- | --- | --- | --- |
| DeiT-B | - | 86.6M | 17.6G | 1 $\times$ | 81.8% |
| DPS-DeiT-B [33] | Unstructured (Token) | 87M | 9.4G | 1.4 $\times$ | 81.5% |
| DynamicVIT-B/08 [166] | Unstructured (Token) | - | 13.3G | 1.2 $\times$ | 81.6% |
| S ${}^{2}$ ViTE-B [167] | Un- and Structured | 56.8M | 11.7G | 1.33 $\times$ | 82.2% |
| SAViT-50 [168] | Structured | 42.6M | 8.8G | 1.55 $\times$ | 82.5% |
| SAViT-70 [168] | Structured | 25.4M | 5.3G | 2.05 $\times$ | 81.7% |
| ViT-Slim [32] | Structured | 52.6M | 10.6G | - | 82.4% |
| WD-prining [169] | Structured and Early-Ex | 55.3M | 9.9G | 1.18 $\times$ | 80.76% |
| X-pruner [34] | Structured | 87M ${}^{\dagger}$ | 8.5G | - | 81.02% |
- ${}^{\dagger}$ The corresponding data is from [170].
TABLE VI: comparison of representative pruning methods on large language Transformer.
| Method | Pruning Type | Source Model | Compression | Speed | Comparison with |
| --- | --- | --- | --- | --- | --- |
| Rate | Up | Source Model | | | |
| SparseGPT [143] ${}^{\dagger}$ | Un- and semi-sturctured | OPT-175B [146] | 50-60% | 1.54-1.79 $\times$ | 8.38/8.34 |
| (WikiText2 ppl) | | | | | |
| Sheared LLama [30] | Structured | LLaMA2-7B [1] | 61.4% | - | 56.7%/64.6% |
| (11 datasets) | | | | | |
| LLM Pruner [30] | Structured | LLaMA2-7B [1] | 20% | 1.18 $\times$ | 60.07%/68.59% |
| (7 datasets) | | | | | |
| Dyna. Context Pruning [31] | Context | GPT-2 [3] | 80.35% ${}^{\star}$ | 1.2 $\times$ | +0.085 ppl |
| (Wiki and bookcorpus) | | | | | |
- ${}^{\dagger}$ Data is from 2:4 partial sparsity setting with GPU speedup [143].
- ${}^{\star}$ Compression rate refers to context tokens instead of parameters.
Token and feature pruning. The early exploration of pruning for vision Transformer pruning focuses on reducing information redundancy in feature space. In a preliminary study, Zhu et al. [171] propose learning the most informative feature dimensions and neglecting the remaining dimensions in the inference time. This is achieved by learning a diagonal real-value feature mask during pre-training with a sparsity regularizer and obtaining the hard feature mask using a defined threshold. From a different perspective, given the fact that token features in late ViT layers are quite similar in feature embedding, patch slimming [33] method is proposed to prune the patches from the output layer to the input layer. In particular, for each layer, it evaluates the impact of a patch on the final output feature and removes less important ones. Besides determining the patch importance by training dataset statistics, token sparsification can also be achieved by inserting a light-weight prediction network to indicate the importance of the input patches [33, 166, 172, 173, 167]. Note that since the self-attention can accept the token sequence regardless of its length, unstructured pruning of input tokens can be easily implemented in a hardware-friend manner for these token pruning methods. Furthermore, similar techniques are adopted in other Transformer-based vision models such as DETR [174] to promote focus on object tokens [175, 176].
Structured pruning. Apart from the redundancy in feature dimension and tokens, the heavy burden of matrix computation is another important axis to improve model efficiency. In an early work, [167] proposes to selectively optimize for sparsity by obtaining unstructured subnetworks and structured pruning for attention heads, besides importance-based token selection. [169] develops a structural pruning method to reduce the network’s width by pruning linear projection matrices as well as attention heads, together with depth pruning with inserted early-exit classifiers at each layer. Later on, more works emerged to explore the collaborative effect of parameter pruning across different Transformer modules. In [177], a Hessian-based criteria is proposed to evaluate the parameter importance globally over learnable parameters. Together with the parameter redistribution strategy that promotes shorter latency, this method can also introduce novel efficient ViT structures named NViT. With an emphasis on speeding up continuous structure search, Chavan et al. [32] perform a single-shot architecture search for all the ViT components with significantly fewer GPU hours needed. Also, by incorporating the interactions between the components, Zheng et al. [168] manage to introduce adaptively the pruning ratio for each component, adding another degree of freedom in pruning. More recently, Yu et al. [34] propose to use class labels to optimize the fully differentiable pruning operation at each layer.
#### 4.1.4 Discussion
Pruning serves as a fundamental way to reduce the computation burden of the pre-trained Transformer. Traditionally, this is achieved in an orthogonal manner by pruning input features or network parameters. In particular, we notice that different pruning methods can biasedly benefit particular application purposes. For large language models, structured network pruning fits more in the setting when a modest length (e.g., several thousand tokens) is used as input to speed up inference, as the defined hardware-friendly model shape has been proven to realize a good trade-off between acceleration and perplexity decrease [30]. In comparison, context pruning or sparse attention methods deserve more attention when context length gets longer, and the ceiling of pruning ratio for context for LLMs (e.g., 7B or more) has not been determined in a detailed manner, which is an interesting direction to explore. For vision Transformers, the development of pruning methods has systematically tackled each angle of model design, making the current pruning process more flexible and closer to a continuous structural search task, which can be an excellent reference to the development of language domains. The training cost is another critical axis to consider in the pruning of large models, especially when incremental LLM pre-training is considered a component of the pruning process. Therefore, developing training-efficient pruning methods is attracting more attention. This requires the parameter (or block) sensitivity to be estimated more accurately using limited data or the information stored in hidden features during training/inference to be explored more in-depth.
### 4.2 Efficient Architecture
Transformer [11] has become the de facto backbone architecture to develop various large vision and language models (LVMs and LLMs), making it possible to scale to billions of parameters. In this section, we review the efficient architecture design of large Transformer-based models in both vision and language domains, i.e., efficient design on mainstream architecture, attention mechanism and feed forward network. Notably, here we also discuss the recently proposed Mamba [178], a general sequence model backbone that integrates selective state space models (SSMs) into a simplified end-to-end neural network architecture. In this section, we consider it as an efficient design for subquadratic-time architectures.
#### 4.2.1 Representative Transformer-Based Models
Table VII presents the model cards of several representative LLMs and LVMs with public details. Vanilla Transformer [11] is based on the encoder-decoder architecture [133]. In this architecture, the encoder stacks multi-head self-attention layers to encode the input sequence into latent representations, while the decoder performs cross-attention on these representations and generates target sequences in an autoregressive manner. BERT [91] is designed to pre-train deep bidirectional representations from unlabeled text by considering both left and right context in all layers. GPT-series models [4, 2] have effectively demonstrated the power of in-context learning using decoder-only architectures [147, 179, 180]. These decoder architectures incorporate unidirectional attention masks to ensure that each input token can only attend to previous tokens, including itself. The input and output tokens are processed in a similar manner within the decoder. In the following, we will provide separate reviews from the perspectives of CV and NLP on the current structural innovations in Transformer-based architectures.
#### 4.2.2 Language Domain
The self-attention mechanism in conventional Transformer architectures often faces quadratic computational complexity. This poses a challenge for training and making inferences with long input sequences. To enhance efficiency, current innovations in the mainstream can be classified into three categories: optimizations targeting the attention mechanism, the direct replacement of attention with more efficient architectures, and enhancements focused on the FFN component.
Attention innovation. Various efficient Transformer variants have been proposed to reduce the computational complexity of the attention mechanism by incorporating structural priors on attention, such as sparsity [181, 182, 183, 184, 185, 186]. For example, Reformer [37] approximates full attention computation using locality-sensitive hashing, reducing the complexity from $\mathcal{O}(N^{2})$ to $\mathcal{O}(N\log N)$ . Other techniques include pooling-based compression [185], clustering methods [184, 187] that apply k-means clustering to learn dynamic sparse attention regions, and Longformer [188], which combines local windowed attention with task-motivated global attention. Chelba et al. [189] proposes truncating the target-side window based on an $N$ -gram assumption, restricting the self-attention mechanism to use only the previous $N-1$ tokens. Additionally, locally banded sparse attention methods, such as Factorized Attention [183], have been adopted in models like GPT-3 [4].
Other methods have been proposed to reduce the computational complexity of the attention mechanism using linear attention. Performer [190] introduces fast attention via positive orthogonal random features, which is a scalable kernel method for approximating softmax attention kernels. This approach can efficiently model kernelizable attention mechanisms beyond softmax and comes with strong theoretical guarantees, including nearly unbiased estimation of the attention matrix, uniform convergence, and low estimation variance. Linear Transformer [191] reformulates self-attention as a linear dot-product of kernel feature maps. It uses kernels $\frac{\phi(q_{i})\phi(k_{j})}{\sum_{n=1}^{|x|}\phi(q_{i})\phi(k_{n})}$ to replace the softmax function and takes advantage of the associativity property of matrix products to reduce the computational complexity from $\mathcal{O}(N^{2})$ to $\mathcal{O}(N)$ .
Moreover, there have been studies [35, 192, 193] exploring the concept of multi-query attention, where different heads share the same linear transformation matrices for the keys and values, as proposed by Shazeer et al. [193]. This approach offers significant computational savings with only a minimal impact on model quality. To further strike a balance between multi-query attention and multi-head attention, GQA et al. [194] introduces the concept of grouped attention heads, where heads within the same group share identical transformation matrices.
TABLE VII: Model cards of several representative Transformer-based LLMs and LVMs with public configuration details. Here $d_{model}$ is already expanded by expansion ratio.
| Models | Arch. | $\#$ Params | Layer | Head | $d_{model}$ |
| --- | --- | --- | --- | --- | --- |
| Natural Language Processing | | | | | |
| BERT [91] | Encoder | 0.1B-0.3B | 12-24 | 12-16 | 3072-4096 |
| PaLM [35] | Decoder | 9B-540B | 32-118 | 16-48 | 4096-18432 |
| Gopher [179] | Decoder | 4M-280B | 8-80 | 16-128 | 512-16384 |
| BLOOM [147] | Decoder | 0.6B-176B | 24-70 | 16-112 | 1024-14336 |
| GPT3 [4] | Decoder | 175B | 96 | 96 | 12288 |
| GLM-130B [195] | Decoder | 130B | 70 | 96 | 12288 |
| Chinchilla [196] | Decoder | 70B | 80 | 64 | 8192 |
| LLaMA [1] | Decoder | 7B-65B | 32-80 | 32-64 | 4096-8192 |
| Falcon [180] | Decoder | 40B | 60 | 64 | 8192 |
| T5 [133] | En.-De. | 11B | 24 | 128 | 1024 |
| Computer Vision | | | | | |
| ViT [128] | Encoder | 86M-0.6B | 12-32 | 12-16 | 3072-5120 |
| DeiT [127] | Encoder | 5M-86M | 12 | 3-12 | 768-3072 |
| TNT [197] | Encoder | 6M-66M | 12 | 3-10 | 768-2560 |
| PVT [198] | Pyramid | 13M-61M | 12-45 | 8 | 2048 |
| CMT [199] | Pyramid | 10M-75M | 24-40 | 8 | 1324-2432 |
| Swin [38] | Pyramid | 29M-0.2B | 16-28 | 24-48 | 3072-6144 |
FlashAttention [200] offers an innovative approach to optimizing the speed and memory utilization of attention modules on GPUs. It focuses on an Input/Output (IO)-aware perspective, making better use of the fast memory SRAM by dividing the input into blocks. This optimization technique has already been integrated into various platforms [201, 202, 203, 204, 205] developed for training LLMs.
Non-Transformer architecture. Recent research has focused on developing new architectures for language modeling, including parameterized state space models [206, 207, 208, 209], long convolutions [210], and the incorporation of recurrent models [211, 212, 36, 213], which come with strong inductive biases. In particular, Gu et al. [206] introduces the S4 model, based on a novel parameterization for the SSM [214, 215], which involves conditioning matrix A with a low-rank correction, making it stably diagonalizable. This reduces the SSM to the well-studied computation of a Cauchy kernel [216]. The GSS model [207] further extends the use of gating units to the state space model family, and it is observed that employing gating units leads to dimensionality reduction when performing FFT operations, which addresses a major bottleneck in training speed. Hyena [210] trains a recurrence of gating units and implicitly parametrized long convolutions, which serves as an attention-free drop-in replacement for the traditional Transformer architecture. Ma et al. [213] introduces MEGA, a single-head gated attention mechanism enhanced with exponential moving average. This mechanism aims to integrate the inductive bias of position-aware local dependencies into the position-agnostic attention mechanism.
Peng et al. [217] proposes RWKV based on the fact that Recurrent Neural Networks (RNNs) show linear scaling in memory and computational requirements but encounter limitations in parallelization and scalability. They leverage a linear attention mechanism, specifically replacing the quadratic QK attention with a scalar formulation that has linear cost. Additionally, they redefine recurrence and sequential inductive biases to parallelize computations during training while maintaining constant computational and memory complexity during inference.
<details>
<summary>x13.png Details</summary>
### Visual Description
## Diagram: Modified Attention Mechanism with Group Normalization
### Overview
The image displays a computational graph or data flow diagram illustrating a modified attention mechanism, likely from a neural network architecture. The diagram shows the transformation of an input tensor `X` through a series of operations to produce an output tensor `O`. The core operation appears to be a variant of scaled dot-product attention, incorporating an additional element-wise multiplication with a term `D` and a subsequent Group Normalization (GN) step.
### Components/Axes
The diagram is structured vertically, with data flowing from the bottom to the top, indicated by black arrows.
**1. Input Layer (Bottom):**
* **Component:** A single input labeled `X`.
* **Position:** Centered at the bottom of the diagram.
* **Function:** The source tensor that is fed into the system.
**2. Linear Projection Layer (Middle-Bottom):**
* **Components:** Three parallel branches originating from `X`.
* **Labels:** `Q`, `K`, `V`.
* **Position:** `Q` is on the left, `K` is in the center, and `V` is on the right. They are horizontally aligned above `X`.
* **Function:** Represents the linear projections of the input `X` into Query (`Q`), Key (`K`), and Value (`V`) matrices, a standard step in attention mechanisms.
**3. Core Attention Operation (Center):**
* **Component:** A mathematical expression: `(QKᵀ ⊙ D)V`.
* **Position:** Centered above the `Q`, `K`, `V` layer.
* **Breakdown of the Expression:**
* `QKᵀ`: Matrix multiplication of `Q` and the transpose of `K`.
* `⊙`: The Hadamard (element-wise) product symbol.
* `D`: A matrix or tensor that is multiplied element-wise with the result of `QKᵀ`.
* `( ... )V`: The result of the element-wise product is then matrix-multiplied with `V`.
* **Function:** This is the central computation. It modifies the standard attention score calculation (`QKᵀ`) by incorporating an additional term `D` via element-wise multiplication before applying it to the values `V`.
**4. Normalization Layer (Upper-Middle):**
* **Component:** A blue, rounded rectangular block labeled `GN`.
* **Position:** Centered above the core attention operation.
* **Function:** `GN` most commonly stands for **Group Normalization**. This layer normalizes the output of the attention operation, which can help stabilize training.
**5. Output Layer (Top):**
* **Component:** A single output labeled `O`.
* **Position:** Centered at the very top of the diagram.
* **Function:** The final output tensor of this computational block.
### Detailed Analysis
The diagram defines a precise sequence of tensor operations:
1. An input tensor `X` is projected into three tensors: `Q`, `K`, and `V`.
2. The attention scores are computed as the matrix product `QKᵀ`.
3. These scores are modified by an element-wise multiplication with a tensor `D`. The nature of `D` (learnable parameter, mask, bias, etc.) is not specified in the diagram.
4. The modified scores are applied to the value tensor `V` via matrix multiplication.
5. The resulting tensor undergoes Group Normalization (`GN`).
6. The normalized tensor is the final output `O`.
The flow is strictly feedforward and sequential from `X` to `O`, with the only parallelism occurring in the initial projection to `Q`, `K`, and `V`.
### Key Observations
* **Non-Standard Attention:** The inclusion of the element-wise product with `D` (`⊙ D`) is a key deviation from the vanilla scaled dot-product attention formula (which is typically `softmax(QKᵀ/√d_k)V`).
* **Explicit Normalization:** The diagram explicitly includes a normalization step (`GN`) after the attention computation, which is not always depicted in standard attention diagrams but is a common practical component.
* **Abstract Notation:** The diagram uses abstract mathematical notation (`Q`, `K`, `V`, `D`) without specifying dimensions, activation functions, or scaling factors (like the typical `1/√d_k`).
* **Clear Data Flow:** The arrows provide an unambiguous representation of the data dependency and order of operations.
### Interpretation
This diagram represents a **customized attention module** for a neural network, likely a Transformer variant. The core innovation highlighted is the modulation of the attention score matrix (`QKᵀ`) by an additional term `D` before the value aggregation.
**What the data suggests:**
* The term `D` could serve multiple purposes: it might be a **learnable bias** added to the attention scores, a **mask** (e.g., for causality or relative position encoding), or a **gating mechanism** to dynamically weight attention patterns.
* The application of Group Normalization (`GN`) suggests this module is designed for scenarios where batch normalization is less suitable (e.g., with small batch sizes or in generative models), aiming to improve training stability and convergence.
**How elements relate:**
The architecture follows the classic "project, attend, aggregate" pattern of attention but inserts a novel, element-wise interaction (`⊙ D`) between the score computation and value aggregation. This creates a point of potential architectural innovation where the model can learn to suppress or enhance specific attention relationships. The final `GN` layer acts as a stabilizer for the output of this potentially non-linear interaction.
**Notable implications:**
This module would allow a network to learn more complex relationships than standard attention, as `D` provides an extra degree of freedom to condition the attention scores. The diagram serves as a blueprint for implementing this specific layer, clearly defining the required operations and their order. The absence of a softmax operation in the diagram is notable; it may be implied within the `QKᵀ` step or omitted for simplicity, though its absence would be highly unusual for an attention mechanism.
</details>
(a) Parallel representation.
<details>
<summary>x14.png Details</summary>
### Visual Description
## Diagram: Recurrent Attention Unit Block Diagram
### Overview
The image displays a technical block diagram of a recurrent neural network component, specifically illustrating a unit that processes an input \(X_n\) and a previous recurrent state \(S_{n-1}\) to produce an updated state \(S_n\) and an output \(O_n\). The diagram uses standard mathematical notation and flow arrows to depict data flow and operations within a defined computational block (light blue rectangle).
### Components/Axes
The diagram is composed of labeled signals, operational nodes, and a processing block. There are no traditional chart axes. The key components are:
**Inputs:**
* **\(S_{n-1}\)**: Labeled "Recurrent State". Positioned at the top-left, entering the main block from the left.
* **\(X_n\)**: Labeled "Input". Positioned at the bottom-center, entering the main block from below.
**Internal Signals (within the light blue block):**
* **\(V_n\)**: Derived from \(X_n\). Positioned in the lower-left quadrant of the block.
* **\(K_n\)**: Derived from \(X_n\). Positioned in the lower-center of the block.
* **\(Q_n\)**: Derived from \(X_n\). Positioned in the lower-right quadrant of the block.
* **\(\gamma\)**: A parameter or scaling factor. Positioned in the top-left area of the block, above the main data path.
**Outputs:**
* **\(S_n\)**: The updated recurrent state. Positioned at the top-right, exiting the main block to the right.
* **\(O_n\)**: Labeled "Output". Positioned at the far right, exiting a separate blue block labeled "GN".
**Operational Nodes:**
* **Addition Node (Circle with +)**: Located on the top data path inside the main block. It receives \(S_{n-1}\) (scaled by \(\gamma\)) and the output of a multiplication node.
* **Multiplication Node 1 (Circle with ×)**: Located in the center-left of the main block. It receives \(V_n\) and \(K_n\) as inputs. Its output feeds into the addition node.
* **Multiplication Node 2 (Circle with ×)**: Located in the center-right of the main block. It receives \(Q_n\) and the updated state signal (post-addition) as inputs. Its output feeds into the "GN" block.
* **GN Block**: A rounded blue rectangle labeled "GN". Positioned to the right of the main block. It receives the output from Multiplication Node 2 and produces the final output \(O_n\).
### Detailed Analysis
**Data Flow and Operations:**
1. **State Update Path:** The previous state \(S_{n-1}\) flows horizontally from the left. It is multiplied by a factor \(\gamma\) (implied by proximity, though no explicit multiplication symbol is shown on that line). Concurrently, the input \(X_n\) is projected into three vectors: \(V_n\), \(K_n\), and \(Q_n\). \(V_n\) and \(K_n\) are combined via a multiplication operation (likely a dot-product or element-wise product). The result of this \(V_n \times K_n\) operation is then added to the scaled previous state (\(\gamma \cdot S_{n-1}\)) to form the new state \(S_n\).
2. **Output Generation Path:** The newly computed state \(S_n\) is tapped and fed into a second multiplication operation along with the query vector \(Q_n\). The result of \(Q_n \times S_n\) is passed through a block labeled "GN" (likely standing for "Group Normalization" or "Gain Normalization") to produce the final output \(O_n\).
**Spatial Grounding:**
* The main computational block is a light blue rectangle occupying the center of the image.
* Inputs enter from the left (\(S_{n-1}\)) and bottom (\(X_n\)).
* Outputs exit to the right (\(S_n\) and \(O_n\)).
* The "GN" block is positioned outside and to the right of the main blue rectangle.
### Key Observations
* The diagram represents a **gated or attention-based recurrent unit**. The structure resembles a linear attention mechanism or a simplified transformer cell unrolled over time step \(n\).
* The operations suggest that the new state \(S_n\) is a function of the previous state and a key-value style interaction (\(V_n \times K_n\)) derived from the current input.
* The output \(O_n\) is generated by attending the current query \(Q_n\) to the *updated* state \(S_n\), followed by normalization.
* The parameter \(\gamma\) likely controls the contribution of the previous state, acting as a forget gate or decay factor.
* The use of distinct \(V\), \(K\), and \(Q\) projections from a single input \(X_n\) is a hallmark of attention mechanisms.
### Interpretation
This diagram illustrates the core computational logic of a recurrent layer that incorporates attention-like mechanisms. It is designed to process sequential data (\(X_n\)) while maintaining a memory state (\(S_{n-1}\)).
* **Function:** The unit performs two main functions: (1) **State Memory Update**, where it integrates new information from the input (via \(V_n\) and \(K_n\)) with its existing memory, and (2) **Contextual Output Generation**, where it produces an output based on a query (\(Q_n\)) applied to its current memory state.
* **Relationships:** The input \(X_n\) is central, being transformed into three distinct functional roles (Value, Key, Query). The state \(S_n\) serves as a dynamic memory that is both an output for the next time step and a context for generating the current output \(O_n\).
* **Notable Design:** The flow where the output \(O_n\) is derived from \(Q_n\) interacting with the *updated* state \(S_n\) (rather than the previous state \(S_{n-1}\)) is significant. This means the output for time step \(n\) immediately reflects the newly integrated information from input \(X_n\), allowing for very responsive sequence processing. The "GN" block suggests a normalization step is critical for stabilizing the output activations.
* **Potential Application:** This architecture is characteristic of modern recurrent units like **Linear Transformers**, **Recurrent Memory Networks**, or **Gated Attention Cells**, which aim to combine the parallelizability of feed-forward models with the memory capabilities of RNNs.
</details>
(b) Recurrent representation.
Figure 6: Dual form of RetNet (image from [36]).
Furthermore, Retentive Network (RetNet) [36] theoretically derives the connection between recurrence and attention and proposes the retention mechanism for sequence modeling. This mechanism supports three computation paradigms, namely parallel, recurrent, and chunkwise recurrent, achieving training parallelism, low-cost inference, and good performance simultaneously. The parallel representation (also depicted in Figure 5(a)) of retention is:
$$
\displaystyle Q=(XW_{Q})\odot\Theta, \displaystyle\quad K=(XW_{K})\odot\overline{\Theta},\quad V=XW_{V} \displaystyle\Theta_{n}=e^{in\theta}, \displaystyle\quad D_{nm}=\left\{\begin{aligned} &\gamma^{n-m},&n\geq m\\
&0,&n<m\\
\end{aligned}\right. \displaystyle\mathrm{Rete} \displaystyle\mathrm{ntion}(X)=(QK^{\intercal}\odot D)V \tag{10}
$$
where $\overline{\Theta}$ is the complex conjugate of $\Theta$ , and $D\in\mathbb{R}^{|x|\times|x|}$ combines causal masking and exponential decay along relative distance as one matrix. This parallel representation enables RetNet to train the models with GPUs efficiently. The recurrent version of retention (Figure 5(b)) is favorable for inference and can be written as (for $n$ -th timestep):
$$
\displaystyle S_{n}=\gamma S_{n-1}+K_{n}^{\intercal}V_{n} \displaystyle\mathrm{Rete}\mathrm{ntion}(X_{n})=Q_{n}S_{n},\quad n=1,\cdots,|x| \tag{11}
$$
where $Q,K,V,\gamma$ are the same as in Eq. 10.
More recently, Gu et al. [178] have highlighted the limitations of previous structured SSMs in effectively modeling discrete and information-dense data, such as text. In response, they propose Mamba, consisting a novel class of selective mechanisms and hardware-aware designs that enable linear scalability to billions of parameters, while maintaining computational efficiency and achieving strong performance. Specifically, in comparison to the H3 [208] block, the Mamba model replaces the first multiplicative gate with an activation function. Additionally, Mamba incorporates an SSM into the main branch, distinguishing it from the MLP block in transformer. Building upon the foundation of Mamba, researchers are further unlocking the potential of SSMs for scaling by exploring new techniques, such as combining SSMs with Mixture of Experts (MoE) [218] or introducing methods to enhance the flow of hidden information [219] between layers in SSMs.
FFN innovation. Large and sparse FFNs such as Mixture-of-Experts (MoE) [220, 221, 222, 223, 224, 225] have been effective in scaling up Transformer-based models for pre-training LLMs. They replace a single FFN module with multiple equally-sized modules (experts) and activate only a few experts based on the input. This selective activation of FFN improves generalization performance while maintaining fixed training and inference costs. Liu et al. [226] further finds a simpler selection method known as Avg-K, which selects blocks based on their mean aggregated hidden states. This method achieves lower perplexity in language model pre-training compared to existing MoE architectures such as Switch Transformer [222] and HashLayer [223].
Romero et al. [227] presents Continuous Kernel Convolution (CKConv) for handling arbitrarily long sequences in a parallel manner within a single operation. CKConv formulates convolutional kernels as vector-valued continuous functions parameterized by a small MLP instead of a sequence of independent weights. This MLP takes a time-step as input and generates the value of the convolutional kernel at the corresponding position, allowing the generation of kernels at different resolutions and arbitrarily sizes.
PaLM [35] and LLaMA [1] both utilize the SwiGLU Activation (Swish( $xW$ ) $\dot{x}V$ ) for original FFN intermediate activations. This choice is based on the observation that SwiGLU activations have been shown to substantially enhance quality in compute-equivalent experiments [228] when compared to standard activation functions such as ReLU, GeLU, or Swish [228]. It’s important to note that using SwiGLU does require three matrix multiplications in the MLP instead of two.
TABLE VIII: Model comparison from training parallelization (TP), inference cost (Time), and memory complexity (Memory). Here $N$ and $d$ denote the sequence length and feature dimension, respectively.
| Architectures | TP | Time | Memory |
| --- | --- | --- | --- |
| RNN [229] | ✘ | $O(Nd)$ | $O(d)$ |
| Transformer [11] | ✔ | $O(N^{2}d)$ | $O(N^{2}+Nd)$ |
| Linear Attention [191] | ✔ | $O(Nd^{2})$ | $O(Nd+d^{2})$ |
| H3 [208] | ✔ | $O(Nd(\log N+d))$ | $O(Nd)$ |
| S4 [206] | ✔ | $O(Nd^{2})$ | $O(Nd)$ |
| Hyena [210] | ✔ | $O(Nd(\log N+d))$ | $O(N\log N\cdot d)$ |
| Reformer [37] | ✔ | $O(N\log N\cdot d)$ | $O(N\log N+Nd)$ |
| Performer [190] | ✔ | $O(Nd^{2}\log d)$ | $O((Nd+d^{2})\cdot\log d)$ |
| RWKV [217] | ✔ | $O(Nd)$ | $O(d)$ |
| RetNet [36] | ✔ | $O(Nd^{2})$ | $O(d^{2})$ |
| Mamba [178] | ✔ | $O(Nd)$ | $O(d)$ |
#### 4.2.3 Vision Domain
Dosovitskiy et al. [128] first introduces the Vision Transformer (ViT) primarily designed for image classification tasks. In the ViT model, the input image is initially divided into a sequence of fixed-length tokens, which are then processed through multiple Transformer layers to capture global relationships among the tokens. Similar to the challenges faced in the language domain, the self-attention mechanism with quadratic computational complexity is one of the major hurdles for training and inference in Transformer-based architectures applied to vision tasks. Many dense prediction tasks, such as object detection and image segmentation, often require higher-resolution inputs, necessitating the manual design of efficient architectures to handle these challenges. To address these issues, novel architectures like PVT [198] and Swin [198] have been proposed. These architectures incorporate hierarchical (pyramid) structures with multiple stages, effectively overcoming the challenges of adapting the original isotropic ViT [128] to various dense prediction tasks in computer vision.
In the following, we will delve into papers that focus on improving the efficiency of ViTs, including efforts to enhance local information processing [199, 230, 231, 232, 233, 234, 235], simplify the attention mechanism [236, 237, 238, 239, 240], and explore alternative modules that can replace or work alongside attention mechanisms [241, 39, 242, 243, 244], etc.
Enhancing locality. T2T-ViT [245] leverages token transformations to reduce token length by iteratively aggregating neighboring tokens to one token. LeViT [246] and MobileViT [247] employ hybrid architectures with stacked convolution layers, efficiently reducing the number of features through the first layer. TNT [197] further divides original ViT’s 16 $\times$ 16 patches into smaller 4 $\times$ 4 patches for capturing local information. CrossViT [234] processes small-patch and large-patch tokens separately, fusing them through multiple attention mechanisms. Twins [248] alternates between local and global attention layers for improved performance. RegionViT [249] introduces regional tokens and local tokens, enhancing local context with global information. KVT [250] introduces the k-NN attention to utilize locality of images patches and ignore noisy tokens by only computing attentions with top-k similar tokens. CMT [199] applies depth-wise convolutions to augment local patterns in the attention map and intermediate activation of the FFN. Pan et al. [251] proposes HiLo attention to disentangle the high/low frequency patterns in an attention layer by separating the heads into two groups, each equipped with specialized operations that focus on local windows.
Faster attention. Parmar et al. [236] introduces restrictions on the attention mechanism to focus on local neighborhoods. Swin [38] and Cswin [252] incorporate local attention within a window and introduce a shifted window partitioning method to enable cross-window connections. Shuffle Transformer [253] and Msg-Transformer [254] employ spatial shuffle operations as alternatives to shifted window partitioning, facilitating cross-window connections. FasterViT [237] introduces hierarchical attention, breaking down global self-attention into multi-level attention components. This approach combines local window attention and hierarchical attention to achieve global information propagation while reducing computational costs. FLatten Transformer [255] integrates depth-wise convolution in conjunction with linear attention mechanisms to address the challenge of maintaining diversity in output features across different positions.
Attention-free architecture. AFT [256] pioneers an innovative approach by combining key and value elements with learned position biases, followed by element-wise multiplication with the query. This operation’s distinctive feature is its linear memory complexity concerning both context size and feature dimension, making it compatible with large input and model sizes. GFnet [257] presents an alternative by replacing traditional attention mechanisms with Fast Fourier Transform (FFT), frequency gating, and inverse FFT for rapid token mixing. Additionally, some architectures have been entirely based on pure multi-layer perceptrons (MLPs) [258, 40, 259, 260, 243, 244], without using convolutions or self-attention mechanisms. Yu et al. [39] proposes MetaFormer as a general architecture abstracted from ViTs without specifying the token mixer. By employing basic token mixing, primarily through non-parametric pooling, MetaFormer achieves satisfactory performance.
Inspired by the efficient hardware-aware designs of Mamba in NLP, Zhu et al. [261] propose a novel generic vision backbone called Vim with bidirectional Mamba blocks. Different from the sequential nature of language, representing visual data in SSMs presents challenges primarily due to their sensitivity to spatial position. To address this, Vim introduces position embeddings to mark image sequences and compresses visual representations using bidirectional SSM. Following the architectures of ViT [12] and DeiT [26], Vim initially employs a 16×16 kernel size projection layer to obtain a 1-D sequence of non-overlapping patch embeddings. Simultaneously, Liu et al. [262] propose the visual state space model (VMamba), which introduces the Cross-Scan Module (CSM) to traverse the spatial domain and convert any non-causal visual image into ordered patch sequences, thereby addressing the inherent direction-sensitive issue. Following the architecture of Swin [136], VMamba begins the process by partitioning the input image into patches using a stem module, without further flattening the patches into a 1-D sequence.
NAS. Indeed, the quest for optimizing Transformer architectures extends to the realm of neural architecture search (NAS). Various models, including Scaling-ViT [263], ViTAS [264], AutoFormer [265] and GLiT [266], have emerged as products of NAS-driven design, demonstrating the potential for more efficient and effective Transformer architectures.
#### 4.2.4 Discussion
Transformer models consistently achieve state-of-the-art results across various tasks, but the cost of training these models, especially on long sequences, can be prohibitive. Table VIII concludes several papers that propose improvements to the attention mechanism from three different dimensions. While using linear attention can help reduce the computational complexity of the attention mechanism, it struggles to effectively encode position information, leading to a potential decrease in model performance. Models like RWKV [217], RetNet [36] and Mamba [178], which generate outputs recursively similar to RNNs, present a promising direction for further exploration. These models only need to refer to the previous state during decoding, significantly improving decoding efficiency by avoiding the need to revisit all previous states, a characteristic of conventional Transformers. Additionally, they can encode entire sentences in parallel, enabling highly parallel and efficient training.
In contrast to NLP tasks, high-level vision tasks place a strong emphasis on the network’s ability to capture local details, contextual information, and multi-scale features for dense predictions [198]. Several studies [267, 39] have also demonstrated that the self-attention mechanism is not always indispensable for feature extraction in vision tasks. In the exploration of pure vision tasks, the possibility of directly omitting the attention module is a worthwhile avenue for further research. However, it’s worth noting that attention and cross-attention modules still play a crucial role in integrating visual features with other modalities [15]. Therefore, the quest for faster attention algorithms remains a valuable research direction.
## 5 Specialized Approaches
In addition to quantization, distillation, pruning, and novel network architectures, there are several other model compression and acceleration approaches including tensor decomposition, early exiting, and speculative sampling.
Tensor decomposition. Tensor or matrix decomposition aims to decompose a large tensor or matrix into smaller ones in order to reduce the number of parameters and computational costs. This approach was first introduced into the compression of fully-connected layers and convolutional networks [268, 269]. In terms of large language models, Tensor decomposition is utilized for simplifying model weight or embedding layers. Edalati et.al [270] is the first attempt to use Kronecker decomposition for compressing generative language models. This work represent weight matrix with two smaller matrices using Kronecker decomposition and achieves lossless $1.5\times$ compression of GPT2. LoRD [271] further uses low rank decomposition to compress code LLMs. And TSVD [272] compresses linear mappings in Transformers using SVD and constrain $U$ and $V$ matrices in ternary format. In addition to model weights, TensorGPT [273] proposes to compress embedding layers and achieves more than $3\times$ compression factor with lossless performance.
Early exiting. Early exiting can dynamically allocate different resources for each input sample and maintain the original performance. This technology has been successfully used in information retrieval system [274] and convolutional networks [275]. Many early exiting technologies have been proposed for encoder-only Transformers [276, 277, 278, 279]. The key problem in early exiting is determining when to exit. The existing works mainly utilize intrinsic confidence measures [276], routing in advance [280], or training a early-exit classifier [281]. For Transformers with decoders, several works have been proposed. For example, Elbayad et.al [282] introduces the depth-adaptive Transformer for accelerating machine translation. CALM [283] proposes to dynamically allocate different computational resources for different input and timesteps, achieving speedups of up to $3\times$ on T5 encoder-decoder model. Additionally, SkipDecoder [284] bypasses tokens in lower layers to middle layers in order to enable batch inference and reuse KV caching, achieving inference speedups of $2\times$ to $5\times$ on OPT models.
Speculative sampling. Speculative sampling, a special acceleration approach for Transformer decoding, computes several tokens in parallel [285, 286]. In large language models, decoding $K$ tokens requires $K$ runs of the model – a slow process. Taking the reference tokens generated from smaller models, speculative sampling runs these tokens in parallel to significantly accelerate the decoding process. Moreover, the rejection scheme [286] can preserve the distribution of the original LLM so that speculative sampling is theoretically lossless. For example, Yang et.al [287] further take the input text as reference tokens without introducing extra models. And LLMCad [288] introduces an on-device inference engine for LLMs with speculative sampling as a key technology.
Discussion. This section has reviewed the methods that differ from those mentioned earlier, covering tensor decomposition, early exiting, and speculative sampling. These methods do not change the basic operations but instead accelerate model inference by modifying the network connections. Tensor decomposition approximates the original weights with low-rank ones, potentially leading to performance degradation. Early exiting makes predictions by adaptively allocating resources on a per sample basis. And speculative sampling considers the specialization of decoders and can accelerate decoding in a lossless manner. These special methods inspire future development of new algorithms for accelerating LLMs or LVMs.
## 6 Conclusions and Future Directions
In this survey, we systematically investigate the compression methods for Transformer models. Compared with compression method for conventional models, there are unique considerations when compressing Transformer models. Unlike other architectures such as CNN or RNN, the Transformer adopts a unique architecture design with alternative attention and FFN modules. This necessitates specifically tailored compression methods for optimal compression rates. Moreover, the efficiency of compression methods becomes particularly crucial for large models. Certain model compression methods require substantial amounts of computational resources, making them potentially prohibitive for such models. This survey aims to encompass the majority of recent works pertaining to Transformers and articulate a comprehensive roadmap for their compression. Subsequently, we delve into the interconnections among various methods, address post-challenges, and outline directions for future research.
Relationship between different compression methods. Different compression methods can be used together to obtain an extremely efficient architecture. A conventional sequence is to firstly define a new architecture with efficient operations. Any redundant components (e.g., attention head, layers) are then removed to reduce the model size. For practical hardware implementation, quantizing weights or activations to lower bits is imperative. The choice of required bits depend on not only the tolerance of error, but also hardware design. As an illustration, Int8 computation is efficiently optimized on Nvidia A100 GPU, but it lacks support on the older Tesla P100 GPU. Distillation commonly serves as a training strategy that applies during the fine-tuning phases of both pruning and quantization. Combining different compression strategies is a promising direction to explore for achieving extremely high compression rates. Although detailed exploration has been conducted into conventional models like CNN [9, 289], Transformer models have more complicated architectures and higher computational costs. Finding a suitable combination strategy via joint search is a challenging task.
Training-efficient compression strategy. In contrast to compressing conventional models, there is a heightened emphasis on the computational cost of compression methods.
Different from compression for conventional models, the cost of compressing large Transformers has been put on higher priority. Large Transformers are currently trained on vast datasets using significant computational resources. For instance, Llama2 is trained on 2 trillion tokens with thousands of GPUs over several months [1]. It is impractical to fine-tune a model with comparable computational resources during pre-training, especially when the original data is often inaccessible. Therefore, the feasibility of efficient compression methods after training becomes more viable.
Initially developed for conventional small models, a series of works have extensively researched post-training quantization [21, 41, 22, 42], and these methods have seamlessly transitioned to Transformers. With only several GPU hours, some recent works GPTQ [72],and SmoothQuant [18] quantize an FP16 model to Int8 without significant performance loss. However, for lower bits (e.g., 4 bit), the quantized model still surfers significant performance degradation [72, 18]. It is worth noting that extremely-low-bit models, such as binary Transformers, have been extensively explored in conventional small models, yet they remain relatively unexplored in the context of large models.
For pruning, the challenge of post-training is intricately linked to the pruning granularity. Although nonstructural sparsity can achieve a high compression rate with minimal fine-tuning requirements [143], a similar strategy is hard to transfer to the structural pruning. Directly removing the entire attention heads or layers will result in a substantial alteration of the model’s architecture and a significant reduction in accuracy. How to recognize effective weights and how to effectively recover the performance are both important directions for exploration. The efficient strategy to identify effective weights and recovery representation ability are key research directions in addressing these challenges.
Efficient architectures beyond Transformer. In real-world applications, the input context for a Transformer architecture can be extremely long, encompassing sequence texts (e.g., a book with hundreds of thousands of words) in NLP or high-resolution images in CV. The vanilla attention mechanism exhibits quadratic complexity regarding the length of the input sequence, posing a significant computational challenge for long-sequence inputs. Numerous studies have addressed this issue by mitigating the computational cost of attention, employing techniques such as sparse attention, local attention, etc (See Section 4.2). However, these attention compression strategies often compromise the representation ability, leading to diminished performance.
Emerging architectures such as RWKV [217] and RetNet [36] adopt a recursive output generation akin to RNNs, effectively reducing computational complexity to $\mathcal{O}(N)$ . This development holds promise for further exploration in the quest for more efficient models. For computer vision tasks, even a pure MLP architecture without an attention module can achieve SOTA performance [40, 244, 260, 258, 259]. Beyond the widely used Transformer architecture, it is promising to explore new efficient architectures by carefully investigating their efficiency, generalization and scalability.
## References
- [1] H. Touvron et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
- [2] OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
- [3] A. Radford et al. Language models are unsupervised multitask learners. 2019.
- [4] T. Brown et al. Language models are few-shot learners. Advances in neural information processing systems, 2020.
- [5] W. Zeng et al. Pangu- $\alpha$ : Large-scale autoregressive pretrained chinese language models with auto-parallel computation. arXiv preprint arXiv:2104.12369, 2021.
- [6] X. Ren et al. Pangu- $\sigma$ : Towards trillion parameter language model with sparse heterogeneous computing. arXiv preprint arXiv:2303.10845, 2023.
- [7] Y. Jiang et al. Lion: Adversarial distillation of closed-source large language model. arXiv preprint arXiv:2305.12870, 2023.
- [8] S. Li et al. Explanations from large language models make small reasoners better. arXiv preprint arXiv:2210.06726, 2022.
- [9] S. Han et al. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.
- [10] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
- [11] A. Vaswani et al. Attention is all you need. Advances in neural information processing systems, 2017.
- [12] A. Dosovitskiy et al. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv, abs/2010.11929, 2020.
- [13] H. Chen et al. Pre-trained image processing transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12299–12310, 2021.
- [14] Y. Bai et al. Sequential modeling enables scalable learning for large vision models, 2023.
- [15] A. Radford et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, 2021.
- [16] J. Li et al. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp. 12888–12900. PMLR, 2022.
- [17] H. Liu et al. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023.
- [18] G. Xiao et al. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087–38099. PMLR, 2023.
- [19] W. Shao et al. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13137, 2023.
- [20] T. Dettmers et al. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023.
- [21] Z. Liu et al. Post-training quantization for vision transformer. Advances in Neural Information Processing Systems, 34:28092–28103, 2021.
- [22] Y. Lin et al. Fq-vit: Post-training quantization for fully quantized vision transformer. arXiv preprint arXiv:2111.13824, 2021.
- [23] S.-Y. Liu et al. Oscillation-free quantization for low-bit vision transformers. arXiv preprint arXiv:2302.02210, 2023.
- [24] V. Sanh et al. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
- [25] W. Wang et al. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 2020.
- [26] H. Touvron et al. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, 2020.
- [27] K. Wu et al. Tinyvit: Fast pretraining distillation for small vision transformers. In European Conference on Computer Vision, 2022.
- [28] Z. Hao et al. Learning efficient vision transformers via fine-grained manifold distillation. Advances in Neural Information Processing Systems, 2022.
- [29] X. Ma et al. Llm-pruner: On the structural pruning of large language models. ArXiv, abs/2305.11627, 2023.
- [30] M. Xia et al. Sheared llama: Accelerating language model pre-training via structured pruning. ArXiv, abs/2310.06694, 2023.
- [31] S. Anagnostidis et al. Dynamic context pruning for efficient and interpretable autoregressive transformers. ArXiv, abs/2305.15805, 2023.
- [32] A. Chavan et al. Vision transformer slimming: Multi-dimension searching in continuous optimization space. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4921–4931, 2022.
- [33] Y. Tang et al. Patch slimming for efficient vision transformers. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12155–12164, 2021.
- [34] L. Yu and W. Xiang. X-pruner: explainable pruning for vision transformers. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24355–24363, 2023.
- [35] A. Chowdhery et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
- [36] Y. Sun et al. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023.
- [37] N. Kitaev et al. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020.
- [38] Z. Liu et al. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, 2021.
- [39] W. Yu et al. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022.
- [40] I. O. Tolstikhin et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems, 2021.
- [41] Z. Yuan et al. Ptq4vit: Post-training quantization for vision transformers with twin uniform quantization. In European Conference on Computer Vision, pp. 191–207. Springer, 2022.
- [42] Y. Ding et al. Towards accurate post-training quantization for vision transformer. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 5380–5388, 2022.
- [43] Z. Lit et al. Auto-vit-acc: An fpga-aware automatic acceleration framework for vision transformer with mixed-scheme quantization. In 2022 32nd International Conference on Field-Programmable Logic and Applications (FPL), pp. 109–116. IEEE, 2022.
- [44] Y. Liu et al. Noisyquant: Noisy bias-enhanced post-training activation quantization for vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20321–20330, 2023.
- [45] Z. Li et al. Repq-vit: Scale reparameterization for post-training quantization of vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17227–17236, 2023.
- [46] Z. Wang et al. Quantformer: Learning extremely low-precision vision transformers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
- [47] Y. Li et al. Q-vit: Accurate and fully quantized low-bit vision transformer. Advances in Neural Information Processing Systems, 35:34451–34463, 2022.
- [48] Z. Li et al. Q-vit: Fully differentiable quantization for vision transformer. arXiv preprint arXiv:2201.07703, 2022.
- [49] J. Chen et al. Data-free quantization via mixed-precision compensation without fine-tuning. Pattern Recognition, pp. 109780, 2023.
- [50] Z. Li and Q. Gu. I-vit: Integer-only quantization for efficient vision transformer inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17065–17075, 2023.
- [51] N. Frumkin et al. Jumping through local minima: Quantization in the loss landscape of vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16978–16988, 2023.
- [52] J. Xiao et al. Patch-wise mixed-precision quantization of vision transformer. arXiv preprint arXiv:2305.06559, 2023.
- [53] Z. Li et al. Patch similarity aware data-free quantization for vision transformers. In European Conference on Computer Vision, pp. 154–170. Springer, 2022.
- [54] Z. Li et al. Psaq-vit v2: Toward accurate and general data-free quantization for vision transformers. IEEE Transactions on Neural Networks and Learning Systems, 2023.
- [55] S. Xu et al. Q-detr: An efficient low-bit quantized detection transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3842–3851, 2023.
- [56] X. Huang et al. Variation-aware vision transformer quantization. arXiv preprint arXiv:2307.00331, 2023.
- [57] S. Shen et al. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8815–8821, 2020.
- [58] Y. Bondarenko et al. Understanding and overcoming the challenges of efficient transformer quantization. arXiv preprint arXiv:2109.12948, 2021.
- [59] H. Bai et al. Towards efficient post-training quantization of pre-trained language models. Advances in Neural Information Processing Systems, 35:1405–1418, 2022.
- [60] O. Zafrir et al. Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS), pp. 36–39. IEEE, 2019.
- [61] H. Tang et al. Mkq-bert: Quantized bert with 4-bits weights and activations. arXiv preprint arXiv:2203.13483, 2022.
- [62] S. Kim et al. I-bert: Integer-only bert quantization. In International conference on machine learning, pp. 5506–5518. PMLR, 2021.
- [63] H. Bai et al. Binarybert: Pushing the limit of bert quantization. arXiv preprint arXiv:2012.15701, 2020.
- [64] H. Qin et al. Bibert: Accurate fully binarized bert. arXiv preprint arXiv:2203.06390, 2022.
- [65] W. Zhang et al. Ternarybert: Distillation-aware ultra-low bit bert. arXiv preprint arXiv:2009.12812, 2020.
- [66] C. Zhao et al. Automatic mixed-precision quantization search of bert. arXiv preprint arXiv:2112.14938, 2021.
- [67] L. Ouyang et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
- [68] T. Dettmers et al. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339, 2022.
- [69] G. Park et al. nuqmm: Quantized matmul for efficient inference of large-scale generative language models. arXiv preprint arXiv:2206.09557, 2022.
- [70] Z. Yao et al. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168–27183, 2022.
- [71] X. Wei et al. Outlier suppression: Pushing the limit of low-bit transformer language models. Advances in Neural Information Processing Systems, 35:17402–17414, 2022.
- [72] E. Frantar et al. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022.
- [73] J. Lin et al. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978, 2023.
- [74] X. Wei et al. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint arXiv:2304.09145, 2023.
- [75] J. Liu et al. Qllm: Accurate and efficient low-bitwidth quantization for large language models. arXiv preprint arXiv:2310.08041, 2023.
- [76] Z. Yuan et al. Rptq: Reorder-based post-training quantization for large language models. arXiv preprint arXiv:2304.01089, 2023.
- [77] X. Ding et al. Cbq: Cross-block quantization for large language models. arXiv preprint arXiv:2312.07950, 2023.
- [78] S. Kim et al. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629, 2023.
- [79] W. Cheng et al. Optimize weight rounding via signed gradient descent for the quantization of llms. arXiv preprint arXiv:2309.05516, 2023.
- [80] Z. Liu et al. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888, 2023.
- [81] J. Kim et al. Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization. arXiv preprint arXiv:2305.14152, 2023.
- [82] Z. Dong et al. Hawq: Hessian aware quantization of neural networks with mixed-precision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 293–302, 2019.
- [83] J. Ba and R. Caruana. Do deep nets really need to be deep? Advances in neural information processing systems, 2014.
- [84] C. Bucilua et al. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, 2006.
- [85] T. Wu et al. Weight-inherited distillation for task-agnostic bert compression. arXiv preprint arXiv:2305.09098, 2023.
- [86] G. Hinton et al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
- [87] T. Wu et al. Rethinking kullback-leibler divergence in knowledge distillation for large language models. arXiv preprint arXiv:2404.02657, 2024.
- [88] R. Tang et al. Distilling task-specific knowledge from bert into simple neural networks. arXiv preprint arXiv:1903.12136, 2019.
- [89] J. Ko et al. Distillm: Towards streamlined distillation for large language models. arXiv preprint arXiv:2402.03898, 2024.
- [90] A. Romero et al. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.
- [91] J. Devlin et al. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
- [92] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 1997.
- [93] M. E. Peters et al. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018.
- [94] Y. Kim and A. M. Rush. Sequence-level knowledge distillation. arXiv preprint arXiv:1606.07947, 2016.
- [95] K. J. Liang et al. Mixkd: Towards efficient distillation of large-scale language models. arXiv preprint arXiv:2011.00593, 2020.
- [96] H. Zhang et al. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.
- [97] I. Turc et al. Well-read students learn better: The impact of student initialization on knowledge distillation. arXiv preprint arXiv:1908.08962, 2019.
- [98] Y. Gu et al. Knowledge distillation of large language models. arXiv preprint arXiv:2306.08543, 2023.
- [99] R. Agarwal et al. Gkd: Generalized knowledge distillation for auto-regressive sequence models. arXiv preprint arXiv:2306.13649, 2023.
- [100] Y. Huang et al. In-context learning distillation: Transferring few-shot learning ability of pre-trained language models. arXiv preprint arXiv:2212.10670, 2022.
- [101] S. Sun et al. Patient knowledge distillation for bert model compression. arXiv preprint arXiv:1908.09355, 2019.
- [102] S. Bae et al. Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding. arXiv preprint arXiv:2310.05424, 2023.
- [103] Z. Sun et al. Mobilebert: a compact task-agnostic bert for resource-limited devices. arXiv preprint arXiv:2004.02984, 2020.
- [104] X. Jiao et al. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351, 2019.
- [105] Z. Li et al. Hint-based training for non-autoregressive machine translation. arXiv preprint arXiv:1909.06708, 2019.
- [106] S. Mukherjee and A. Awadallah. Xtremedistil: Multi-stage distillation for massive multilingual models. arXiv preprint arXiv:2004.05686, 2020.
- [107] H. Tsai et al. Small and practical bert models for sequence labeling. arXiv preprint arXiv:1909.00100, 2019.
- [108] J. Wei et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022.
- [109] R. Schaeffer et al. Are emergent abilities of large language models a mirage? arXiv preprint arXiv:2304.15004, 2023.
- [110] N. Ho et al. Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071, 2022.
- [111] C.-Y. Hsieh et al. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02301, 2023.
- [112] L. C. Magister et al. Teaching small language models to reason. arXiv preprint arXiv:2212.08410, 2022.
- [113] P. Wang et al. Scott: Self-consistent chain-of-thought distillation. arXiv preprint arXiv:2305.01879, 2023.
- [114] S. Saha et al. Can language models teach weaker agents? teacher explanations improve students via theory of mind. arXiv preprint arXiv:2306.09299, 2023.
- [115] M. Wu et al. Lamini-lm: A diverse herd of distilled models from large-scale instructions. arXiv preprint arXiv:2304.14402, 2023.
- [116] P. West et al. Symbolic knowledge distillation: from general language models to commonsense models. arXiv preprint arXiv:2110.07178, 2021.
- [117] J. Wei et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 2022.
- [118] X. Zhu et al. Pad: Program-aided distillation specializes large models in reasoning. arXiv preprint arXiv:2305.13888, 2023.
- [119] K. Shridhar et al. Distilling reasoning capabilities into smaller language models. In Findings of the Association for Computational Linguistics: ACL 2023, 2023.
- [120] Y. Fu et al. Specializing smaller language models towards multi-step reasoning. arXiv preprint arXiv:2301.12726, 2023.
- [121] C. Xu et al. Bert-of-theseus: Compressing bert by progressive module replacing. arXiv preprint arXiv:2002.02925, 2020.
- [122] A. H. Jha et al. Large language model distillation doesn’t need a teacher. arXiv preprint arXiv:2305.14864, 2023.
- [123] T. Li et al. A short study on compressing decoder-based language models. arXiv preprint arXiv:2110.08460, 2021.
- [124] X. Qiu et al. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 2020.
- [125] S. Yu et al. Unified visual transformer compression. arXiv preprint arXiv:2203.08243, 2022.
- [126] Z. Hao et al. Vanillakd: Revisit the power of vanilla knowledge distillation from small scale to large scale. arXiv preprint arXiv:2305.15781, 2023.
- [127] H. Touvron et al. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, 2021.
- [128] A. Dosovitskiy et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
- [129] S. Ren et al. Co-advise: Cross inductive bias distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, 2022.
- [130] B. Zhao et al. Cumulative spatial knowledge distillation for vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
- [131] X. Chen et al. Dearkd: data-efficient early knowledge distillation for vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
- [132] Z. Yang et al. Vitkd: Practical guidelines for vit feature knowledge distillation. arXiv preprint arXiv:2209.02432, 2022.
- [133] C. Raffel et al. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2020.
- [134] H. Cheng et al. A survey on deep neural network pruning-taxonomy, comparison, analysis, and recommendations. ArXiv, abs/2308.06767, 2023.
- [135] J. Devlin et al. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics, 2019.
- [136] Z. Liu et al. Swin transformer: Hierarchical vision transformer using shifted windows. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9992–10002, 2021.
- [137] A. Renda et al. Comparing rewinding and fine-tuning in neural network pruning. ArXiv, abs/2003.02389, 2020.
- [138] N. Lee et al. Snip: Single-shot network pruning based on connection sensitivity. ArXiv, abs/1810.02340, 2018.
- [139] T. Zhang et al. A systematic dnn weight pruning framework using alternating direction method of multipliers. In European Conference on Computer Vision, 2018.
- [140] Z. You et al. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. In Neural Information Processing Systems, 2019.
- [141] A. Zhou et al. Learning n: M fine-grained structured sparse neural networks from scratch. ArXiv, abs/2102.04010, 2021.
- [142] S. Goyal et al. Power-bert: Accelerating bert inference via progressive word-vector elimination. In International Conference on Machine Learning, 2020.
- [143] E. Frantar and D. Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. ArXiv, abs/2301.00774, 2023.
- [144] M. Sun et al. A simple and effective pruning approach for large language models. ArXiv, abs/2306.11695, 2023.
- [145] A. Syed et al. Prune and tune: Improving efficient pruning techniques for massive language models. In Tiny Papers @ ICLR, 2023.
- [146] S. Zhang et al. Opt: Open pre-trained transformer language models. ArXiv, abs/2205.01068, 2022.
- [147] T. L. Scao et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022.
- [148] P. Michel et al. Are sixteen heads really better than one? In Neural Information Processing Systems, 2019.
- [149] A. Fan et al. Reducing transformer depth on demand with structured dropout. ArXiv, abs/1909.11556, 2019.
- [150] M. Santacroce et al. What matters in the structured pruning of generative language models? ArXiv, abs/2302.03773, 2023.
- [151] F. Lagunas et al. Block pruning for faster transformers. In M.-F. Moens et al., editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10619–10629, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
- [152] M. Xia et al. Structured pruning learns compact and accurate models. In Annual Meeting of the Association for Computational Linguistics, 2022.
- [153] E. Frantar and D. Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. ArXiv, abs/2208.11580, 2022.
- [154] M. Zhu and S. Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. ArXiv, abs/1710.01878, 2017.
- [155] I. Hubara et al. Accelerated sparse neural training: A provable and efficient method to find n: M transposable masks. ArXiv, abs/2102.08124, 2021.
- [156] Y. He et al. Channel pruning for accelerating very deep neural networks. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1398–1406, 2017.
- [157] E. Voita et al. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. ArXiv, abs/1905.09418, 2019.
- [158] C. Tao et al. Structured pruning for efficient generative pre-trained language models. In Annual Meeting of the Association for Computational Linguistics, 2023.
- [159] J. E. Hu et al. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.09685, 2021.
- [160] S. Kim et al. Learned token pruning for transformers. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2021.
- [161] R. Child et al. Generating long sequences with sparse transformers. ArXiv, abs/1904.10509, 2019.
- [162] M. Zaheer et al. Big bird: Transformers for longer sequences. ArXiv, abs/2007.14062, 2020.
- [163] A. Roy et al. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53–68, 2020.
- [164] Q. Guo et al. Star-transformer. ArXiv, abs/1902.09113, 2019.
- [165] H. Lee et al. Sparse token transformer with attention back tracking. In International Conference on Learning Representations, 2023.
- [166] Y. Rao et al. Dynamicvit: Efficient vision transformers with dynamic token sparsification. ArXiv, abs/2106.02034, 2021.
- [167] T. Chen et al. Chasing sparsity in vision transformers: An end-to-end exploration. In Neural Information Processing Systems, 2021.
- [168] C. Zheng et al. Savit: Structure-aware vision transformer pruning via collaborative optimization. In Neural Information Processing Systems, 2022.
- [169] F. Yu et al. Width & depth pruning for vision transformers. In AAAI Conference on Artificial Intelligence, 2022.
- [170] L. Papa et al. A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking. ArXiv, abs/2309.02031, 2023.
- [171] M. Zhu et al. Vision transformer pruning. 2021.
- [172] Y. Wang et al. Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition. In Neural Information Processing Systems, 2021.
- [173] Y. Liang et al. Not all patches are what you need: Expediting vision transformers via token reorganizations. ArXiv, abs/2202.07800, 2022.
- [174] N. Carion et al. End-to-end object detection with transformers. ArXiv, abs/2005.12872, 2020.
- [175] B. Roh et al. Sparse detr: Efficient end-to-end object detection with learnable sparsity. ArXiv, abs/2111.14330, 2021.
- [176] D. Zheng et al. Less is more: Focus attention for efficient detr. ArXiv, abs/2307.12612, 2023.
- [177] H. Yang et al. Global vision transformer pruning with hessian-aware saliency. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18547–18557, 2021.
- [178] A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
- [179] J. W. Rae et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021.
- [180] G. Penedo et al. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116, 2023.
- [181] H. Peng et al. Random feature attention. arXiv preprint arXiv:2103.02143, 2021.
- [182] M. Zaheer et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 2020.
- [183] R. Child et al. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.
- [184] A. Roy et al. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 2021.
- [185] J. W. Rae et al. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05507, 2019.
- [186] G. Xiao et al. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023.
- [187] U. Khandelwal et al. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00172, 2019.
- [188] I. Beltagy et al. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.
- [189] C. Chelba et al. Faster transformer decoding: N-gram masked self-attention. arXiv preprint arXiv:2001.04589, 2020.
- [190] K. Choromanski et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794, 2020.
- [191] A. Katharopoulos et al. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, 2020.
- [192] R. Li et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, 2023.
- [193] N. Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019.
- [194] J. Ainslie et al. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023.
- [195] A. Zeng et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414, 2022.
- [196] J. Hoffmann et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.
- [197] K. Han et al. Transformer in transformer. Advances in Neural Information Processing Systems, 2021.
- [198] W. Wang et al. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF international conference on computer vision, 2021.
- [199] J. Guo et al. Cmt: Convolutional neural networks meet vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
- [200] T. Dao et al. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 2022.
- [201] J. Rasley et al. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.
- [202] M. Shoeybi et al. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.
- [203] A. Paszke et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 2019.
- [204] S. Li et al. Colossal-ai: A unified deep learning system for large-scale parallel training. In Proceedings of the 52nd International Conference on Parallel Processing, 2023.
- [205] T. Wolf et al. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.
- [206] A. Gu et al. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021.
- [207] H. Mehta et al. Long range language modeling via gated state spaces. arXiv preprint arXiv:2206.13947, 2022.
- [208] T. Dao et al. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212.14052, 2022.
- [209] A. Gupta et al. Diagonal state spaces are as effective as structured state spaces. Advances in Neural Information Processing Systems, 2022.
- [210] M. Poli et al. Hyena hierarchy: Towards larger convolutional language models. arXiv preprint arXiv:2302.10866, 2023.
- [211] Z. Dai et al. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019.
- [212] P. H. Martins et al. $\infty$ -former: Infinite memory transformer. arXiv preprint arXiv:2109.00301, 2021.
- [213] X. Ma et al. Mega: moving average equipped gated attention. arXiv preprint arXiv:2209.10655, 2022.
- [214] C.-T. Chen. Linear system theory and design. Saunders college publishing, 1984.
- [215] A. Gu et al. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 2020.
- [216] V. Pan. Fast approximate computations with cauchy matrices and polynomials. Mathematics of Computation, 2017.
- [217] B. Peng et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048, 2023.
- [218] M. Pióro et al. Moe-mamba: Efficient selective state space models with mixture of experts. arXiv preprint arXiv:2401.04081, 2024.
- [219] W. He et al. Densemamba: State space models with dense hidden connection for efficient large language models. arXiv preprint arXiv:2403.00818, 2024.
- [220] N. Du et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, 2022.
- [221] D. Lepikhin et al. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.
- [222] W. Fedus et al. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 2022.
- [223] S. Roller et al. Hash layers for large sparse models. Advances in Neural Information Processing Systems, 2021.
- [224] Z. Chi et al. On the representation collapse of sparse mixture of experts. Advances in Neural Information Processing Systems, 2022.
- [225] M. Lewis et al. Base layers: Simplifying training of large, sparse models. In International Conference on Machine Learning, 2021.
- [226] L. Z. Liu et al. Towards a unified view of sparse feed-forward network in pretraining large language model. arXiv preprint arXiv:2305.13999, 2023.
- [227] D. W. Romero et al. Ckconv: Continuous kernel convolution for sequential data. arXiv preprint arXiv:2102.02611, 2021.
- [228] N. Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020.
- [229] D. E. Rumelhart et al. Learning representations by back-propagating errors. nature, 1986.
- [230] B. Heo et al. Rethinking spatial dimensions of vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
- [231] J. Yang et al. Focal self-attention for local-global interactions in vision transformers. arXiv preprint arXiv:2107.00641, 2021.
- [232] S. Tang et al. Quadtree attention for vision transformers. arXiv preprint arXiv:2201.02767, 2022.
- [233] Z. Pan et al. Scalable vision transformers with hierarchical pooling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
- [234] C.-F. R. Chen et al. Crossvit: Cross-attention multi-scale vision transformer for image classification. In Proceedings of the IEEE/CVF international conference on computer vision, 2021.
- [235] W. Xu et al. Co-scale conv-attentional image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
- [236] N. Parmar et al. Image transformer. In International conference on machine learning, 2018.
- [237] A. Hatamizadeh et al. Fastervit: Fast vision transformers with hierarchical attention. arXiv preprint arXiv:2306.06189, 2023.
- [238] X. Liu et al. Efficientvit: Memory efficient vision transformer with cascaded group attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
- [239] H. You et al. Castling-vit: Compressing self-attention via switching towards linear-angular attention during vision transformer inference. arXiv preprint arXiv:2211.10526, 2022.
- [240] Y. Xiong et al. Nyströmformer: A nyström-based algorithm for approximating self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
- [241] M.-H. Guo et al. Beyond self-attention: External attention using two linear layers for visual tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
- [242] J. Wang et al. Riformer: Keep your vision backbone effective but removing token mixer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
- [243] D. Lian et al. As-mlp: An axial shifted mlp architecture for vision. arXiv preprint arXiv:2107.08391, 2021.
- [244] S. Chen et al. Cyclemlp: A mlp-like architecture for dense prediction. arXiv preprint arXiv:2107.10224, 2021.
- [245] L. Yuan et al. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference on computer vision, 2021.
- [246] B. Graham et al. Levit: a vision transformer in convnet’s clothing for faster inference. In Proceedings of the IEEE/CVF international conference on computer vision, 2021.
- [247] S. Mehta and M. Rastegari. Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer. arXiv preprint arXiv:2110.02178, 2021.
- [248] X. Chu et al. Twins: Revisiting the design of spatial attention in vision transformers. Advances in Neural Information Processing Systems, 2021.
- [249] C.-F. Chen et al. Regionvit: Regional-to-local attention for vision transformers. arXiv preprint arXiv:2106.02689, 2021.
- [250] P. Wang et al. Kvt: k-nn attention for boosting vision transformers. In European conference on computer vision, 2022.
- [251] Z. Pan et al. Fast vision transformers with hilo attention. Advances in Neural Information Processing Systems, 2022.
- [252] X. Dong et al. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
- [253] Z. Huang et al. Shuffle transformer: Rethinking spatial shuffle for vision transformer. arXiv preprint arXiv:2106.03650, 2021.
- [254] J. Fang et al. Msg-transformer: Exchanging local spatial information by manipulating messenger tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
- [255] D. Han et al. Flatten transformer: Vision transformer using focused linear attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
- [256] S. Zhai et al. An attention free transformer. arXiv preprint arXiv:2105.14103, 2021.
- [257] Y. Rao et al. Global filter networks for image classification. Advances in neural information processing systems, 2021.
- [258] H. Touvron et al. Resmlp: Feedforward networks for image classification with data-efficient training. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
- [259] J. Guo et al. Hire-mlp: Vision mlp via hierarchical rearrangement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 826–836, 2022.
- [260] Y. Tang et al. An image patch is a wave: Phase-aware vision mlp. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10935–10944, 2022.
- [261] L. Zhu et al. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417, 2024.
- [262] Y. Liu et al. Vmamba: Visual state space model. arXiv preprint arXiv:2401.10166, 2024.
- [263] X. Zhai et al. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
- [264] X. Su et al. Vitas: Vision transformer architecture search. In European Conference on Computer Vision, 2022.
- [265] M. Chen et al. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF international conference on computer vision, 2021.
- [266] B. Chen et al. Glit: Neural architecture search for global and local image transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
- [267] H. Chen et al. Vanillanet: the power of minimalism in deep learning. arXiv preprint arXiv:2305.12972, 2023.
- [268] M. Denil et al. Predicting parameters in deep learning. Advances in neural information processing systems, 26, 2013.
- [269] M. Jaderberg et al. Speeding up convolutional neural networks with low rank expansions. In BMVC 2014-Proceedings of the British Machine Vision Conference 2014. British Machine Vision Association, 2014.
- [270] A. Edalati et al. Kronecker decomposition for gpt compression. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 219–226, 2022.
- [271] A. Kaushal et al. Lord: Low rank decomposition of monolingual code llms for one-shot compression. arXiv preprint arXiv:2309.14021, 2023.
- [272] B. Chen et al. Ternary singular value decomposition as a better parameterized form in linear mapping. arXiv preprint arXiv:2308.07641, 2023.
- [273] M. Xu et al. Tensorgpt: Efficient compression of the embedding layer in llms based on the tensor-train decomposition. arXiv preprint arXiv:2307.00526, 2023.
- [274] B. B. Cambazoglu et al. Early exit optimizations for additive machine learned ranking systems. In Proceedings of the third ACM international conference on Web search and data mining, pp. 411–420, 2010.
- [275] S. Teerapittayanon et al. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd international conference on pattern recognition (ICPR), pp. 2464–2469. IEEE, 2016.
- [276] A. C. Stickland and I. Murray. Bert and pals: Projected attention layers for efficient adaptation in multi-task learning. In International Conference on Machine Learning, pp. 5986–5995. PMLR, 2019.
- [277] R. Schwartz et al. The right tool for the job: Matching model and instance complexities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6640–6651, 2020.
- [278] W. Liu et al. Fastbert: a self-distilling bert with adaptive inference time. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6035–6044, 2020.
- [279] L. Hou et al. Dynabert: Dynamic bert with adaptive width and depth. Advances in Neural Information Processing Systems, 33:9782–9793, 2020.
- [280] Y. Liu et al. Faster depth-adaptive transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 13424–13432, 2021.
- [281] T. Schuster et al. Consistent accelerated inference via confident adaptive transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 4962–4979, 2021.
- [282] M. Elbayad et al. Depth-adaptive transformer. In ICLR, pp. 1–14, 2020.
- [283] T. Schuster et al. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 35:17456–17472, 2022.
- [284] L. Del Corro et al. Skipdecode: Autoregressive skip decoding with batching and caching for efficient llm inference. arXiv preprint arXiv:2307.02628, 2023.
- [285] Y. Leviathan et al. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274–19286. PMLR, 2023.
- [286] C. Chen et al. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023.
- [287] N. Yang et al. Inference with reference: Lossless acceleration of large language models. arXiv preprint arXiv:2304.04487, 2023.
- [288] D. Xu et al. Llmcad: Fast and scalable on-device large language model inference. arXiv preprint arXiv:2309.04255, 2023.
- [289] T. Wang et al. Apq: Joint search for network architecture, pruning and quantization policy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2078–2087, 2020.