# XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression
## Abstract
Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks. However, their extensive memory requirements, particularly due to KV cache growth during long-text understanding and generation, present significant challenges for deployment in resource-constrained environments. Quantization has emerged as a promising solution to reduce memory consumption while preserving historical information. We propose XQuant, a training-free and plug-and-play framework that achieves ultra-low equivalent bit-width KV cache quantization. XQuant introduces two key innovations: a computationally negligible data-free calibration method and cross-layer KV cache compression, enabling quantization to sub-1.4 bits. Extensive experiments on TruthfulQA and LongBench demonstrate that XQuant outperforms state-of-the-art methods (e.g., KIVI-2bit and AsymKV-1.5bit) by achieving lower bit-width while maintaining superior performance, establishing a better trade-off between memory efficiency and model accuracy. The source code is available at https://github.com/brinenick511/XQuant.
XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression
Haoqi Yang 2, Yao Yao 3, Zuchao Li 1 Corresponding author., Baoyuan Qi 4, Guoming Liu 4, Hai Zhao 3 1 School of Artificial Intelligence, Wuhan University, Wuhan, China, 2 School of Computer Science, Wuhan University, Wuhan, China, 3 School of Computer Science, Shanghai Jiao Tong University, Shanghai, China, 4 Xiaomi Inc., Beijing, China {yanghq, zcli-charlie}@whu.edu.cn, yaoyao27@sjtu.edu.cn, {qibaoyuan, liuguoming}@xiaomi.com, zhaohai@cs.sjtu.edu.cn
## 1 Introduction
The rapid advancement of Large Language Models (LLMs) has propelled significant progress in a wide array of natural language processing (NLP) applications, including code generation, search systems, and many others Ouyang et al. (2023); Sharma et al. (2024); Ma et al. (2024). The exceptional performance of LLMs is primarily driven by their immense parameter scales, which enable them to excel across diverse tasks. However, this remarkable success comes with substantial costs: the computational and memory demands associated with deploying LLMs have increased exponentially due to increasing models parameters and growing input and output, posing a formidable bottleneck for practical deployment. In particular, GPU memory consumption has surged to levels that frequently surpass the capacities of current hardware infrastructures, making large-scale deployment increasingly challenging Shi et al. (2024).
To mitigate this challenge, the Key-Value (KV) cache mechanism has been widely adopted Yao et al. (2024); Yang et al. (2024d); Ainslie et al. (2023); Kwon et al. (2023). The KV cache optimizes memory efficiency by storing and reusing previously computed keys and values in the attention mechanism, thereby reducing redundant computations and GPU memory usage. Despite its advantages, as model sizes and the input/output sequence lengths continue to grow, the storage overhead of the KV cache itself becomes increasingly significant Shi et al. (2024). For instance, a 30-billion-parameter language model with a batch size of 128 and a sequence length of 1024 may require up to 180 GB of memory solely for storing the KV cache Zhang et al. (2023). Although the computational and memory requirements are reduced compared to not using it, such escalating demands still pose substantial challenges for deploying LLMs with constrained hardware resources.
To address this problem, prior works have explored various strategies from different perspectives. Some studies Sheng et al. (2023); Hooper et al. (2024); Liu et al. (2024b); Tao et al. (2024) focus on quantizing the floating-point KV cache (and, in some cases, model weights) to lower precision. However, these approaches often experience performance degradation under extreme compression ratios, particularly around 2-bit precision. Alternatively, other methods Xiao et al. (2023); Zhang et al. (2023); Li et al. (2024); Cai et al. (2024) aim to alleviate the storage burden by evicting unimportant tokens. These methods dynamically or statically identify and discard less critical tokens to reduce memory usage. Nevertheless, these methods inherently introduce information loss, resulting in reduced memory retention and severe forgetting issues, which can undermine the model’s ability to maintain consistent performance on longer sequences. Existing KV cache quantization methods, due to inherent architectural constraints, fail to mitigate the severe performance degradation when operating under ultra-low-bit settings.
To address these limitations, this paper focuses on training-free KV cache quantization scenarios under extreme compression ratios and introduces XQuant, a plug-and-play framework for ultra-low-bit KV cache quantization. XQuant delivers two key improvements over existing quantization methods: (1) Data-Free Calibration: Traditional quantization methods often face significant limitations when mapping values to low-bit precision. Specifically, they tend to use the two endpoint values (e.g., 0 and 1 in 1-bit quantization) as representative values, which can result in substantial quantization errors, particularly under low bit-width settings. To address this issue, XQuant introduces a parameterized calibration scheme that allows for more fine-grained mapping of values. By adjusting the representative values to better reflect the actual data distribution, this method significantly reduces quantization errors and minimizes performance loss without the need for additional data. (2) Cross-Layer KV Cache Compression: We observe enhanced KV cache similarity between adjacent layers after quantization - a previously overlooked phenomenon. This enables effective cross-layer compression, where the quantized KV cache of one layer is shared across subsequent layers, significantly reducing computational and memory costs. Meanwhile, a subset of layer-specific parameters is preserved to retain the unique characteristics of each layer, ensuring minimal loss of model performance.
To evaluate the effectiveness of XQuant, we conduct extensive experiments on a consumer-grade NVIDIA GeForce RTX 3090 GPU (24GB) across diverse datasets, including TruthfulQA Lin et al. (2022) and subsets of LongBench Bai et al. (2024). Experimental results demonstrate that XQuant achieves an equivalent bit-width of less than 1.4-bit across various LLMs, outperforming existing methods such as KIVI-2bit Liu et al. (2024b) and AsymKV-1.5bit Tao et al. (2024). Notably, XQuant achieves comparable performance to full-precision baselines while offering a significantly improved trade-off between model performance and compression ratio.
## 2 Related Work
Two mainstream approaches for addressing KV cache challenges are Quantization and Eviction methods Shi et al. (2024).
Quantization has emerged as a prominent technique for compressing large-scale models by mapping high-precision data to lower-precision formats (e.g., 16-bit, 8-bit, or even 4-bit integers). This significantly reduces memory footprints while maintaining acceptable levels of model performance. A substantial body of work focuses on quantizing model weights. AWQ Lin et al. (2024) optimizes neural network weight quantization by dynamically adapting the bit-width based on the weights’ significance. By retaining higher precision for more impactful weights and reducing precision for less critical ones, AWQ minimizes performance loss while achieving compression. However, aggressive compression is constrained by "model hemorrhage" Ma et al. (2025), a phenomenon identifying that models possess inherent robustness thresholds beyond which performance degrades sharply. This makes maintaining stability in the ultra-low-bit regime a critical challenge.
Another line of research concentrates on the quantization of the KV cache. KVQuant, introduced by Hooper et al. (2024), employs distinct quantization strategies for keys and values. It applies per-channel quantization to the keys—particularly before Rotary Positional Embeddings (RoPE)—and per-token quantization to the values, effectively managing outliers and minimizing RoPE-induced distortions. Similarly, MiKV Yang et al. (2024c) introduces a mixed-precision KV-cache strategy that retains important KV pairs in high precision. Concurrently, KIVI Liu et al. (2024b) develops a tuning-free 2-bit KV cache quantization scheme, where the key cache is quantized per-channel, and the value cache is quantized per-token. Building on this, AsymKV Tao et al. (2024) further combines 1-bit and 2-bit representations through an asymmetric and layer-wise quantization configuration, achieving a better trade-off between precision and compression ratio.
In contrast, some works simultaneously quantize both the model weights and the attention cache. For example, FlexGen Sheng et al. (2023) introduces a high-throughput inference framework that applies group-wise 4-bit quantization to compress both the model weights and KV cache. FlexGen divides tensors into small groups, computes the minimum and maximum values within each group, and performs asymmetric quantization. The resulting tensors are stored in 4-bit format and later dequantized to FP16 during computation, achieving a reduction in memory usage and I/O costs with minimal accuracy degradation. Despite the advancements of these methods, significant performance degradation remains a challenge when quantizing KV cache activations to extremely low-precision levels, particularly below 2-bit.
Eviction methods aim to discard unnecessary tokens during inference to reduce memory usage. StreamingLLM Xiao et al. (2023) identifies the phenomenon of attention sinks, where initial tokens are retained to stabilize attention computations. StreamingLLM combines these attention sinks with a sliding window of recent tokens to introduce a rolling KV cache, effectively balancing memory efficiency and model performance. Building on this, SirLLM Yao et al. (2024) uses token entropy to preserve critical tokens’ KV cache and incorporates a memory decay mechanism to enhance LLMs’ long-term memory while maintaining short-term reasoning abilities.
Other methods, such as H2O Zhang et al. (2023) and SnapKV Li et al. (2024), dynamically identify and evict non-important tokens based on attention scores. PyramidKV Cai et al. (2024); Yang et al. (2024a) observes that attention scores are more sparse in higher layers and accordingly allocates different memory budgets across layers. SpindleKV Tang et al. (2025) further develops a hybrid approach to balance reduction across layers, combining attention-based eviction in deep layers with a codebook-based replacement strategy for shallow layers. However, most existing KV eviction methods depend on attention scores to identify non-important tokens, which limits their compatibility with common optimizations like FlashAttention Dao (2023), reducing their practical usability.
Structural Approaches modify the model’s architecture, in contrast to post-hoc data compression. For instance, some methods cache only partial layers of the KV cache Wu and Tu (2024); Sun et al. (2024); Brandon et al. (2024), while KV-Latent Luohe et al. (2025) reduces the dimensionality of K and V vectors. A key characteristic of these approaches is that they all require additional training, which contrasts with our plug-and-play framework. We further clarify the key differences and highlight our contributions in Appendix G.
Compared to existing methods, we introduce XQuant with two key innovations: (1) A novel, simple yet effective data-free calibration method that achieves superior compression performance even under ultra-low-bit settings, eliminating the need for additional calibration data. (2) cross-layer KV cache compression that leverages previously overlooked quantization-enhanced layer similarities to achieve significant memory and computational savings. While prior work has studied layer representation similarities, our approach uniquely exploits the quantization-enhanced similarities to enable effective ultra-low-bit compression.
## 3 XQuant
<details>
<summary>x1.png Details</summary>
### Visual Description
## Diagram: Attention Layer Quantization and Speedup Process
### Overview
The diagram illustrates a technical workflow for optimizing attention layers in a neural network through quantization, calibration, and compression. It compares two paths: a standard process for "Original KV L" and an optimized "Speedup" path for "Original KV L+1". A key-value (KV) cache sharing mechanism is highlighted on the right.
### Components/Axes
1. **Left Side (Process Flow)**:
- **Top Branch**:
- **Original KV L** → Quantization (blue block) → Calibration (green block) → **Attention L**
- **Bottom Branch (Speedup)**:
- **Original KV L+1** → Quantization (blue block) → Calibration (green block) → Compression (green block) → **Attention L+1**
- **Dashed Line**: Indicates a shortcut bypassing compression in the speedup path.
2. **Right Side (KV Cache Details)**:
- **Boxed Section**:
- **Original scaling factor and zero point** (blue blocks)
- **Calibrated scaling factor and zero point** (blue blocks)
- **Quantized cache shared between layers** (green block)
### Detailed Analysis
- **Quantization**: Both branches use blue blocks to represent quantization of key-value pairs.
- **Calibration**: Green blocks follow quantization in both paths, adjusting scaling factors and zero points.
- **Compression**: Only present in the standard path (solid line), absent in the speedup path (dashed line).
- **Attention Layers**: Outputs from both paths feed into "Attention L" and "Attention L+1".
- **KV Cache**: The right box shows shared quantized cache (green) between layers, with original/calibrated scaling factors (blue).
### Key Observations
1. **Speedup Mechanism**: The bottom branch skips the compression step (dashed line), reducing computational overhead.
2. **Cache Sharing**: The green "Quantized cache" block is shared between layers, minimizing redundancy.
3. **Calibration Consistency**: Both paths use identical calibration (green blocks), ensuring uniformity in scaling adjustments.
4. **Color Coding**: Blue blocks (quantization) and green blocks (calibration/compression) are consistently applied across components.
### Interpretation
This diagram demonstrates a method to accelerate attention layers by:
1. **Quantizing and calibrating** key-value pairs to reduce memory usage and computational cost.
2. **Sharing quantized caches** between layers to avoid redundant computations, enabling the speedup path.
3. **Bypassing compression** in the optimized path, trading minor accuracy tradeoffs for faster inference.
The process balances efficiency and accuracy, with calibration ensuring post-quantization adjustments maintain model performance. The shared cache and skipped compression steps highlight tradeoffs between speed and precision in neural network optimization.
</details>
Figure 1: The illustration of XQuant workflow. XQuant partitions the KV cache into layer-wise pairs. For every higher layer in a pair, XQuant only computes and stores the scaling factors and zero-points during quantization phase, and then fetches the quantized cache from the lower layer during dequantization phase.
In this section, we present XQuant, a novel quantization framework for efficient KV cache compression. As illustrated in Figure 1, our framework introduces two key innovations: a data-free calibration technique that asymmetrically adjusts quantization parameters without additional calibration data, and a cross-layer KV cache compression mechanism that leverages the similarity of quantized caches between adjacent layers to effectively reduce both computational and memory overhead.
### 3.1 Background
To formalize KV cache quantization, we consider a group of floating-point keys or values $X$ . The quantization process transforms $X$ into three components: a B-bit quantized cache $X_Q$ , a zero-point $z$ , and a scaling factor $s$ Liu et al. (2024b):
Quantization Phase:
$$
z=min(X),s=\frac{max(X)-min(X)}{(2^B-1)} \tag{1}
$$
$$
X_T=(X-z)/s,X_Q=\lceilX_T\rfloor \tag{2}
$$
Dequantization Phase:
$$
\hat{X}=X_Q*s+z \tag{3}
$$
where $X^*$ is the dequantized counterpart and $\lceil·\rfloor$ is the rounding function. $X_T$ , the transformed matrix, is not explicitly cached but is introduced as an intermediate variable to facilitate subsequent mathematical derivations.
Building upon this framework, prior works introduce various configurations to enhance performance. For example, Liu et al. (2024b) focuses on the element-wise distribution within the KV cache, adopting per-channel quantization for the key cache and per-token quantization for the value cache. Similarly, Tao et al. (2024) introduces layer-wise quantization configurations, employing asymmetric bit-widths for the key and value caches across different layers. While effective, these approaches often suffer from significant performance degradation under low-bit quantization settings, particularly around 2-bit precision. This limitation motivates the need for further advancements in KV cache compression techniques.
### 3.2 Data-Free Calibration
Since existing quantization methods often experience significant performance degradation at 2-bit precision, achieving ultra-low-bit compression first requires bridging this performance gap. In this section, we propose a data-free calibration method that effectively preserves model performance, enabling more aggressive compression ratios.
To analyze extreme quantization scenarios, we start with 1-bit quantization where each parameter is constrained to a binary state. Formally, the round-to-nearest operation $\lceil·\rfloor$ is defined as:
$$
\lceil e\rfloor=\begin{cases}0&if e∈[0,0.5],\\
1&if e∈(0.5,1].\end{cases} \tag{4}
$$
where $e$ denotes an element of the transformed matrix. For any bit-width $B$ , this rounding operation maps values to a discrete set within $[0,2^B-1]$ , where each original value is assigned to its nearest representative in the quantized space. As shown in Figure 2 (a), fixed representative values at endpoints (0 and 1) yield substantial quantization error for 1-bit quantization. We therefore introduce a relaxed-constraint mapping function that adaptively determines the quantization levels, formulated as:
$$
f(e,η)=\begin{cases}η&if e∈[0,0.5],\\
1-η&if e∈(0.5,1].\end{cases} \tag{5}
$$
where $η∈[0,0.5]$ serves as a calibration parameter for determining quantization tendencies. Clearly, $f(e,0)$ is equivalent to the round-to-nearest function $\lceil e\rfloor$ . We extend this formulation to the general case of $B$ -bit quantization and denote the corresponding parameter as $η_B$ .
We relax the constraint that quantized values must be integers and apply fake quantization as a preliminary experiment. Table 7 shows that using this constraint-relaxed mapping function improves model performance, validating our proposed insight.
However, storing floating-point numbers as so-called quantized caches is impractical, as shown in Figure 2 (b). To address the aforementioned problem, we establish an equivalent implementation, with the mathematical proof provided below. We formalize the final data-free calibration approach as:
Consider a group of floating-point keys or values $X∈R^g$ , where $g$ stands for the group size. Note that $X∈[min(X),max(X)]^g=[z,s*(2^B-1)+z]^g$ , we can deduce:
$$
X_Q∈[0,2^B-1]^g \tag{6}
$$
from Equation 1 and Equation 2. If we choose $η*(2^B-1)$ and $(1-η)*(2^B-1)$ generalized from Equation 5 as two endpoints, it is equivalent to calibrate the zero-point and scaling factor to $\hat{z}$ and $\hat{s}$ , and then dequantize with them. Note that the dequantized matrix
$$
\hat{X}=X_Q*\hat{s}+\hat{z}∈[\hat{s}*0+\hat{z},\hat{s}*(2^B-1)+\hat{z}]^g \tag{7}
$$
and the corresponding interval given by two endpoints:
$$
[z+η s(2^B-1),z+s(2^B-1)(1-η)] \tag{8}
$$
By calculation we get the final operations for calibration:
$$
\hat{z}=z+η s(2^B-1),\hat{s}=(1-2η)s \tag{9}
$$
<details>
<summary>x2.png Details</summary>
### Visual Description
## Diagram: Quantization Methods Comparison
### Overview
The diagram illustrates three quantization techniques:
1. **Standard Quantization** (a)
2. **Relaxed-Constraint Quantization** (b)
3. **Quantization with Calibration** (c)
Each method maps original data (`X`) to quantized values (`X_Q`) and back to reconstructed data (`X̂`), with trade-offs in hardware compatibility and quantization error.
---
### Components/Axes
- **Axes**:
- Horizontal axis labeled `X` (original data range: `X_min = z` to `X_max`).
- Horizontal axis labeled `X̂` (reconstructed data range: `0` to `1`).
- **Key Elements**:
- **Quantization**: Arrows from `X` to `X_Q` (quantized value).
- **Dequantization**: Arrows from `X_Q` to `X̂`.
- **Shaded Regions**: Represent mapping ranges (e.g., triangular/trapezoidal shapes).
- **Checkmarks/Crosses**:
- Green check: High hardware compatibility / Low quantization error.
- Red cross: Low hardware compatibility / High quantization error.
- **Legend**:
- Green: High hardware compatibility, low error.
- Red: High error.
- Blue: Quantization range (`X_min` to `X_max`).
---
### Detailed Analysis
#### (a) Standard Quantization
- **Quantization**: Maps `X` to `X_Q` using `z` and `s` (step size).
- **Dequantization**: Reconstructs `X̂` with `z` and `s`.
- **Shaded Region**: Triangular, indicating linear mapping.
- **Hardware Compatibility**: ✅ High (green check).
- **Quantization Error**: ❌ High (red cross).
#### (b) Relaxed-Constraint Quantization
- **Quantization**: Maps `X` to `X_Q` with `z` and `s`.
- **Dequantization**: Reconstructs `X̂` with `z` and `s`.
- **Shaded Region**: Trapezoidal, with floating-point storage (`η`).
- **Hardware Compatibility**: ❌ Low (red cross).
- **Quantization Error**: ✅ Low (green check).
#### (c) Quantization with Calibration
- **Quantization**: Maps `X` to `X_Q` with `z` and `s`.
- **Calibration**: Adjusts parameters (`ẑ`, `ŝ`).
- **Dequantization**: Reconstructs `X̂` with `z` and `s`.
- **Shaded Region**: Triangular, with integer-only storage.
- **Hardware Compatibility**: ✅ High (green check).
- **Quantization Error**: ✅ Low (green check).
---
### Key Observations
1. **Trade-offs**:
- Standard quantization prioritizes hardware compatibility but introduces high error.
- Relaxed-constraint quantization sacrifices compatibility for lower error.
- Calibration balances both, achieving high compatibility and low error.
2. **Storage**:
- (a) and (b) use floating-point storage (`η`), while (c) uses integer-only storage.
3. **Error Distribution**:
- Shaded regions’ shapes correlate with error magnitude (e.g., trapezoidal regions in (b) suggest broader error tolerance).
---
### Interpretation
The diagram highlights the **trade-off between hardware efficiency and accuracy** in quantization.
- **Standard Quantization** (a) is hardware-friendly but error-prone, likely due to simplistic mapping.
- **Relaxed-Constraint Quantization** (b) uses floating-point storage to reduce error but requires less compatible hardware.
- **Calibration** (c) optimizes both by adjusting parameters (`ẑ`, `ŝ`) and using integer storage, suggesting it is the most efficient method for systems prioritizing accuracy and compatibility.
The use of **integer-only storage** in (c) likely reduces computational overhead, aligning with high hardware compatibility. The **floating-point storage** in (b) enables finer granularity, lowering error but increasing hardware demands.
This analysis underscores the importance of calibration in balancing practical constraints and performance in quantization systems.
</details>
Figure 2: The illustration of the proposed data-free calibration method.
Since $X_T=(X-z)/s$ , the reconstruction loss $MSE(X,\hat{X})=s^2· MSE(X_T,f(X_T,η))$ . For analytical tractability, particularly for 1-bit quantization within small group sizes, we can assume that $X_T∼ U(0,1)$ . Thus the expected MSE in the transformed space can be formulated as:
| | $\displaystyle MSE(X_T,f(X_T,η))$ | |
| --- | --- | --- |
Since the standard quantization scheme is equivalent to setting $η=0$ , this result confirms that any value of $η∈(0,1/2)$ will strictly reduce the theoretical reconstruction error.
As shown in Figure 2 (c), we propose the improved quantization scheme with this data-free calibration as follows:
Quantization Phase with Calibration:
$$
z=min(X),s=\frac{max(X)-min(X)}{(2^B-1)} \tag{10}
$$
$$
X_T=(X-z)/s,X_Q=\lceilX_T\rfloor \tag{11}
$$
$$
\hat{z}=z+η s(2^B-1),\hat{s}=(1-2η)s \tag{12}
$$
Dequantization Phase with Calibration:
$$
\hat{X}=X_Q*\hat{s}+\hat{z} \tag{13}
$$
<details>
<summary>src/delta.png Details</summary>
### Visual Description
## Stacked Bar Chart: Layer Distribution by Delta Value
### Overview
The image displays a stacked bar chart with four horizontal bars representing layers 0 to 30 on the x-axis. Each bar is divided into four color-coded segments corresponding to Delta values (0-3), with percentages summing to 100% for each layer. The chart emphasizes proportional distribution across layers.
### Components/Axes
- **X-axis (Layer)**: Labeled "Layer," with integer markers from 0 to 30 in increments of 5.
- **Y-axis (Percentage)**: Labeled "Percentage (%)" with markers from 0 to 100 in increments of 20.
- **Legend**: Located in the bottom-right corner, mapping:
- **Dark blue**: Delta=3
- **Teal**: Delta=2
- **Light blue**: Delta=1
- **Yellow**: Delta=0
### Detailed Analysis
- **Delta=0 (Yellow)**: Dominates each bar, occupying ~30-35% of the total height across all layers.
- **Delta=1 (Light blue)**: Second-largest segment, consistently ~45-50% of each bar.
- **Delta=2 (Teal)**: Third segment, ~15-20% of each bar.
- **Delta=3 (Dark blue)**: Smallest segment, ~5-10% of each bar.
- **Consistency**: All layers (0-30) exhibit identical proportional distributions, with no visible variation between bars.
### Key Observations
1. **Uniform Distribution**: Proportions of Delta values remain constant across all layers, suggesting no layer-specific anomalies or trends.
2. **Dominance of Delta=0**: The yellow segment (Delta=0) is the largest contributor in every layer, indicating it represents the most prevalent state.
3. **Stable Hierarchy**: The order of segment sizes (Delta=1 > Delta=0 > Delta=2 > Delta=3) is preserved uniformly.
### Interpretation
The chart demonstrates a stable, unchanging distribution of Delta values across all layers. This implies:
- **System Stability**: The underlying process or dataset maintains consistent proportions of Delta categories, with no significant shifts over time or conditions.
- **Baseline Representation**: Delta=0 likely represents a baseline or default state, while higher Deltas (1-3) denote incremental deviations or variations.
- **Predictability**: The uniformity suggests the system or data source exhibits predictable behavior, with no outliers or unexpected fluctuations.
### Spatial Grounding & Verification
- **Legend Placement**: Bottom-right corner, clearly separated from the chart.
- **Color Consistency**: All segments match legend colors exactly (e.g., dark blue for Delta=3 in all bars).
- **Axis Alignment**: X-axis labels are centered below bars; Y-axis percentages are left-aligned.
### Content Details
- **Layer 0**: Yellow (~35%), Light Blue (~50%), Teal (~15%), Dark Blue (~5%).
- **Layer 30**: Identical proportions to Layer 0, confirming uniformity.
- **No Intermediate Variations**: All layers replicate the same distribution pattern.
### Final Notes
The absence of numerical labels on the bars necessitates reliance on visual estimation. The chart’s simplicity emphasizes proportional relationships rather than absolute values, prioritizing comparative analysis over precise quantification.
</details>
Figure 3: Layer-wise analysis of absolute differences between adjacent layers in quantized KV Cache matrices. Here, delta represents the absolute difference of quantized values between consecutive layers.
### 3.3 Cross-Layer Compression
#### 3.3.1 Motivation
Building upon Tao et al. (2024) ’s investigation of ultra-low-bit KV cache asymmetric quantization, our reproduction experiments on LongBench Bai et al. (2023) with Mistral Jiang et al. (2023) demonstrate severe limitations of existing approaches, as shown in Table 8.
We found that 1-bit asymmetric quantization of the key cache is practically infeasible. Even when restricting 1-bit quantization to the top 8 layers (AsymKV-24/32), significant performance degradation occurs. Given the limitations of further key cache quantization, we turn to cross-layer compression techniques as a viable alternative to achieve comparable ultra-low-bit quantization without compromising performance.
#### 3.3.2 Analysis on Quantized KV Cache
To enable cross-layer compression, we first analyze the characteristics of quantized KV caches by examining inter-layer similarities. We hypothesize that significant redundancy between adjacent layers could create opportunities for more aggressive compression. Using the KIVI-2 framework Liu et al. (2024b), we conduct preliminary experiments on the Mistral-7B-Instruct-v0.2 model Jiang et al. (2023) with random samples from LongBench Bai et al. (2023).
Under the 2-bit quantization scheme in KIVI-2, quantized cache values are restricted to {0, 1, 2, 3}, naturally constraining element-wise absolute differences to the same range. Our analysis, illustrated in Figure 3, reveals a striking pattern: over 80% of positions between adjacent layers exhibit minimal differences (0 or 1), while extreme differences (3) occur in less than 5% of positions. This pattern becomes even more pronounced in the 1-bit scenario, where mapping {0,1} to 0 and {2,3} to 1 maintains identical values in over 80% of positions between adjacent layers. These empirical findings demonstrate substantial redundancy in quantized KV caches between adjacent layers, suggesting significant potential for further compression.
#### 3.3.3 Compression Algorithm
Leveraging these insights into inter-layer similarities, we propose a novel cross-layer compression method that decomposes KV caches into two components: shared quantized caches and layer-specific parameters. Specifically, adjacent layers share a common set of quantized value caches ( $X_Q$ ), while maintaining their individual scaling factors and zero-points for dequantization. This decomposition enables efficient compression by allowing each layer to reuse the merged cache from its group, while preserving the layer-specific characteristic through its unique quantization parameters, namely zero-points and scaling factors.
In the implementation, for a model with $L$ layers, we organize the layers into groups of size $G$ . Within each group, KV caches are compressed using weighted averaging, where each layer $l$ ( $0≤ l≤ L$ ) is assigned a weight $γ_l$ , subject to the constraint $∑γ_l=1$ .
Formally, for every layer $l$ in a group $G$ , the quantization workflow with cross-layer compression and calibration is utilized as follows:
Quantization Phase with Cross-Layer Compression and Calibration:
$$
∀ l∈G,
$$
$$
z_l=min(X_l), s_l=\frac{max(X_l)-min(X_l)}{(2^B-1)}
$$
$$
\hat{z}_l=z_l+η s_l(2^B-1), \hat{s}_l=(1-2η)s_l
$$
$$
X_Q=∑_l∈Gγ_l\bigg\lceil\frac{X_l-z_l}{s_l}\bigg\rfloor
$$
Dequantization Phase with Cross-Layer Compression and Calibration:
$$
\hat{X}_l=X_Q*\hat{s}_l+\hat{z}_l
$$
We present the pseudo code for the whole workflow as shown in Appendix J.
| Model Mistral-7b | Method Full Cache | Bit-width 16 | TruthfulQA 32.09 |
| --- | --- | --- | --- |
| KIVI | 2 | 32.17 | |
| AsymKV | 1.5 | 32.80 | |
| XQuant | 1.38 | 34.93 | |
| Llama2-7b | Full Cache | 16 | 30.77 |
| KIVI | 2 | 33.92 | |
| AsymKV | 1.5 | 33.84 | |
| XQuant | 1.4 | 34.22 | |
Table 1: Evaluation on TruthfulQA task with normal context length.
| Model | Method | Bit-width | HQA | 2Wiki | MSQ | TREC | TQA | SAMS | PC | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Mistral-7b-ins | Full Cache | 16 | 43.02 | 27.10 | 18.78 | 71.00 | 86.23 | 42.75 | 2.75 | 41.66 |
| PyramidInfer | / | 35.08 | 23.92 | 16.90 | 62.00 | 85.06 | 41.45 | 1.04 | 32.55 | |
| KIVI | 2 | 41.96 | 26.08 | 18.13 | 71.00 | 86.00 | 43.70 | 2.78 | 41.38 | |
| AsymKV | 1.5 | 37.17 | 22.77 | 15.76 | 70.50 | 86.25 | 43.44 | 3.16 | 39.86 | |
| XQuant | 1.38 | 42.90 | 26.65 | 17.44 | 71.50 | 84.50 | 45.18 | 5.71 | 41.98 | |
| Llama2-7b-chat | Full Cache | 16 | 30.09 | 26.48 | 9.98 | 63.00 | 84.19 | 41.22 | 4.50 | 37.07 |
| PyramidInfer | / | 29.14 | 24.53 | 7.49 | 54.00 | 81.79 | 40.71 | 4.00 | 34.52 | |
| KIVI | 2 | 29.10 | 25.12 | 9.86 | 63.00 | 84.98 | 40.18 | 4.00 | 36.61 | |
| AsymKV | 1.5 | 27.75 | 24.82 | 8.45 | 62.00 | 84.21 | 41.22 | 2.75 | 35.89 | |
| XQuant | 1.4 | 29.21 | 25.56 | 9.69 | 62.50 | 84.57 | 40.01 | 4.00 | 36.51 | |
Table 2: Evaluation of different KV cache compression methods on LongBench tasks.
#### 3.3.4 Speedup through Cross-layer Compression
While our previous discussion introduced weighted averaging with the weight $γ$ for compressing $X_Q$ within a group, we can further optimize the computation by setting $γ_k=1$ for a chosen dominant layer $k$ , which consequently forces all other $γ$ values within the group to zero. In this accelerated configuration, each subordinate layer only needs to compute and store its own scaling factors and zero-points, significantly reducing computational overhead. Specifically,
$$
X_Q=\bigg\lceil\frac{X_k-z_k}{s_k}\bigg\rfloor\\
$$
As illustrated in Figure 1, this optimization eliminates the computations shown in the dashed line, effectively streamlining the process. Experimental results show that selecting the first layer within the group as the dominant layer yields optimal performance, as demonstrated in Table 4 and Table 5.
## 4 Evaluation
### 4.1 Experimental Setup
Models.
We evaluate our XQuant on Llama-2-7b / Llama-2-7b-chat Touvron et al. (2023) and Mistral-7B-v0.3 / Mistral-7B-instruct-v0.2 Jiang et al. (2023).
Tasks.
For the normal context length task, we choose TruthfulQA (BLEU score) from LM-Eval Gao et al. (2021). We also select several subsets from LongBench Bai et al. (2023) for the long context length tasks, including HotpotQA (F1 score), 2WikiMultihopQA (F1 score), MuSiQue (F1 score), TREC (classification accuracy), TriviaQA (F1 score), SAMSum (Rouge-L) and PassageCount (Exact match accuracy). MultiFieldQA-Zh (F1 score) is selected for some ablation studies as well.
Baselines and Implementations.
We compare our framework with previous works, including original 16-bit floating implementation, KIVI-2 Liu et al. (2024b) and AsymKV Tao et al. (2024). All relevant configurations adhere as in KIVI, i.e., quantizing key cache per-channel and value cache per-token, and with a group size of 32 and a residual length of 128. We reproduce AsymKV based on the official implementation of KIVI, with a typical configuration (AsymKV-32/0) selected from the original paper, i.d., quantizing all the key cache into 2-bit and value cache into 1-bit, which corresponds to an equivalent bit-width of 1.5.
A token eviction method Yang et al. (2024b), configured with a 40% KV cache budget, is also included as a baseline for the LongBench tasks.
We set the maximum sequence length to 30000 for the Mistral model to conduct our experiments with a single NVIDIA GeForce RTX 3090 GPU (24GB), and 8192 for the Llama model as default. We do not consider SLERP Shoemake (1985); Liu et al. (2024a) because of the incompatibility between rescale-recover operations and quantized cache.
### 4.2 Performance Comparison
LM-Eval Results.
Table 1 presents the evaluation of different quantization methods on the TruthfulQA task with a standard context length. XQuant not only achieves competitive performance but surpasses the full cache baseline, with a TruthfulQA score of 34.93 on Mistral-7b and 34.22 on Llama2-7b, outperforming all other methods at significantly lower bit-widths. These results highlight that XQuant provides superior performance in conventional context length settings.
LongBench Results.
We evaluate XQuant on the LongBench benchmark using two widely adopted models: Mistral-7b-Instruct-v0.2 and Llama-2-7b-chat. As shown in Table 2, XQuant achieves significant improvements over other KV cache compression methods, particularly under ultra-low-bit settings.
In all datasets of LongBench, XQuant achieves performance comparable to the full cache baseline while reducing bit-width by 31% compared to KIVI-2bit. Notably, XQuant achieves an average score of 41.98 for Mistral, surpassing KIVI-2bit while maintaining a significantly lower bit-width of 1.38. Moreover, XQuant outperforms AsymKV on nearly all datasets while simultaneously reducing bit-width by 8% relative to AsymKV. Additionally, compared to PyramidInfer, which sacrifices precision to reduce storage overhead, XQuant demonstrates clear advantages in maintaining high accuracy across tasks while achieving lower bit-width.
### 4.3 Ablation and Analysis
In this section, we conduct ablation studies in some randomly selected lightweight LongBench subsets.
| Method Full Cache KIVI | Bit-width 16 2 | $η_1$ / / | $η_2$ / 0 | MFQA-Zh 48.26 42.27 |
| --- | --- | --- | --- | --- |
| AsymKV | 1.5 | 0 | 0 | 36.30 |
| XQuant | 1.375 | 0 | 0 | 37.20 |
| 0 | 0.05 | 40.32 | | |
| 0.2 | 0 | 41.98 | | |
| 0.2 | 0.05 | 44.20 | | |
Table 3: Ablation study on the effect of data-free calibration in XQuant on the MultiFieldQA-Zh benchmark from LongBench.
Calibration Parameter.
Table 3 presents an ablation study on the impact of data-free calibration in XQuant on the MultiFieldQA-Zh benchmark. The results indicate that applying calibration ( $η_1≠ 0$ or $η_2≠ 0$ ) significantly improves XQuant’s performance, reducing the performance gap with the full cache baseline.
| Method Full Cache KIVI | Bit-width 16 2 | $γ_0$ / / | MuSiQue 18.78 18.13 |
| --- | --- | --- | --- |
| Flooring | 1.63 | / | 16.79 |
| Ceiling | 1.63 | / | 16.36 |
| Weighted Average | 1.63 | [0,1/6) | 12.20 |
| 1.63 | (1/6,1/4) | 14.05 | |
| 1.63 | (1/4,1/2) | 16.84 | |
| 1.63 | (1/2,3/4) | 17.32 | |
| 1.63 | (3/4,5/6) | 17.60 | |
| 1.63 | (5/6,1] | 17.32 | |
Table 4: The comparison between different cross-layer compression method with group size $G=2$ , where $γ_0,γ_1$ stands for the coefficient in the weighted average $(γ_1+γ_0=1)$ .
Cross-Layer Compression Method.
We further explore the weighted average with a group size $G=2$ and coefficients $γ_0,γ_1=1-γ_0$ , where $γ_0$ falls into six intervals derived in Appendix F. Notably, when $γ_0∈[0,1/6)$ or $γ_0∈(5/6,1]$ , the operation is optimized to directly sharing the quantized cache. We evaluate KIVI-2 on Mistral-7B-Instruct-v0.2 without our proposed calibration methods starting from the 8-th layer. As summarized in Table 4, the accelerated compression methods ( $γ_0∈[0,1/6)∪(5/6,1]$ ) avoid redundant operations seen in the workflow of Liu et al., 2024b, which rounds quantized integers into floating-point numbers. As shown in Table 4, the accelerated compression operation demonstrates its effectiveness in maintaining sufficient information for model performance, particularly when $γ_0∈(5/6,1]$ . This configuration effectively allows odd-numbered layers to reuse the quantized cache from the preceding even-numbered layers without requiring additional quantization or storage overhead for odd-numbered layers.
We adopt this accelerated compression strategy across all experiments due to its favorable balance between computational efficiency and information preservation.
| Method | Bit-width | $G$ | $k$ | MSQ | MFQA-Zh |
| --- | --- | --- | --- | --- | --- |
| Full Cache | 16 | / | / | 18.78 | 48.26 |
| KIVI | 2 | / | / | 18.13 | 42.27 |
| 0 | 17.32 | 37.44 | | | |
| 2 | 1 | 12.20 | 20.48 | | |
| 0 | 14.92 | 17.53 | | | |
| 1 | 16.97 | 37.37 | | | |
| 3 | 2 | 13.21 | 20.80 | | |
| 0 | 14.82 | 23.53 | | | |
| 1 | 12.44 | 18.68 | | | |
| 2 | 16.12 | 35.48 | | | |
| XQuant | 1.63 | 4 | 3 | 15.39 | 20.32 |
Table 5: The comparison of different group sizes $G$ and selection indices $k$ within each group, where XQuant is employed without the calibration step for a clearer analysis.
Group Size.
After optimizing the cross-layer compression method, another factor is the group size. To investigate the effects of layer grouping, we partition the total $L$ layers of a model (where $L=32$ for Mistral-7B and Llama 2-7B) into $L/G$ contiguous groups of size $G$ . The parameter $k$ indicates that we store and share the quantized cache only in the $k$ -th layer of each group. We evaluate group sizes $G∈\{2,3,4\}$ . This range is motivated by the empirical observation that while adjacent layers exhibit high similarity in their quantized representations (i.e., $G=2$ , as shown in Figure 3), this similarity diminishes gradually for layer distances greater than three. For models with $L=32$ layers, $G=4$ thus serves as a sufficient upper bound for investigation due to this diminishing similarity. We set all configurations under the same compression ratio, namely keep all layers in key cache and 20 layers in value cache based on KIVI-2bit framework, using Mistral-7b-instruct-v0.2. As shown in Table 5, the model achieves the best performance with the configuration of $G=2$ and $k=0$ .
| Method | Bit-width | TREC | SAMS |
| --- | --- | --- | --- |
| Full Cache | 16 | 71 | 42.75 |
| KIVI | 2 | 71 | 43.7 |
| AsymKV | 1.5 | 70.5 | 43.44 |
| AsymKV | 1.375 | 69.5 | 42.76 |
| XQuant | 1.375 | 71.5 | 45.18 |
| AsymKV | 1.28 | 58.5 | 37.41 |
| XQuant | 1.28 | 68.5 | 39.84 |
| AsymKV | 1.15625 | 41 | 23.47 |
| XQuant | 1.15625 | 68.5 | 39.47 |
Table 6: The comparison of different configurations under extremely-low compression ratio.
Performance-Compression Trade-offs.
Table 6 evaluates the trade-offs between bit-width reduction and performance degradation across different quantization methods. As shown in Table 6, XQuant consistently outperforms other methods at the same bit-width, achieving higher scores on both TREC and SAMS benchmarks. Notably, even at an extremely low bit-width of 1.15625, XQuant preserves a significant portion of the model’s performance, maintaining a TREC score of 68.5 compared to the full-cache baseline of 71. These results demonstrate that XQuant effectively balances performance retention and compression, achieving state-of-the-art trade-offs in ultra-low-bit KV cache quantization.
## 5 Conclusion
To alleviate the growing memory overhead in LLM inference, we propose XQuant, a plug-and-play framework that quantizes KV cache at an extreme compression ratio. Based on our observations on classical training-free quantization and the distributions of quantized integers, we propose a data-free calibration method and a compute-efficient cross-layer compression method. Extensive experiments show that XQuant achieves state-of-the-art trade-offs between performance degradation and compression ratio, without sacrificing computational efficiency. Integrating these two novel methods, our XQuant achieves comparable performance with full-precision baseline under 1.4-bit quantization, and still maintains competitive performance for some tasks around an extremely 1.16-bit quantization.
## Limitations and Future Work
Our work presents several avenues for future exploration. First, while XQuant demonstrates promising results on representative models and benchmarks, its robustness and generalizability could be further validated by extending evaluations to a wider range of newer-generation or larger-scale models and more diverse downstream scenarios. Second, our current work relies on task-specific configurations. Although a unified setting proves robust (as shown in Appendix E), the development of an automated method to search for optimal configurations presents a valuable direction for future research. Finally, the key innovations of XQuant — Data-Free Calibration and Cross-layer Compression — are in principle orthogonal to other KV cache compression paradigms. A fruitful area for future work would be to investigate their compatibility and potential synergies with these existing methods, potentially yielding even greater efficiency gains.
## Acknowledgements
This work was supported by the National Natural Science Foundation of China (Grant No. 62306216) and the Natural Science Foundation of Hubei Province of China (Grant No. 2023AFB816).
Hai Zhao’s contribution was funded by the Major Program of the Chinese National Foundation of Social Sciences under Grant "The Challenge and Governance of Smart Media on News Authenticity" [No. 23&ZD213].
The authors also gratefully acknowledge support from the Xiaomi Open-Competition Research Program.
## References
- Ainslie et al. (2023) Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. 2023. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895–4901, Singapore. Association for Computational Linguistics.
- Bai et al. (2024) Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 3119–3137. Association for Computational Linguistics.
- Bai et al. (2023) Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508.
- Brandon et al. (2024) William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan Kelly. 2024. Reducing transformer key-value cache size with cross-layer attention. arXiv preprint arXiv:2405.12981.
- Cai et al. (2024) Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, et al. 2024. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02069.
- Dao (2023) Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691.
- Frantar et al. (2022) Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. GPTQ: Accurate post-training compression for generative pretrained transformers. arXiv preprint arXiv:2210.17323.
- Gao et al. (2021) Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. 2021. A framework for few-shot language model evaluation. Version v0. 0.1. Sept, 10:8–9.
- Hooper et al. (2024) Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length llm inference with kv cache quantization. arXiv preprint arXiv:2401.18079.
- Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.
- Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626.
- Li et al. (2024) Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469.
- Lin et al. (2024) Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87–100.
- Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics.
- Liu et al. (2024a) Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. 2024a. Minicache: Kv cache compression in depth dimension for large language models. arXiv preprint arXiv:2405.14366.
- Liu et al. (2024b) Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024b. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. ArXiv, abs/2402.02750.
- Luohe et al. (2025) Shi Luohe, Zuchao Li, Lefei Zhang, Baoyuan Qi, Liu Guoming, and Hai Zhao. 2025. KV-latent: Dimensional-level KV cache reduction with frequency-aware rotary positional embedding. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1535–1550, Vienna, Austria. Association for Computational Linguistics.
- Ma et al. (2024) Xinbei Ma, Zhuosheng Zhang, and Hai Zhao. 2024. Comprehensive cognitive llm agent for smartphone gui automation. arXiv preprint arXiv:2402.11941.
- Ma et al. (2025) Ziyang Ma, Zuchao Li, Lefei Zhang, Gui-Song Xia, Bo Du, Liangpei Zhang, and Dacheng Tao. 2025. Model hemorrhage and the robustness limits of large language models. arXiv preprint arXiv:2503.23924.
- Ouyang et al. (2023) Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. 2023. Llm is like a box of chocolates: the non-determinism of chatgpt in code generation. arXiv preprint arXiv:2308.02828.
- Qwen et al. (2025) Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115.
- Sharma et al. (2024) Nikhil Sharma, Q Vera Liao, and Ziang Xiao. 2024. Generative echo chamber? effect of llm-powered search systems on diverse information seeking. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–17.
- Sheng et al. (2023) Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, pages 31094–31116. PMLR.
- Shi et al. (2024) Luohe Shi, Hongyi Zhang, Yao Yao, Zuchao Li, and Hai Zhao. 2024. Keep the cost down: A review on methods to optimize llm’s kv-cache consumption. arXiv preprint arXiv:2407.18003.
- Shoemake (1985) Ken Shoemake. 1985. Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques, pages 245–254.
- Sun et al. (2024) Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. 2024. You only cache once: Decoder-decoder architectures for language models. arXiv preprint arXiv:2405.05254.
- Tang et al. (2025) Zicong Tang, Shi Luohe, Zuchao Li, Baoyuan Qi, Liu Guoming, Lefei Zhang, and Ping Wang. 2025. SpindleKV: A novel KV cache reduction method balancing both shallow and deep layers. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 28428–28442, Vienna, Austria. Association for Computational Linguistics.
- Tao et al. (2024) Qian Tao, Wenyuan Yu, and Jingren Zhou. 2024. Asymkv: Enabling 1-bit quantization of kv cache with layer-wise asymmetric quantization configurations. arXiv preprint arXiv:2410.13212.
- Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
- Wu and Tu (2024) Haoyi Wu and Kewei Tu. 2024. Layer-condensed kv cache for efficient inference of large language models. arXiv preprint arXiv:2405.10637.
- Xiao et al. (2023) Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks. arXiv.
- Yang et al. (2024a) Dongjie Yang, Xiaodong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. 2024a. Pyramidinfer: Pyramid KV cache compression for high-throughput LLM inference. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 3258–3270. Association for Computational Linguistics.
- Yang et al. (2024b) Dongjie Yang, XiaoDong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. 2024b. Pyramidinfer: Pyramid kv cache compression for high-throughput llm inference. arXiv preprint arXiv:2405.12532.
- Yang et al. (2024c) June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee. 2024c. No token left behind: Reliable KV cache compression via importance-aware mixed precision quantization. CoRR, abs/2402.18096.
- Yang et al. (2024d) Yifei Yang, Zouying Cao, Qiguang Chen, Libo Qin, Dongjie Yang, Hai Zhao, and Zhi Chen. 2024d. Kvsharer: Efficient inference via layer-wise dissimilar kv cache sharing. arXiv preprint arXiv:2410.18517.
- Yao et al. (2024) Yao Yao, Zuchao Li, and Hai Zhao. 2024. Sirllm: Streaming infinite retentive llm. arXiv preprint arXiv:2405.12528.
- Zhang et al. (2023) Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661–34710.
## Appendix A Preliminary Study on Relaxed-Contraint Mapping
| Method Full Cache KIVI | Bit-width 16 2 | $η_1$ / / | $η_2$ / 0 | MFQA-Zh 48.26 42.27 |
| --- | --- | --- | --- | --- |
| KIVI | 2 | / | 0.05 | 44.34 |
| AsymKV | 1.5 | 0 | 0 | 36.30 |
| AsymKV | 1.5 | 0 | 0.05 | 41.28 |
| AsymKV | 1.5 | 0.2 | 0 | 42.78 |
| AsymKV | 1.5 | 0.2 | 0.05 | 43.81 |
Table 7: The comparison using different quantization methods with and without our calibration method in MultiFieldQA-Zh tasks from LongBench.
As demonstrated in Figure 2, the traditional quantization workflow faces higher quantization error in low-bit scenarios. In Section 3.2, we propose a flexible mapping to mitigate the quantization error in this aspect. Moreover, to provide empirical evidence supporting the effectiveness of the flexible mapping in the proposed calibration method, we employ its generalized form and conduct a preliminary study on the default KIVI-2bit and AsymKV-32/0 configurations. We extend this approach to a generalized B-bit quantization mechanism, where $η_B$ serves as the corresponding parameter. Notably, when $η_B=0$ , the B-bit quantization operates without the flexible mapping.
The results in Table 7 demonstrate that incorporating the flexible mapping function enhances model performance across different quantization settings.
## Appendix B Preliminary Experiment on Layer-Wise Asymmetric Quantization
In the existing method Tao et al. (2024), the KV cache for each layer is quantized using either 1-bit or 2-bit precision. A straightforward strategy to maximize the compression ratio is to apply 1-bit quantization to a greater number of layers.
However, a significant bottleneck arises, as it is nearly impossible to quantize the key cache at 1-bit precision without compromising performance. As shown in Table 8, further compression by increasing the number of 1-bit quantized key cache layers is not feasible, as it leads to substantial performance degradation. This observation motivates us to explore alternative compression methodologies.
| Method | Bit-width | # Key Layers in 1-bit | MFQA-Zh |
| --- | --- | --- | --- |
| Full Cache | 16 | / | 48.26 |
| KIVI (32/32) | 2 | 0 | 42.27 |
| AsymKV-24/32 | 1.875 | 8 | 37.10 |
| AsymKV-16/32 | 1.75 | 16 | 21.36 |
| AsymKV-8/32 | 1.625 | 24 | 13.16 |
| AsymKV-0/32 | 1.5 | 32 | 7.66 |
Table 8: Evaluation on LongBench based on AsymKV shows that the key cache is nearly impossible to quantized under 1-bit.
## Appendix C Equivalent Bit-width Analysis
Formally, let $b,h,s,d$ be the batch size, the number of heads in GQA Ainslie et al. (2023), the sequence length and the dimension per head. The original $L$ layers of KV cache occupies $2L*bhsd*16 bit$ , which equals to $2L*n*16 bit$ if we set $n=bhsd$ for convenience.
Consider a typical KV cache quantization scheme Liu et al. (2024b). If we quantize all $L$ layers of key cache and value cache into $b$ -bit, the quantized KV cache memory usage is $2L*n*b bit$ . Tao et al., 2024 uses a asymmetrical configurations for key and value caches across different layers. In their paper, Asym- $l_k$ / $l_v$ means quantizing the initial $l_k$ layers of key cache and $l_v$ of value cache into 2-bit, and quantizating 1-bit for others. So the quantized KV cache memory usage is $(2*l_k+(32-l_k)+2*l_v+(32-l_v))*n bit$ . For example, Asym-1.5bit stands for Asym-32/0 in our paper, which can be calculated to $3L*n bit$ and can be equivalently considered as a 1.5-bit symmetrical quantization for better understanding of the compression ratio.
The related parameters in XQuant are $kq$ , $vq$ , $km$ , and $vm$ . The equivalent bit-width $B$ can be expressed as follows: $B=((32-max(kq,km))/2+(max(kq,km)-min(kq,km))+(max(kq,km)+min(kq,km))*2+(32-max(vq,vm))/2+(max(vq,vm)+min(vq,vm))+(max(vq,vm)+min(vq,vm))*2)/64$ .
In the classical configuration in our paper, $kq=30$ , $vq=2$ , $km=32$ , and $vm=16$ , in key cache we apply 2-bit quantization to the layers $[0,kq)$ and 1-bit quantization to the layers $[kq,32)$ , and cross-layer compression to the layers $[km,32)$ . The value cache is processed in the same manner. Therefore, the equivalent bit-widths of the key and value caches are computed as follows:
$$
B_k=\frac{(32-30)+30*2}{32}=1.9375
$$
$$
B_v=\frac{(32-16)/2+(16-2)+2*2}{32}=0.8125
$$
The average bit-width is therefore 1.375, which appears as 1.38 in most parts of this paper. More parameter sets used in our experiments are listed in Appendix I.
To maintain consistency with seminal works (e.g., KIVI Liu et al. (2024b) and GPTQ Frantar et al. (2022)), our reported "equivalent bit-width" for asymmetrical quantization methods considers only the quantized integer tensors, excluding metadata overhead like scaling factors and zero-points. The comparisons remain rigorous, as all evaluated quantization methods were implemented with identical group sizes and residual lengths. This ensures the unaccounted overhead is uniform across all methods and does not affect their relative performance rankings.
<details>
<summary>x3.png Details</summary>
### Visual Description
## Bar Chart: Performance Comparison of KIVI (2-bit) and Xquant (1.38-bit)
### Overview
The chart compares the performance of two technologies, KIVI (2-bit) and Xquant (1.38-bit, labeled as "Ours"), using a vertical bar chart. The y-axis represents time in milliseconds (ms), while the x-axis lists the two technologies. The chart emphasizes a significant performance difference between the two.
### Components/Axes
- **X-axis**:
- Categories: "KIVI (2-bit)" and "Xquant (1.38-bit, Ours)".
- Positioning: Centered labels below each bar.
- **Y-axis**:
- Scale: Linear, ranging from 18,000 ms (bottom) to 22,000 ms (top), with increments of 500 ms.
- Labels: Numerical values (e.g., 18,000, 18,500, ..., 22,000) aligned to the left of the axis.
- **Legend**:
- Located at the bottom of the chart.
- Colors:
- Dark green for "KIVI (2-bit)".
- Light green for "Xquant (1.38-bit, Ours)".
- **Bars**:
- Two vertical bars, one for each technology.
- Heights correspond to their respective performance values.
### Detailed Analysis
- **KIVI (2-bit)**:
- Bar height: 21,842 ms.
- Position: Leftmost bar, dark green.
- **Xquant (1.38-bit, Ours)**:
- Bar height: 18,807 ms.
- Position: Rightmost bar, light green.
- **Y-axis Alignment**:
- The KIVI bar extends to ~21,842 ms, nearly reaching the top of the chart.
- The Xquant bar aligns closer to the 18,800 ms mark, significantly lower than KIVI.
### Key Observations
1. **Performance Gap**: Xquant (1.38-bit) outperforms KIVI (2-bit) by **3,035 ms** (21,842 ms vs. 18,807 ms).
2. **Color Coding**: The legend clearly distinguishes the two technologies via dark and light green.
3. **Axis Scaling**: The y-axis starts at 18,000 ms, ensuring both bars are visible above the baseline.
### Interpretation
The data suggests that the 1.38-bit Xquant technology is **~13% faster** than the 2-bit KIVI, as measured by the time metric (lower ms = better performance). The use of "Ours" implies Xquant is the proprietary or preferred solution of the creators. The stark difference in performance could indicate that reducing bit precision (from 2-bit to 1.38-bit) improves efficiency without compromising functionality. This aligns with trends in computational optimization, where lower precision often enhances speed. The chart’s design effectively highlights this trade-off, making it suitable for technical audiences evaluating algorithmic or hardware performance.
</details>
Figure 4: Comparison of Execution Time.
## Appendix D Efficiency analysis
Using Mistral-7B as an example, we theoretically analyze the computational cost of our two key improvements. During the calibration step, generating each token incurs only 64 additional floating-point multiplications and 32 additions (Equation 12), which are negligible in practice. Moreover, as described in Section 3.3.4, the cross-layer compression step optimizes efficiency by skipping certain parts of the quantization process (Equation 2).
To evaluate inference efficiency, we adopt the same experimental setup as implemented in KIVI’s repository, using a batch size of 16, a prompt length of 1024, and an output length of 128. As shown in Figure 4, XQuant, by leveraging its unique speedup mechanism, demonstrates competitive inference efficiency.
## Appendix E Hyperparameter
The related parameters in XQuant are $kq$ , $vq$ , $km$ , and $vm$ . In XQuant, we quantize the lower $kq$ , $vq$ layers of key and value cache into 2-bit, while quantizing others into 1-bit. We apply cross-layer compression from the $km$ th, $vm$ th layer of key and value cache. All the configurations are summarized in Table 11.
As demonstrated in Table 9, additional experiments on the Mistral-7B-Instruct model using the LongBench benchmark show that XQuant, with a fixed $η_1=1/6$ and $η_2=0.045$ , consistently delivers strong performance as well. These results suggest that this fixed set of hyperparameters are robust and can generalize effectively across different datasets. Therefore, task-specific hyperparameter tuning is superior but not necessary, and the method can achieve reliable performance with a fixed, pre-selected set of hyperparameters.
| Full Cache AsymKV XQuant | 16 1.5 1.38 | / / Task-specific | 43.02 37.17 42.90 | 27.10 22.77 26.65 | 18.78 15.76 17.44 | 71.00 70.50 71.50 | 86.23 86.25 84.50 | 42.75 43.44 45.18 | 2.75 3.16 5.71 | 41.66 39.86 41.98 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| XQuant | 1.38 | Static | 42.64 | 25.16 | 16.91 | 70.50 | 84.50 | 42.64 | 4.57 | 40.99 |
Table 9: Evaluation of different KV cache compression methods using static hyperparameters setting.
## Appendix F Cross-Layer Compression Strategy
Under 2-bit quantization, the values in the KV cache are restricted to the discrete integer set $\{i∈Z\mid 0≤ i≤ 3\}$ . Therefore, a rounding operation is required after weighted averaging. If standard rounding-to-nearest is applied, the range of $γ_0$ can be divided into six disjoint intervals, as summarized in Table 4. The derivation is as follows:
Let $e_0$ and $e_1$ denote the $B$ -bit quantized values at the same position in adjacent layers of $X_Q$ . Then the merged value $e_m$ after cross-layer compression is computed as:
| | $\displaystyle e_m$ | $\displaystyle=≤ft\lfloor\frac{γ_0e_0+γ_1e_1}{γ_0+γ_1}\right\rceil$ | |
| --- | --- | --- | --- |
Without loss of generality, assume $e_0≥ e_1$ and define $δ=e_0-e_1≥ 0$ . Then we have:
$$
e_m=e_1+≤ft\lfloorγ_0δ\right\rceil, \tag{14}
$$
where $γ_0∈[0,1]$ and $δ∈Z∩[0,3]$ . Since $γ_0δ∈[0,δ]$ , the rounding term $≤ft\lfloorγ_0δ\right\rceil$ in Eq. 14 can only take $δ+1$ discrete values. Let $≤ft\lfloorγ_0δ\right\rceil=c$ , where $c∈Z∩[0,δ]$ . Then:
$$
γ_0δ∈≤ft(c-\frac{1}{2},c+\frac{1}{2}\right)∩[0,δ], \tag{15}
$$
which yields the following constraint for $γ_0$ , when $δ>0$ :
$$
γ_0∈≤ft(\frac{c-1/2}{δ},\frac{c+1/2}{δ}\right)∩[0,1]. \tag{16}
$$
We now enumerate all valid combinations of $δ$ and $c$ from Equation 16:
- $δ=0$ : Only one possible value exists; trivial case omitted.
- $δ=1$ :
- $c=0$ : $γ_0∈[0,1/2)$
- $c=1$ : $γ_0∈(1/2,1]$
- $δ=2$ :
- $c=0$ : $γ_0∈[0,1/4)$
- $c=1$ : $γ_0∈(1/4,3/4)$
- $c=2$ : $γ_0∈(3/4,1]$
- $δ=3$ :
- $c=0$ : $γ_0∈[0,1/6)$
- $c=1$ : $γ_0∈(1/6,1/2)$
- $c=2$ : $γ_0∈(1/2,5/6)$
- $c=3$ : $γ_0∈(5/6,1]$
Collectively, this yields six effective intervals of $γ_0$ , as summarized in Table 4.
## Appendix G Comparison with Other Cross-Layer Compression Methods
Several prior works have explored inter-layer redundancy from different perspectives. To eliminate potential confusion, we clarify several key distinctions and highlight innovations as follows: (a) Most existing methods compute KV caches at a subset of layers. However, these approaches require additional training steps and, in some cases, even full retraining, significantly limiting scalability. In contrast, XQuant is designed as a plug-and-play solution that leverages deeper insights to enable effective redundancy reduction without any additional training. (b) XQuant is the only method that explicitly considers inter-layer redundancy through the lens of quantization. After quantization, the KV cache is decomposed into three components: the quantized cache, zero-points, and scaling factors. We demonstrate that the quantized cache, consisting solely of integers, exhibits substantial inter-layer similarity. Meanwhile, the zero-points and scaling factors, which require minimal storage, are retained individually to preserve per-layer characteristics without being compressed. (c) MiniCache Liu et al. (2024a) is another training-free method that primarily introduces a retention-recovery mechanism for cache magnitudes and unmergable tokens. However, such operations are not directly compatible in mainstream open-source KV quantization frameworks. Furthermore, its use of the SLERP function imposes several constraints, making it inapplicable to quantized caches, which fundamentally differs from XQuant.
| Full Cache AsymKV XQuant | 16 1.4 1.4 | 58.20 38.55 54.16 | 61.88 44.69 57.44 |
| --- | --- | --- | --- |
Table 10: Comparison of XQuant with Full Cache and AsymKV on the Qwen2.5-14B model using the LongBench benchmark.
## Appendix H Evaluation on Qwen2.5-14B
As shown in Table 10, we evaluated XQuant on a larger-scale and newer-generation model, Qwen2.5-14B Qwen et al. (2025), using the LongBench benchmark. The results demonstrate that XQuant generalizes well to different models, maintaining a superior trade-off between model performance and compression ratio.
| Mistral-7b-instruct-v0.2 | HQA | 30 | 2 | 32 | 16 | 1/6 | 0.045 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 2Wiki | 32 | 0 | 32 | 16 | 0 | 0.09 | |
| MSQ | 32 | 0 | 32 | 16 | 1/6 | 0 | |
| TREC | 30 | 2 | 32 | 16 | 1/6 | 0 | |
| TQA | 30 | 2 | 32 | 16 | 1/6 | 0.09 | |
| SAMS | 30 | 2 | 32 | 16 | 0 | 0 | |
| PC | 32 | 0 | 32 | 16 | 0 | 0.045 | |
| Llama2-7b | TruthfulQA | 28 | 0 | 32 | 28 | 1/3 | 0 |
| Llama2-7b-chat | HQA | 28 | 0 | 32 | 28 | 1/6 | 0.045 |
| 2Wiki | 28 | 0 | 32 | 28 | 1/3 | 0.045 | |
| MSQ | 28 | 0 | 32 | 28 | 1/3 | 0 | |
| TREC | 32 | 0 | 32 | 20 | 1/6 | 0 | |
| TQA | 32 | 0 | 32 | 20 | 1/6 | 0 | |
| SAMS | 32 | 0 | 32 | 20 | 0 | 0 | |
| PC | 32 | 0 | 32 | 20 | 1/3 | 0.045 | |
Table 11: The configurations of our main experiments.
## Appendix I Configurations
The Configurations of XQuant in our main experiments are summarized in Table 11
## Appendix J XQuant Pseudo Code
The pseudo code for the whole workflow is provided in Algorithm 1 and 2.
1
2
Input : $kq$ , $vq$ , $km$ , $vm$ , $η[2]$
Output : Optimized Quantized Cache
3
4 for $l← 0$ to $31$ do
5 if $l<vm$ or $l\bmod 2==0$ then
6 $KeyCache[l]←\textbf{Quantize}\big(X^l_k, 2 \textbf{if }l<kq \textbf{else }1\big)$
7
8 else
9 $KeyCache[l]←\textbf{PseudoQuantize}\big(X^l_k, 2 \textbf{if }l<kq \textbf{else }1\big)$
10
11 if $l<vq$ or $l\bmod 2==0$ then
12 $ValueCache[l]←\textbf{Quantize}\big(X^l_v, 2 \textbf{if }l<vq \textbf{else }1\big)$
13
14 else
15 $ValueCache[l]←\textbf{PseudoQuantize}\big(X^l_v, 2 \textbf{if }l<vq \textbf{else }1\big)$
16
17 for $l← 0$ to $31$ do
18 if $l<km$ or $l\bmod 2==0$ then
19 $DequantizedKey←\textbf{Dequantize}\big($
20 $KeyCache[l][0],$
21 $KeyCache[l][1],$
22 $KeyCache[l][2]\big)$
23
24 else
25 $DequantizedKey←\textbf{Dequantize}\big($
26 $KeyCache[l-1][0],$
27 $KeyCache[l-1][1],$
28 $KeyCache[l][2]\big)$
29
30 if $l<vm$ or $l\bmod 2==0$ then
31 $DequantizedValue←\textbf{Dequantize}\big($
32 $ValueCache[l][0],$
33 $ValueCache[l][1],$
34 $ValueCache[l][2]\big)$
35
36 else
37 $DequantizedValue←\textbf{Dequantize}\big($
38 $ValueCache[l-1][0],$
39 $ValueCache[l-1][1],$
40 $ValueCache[l][2]\big)$
41
Algorithm 1 XQuant Procedure
1
2
3 Function PseudoQuantize( $X$ , $n\_bits$ ):
4 $zero\_point←\min(X)$ // Find the minimum value of $X$
5 $scaling\_factor←\frac{\max(X)-\min(X)}{2^n\_bits-1}$ // Calculate scaling factor
6 return
7 $\textbf{Calibrate}(zero\_point,$ $scaling\_factor, n\_bits)$ ,
8 None
9
10
11 Function Quantize( $X$ , $n\_bits$ ):
12 $zero\_point←\min(X)$
13 $scaling\_factor←\frac{\max(X)-\min(X)}{2^n\_bits-1}$
14 $quantized\_cache←\textbf{round}≤ft(\frac{X-zero\_point}{scaling\_factor}\right)$ // Round to nearest quantized value
15 return
16 $\textbf{Calibrate}(zero\_point,$ $scaling\_factor, n\_bits)$ ,
17 $quantized\_cache$
18
19
20 Function Dequantize( $zero\_point$ , $scaling\_factor$ , $quantized\_cache$ ):
21 return $quantized\_cache· scaling\_factor+zero\_point$ // Reconstruct original value
22
23
24 Function Calibrate( $zero\_point$ , $scaling\_factor$ , $n\_bits$ ):
25 $zero\_point\_cali← zero\_point+scaling\_factor·η[n\_bits]$ // Adjust zero point based on $η$
26
27 $scaling\_factor\_cali← scaling\_factor·\big(1-2·η[n\_bits]\big)$ // Adjust scaling factor based on $η$
28
29 return
30 $zero\_point\_cali, scaling\_factor\_cali$ // Return calibrated values
31
32
Algorithm 2 Supporting Functions