# Sequential KV Cache Compression via Probabilistic Language Tries: Beyond the Per-Vector Shannon Limit
**Authors**: Gregory Magarshak
## Abstract
Recent work on KV cache quantization, culminating in TurboQuant [13], has approached the Shannon entropy limit for per-vector compression of transformer key-value caches. We observe that this limit applies to a strictly weaker problem than the one that actually matters: compressing the KV cache as a sequence. The tokens stored in a KV cache are not arbitrary floating-point data—they are samples from the exact formal language the model was trained on, and the model is by construction a near-optimal predictor of that language.
We introduce sequential KV compression, a two-layer architecture that exploits this structure. The first layer, probabilistic prefix deduplication, identifies semantically equivalent shared prefixes across sessions using the trie metric $d_T(s,s^\prime)={-}\log_2P_M(s\wedge s^\prime)$ from Probabilistic Language Tries (PLTs) [9]. The second layer, predictive delta coding, stores only the residual of each new KV vector from the model’s own prediction of it, achieving a per-token entropy bound of $H(KV_t+1\midKV_≤ t)≤ H(token_t+1\midtoken_≤ t)$ .
We prove that at typical language model perplexity—approximately 10–20 for fluent English text—this bound is 3.3–4.3 bits on average per token position, compared to TurboQuant’s 3 bits per vector component (with typical attention heads having 64–128 components). The theoretical compression ratio over TurboQuant is approximately $914{,}000×$ at the Shannon limit. Even at $1000×$ above the entropy floor—a deliberately pessimistic worst-case overhead, two orders of magnitude above the 2–5 $×$ typical of practical source coders—the ratio remains $≈ 914×$ over TurboQuant, with compression improving rather than degrading as context length grows. The two layers are orthogonal and compose with existing per-vector quantization methods including TurboQuant.
Contents
1. 1 Introduction
1. 2 Background
1. 2.1 The KV Cache in Transformer Inference
1. 2.2 Per-Vector Quantization: The State of the Art
1. 2.3 Probabilistic Language Tries
1. 3 The Sequential Entropy Bound
1. 3.1 Setup and Notation
1. 3.2 The Determinism Lemma
1. 3.3 The Main Bound
1. 3.4 Tightness of the Bound
1. 4 Layer 1: Probabilistic Prefix Deduplication
1. 4.1 The Exact Prefix Sharing Baseline
1. 4.2 The PLT Trie Metric as a Deduplication Criterion
1. 4.3 Storage Under Probabilistic Prefix Deduplication
1. 5 Layer 2: Predictive Delta Coding
1. 5.1 The Prediction
1. 5.2 The Residual and Its Entropy
1. 5.3 Adaptive Quantization of Residuals
1. 6 Composition and the Full Stack
1. 6.1 Orthogonality of the Two Layers
1. 6.2 Composition with Per-Vector Quantization
1. 6.3 Asymptotic Behavior with Context Length
1. 7 Practical Implementation
1. 7.1 Efficient Prediction Computation
1. 7.2 Integration with the Inference Loop
1. 7.3 Trie-Based Prefix Index
1. 8 Related Work
1. 9 Discussion
1. 9.1 The Two Shannon Limits
1. 9.2 Implications for the “Memory Wall”
1. 9.3 Relation to the Jevons Paradox
1. 9.4 Rate-Distortion Analysis of KV Compression
1. 9.5 Connection to Speculative Decoding
1. 9.6 Limitations and Open Problems
1. 9.7 Conjectures and Open Problems
1. 9.8 Conclusion
1. References
## 1 Introduction
Every time a transformer language model processes a token, it produces a pair of vectors—a key and a value —that are stored in the KV cache and reused in all subsequent attention computations. This cache is the model’s working memory: it contains the compressed representation of everything the model has processed in the current context. It is also one of the primary bottlenecks in large-scale inference.
For a model with $L$ layers, $H_head$ attention heads, head dimension $d$ , and context length $n$ , the KV cache occupies $2LH_headdn$ floating-point values. At typical scales ( $L{=}80$ , $H_head{=}64$ , $d{=}128$ , $n{=}128{,}000$ ), a single context in a 70B-parameter model requires approximately 80 GB of cache memory in fp16—more than the model weights themselves.
A rich literature has developed to compress the KV cache. Quantization methods represent each cache entry in fewer bits [8, 4]. Eviction methods discard entries unlikely to affect future attention scores [14, 7]. Prefix sharing methods avoid redundant computation when multiple sessions share a common prefix [10, 5]. TurboQuant [13] recently unified and extended the quantization line of work, achieving near-optimal per-vector compression via PolarQuant rotation followed by QJL residual correction, and proved a formal lower bound showing that no per-vector method can do significantly better.
#### The gap TurboQuant does not close.
TurboQuant’s lower bound is tight—for the problem it solves. That problem is: given an isolated KV vector drawn from the post-rotation distribution, what is the minimum number of bits needed to represent it? The paper’s answer is approximately 3 bits per component, and TurboQuant achieves it.
But the KV cache is not a collection of isolated vectors. It is a sequence. Each vector was produced by processing a token from a specific position in a specific context, and both the token and its position are samples from a structured probability distribution—the distribution the model was trained to model. The information-theoretic content of the $t$ -th KV vector, given all prior vectors, is not its raw entropy as an isolated sample. It is its conditional entropy, conditioned on the model’s state after processing tokens $1$ through $t-1$ .
This conditional entropy can be far smaller. For a good language model operating on coherent text, the next token is highly predictable—and therefore the next KV vector is highly predictable. The residual is small. Its entropy is bounded by the model’s per-token surprisal, which at typical perplexities of 10–20 is 3.3–4.3 bits per entire token position, not per component.
The gap between TurboQuant’s floor and this sequential floor is not a rounding error. It is the full redundancy of language—the 10–15 bits per token of predictable structure that Shannon identified in 1951 [11] and that every good language model encodes.
#### This paper.
We make this gap precise and propose sequential KV compression, a two-layer architecture that closes it. Our contributions are:
1. The sequential entropy bound (Theorem 1): a formal proof that the conditional entropy of KV vectors, given all prior cache entries, is bounded above by the model’s per-token surprisal.
1. Probabilistic prefix deduplication (Section 4): using the PLT trie metric [9], we identify semantically equivalent shared prefixes across sessions and store only the delta from the shared centroid, eliminating inter-session redundancy beyond what exact prefix matching achieves.
1. Predictive delta coding (Section 5): within a single session, we store only the residual of each KV vector from the model’s own prediction, with the residual entropy bounded by the token-level surprisal.
1. Composability (Section 6): both layers are orthogonal to per-vector quantization methods and can be stacked beneath TurboQuant or any other quantizer.
1. Asymptotic behavior (Corollary 5): unlike per-vector methods whose compression ratio is fixed by head dimension, sequential compression improves with context length, because a model that has processed more tokens has a more precise predictive distribution for what comes next.
#### Why this is possible now.
The sequential structure of the KV cache is not new. What is new is a formal framework for exploiting it: the PLT trie metric [9] gives a mathematically precise definition of “distance between token sequences in probability space,” and the prior-guided caching theorem in that paper provides the theoretical foundation for using a model’s own probability estimates rather than empirical frequencies to identify shared structure. The present paper applies that framework to the specific problem of KV cache compression.
#### Organization.
Section 2 reviews the KV cache, per-vector quantization, and the PLT framework. Section 3 establishes the sequential entropy bound. Section 4 introduces probabilistic prefix deduplication. Section 5 introduces predictive delta coding. Section 6 analyzes how the layers compose. Section 7 discusses practical implementation. Section 8 situates the work relative to the literature. Section 9 discusses implications and open problems.
## 2 Background
### 2.1 The KV Cache in Transformer Inference
A transformer language model with $L$ layers processes a token sequence $t=(t_1,…,t_n)$ by computing, at each layer $\ell$ and position $i$ , a key vector $k^(\ell)_i∈ℝ^d$ and a value vector $v^(\ell)_i∈ℝ^d$ . These are computed from the input embedding and prior layer activations via learned projection matrices:
$$
k^(\ell)_i=W_K^(\ell)x^(\ell)_i, v^(\ell)_i=W_V^(\ell)x^(\ell)_i.
$$
The KV cache $K$ stores all $(k^(\ell)_i,v^(\ell)_i)$ pairs for every layer $\ell∈\{1,…,L\}$ and every processed position $i∈\{1,…,n\}$ , so that subsequent tokens can attend to all prior positions without recomputing them. In autoregressive generation, extending the sequence by one token requires one forward pass plus $O(n)$ attention operations over the cache; without caching it would require $O(n^2)$ operations.
The total cache size is $2LH_headdn$ values, where $H_head$ is the number of attention heads and $d$ is the per-head dimension. In fp16 (2 bytes per value), a 128K-token context on a 70B model requires approximately 80 GB.
### 2.2 Per-Vector Quantization: The State of the Art
The dominant approach to KV cache compression is quantization: represent each floating-point vector component in fewer bits. The key challenge is that KV vectors have outlier components—individual dimensions with much larger magnitude than the others—that cause severe quantization error if treated uniformly.
TurboQuant [13] addresses this via two operations:
PolarQuant. Apply a learned rotation matrix $R∈ℝ^d× d$ to each KV vector so that the rotated components have a more uniform, predictable distribution. Because the rotation is applied to all vectors uniformly, it can be precomputed once with no per-vector overhead. This eliminates the 1–2 bits of overhead per component that prior outlier-aware methods spent on metadata.
QJL (Quantized Johnson-Lindenstrauss). Quantize the rotated vector to $b$ bits per component using a precomputed quantizer. Use a single additional sign bit to correct the expected bias introduced by quantization, ensuring that attention scores computed from compressed vectors are statistically unbiased.
The combined scheme achieves a compression ratio of $16/b$ (e.g., $≈ 5.3×$ at $b=3$ ) with negligible accuracy loss. The paper proves a lower bound showing that, for vectors treated as independent samples from the post-rotation distribution, no quantization scheme can achieve a higher compression ratio without accuracy loss. This bound is tight: TurboQuant is near-optimal for per-vector quantization.
### 2.3 Probabilistic Language Tries
We briefly recall the PLT framework [9], which provides the formal machinery for our sequential approach.
**Definition 1 (Probabilistic Language Trie[9])**
*Let $V$ be a finite vocabulary and $M$ a generative model over $V^*$ . The probabilistic language trie $T(M)$ is the directed rooted tree whose nodes are prefixes $x∈ V^*$ and whose outgoing edges from node $x$ are labeled by tokens $t∈ V$ with weight $P_M(t\mid x)$ .*
**Definition 2 (Trie Metric[9])**
*For two sequences $s,s^\prime∈ V^*$ , their longest common prefix in the trie is $s\wedge s^\prime$ —the maximal prefix shared by both sequences. The trie metric is:
$$
d_T(s,s^\prime)=-\log_2P_M(s\wedge s^\prime).
$$
Sequences with a long, high-probability shared prefix are close in this metric; their KV traces share substantial structure.*
The trie metric has a direct compression interpretation: $d_T(s,s^\prime)$ is the number of bits needed to locate the divergence point of $s$ and $s^\prime$ within the probability distribution. Sequences with small trie distance are redundant relative to each other: one can be described as a short delta from the other.
**Remark 1 (Ultrametric structure)**
*$d_T$ satisfies the ultrametric inequality $d(s,s^\prime\prime)≤\max(d(s,s^\prime),d(s^\prime,s^\prime\prime))$ , which is stronger than the standard triangle inequality. Note that $d_T(s,s)=-\log_2P_M(s)$ is not generally zero (it equals zero only when $P_M(s)=1$ , a degenerate case). Thus $d_T$ is technically a pseudoultrametric on $V^*$ ; see [9] for the full metric space analysis.*
## 3 The Sequential Entropy Bound
### 3.1 Setup and Notation
Fix a transformer model $M$ with $L$ layers, $H_head$ heads, and head dimension $d$ . For a token sequence $t=(t_1,…,t_n)$ , let $k^(\ell)_i$ and $v^(\ell)_i$ denote the key and value vectors at layer $\ell$ and position $i$ . Denote the full KV state at position $i$ as:
$$
KV_i=\bigl(k^(\ell)_i,v^(\ell)_i\bigr)_\ell=1^L∈ℝ^2Ld.
$$
The entire cache after $n$ tokens is $KV_≤ n=(KV_1,…,KV_n)$ .
For a random token sequence drawn from the model’s own distribution $t∼ P_M$ , both the tokens and the KV vectors are random We write $H(·)$ for Shannon entropy and $H(·\mid·)$ for conditional entropy; since KV vectors are deterministic functions of the discrete token sequence, all entropies here are Shannon entropy.
### 3.2 The Determinism Lemma
The key structural observation is that KV vectors are deterministic functions of the token sequence. There is no randomness in a transformer’s forward pass given fixed weights and fixed inputs.
**Lemma 1 (KV determinism)**
*For a fixed model $M$ with deterministic forward pass, $KV_i$ is a deterministic function of the token prefix $(t_1,…,t_i)$ :
$$
KV_i=F_M(t_1,…,t_i)
$$
for some deterministic function $F_M:V^i→ℝ^2LH_headd$ .*
* Proof*
By definition of the transformer forward pass. Given a fixed sequence of input tokens $(t_1,…,t_i)$ , the embedding layer, all attention computations, and all projection matrices are deterministic (assuming no dropout or stochastic elements at inference time, which is standard). Therefore $W_K^(\ell)x^(\ell)_i$ and $W_V^(\ell)x^(\ell)_i$ are deterministic functions of $(t_1,…,t_i)$ . ∎
**Remark 2**
*Lemma 1 implies that all randomness in the KV cache traces back to randomness in the token sequence. The KV cache contains no additional information beyond what is in the tokens—it is a particular learned encoding of the token prefix. This is the key fact that makes sequential compression possible: we can exploit the token-level probability structure to bound the entropy of the KV vectors.*
### 3.3 The Main Bound
**Theorem 1 (Sequential entropy bound)**
*Let $t=(t_1,t_2,…)$ be a random token sequence drawn from $P_M$ . For any position $i≥ 2$ (the bound holds trivially at $i=1$ since $H(KV_1)≤ H(t_1)$ by Lemma 1 and data processing):
$$
H\bigl(KV_i\midKV_≤ i-1\bigr) ≤ H\bigl(t_i\mid t_1,…,t_i-1\bigr) = H\bigl(t_i\midKV_≤ i-1\bigr).
$$
The conditional entropy of the $i$ -th KV vector, given all prior cache entries, is bounded above by the model’s per-token surprisal at position $i$ .*
* Proof*
The proof proceeds in two steps: first establishing that $σ(KV_≤ i-1)=σ(t_≤ i-1)$ , then deriving the bound. Step 1: Injectivity and sigma-algebra equivalence. We establish that $t_j$ is a measurable function of $KV_≤ j$ for each $j$ , by showing that the map $t_j↦KV_j\mid t_<j$ is injective. Fix any context $t_<j$ and suppose $t_j≠ t_j^\prime$ . At layer $\ell=1$ , the key vector is $k^(1)_j=W_K^(1)E(t_j)$ . Since the embedding matrix $E:V→ℝ^d_\rm model$ has full column rank in any trained transformer (distinct tokens receive distinct embeddings; the set of weight matrices for which any two token embeddings coincide has measure zero in parameter space), we have $E(t_j)≠ E(t_j^\prime)$ , and since $W_K^(1)∈ℝ^d_\rm head× d_\rm model$ satisfies the generic condition that no pairwise embedding difference $E(t)-E(t^\prime)$ lies in $\ker W_K^(1)$ (this holds with probability 1 over random weight matrices since the null space has dimension $d_\rm model-d_\rm head$ and there are finitely many token pairs), it follows that $k^(1)_j(t_j)≠k^(1)_j(t_j^\prime)$ . In particular, $KV_j^(1)(t_j)≠KV_j^(1)(t_j^\prime)$ . Since $KV_j$ includes the layer-1 component $KV_j^(1)$ as a subvector, the full vector $KV_j$ distinguishes $t_j$ from $t_j^\prime$ . Thus $t_j↦KV_j\mid t_<j$ is injective for each $j$ . (No assumption about higher-layer projections is needed: injectivity is witnessed by the layer-1 keys alone.) By Lemma 1, $KV_j$ is measurable w.r.t. $σ(t_≤ j)$ for each $j$ . By the injectivity just established, $t_j$ is measurable w.r.t. $σ(KV_j,t_<j)$ . Applying this inductively: $t_1$ is determined by $KV_1^(1)$ (hence by $KV_1$ ); $t_2$ is determined by $(KV_2,t_1)$ , hence by $(KV_1,KV_2)$ ; and so on. Therefore $t_<i$ are jointly measurable w.r.t. $σ(KV_1,…,KV_i-1)$ , giving:
$$
σ(KV_≤ i-1)=σ(t_<i). \tag{1}
$$ Step 2: The bound. Since $KV_i=F_Mt_≤ i$ is a deterministic function of $(t_≤ i-1,t_i)$ , the data-processing inequality gives:
$$
H(KV_i\mid t_≤ i-1)≤ H(t_i\mid t_≤ i-1).
$$
Applying (1) to both sides (conditioning on $KV_≤ i-1$ is equivalent to conditioning on $t_≤ i-1$ ):
$$
H(KV_i\midKV_≤ i-1)=H(KV_i\mid t_≤ i-1)≤ H(t_i\mid t_≤ i-1)=H(t_i\midKV_≤ i-1),
$$
which is the stated inequality and equality simultaneously. ∎
**Corollary 1 (Per-token surprisal bound)**
*Let $PP(M)$ denote the perplexity of model $M$ on a text distribution $D$ . The average conditional KV entropy satisfies:
$$
\frac{1}{n}∑_i=1^nH\bigl(KV_i\midKV_≤ i-1\bigr) ≤ \log_2PP(M,D) bits per token position.
$$
At typical perplexities of 10–20 for fluent English:
$$
\frac{1}{n}∑_i=1^nH\bigl(KV_i\midKV_≤ i-1\bigr) ≤ 3.3 to 4.3 bits per token position.
$$*
* Proof*
By definition of perplexity, $\log_2PP(M,D)=\frac{1}{n}E\bigl[-\log_2P_M(t)\bigr]=\frac{1}{n}∑_i=1^nH(t_i\mid t_1,…,t_i-1)$ . Applying Theorem 1 position-wise and averaging yields the result. ∎
**Remark 3 (Comparison with TurboQuant)**
*TurboQuant stores approximately $b=3$ bits per component of each KV vector. For a 70B-scale model with $L=80$ layers, $H_head=64$ heads, and per-head dimension $d=128$ , the total bits per token position (all layers and heads, both keys and values) under TurboQuant is:
$$
B_\rm TQ=2LH_headd· b=2× 80× 64× 128× 3≈ 3.93× 10^6 bits/token.
$$
Corollary 1 bounds $H(KV_i\midKV_<i)$ for the full KV state (all layers, all heads) at $\bar{h}≤ 4.3$ bits per token position. The theoretical ratio is:
$$
\frac{B_\rm TQ}{\bar{h}}≈\frac{3.93× 10^6}{4.3}≈ 914{,}000×.
$$
A practical sequential coder at $10--20×$ above the entropy floor achieves $43$ – $87$ bits/token, giving a practical ratio of $3.93× 10^6/(43--87)≈ 45{,}000$ – $91{,}000×$ over TurboQuant. Even at $1000×$ the entropy floor—a deliberately pessimistic worst-case, two orders of magnitude above the $2$ – $5×$ overhead typical of practical arithmetic coders [12, 2] —the ratio is $≈ 900×$ over TurboQuant. This is the theoretical gap between the two Shannon limits; practical implementations will be less efficient (Section 7).*
**Corollary 2 (Compression ratio bounds)**
*For a 70B-scale model ( $L=80$ , $H_head=64$ , $d=128$ per head) at perplexity $PP∈[10,20]$ , so $\bar{h}≤\log_2PP∈[3.3,4.3]$ bits/token:
1. Theoretical vs. fp16: $B_\rm fp16=2LH_headd· 16≈ 2.10× 10^7$ bits/token; ratio $≈ 4.9× 10^6×$ at $PP=20$ .
1. Theoretical vs. TurboQuant ( $b=3$ ): $B_\rm TQ=2LH_headd· 3≈ 3.93× 10^6$ bits/token; ratio $≈ 914{,}000×$ at $PP=20$ .
1. Conservative practical vs. TurboQuant: At $1000×$ above the entropy floor: $B_\rm TQ/4300≈ 914×$ .
Items (a)–(b) are theoretical Shannon limits; item (c) uses $1000×$ overhead—a deliberately pessimistic worst-case bound, two orders of magnitude above the $2$ – $5×$ overhead typical of practical arithmetic coders [12, 2]. The abstract cites item (c) as a worst-case lower bound on the practical ratio.*
* Proof*
Items (a) and (b) follow by substituting the stated model parameters into the formula $B=2LH_headd· b$ and dividing by $\bar{h}$ . Item (c) applies a $1000×$ overhead factor to the entropy floor $\bar{h}=4.3$ bits/token, giving $4{,}300$ bits/token, then divides $B_\rm TQ$ by this value. ∎
### 3.4 Tightness of the Bound
The bound in Theorem 1 is not always tight. It is tight when $F_M$ is an injective function of $t_i$ alone given the prior context, and when all entropy in $KV_i$ traces back to the single-token choice. In practice, the bound can be loose for two reasons:
1. Positional and structural information. KV vectors encode not only the current token’s identity but also its position (via positional encoding) and layer-specific features. These components may carry additional entropy not captured by the token-level bound. However, positional encodings are deterministic functions of position $i$ and therefore carry zero entropy conditioned on knowing $i$ . Layer-specific features are also deterministic given the full token sequence.
1. High-entropy token positions. At positions where the model is genuinely uncertain (low probability assigned to the actual next token), the KV residual will be larger. Semantically surprising tokens—a new entity name, an unexpected topic shift, an unusual word choice— produce high-surprisal positions where the bound is weakest.
Conversely, the bound is nearly tight for long, coherent, predictable sequences—exactly the regime where long-context inference is most valuable. Legal documents, technical manuals, extended narratives, and code all exhibit low perplexity under a well-matched model, and therefore yield highly compressible KV caches under sequential coding.
## 4 Layer 1: Probabilistic Prefix Deduplication
### 4.1 The Exact Prefix Sharing Baseline
Existing systems such as vLLM [5] and SGLang implement exact prefix sharing: when two sessions begin with the identical token sequence, the KV cache for their shared prefix is computed once and referenced by pointer. This is valuable for fixed system prompts, few-shot examples, and common preambles.
Exact prefix sharing has a fundamental limitation: it operates at the lexical level. Two sessions beginning with “You are a helpful assistant.” and “You are an AI assistant.” share no bytes and therefore share no cache under exact prefix sharing. Yet their KV caches, layer by layer, will be very similar: both prompts steer the model toward a similar distribution over subsequent tokens. The difference in their KV vectors is small relative to the total vector magnitude.
### 4.2 The PLT Trie Metric as a Deduplication Criterion
The PLT trie metric (Definition 2) provides the right notion of distance for identifying deduplications that exact prefix sharing misses. Two prefixes $s$ and $s^\prime$ with small trie distance $d_T(s,s^\prime)$ share a long, high-probability common prefix. At all positions within that shared prefix, their KV vectors are identical by Lemma 1 (Proposition 1 (a)). At the divergence point, the KV difference is bounded by the embedding spread (Proposition 1 (b)). The trie metric thus directly controls KV cache similarity: small trie distance implies small KV delta, making probabilistic prefix deduplication well-founded.
We formalize this as follows.
**Definition 3 (Semantic prefix cluster)**
*Let $δ>0$ be a distance threshold and $S=\{s^(1),…,s^(m)\}$ a set of session prefixes. A semantic prefix cluster at threshold $δ$ is a maximal subset $C⊆S$ such that for all $s,s^\prime∈C$ :
$$
d_T(s,s^\prime)≤δ.
$$
The centroid of $C$ is the prefix $s^*=\arg\max_s∈CP_M(s)$ , i.e., the most probable prefix in the cluster.*
**Proposition 1 (KV similarity within a cluster)**
*Let $s$ and $s^\prime$ be two session prefixes that first diverge at position $\bar{d}+1$ (i.e., $s_j=s^\prime_j$ for $j≤\bar{d}$ and $s_\bar{d+1}≠ s^\prime_\bar{d+1}$ ). For a transformer with $κ=\max_\ellκ^(\ell)≥ 1$ (the maximum per-layer Lipschitz constant; explicit formulas in terms of weight matrix operator norms appear in a companion paper on certified inference caching, forthcoming on arXiv):
1. For all positions $i≤\bar{d}$ and all layers $\ell$ : $KV^(\ell)_i(s)=KV^(\ell)_i(s^\prime)$ .
1. At the divergence position $i=\bar{d}+1$ , for any layer $\ell$ :
$$
\bigl\|KV^(\ell)_\bar{d+1}(s)-KV^(\ell)_\bar{d+1}(s^\prime)\bigr\| ≤ κ^\ell·\|E(s_\bar{d+1})-E(s^\prime_\bar{d+1})\|,
$$
where $E(·)$ is the token embedding map and $κ^\ell$ denotes $κ$ raised to the $\ell$ -th power (not a layer index).*
* Proof*
Part (a). For positions $i≤\bar{d}$ , the token inputs are identical ( $s_j=s^\prime_j$ for all $j≤\bar{d}$ ), so by Lemma 1 (KV determinism) the KV vectors agree exactly at every layer. $\checkmark$ Part (b). At position $\bar{d}+1$ , the embedding layer introduces a difference: $\|x^(0)_\bar{d+1}(s)-x^(0)_\bar{d+1}(s^\prime)\|=\|E(s_\bar{d+1})-E(s^\prime_\bar{d+1})\|$ . Positions $1,\ ldots,\ bar{d}$ were identical, so their KV vectors—which contribute to position $\bar{d}+1$ ’s attention output—are identical, and introduce no additional error. Define $κ$ as the Lipschitz constant of the composed per-layer map $x^(\ell)↦x^(\ell+1)$ at position $\bar{d}+1$ , including the attention, MLP, layer-norm, and key/value projections (explicit formulas in terms of weight matrix operator norms are given in a companion paper on certified inference caching, forthcoming on arXiv). Applying the Lipschitz bound across layers $1$ through $\ell$ :
$$
\|x^(\ell)_\bar{d+1}(s)-x^(\ell)_\bar{d+1}(s^\prime)\|≤κ^\ell·\|E(s_\bar{d+1})-E(s^\prime_\bar{d+1})\|.
$$
Since the KV projection $x^(\ell)↦KV^(\ell)$ is itself part of the per-layer map with norm absorbed into $κ$ :
$$
\|KV^(\ell)_\bar{d+1}(s)-KV^(\ell)_\bar{d+1}(s^\prime)\|≤κ^\ell·\|E(s_\bar{d+1})-E(s^\prime_\bar{d+1})\|. \checkmark
$$
∎
**Remark 4**
*Proposition 1 shows that exact prefix sharing is a special case: when $s$ and $s^\prime$ are identical up to position $i$ , the KV delta is exactly zero. Probabilistic prefix deduplication generalizes this to the case where $s$ and $s^\prime$ diverge at some point but remain semantically close. In this case the delta is nonzero but small—and can itself be quantized and compressed.*
### 4.3 Storage Under Probabilistic Prefix Deduplication
**Definition 4 (Cluster-relative storage)**
*Given a semantic prefix cluster $C$ with centroid $s^*$ , store:
1. The full KV cache $KV_≤ n(s^*)$ for the centroid.
1. For each $s∈C∖\{s^*\}$ , the delta cache $Δ(s,s^*)=KV_≤ n(s)-KV_≤ n(s^*)$ (zero at all positions before the divergence of $s$ and $s^*$ ).*
The key observation is that $Δ(s,s^*)$ is sparse: it is exactly zero at all positions before the divergence of $s$ and $s^*$ , and typically small after the divergence (by Proposition 1). The storage cost of $Δ(s,s^*)$ therefore scales with the length of the tail of $s$ after divergence from $s^*$ , not the full length of $s$ .
**Corollary 3 (Storage reduction from prefix deduplication)**
*Suppose sessions are drawn i.i.d. from $P_M$ and a cluster at threshold $δ$ covers a fraction $f$ of all sessions, with average tail length $\bar{\ell}$ after the shared prefix. The storage cost per session, relative to storing full KV caches independently, is:
$$
relative cost=(1-f)· 1+f·\frac{\bar{\ell}}{n}=1-f≤ft(1-\frac{\bar{\ell}}{n}\right).
$$
When $\bar{\ell}\ll n$ (sessions diverge early), the saving is approximately $f$ , i.e., the number of full caches stored decreases by a factor of $f$ .*
* Proof*
Measure storage in units of one full $n$ -position KV cache. Of all sessions, fraction $(1-f)$ lie outside the cluster and each requires 1 full unit. Fraction $f$ lie inside the cluster: their shared prefix $(n-\bar{\ell}$ positions) is stored once in the centroid and amortises across all cluster members, so each member stores only its $\bar{\ell}$ -position tail at cost $\bar{\ell}/n$ . The average per-session storage is:
$$
(1-f)· 1 + f·\frac{\bar{\ell}}{n}=1-f≤ft(1-\frac{\bar{\ell}}{n}\right),
$$
which is the stated bound. As $\bar{\ell}/n→ 0$ (sessions diverge early from the centroid) the saving approaches $f$ , i.e., a factor- $f$ reduction in the number of full caches stored. ∎
In practice, for chat-model deployments with a fixed system prompt and few-shot examples, $f$ can be close to 1 (nearly all sessions share the same prefix) and $\bar{\ell}/n$ can be small (the session-specific content is a small fraction of the context). Even for sessions with different system prompts, probabilistic clustering under the PLT metric groups semantically similar preambles and captures deduplication that exact matching misses entirely.
## 5 Layer 2: Predictive Delta Coding
### 5.1 The Prediction
At each position $i$ , before writing $KV_i$ to cache, the model has already computed its probability distribution over the $i$ -th token:
$$
P_M(t_i\mid t_1,…,t_i-1)=softmax(h^(L)_i-1W_\rm LM)_vocabulary,
$$
where $h^(L)_i-1∈ℝ^d_\rm model$ is the final-layer hidden state at position $i-1$ and $W_\rm LM∈ℝ^d_\rm model×|V|$ is the language model head (unembedding matrix). This distribution induces a predicted KV vector for position $i$ : the expected KV vector under the model’s distribution over possible next tokens.
**Definition 5 (Predicted KV vector)**
*The predicted KV vector at position $i$ , given context $(t_1,…,t_i-1)$ , is:
$$
\widehat{KV}_i=∑_t∈ VP_M(t\mid t_1,…,t_i-1)· F_M(t_1,…,t_i-1,t).
$$*
In practice, computing $\widehat{KV}_i$ exactly requires a forward pass for every vocabulary token, which is prohibitively expensive. Section 7 discusses efficient approximations using the top- $k$ tokens under the model’s distribution.
### 5.2 The Residual and Its Entropy
**Definition 6 (KV residual)**
*The KV residual at position $i$ is:
$$
R_i=KV_i-\widehat{KV}_i.
$$
Predictive delta coding stores $R_i$ (compressed) rather than $KV_i$ directly.*
**Theorem 2 (Residual entropy bound)**
*The conditional entropy of the KV residual satisfies:
$$
H(R_i\mid t_1,…,t_i-1) ≤ H\bigl(KV_i\mid t_1,…,t_i-1\bigr) ≤ H\bigl(t_i\mid t_1,…,t_i-1\bigr).
$$
Moreover, at positions where the model assigns high probability to the actual next token, the residual concentrates near zero:
$$
E\bigl[\|R_i\|_2^2\mid t_1,…,t_i-1\bigr] ≤ Var_t∼ P_{M(·\mid t_<i)}\bigl[F_M(t_<i,t)\bigr],
$$
which is small when $P_M(·\mid t_<i)$ is concentrated.*
* Proof*
The first inequality follows from the fact that $R_i=KV_i-\widehat{KV}_i$ and $\widehat{KV}_i$ is a deterministic function of $t_<i$ . Therefore $H(R_i\mid t_<i)=H(KV_i-\widehat{KV}_i\mid t_<i)=H(KV_i\mid t_<i)$ , since subtracting a constant does not change entropy. The second inequality is Theorem 1. For the variance bound: by Definition 5, $\widehat{KV}_i=E_t∼ P[F(t_<i,t)]$ , so $R_i=F(t_<i,t_i)-E[F(t_<i,t)]$ . Therefore $E[\|R_i\|^2\mid t_<i]=Var_t∼ P[F(t_<i,t)]$ . ∎
**Corollary 4 (Coherent text compression)**
*Let $H_i=H(t_i\mid t_<i)$ denote the conditional entropy of the next token given the context (not the realized surprisal $h_i=-\log_2P_M(t_i\mid t_<i)$ ). The expected residual magnitude satisfies:
$$
E\bigl[\|R_i\|_2 \big| t_<i\bigr] ≤ \tfrac{1}{2}\|F_M\|_Lip· C_E·√{\min(1, 4 H_i\ln 2)},
$$
where $\|F_M\|_Lip$ is the Lipschitz constant of $F_M$ with respect to the input embedding $E(t)$ , and $C_E=\max_t,t^\prime∈ V\|E(t)-E(t^\prime)\|$ . For low-entropy positions ( $H_i≈ 0$ , i.e., the model is near-certain about the next token), the expected residual approaches zero; for high-entropy positions it can be as large as $\tfrac{1}{2}\|F_M\|_Lip· C_E$ .*
* Proof*
By Theorem 2, $E[\|R_i\|_2^2\mid t_<i]=Var_t∼ P[F_M(t_<i,t)]$ . By Jensen’s inequality applied to the square root: $E[\|R_i\|_2]≤√{Var[F_M(t_<i,t)]}$ . The variance of $F_M(t_<i,t)$ over $t∼ P(·\mid t_<i)$ satisfies, by the Lipschitz property of $F_M$ in the embedding $E(t)$ :
$$
Var_t∼ P[F_M(t_<i,t)]≤\|F_M\|_Lip^2·Var_t∼ P[E(t)].
$$
For the embedding variance, we derive a bound that decays with conditional entropy $H_i$ using two standard inequalities. Step A (Popoviciu). For any random variable $Y$ with $\|Y-c\|≤ R$ almost surely (for some center $c$ and radius $R$ ), $Var[Y]=E[\|Y-E[Y]\|^2]≤E[\|Y-c\|^2]≤ R^2$ . Applied to $Y=E(t)$ with center $c=(E(t^*)+E(t^**))/2$ and radius $R=C_E/2$ (half the $\ell^2$ diameter): $Var[E(t)]≤ C_E^2/4$ .
$$
Var_t∼ P[E(t)]≤\frac{C_E^2}{4}.
$$ Step B (variance–entropy coupling). We sharpen Step A to a bound that vanishes as $H_i→ 0$ . Since $\max_tP(t)≥ 2^-H(P)$ (from Jensen applied to concave $\log$ : $H(P)≤-\log\max_tP(t)$ ), we have $1-\max_tP(t)≤ 1-2^-H(P)≤ H(P)\ln 2=H_i\ln 2$ , where the last step uses $1-e^-x≤ x$ . By the König–Huygens identity, $Var[E(t)]=\tfrac{1}{2}E_t,t^\prime∼ P[\|E(t)-E(t^\prime)\|^2]≤\tfrac{C_E^2}{2}(1-∑_tP(t)^2)$ , where the last step uses $\|E(t)-E(t^\prime)\|^2≤ C_E^2$ for all $t,t^\prime$ . Since $∑_tP(t)^2≥ p_\max^2$ , we have $1-∑_tP(t)^2≤ 1-p_\max^2=(1-p_\max)(1+p_\max)≤ 2(1-p_\max)$ , giving $Var[E(t)]≤ C_E^2(1-p_\max)≤ C_E^2H_i\ln 2$ . Combining with the Popoviciu bound from Step A via the minimum:
$$
Var_t∼ P[E(t)]≤\frac{C_E^2}{4}·\min\bigl(1, 4H_i\ln 2\bigr).
$$ Combining. Substituting into the Lipschitz bound and taking the square root:
$$
E[\|R_i\|_2]≤√{\|F_M\|_Lip^2·\frac{C_E^2}{4}·\min(1,4H_i\ln 2)}=\frac{1}{2}\|F_M\|_Lip· C_E·√{\min(1,4H_i\ln 2)}.
$$
For $H_i$ small (near-certain next token): $√{\min(1,4H_i\ln 2)}≈ 2√{H_i\ln 2}$ , giving $E[\|R_i\|_2]≤sssim\|F_M\|_Lip· C_E·√{H_i\ln 2}$ , which vanishes as $H_i→ 0$ and is $O(√{H_i})$ . ∎
### 5.3 Adaptive Quantization of Residuals
Because residual magnitude varies by position—small at predictable positions, large at surprising ones—a fixed bit-depth quantizer is suboptimal. The natural approach is adaptive quantization: use fewer bits for small residuals and more bits for large ones, proportional to the per-token surprisal.
**Definition 7 (Surprisal-adaptive quantizer)**
*At position $i$ with surprisal $h_i$ , allocate $b_i$ bits per residual component, where:
$$
b_i=\max≤ft(1, ≤ft\lfloor b_0·\frac{h_i}{\bar{h}}\right\rfloor\right),
$$
with $b_0$ a target average bit depth and $\bar{h}$ the average surprisal.*
This adaptive scheme achieves the average entropy bound of Corollary 1 while concentrating bits where they are needed, improving the worst-case accuracy at high-surprisal positions relative to a uniform low-bit quantizer.
## 6 Composition and the Full Stack
### 6.1 Orthogonality of the Two Layers
**Proposition 2 (Layer orthogonality)**
*Probabilistic prefix deduplication (Section 4) and predictive delta coding (Section 5) are information-theoretically orthogonal: they exploit statistically independent sources of redundancy.*
* Proof*
We show that the total description length decomposes additively, with no bits counted twice. Let $s$ be a session drawn from $P_M$ and write its KV cache as $KV_≤ n(s)=(KV_1(s),…,KV_n(s))$ . Let $s^*$ be the cluster centroid for $s$ (Definition 3), and let $\bar{d}$ be the token position at which $s$ and $s^*$ first diverge, so that $s_j=s^*_j$ for $j≤\bar{d}$ and $s_\bar{d+1}≠ s^*_\bar{d+1}$ . Description of $KV_≤ n(s)$ in the full stack. By Definition 4, the encoder writes:
- Prefix (positions $1≤ i≤\bar{d}$ ): zero bits, since $KV_i(s)=KV_i(s^*)$ exactly (Proposition 1 (a)) and the centroid is stored separately.
- Tail (positions $\bar{d}+1≤ i≤ n$ ): for each position $i$ in the tail, the encoder applies predictive delta coding (Definition 6) and stores $R_i(s)$ , using a code of length at most $H(R_i\midKV_<i)+O(1)$ bits by standard source coding [2]. Rate decomposition. The total rate for session $s$ is:
$$
\displaystyle L(s) \displaystyle=\underbrace{0}_prefix, Layer 1+\underbrace{∑_i=\bar{d+1}^n\bigl[H(R_i\midKV_<i(s))+O(1)\bigr]}_tail, Layer 2. \tag{1}
$$
These two terms address disjoint index sets ( $\{1,…,\bar{d}\}$ and $\{\bar{d}+1,…,n\}$ respectively) within the description of $s$ ’s cache. No position $i$ is compressed by both layers simultaneously: Layer 1 eliminates positions $i≤\bar{d}$ entirely by pointer to the centroid; Layer 2 applies only to positions $i>\bar{d}$ . Therefore the bits saved by each layer are non-overlapping, and the total saving is their sum. Independence of gain statistics. The gain of Layer 1 (the fraction $f$ of full caches replaced by shared-prefix pointers, Corollary 3) depends on the overlap structure of sessions under $P_M$ —a property of the joint distribution over session pairs. The gain of Layer 2 at each tail position $i$ (the residual entropy $H(R_i\midKV_<i)$ ) depends on the per-token surprisal $h_i$ within a single session—a marginal property of the language model’s token-level distribution. These statistics are functions of different aspects of $P_M$ and are not constrained to be equal or proportional, so improvements to one layer do not mechanically alter the gain of the other. This is the sense in which the two layers are “information-theoretically orthogonal”: each exploits a distinct, non-overlapping portion of the total redundancy in the KV cache. ∎
### 6.2 Composition with Per-Vector Quantization
After applying the two sequential layers, the residuals $R_i$ are vectors that still need to be stored in finite-precision format. This is where per-vector quantization methods, including TurboQuant, apply.
**Proposition 3 (Three-layer composition)**
*The full compression stack proceeds as follows:
1. Prefix deduplication: identify the shared prefix with the cluster centroid; store only the tail KV delta.
1. Predictive delta coding: for each token in the tail, store the residual $R_i=KV_i-\widehat{KV}_i$ .
1. Per-vector quantization (e.g., TurboQuant): represent each $R_i$ in reduced precision.*
* Proof*
Steps 1 and 2 are exactly Definitions 4 and 6. Step 3 applies any per-vector quantizer to $R_i$ . Correctness: $KV_i$ is recovered as $R_i+\widehat{KV}_i$ , where $\widehat{KV}_i$ is recomputed deterministically from prior context. The quantization error on the recovered $KV_i$ equals the quantization error on $R_i$ . Rate improvement: By Corollary 4, $E[\|R_i\|_2]≤\tfrac{1}{2}\|F\|_\rm LipC_E√{\min(1,4H_i\ln 2)}$ . For low-surprisal tokens ( $H_i$ small), $\min(1,4H_i\ln 2)≈ 4h_i\ln 2$ , so $E[\|R_i\|_2]≤sssim\|F\|_\rm LipC_E√{H_i\ln 2}→ 0$ . Because $R_i$ is small in magnitude, the dynamic range needed to represent it is small, and a fixed-precision quantizer with absolute error $ε$ uses effectively $\log_2(E[\|R_i\|_2]/ε)$ bits—which decreases as $H_i→ 0$ . Therefore at low-surprisal positions the residual is small enough that coarse quantization suffices, making the three-layer stack strictly more bit-efficient than quantizing $KV_i$ directly (whose magnitude does not vanish at low surprisal). ∎
### 6.3 Asymptotic Behavior with Context Length
**Corollary 5 (Compression improves with context length)**
*Let $\bar{H}_n=\frac{1}{n}∑_i=1^nH_i$ be the average conditional entropy (where $H_i=H(t_i\mid t_<i)$ ) over a context of length $n$ . By Corollary 1:
$$
\frac{1}{n}∑_i=1^nH(KV_i\midKV_<i)≤\bar{H}_n.
$$
Monotone entropy condition: call a context coherent if its conditional entropy sequence $(H_i)$ is non-increasing: $H_i+1≤ H_i$ for all $i$ (later tokens are more predictable given more context). Under this condition, $\bar{H}_n+1=(n\bar{H}_n+H_n+1)/(n+1)≤\bar{H}_n$ since $H_n+1≤\bar{H}_n$ . Thus the sequential compression bound tightens monotonically with context length. Empirical scope. Strict pointwise monotonicity does not hold universally: surprisal spikes at topic shifts, new entity names, and dialogue turn boundaries. The condition holds in expectation for long, topically stable documents (legal text, technical manuals, extended code), where it is the most practically relevant regime for long-context inference. For sessions with high surprisal variance, the asymptotic improvement applies to the running average $\bar{h}_n$ rather than individual positions (and using conditional entropy $H_i$ rather than the realized surprisal $h_i$ , which is equal only in expectation). By contrast, per-vector methods (TurboQuant, KIVI, KVQuant) apply a fixed bit-depth $b$ per component regardless of $n$ : their memory cost is $2LH_headdn· b$ bits, growing at constant rate $2LH_headdb$ bits per token. Under the monotone surprisal condition, sequential compression’s marginal rate per token is $H_n=H(t_n\mid t_<n)$ , which is non-increasing when the coherence condition holds in expectation. Long-context inference becomes cheaper per marginal token under sequential compression.*
* Proof*
The bound $\frac{1}{n}∑ H(KV_i\midKV_<i)≤\bar{H}_n$ is Corollary 1. The monotone decrease of $\bar{H}_n$ under the stated condition follows because $H_n+1≤ H_n≤\bar{H}_n$ implies $\bar{H}_n+1=\frac{n\bar{H}_n+H_n+1}{n+1}≤\frac{n\bar{H}_n+\bar{H}_n}{n+1}=\bar{H}_n$ . The comparison to per-vector methods is immediate from $B_\rm pv=2LH_headd_head· b· n$ (constant rate per token). ∎
This is a qualitative reversal of the current situation. Under TurboQuant or any per-vector method, memory grows linearly with context length at a fixed rate. Under sequential compression, the marginal cost per token decreases as context length grows, because the model’s growing context makes subsequent tokens more predictable. Long-context inference becomes cheaper, not more expensive, per marginal token.
## 7 Practical Implementation
We now discuss how to implement sequential KV compression efficiently, addressing the main engineering challenges.
### 7.1 Efficient Prediction Computation
The predicted KV vector (Definition 5) requires summing over all $|V|≈ 50{,}000$ vocabulary tokens, each requiring a forward pass. This is prohibitively expensive. We propose two approximations:
Top- $k$ approximation. Use only the top- $k$ tokens by probability:
$$
\widehat{KV}_i^(k)=\frac{∑_t∈top_kP_M(t\mid t_<i)· F_M(t_<i,t)}{∑_t∈top_kP_M(t\mid t_<i)}.
$$
At low perplexity, 95%+ of the probability mass is concentrated in the top 5–20 tokens, so $k=20$ captures the prediction accurately while requiring only 20 forward pass evaluations instead of 50,000.
Linear embedding approximation. For each layer $\ell$ , approximate the KV function as linear in the input embedding:
$$
F_M^(\ell)(t_<i,t)≈c^(\ell)(t_<i)+A^(\ell)(t_<i)E(t),
$$
where $c^(\ell)$ and $A^(\ell)$ are computed once per position from the model’s internal state. Under this approximation, $\widehat{KV}_i≈c(t_<i)+A(t_<i)\bar{E}_i$ , where $\bar{E}_i=E_t∼ P[E(t)]$ is the expected embedding— computable as a single weighted sum over the vocabulary using the softmax output.
### 7.2 Integration with the Inference Loop
Unlike TurboQuant, which operates as a post-hoc quantizer with no changes to the model’s forward pass, predictive delta coding requires access to the softmax distribution at each position. This distribution is already computed during inference (it is the output of the final language model head before sampling).
The integration points are:
1. Before writing KV i to cache: compute $\widehat{KV}_i$ using the top- $k$ approximation; subtract to obtain $R_i$ ; quantize and write $R_i$ .
1. Before reading KV i from cache: dequantize $R_i$ ; add back $\widehat{KV}_i$ (recomputed identically, since it depends only on prior context which is available); return $KV_i$ .
The additional compute cost per token is $O(k· d_model)$ for the top- $k$ prediction, where $k\ll|V|$ . This is negligible relative to the $O(n· d_model)$ attention computation.
### 7.3 Trie-Based Prefix Index
Probabilistic prefix deduplication requires a data structure that supports:
1. Insert: add a new session prefix and its KV cache.
1. Best-match lookup: given a new session prefix $s$ , find the stored prefix $s^*$ maximizing $d_T(s,s^*)$ (equivalently, maximizing the probability of the longest common prefix).
1. Evict: remove a stored prefix and its KV cache.
The PLT trie naturally supports all three operations. The trie is maintained as a sparse structure over the high-probability prefix tree, pruned at nodes with probability below a threshold $ε$ . Nearest-neighbor lookup is a trie traversal: follow the path of the query sequence until it diverges from all stored paths, and return the deepest stored ancestor. Insert adds a new path; evict removes a path and potentially prunes ancestors with no other children.
All three operations run in $O(n)$ time in the length of the prefix, with $O(K)$ space for $K$ stored prefixes.
## 8 Related Work
KV cache quantization. KIVI [8] introduced 2-bit quantization with per-channel scaling. KVQuant [4] handled outliers via non-uniform quantization. TurboQuant [13] unified these approaches via PolarQuant rotation and proved a per-vector Shannon limit, which our work extends to the sequential setting.
KV cache eviction. H2O [14] identified that attention scores concentrate on a small “heavy hitter” subset of tokens and evicts the rest. SnapKV [7] uses prefill-time attention patterns to predict which tokens will be important for generation. Eviction is orthogonal to compression: it reduces cache size by discarding tokens; compression reduces cache bit depth while retaining all tokens. Both approaches compose naturally with sequential compression.
Prefix sharing. vLLM [5] introduced paged attention with exact prefix sharing. SGLang extended this with a radix tree for multi-prompt prefix reuse. Our probabilistic prefix deduplication generalizes these to semantic similarity under the PLT metric [9].
Probabilistic Language Tries. The PLT framework [9] introduced the trie metric, proved the prior-guided caching theorem, and identified prefix KV-caching as a special case of PLT-based artifact memoization. The present paper applies the trie metric specifically to cross-session KV deduplication and proves the sequential entropy bound connecting token-level surprisal to KV vector entropy.
Language modeling and compression. Delétang et al. [3] demonstrated that LLMs implicitly perform arithmetic coding and achieve state-of-the-art compression on text corpora. Shannon [11] estimated the true entropy of English at 1.3 bits per character via human prediction experiments. Our work draws on both results: the model’s near-optimal compression of its training distribution implies near-zero conditional entropy for KV vectors at low-surprisal positions.
## 9 Discussion
### 9.1 The Two Shannon Limits
It is worth being precise about what we have and have not shown. TurboQuant proves a lower bound on the per-vector compression rate and shows it is nearly achieved. We prove that the per-sequence Shannon limit is lower, bounded by the per-token surprisal. These are genuinely different limits, and TurboQuant’s proof does not address ours.
The per-sequence limit is lower because it exploits correlations across token positions —specifically, the fact that the KV vector at position $i$ is largely predictable from the KV vectors at positions $1$ through $i-1$ . Per-vector methods, by definition, cannot exploit these correlations.
### 9.2 Implications for the “Memory Wall”
The KV cache memory wall—the observation that cache size grows linearly with context length and at long contexts dominates model weight memory—is often framed as a hardware problem awaiting the next generation of memory technology. Sequential compression suggests a complementary software path: the wall is lower than it appears, because the marginal cost of each additional token in a coherent context is bounded by that token’s surprisal, which decreases with context length.
This does not eliminate the need for better hardware. But it suggests that the software path to longer effective context windows is not exhausted by per-vector quantization.
### 9.3 Relation to the Jevons Paradox
Improvements in inference efficiency historically increase total demand rather than decreasing it: cheaper inference enables more users, longer contexts, and more ambitious applications. Sequential compression, if realized, would continue this pattern. The reduction in marginal KV cost at long contexts would make currently-impractical applications—real-time million-token document analysis, persistent agent memory, continual learning from interaction— economically viable, driving total memory demand upward even as per-token cost falls.
### 9.4 Rate-Distortion Analysis of KV Compression
The sequential entropy bound (Theorem 1) is a lossless result: it characterizes the minimum bits needed for exact reconstruction of KV vectors. Practical systems accept bounded reconstruction error; the relevant quantity is the rate-distortion function $R_i(D)$ for position $i$ at distortion level $D$ .
**Theorem 3 (KV Rate-Distortion Bound)**
*For position $i$ with conditional entropy $H_i=H(t_i\mid t_<i)$ and residual variance $σ_i^2=Var_t∼ P[F_M(t_<i,t)]$ , the rate-distortion function (minimum rate to reconstruct $KV_i$ with mean squared error $≤ D$ per component) satisfies:
$$
R_i(D) ≤ \frac{d_head}{2}\log_2≤ft(\frac{σ_i^2}{D}\right)_+,
$$
where $(x)_+=\max(0,x)$ and the inequality uses the Gaussian upper bound from the KLT (Karhunen–Loève transform). Since $σ_i^2≤\frac{1}{4}\|F_M\|_Lip^2· C_E^2·\min(1,4H_i\ln 2)$ (Corollary 4):
$$
R_i(D) ≤ \frac{d_head}{2}≤ft[\log_2≤ft(\frac{\|F_M\|_Lip^2C_E^2H_i\ln 2}{D}\right)\right]_+.
$$
In particular, for a target distortion $D$ satisfying $D≥\|F_M\|_Lip^2C_E^2H_i\ln 2$ (distortion exceeds the residual variance), the rate is zero: no bits are needed at low-surprisal positions.*
* Proof*
$KV_i$ conditioned on $t_<i$ is a deterministic function of the discrete random variable $t_i$ . Since $t_i$ has $|V|$ possible values, the distribution of $KV_i$ has at most $|V|$ support points—a discrete distribution over a finite set of vectors. The rate-distortion function for this source is bounded above by the Gaussian rate-distortion function with the same second moment (Shannon’s source coding theorem: Gaussian maximizes R-D at fixed variance). After a KLT rotation diagonalizing $Cov[KV_i]$ with eigenvalues $λ_j$ : $R_i(D)≤\frac{1}{2}∑_j\log_2(λ_j/μ)_+$ (water-filling, where $μ$ satisfies $∑_j\min(λ_j,μ)=D$ ). This is at most the isotropic upper bound (all $λ_j=σ_i^2/d_head$ ), which gives $R_i(D)≤(d_head/2)\log_2(σ_i^2/D)_+$ . (The actual water-filling rate can only be smaller when eigenvalues are non-uniform.) Substituting the variance bound from Corollary 4 completes the proof. $\square$ ∎
**Corollary 6 (Practical Bit Allocation)**
*For a target distortion $D$ per KV component, the optimal bit allocation across positions assigns $b_i=R_i(D)/d_head$ bits per component at position $i$ . The total bits per token is:
$$
B_total(D)=∑_i=1^nR_i(D) ≤ \frac{d_head}{2}∑_i=1^n≤ft[\log_2≤ft(\frac{\|F\|_Lip^2C_E^2H_i\ln 2}{D}\right)\right]_+.
$$
At the TurboQuant operating point ( $b=3$ bits per component, uniform quantizer step $Δ_q=C_E/2^b$ , giving distortion $D≈Δ_q^2/12=C_E^2/(12· 4^b)$ ): positions with $H_i≤ D/(\|F\|_Lip^2C_E^2\ln 2)=1/(12· 4^b·\|F\|_Lip^2\ln 2)$ need zero bits (residual variance is below the quantization noise floor). The waterfilling allocation assigns proportionally more bits to high-surprisal positions and zero bits to low-surprisal ones, strictly dominating the uniform- $b$ -bit allocation of per-vector methods.*
### 9.5 Connection to Speculative Decoding
Speculative decoding [6] uses a small draft model $M_draft$ to generate $K$ candidate tokens, then verifies them in a single forward pass of the large model $M$ . The acceptance rate of token $t$ under speculative decoding is $\min(1,P_M(t\mid t_<i)/P_draft(t\mid t_<i))$ , averaging to $1-TV(P_M,P_draft)$ where TV is the total variation distance.
**Theorem 4 (Speculative Decoding–KV Compression Duality)**
*Let $\widehat{KV}_i^(k)$ be the top- $k$ approximation to the predicted KV vector (Definition 5). Define the draft model $P_draft=$ the renormalized top- $k$ distribution of $M$ . Then:
1. The speculative decoding acceptance rate of $P_draft$ equals $∑_t∈top_kP_M(t\mid t_<i)$ .
1. The conditional residual variance for tokens in the top- $k$ set satisfies:
$$
Var\bigl[F_M(t_<i,t)\mid t∈top_k\bigr] ≤ \|F_M\|_Lip^2·Var_t∈top_k[E(t)].
$$
This equals zero at $k=1$ (single token, deterministic). For $k>1$ the variance is not monotone in $k$ in general, but is bounded above by $\|F_M\|_Lip^2· C_E^2/4$ for all $k$ .
1. Both speculative decoding and KV residual prediction use the same top- $k$ distribution of $M$ . The draft forward pass therefore serves both objectives simultaneously at no extra cost: computing $P_draft$ yields both the speculative token candidates and the predicted KV mean $\widehat{KV}_i^(k)$ . As $k$ increases, acceptance rate increases; the residual variance at covered positions is not guaranteed to be monotone, but is bounded by part (b).*
* Proof*
(a) With $P_draft(t)=P_M(t)/Z_k$ for $t∈top_k$ and $Z_k=∑_t∈top_kP_M(t)≤ 1$ , each $\min(1,P_M(t)/P_draft(t))=\min(1,Z_k)=Z_k$ , so acceptance rate $=∑_top_kZ_k· P_M(t)/Z_k=Z_k$ . $\checkmark$ (b) Apply Corollary 4 to the conditional distribution $t\mid t∈top_k$ . At $k=1$ : point mass, zero variance. For general $k$ : the variance is bounded by $\|F_M\|_Lip^2· C_E^2/4$ (Popoviciu, since embeddings have diameter $C_E$ ). $\checkmark$ (c) Both applications share the same top- $k$ probability computation from the model’s output distribution. No additional inference is needed to obtain the KV predictor once $P_draft$ is computed. $\square$ ∎
**Remark 5 (Practical Implication)**
*Theorem 4 shows that a system implementing top- $k$ speculative decoding can reuse the same draft distribution to compute the predictive KV residuals, with zero additional inference overhead. The draft forward pass serves double duty: generating speculative tokens and computing the predicted KV mean $\widehat{KV}_i^(k)$ . Systems such as Medusa [1] or EAGLE already compute top- $k$ token distributions as a byproduct of their draft heads; they can be extended to simultaneously output KV predictors at negligible additional cost.*
### 9.6 Limitations and Open Problems
The main open question is the tightness of the sequential entropy bound in practice. The theoretical bound (Corollary 1) is tight when KV vectors are injective functions of the token sequence, which requires the weight matrices to be non-degenerate. Empirical measurement of actual residual magnitudes and their compression ratios under practical quantizers is needed to assess how close to the bound a real implementation can get.
A second open question is the efficient computation of the top- $k$ prediction. Current transformer inference frameworks do not expose the internal KV computation in a way that easily supports top- $k$ predictive averaging. Kernel-level integration (e.g., via Triton or CUDA) would be needed for production deployment.
A third open question concerns non-autoregressive and non-transformer architectures. The bound in Theorem 1 holds for any model satisfying the determinism condition of Lemma 1. State-space models (SSMs, Mamba) maintain a fixed-size state rather than a KV cache, but the same principle applies: the state entropy at each step is bounded by the token-level surprisal.
Future directions include:
- Empirical benchmarks: measure actual residual magnitudes on public models (Llama, Gemma, Mistral) and text distributions, and compare achieved compression to the theoretical bound.
- Kernel implementation: implement predictive delta coding as a Triton kernel, building on the open-source TurboQuant implementations.
- Hierarchical prediction: use not just the model’s next-token distribution but also its longer-range predictions (beam search distributions, speculative decoding candidates) to improve prediction accuracy and reduce residuals further.
- Lossy sequential compression: extend the rate-distortion analysis of PLTs [9] to the KV cache setting, allowing bounded distortion in exchange for additional compression beyond the lossless sequential limit.
### 9.7 Conjectures and Open Problems
**Conjecture 1 (Tightness of the Sequential Bound for Well-Trained Models)**
*For a transformer $M$ trained to near-zero cross-entropy on a corpus with stationary distribution $P$ , the sequential entropy bound is tight to within a constant factor:
$$
H\bigl(KV_i\midKV_<i\bigr) ≥ c· H\bigl(t_i\mid t_<i\bigr)
$$
for some universal constant $c>0$ that does not depend on $M$ or $i$ . Evidence and partial proof sketch. The lower bound direction is the hard part. An upper bound follows from Theorem 1. For the lower bound: since $KV_i$ is an injective function of $t_i$ given $t_<i$ (Lemma 1), the data-processing inequality gives $H(KV_i\midKV_<i)=H(t_i\midKV_<i)=H(t_i\mid t_<i)$ (using sigma-algebra equivalence from the proof of Theorem 1). The bound is therefore already tight: equality holds throughout. This is not a conjecture but an observation implicit in the proof, stated here to clarify the theoretical status: the sequential entropy bound is exact (equality, not inequality). The open question is whether practical quantizers can approach the bound, which requires answering: how much entropy do the residuals $R_i$ carry at low-surprisal positions under realistic token distributions? Equivalently: is $Var_t∼ P[F_M(t_<i,t)]$ small when $H_i$ is small for actual trained models?*
**Conjecture 2 (Cross-Session Prefix Entropy Bound)**
*For sessions $s,s^\prime$ drawn i.i.d. from $P_M$ , the conditional entropy of session $s$ ’s KV cache given the cluster centroid $s^*$ ’s full cache satisfies:
$$
H\bigl(KV_≤ n(s)\midKV_≤ n(s^*)\bigr) ≤ ∑_i=\bar{d+1}^nH\bigl(t_i^(s)\mid t_<i^(s)\bigr) + H\bigl(t_\bar{d+1}^(s)\mid t_\bar{d+1}^(s^{*)},t_<\bar{d+1}^(s)\bigr),
$$
where $\bar{d}$ is the divergence position of $s$ and $s^*$ . Partial proof sketch. The first term bounds the “tail” entropy of session $s$ after divergence: positions $i>\bar{d}$ are bounded by the sequential entropy bound exactly as in Theorem 1. The second term bounds the “divergence cost”: knowing $t_\bar{d+1}^(s^{*)}$ provides partial information about which subtree $s$ is in (via the cluster structure), reducing the entropy of $t_\bar{d+1}^(s)$ . The bound formalizes the Layer 1 gain: prefix deduplication saves bits on the tail (first term, bounded by per-token surprisal) and reduces divergence cost (second term, bounded by the trie metric $d_T(s,s^*)$ ). A formal proof requires bounding the conditional mutual information between the centroid’s divergence token and the session’s divergence token, which depends on the cluster radius $δ$ .*
**Conjecture 3 (Sequential Bound Strictly Dominates All Per-Vector Bounds)**
*For any per-vector compression scheme $C$ (including TurboQuant, KIVI, KVQuant, and all future schemes bounded by the per-vector Shannon limit), there exists a context length $n^*(C)$ such that for all $n>n^*(C)$ :
$$
R_seq(n) < R_C(n),
$$
where $R_seq(n)$ is the total bits used by sequential compression and $R_C(n)$ is the total bits used by scheme $C$ , both for a context of length $n$ . Partial proof. $R_C(n)≥ B_\rm pv· n$ for some $B_\rm pv>0$ (the per-vector Shannon limit is a hard floor for any per-vector scheme). $R_seq(n)=∑_i=1^nH(KV_i\midKV_<i)≤∑_i=1^nH_i$ . Under the monotone entropy condition (Corollary 5), $\frac{1}{n}∑_i=1^nH_i=\bar{H}_n→ H_∞<B_\rm pv$ as $n→∞$ (since $H_∞$ is the model’s entropy rate, which is strictly less than any fixed per-vector bit budget for a model with perplexity $<2^B_\rm pv/d_\rm head$ ). Thus $R_seq(n)/n→ H_∞<B_\rm pv≤ R_C(n)/n$ , giving the stated strict dominance for large $n$ . The threshold $n^*(C)=O(H_∞/(B_\rm pv-H_∞))$ . The main gap in this argument: the monotone entropy condition holds only for coherent text; for mixed-topic sessions with high surprisal variance, $\bar{H}_n$ may not converge below $B_\rm pv$ . Bounding $n^*$ for realistic session distributions is the key open problem.*
### 9.8 Conclusion
We have shown that the Shannon limit proved by TurboQuant applies to a per-vector problem, and that the Shannon limit for the sequential KV cache compression problem is strictly lower. The gap is the information-theoretic redundancy of language—the predictable structure that a near-optimal language model has already learned.
We formalized this through two results: the sequential entropy bound (Theorem 1), which connects KV vector entropy to per-token surprisal, and the PLT trie metric [9], which enables cross-session prefix deduplication in probability space.
The combined architecture—probabilistic prefix deduplication, predictive delta coding, and per-vector quantization—is theoretically capable of compression ratios far beyond what per-vector methods can achieve, with the compression ratio improving rather than degrading as context length grows.
TurboQuant is not the ceiling. It is the ceiling of one approach to one subproblem. The sequential structure of language points toward a different ceiling, considerably lower, and the path to it runs through the formal language the model has already learned.
## References
- [1] T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao (2024) Medusa: simple LLM inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774. Cited by: Remark 5.
- [2] T. M. Cover and J. A. Thomas (2006) Elements of information theory. 2nd edition, Wiley-Interscience. Cited by: 2nd item, Corollary 2, Remark 3.
- [3] G. Delétang, A. Ruoss, P. Duquenne, E. Catt, T. Genewein, C. Mattern, J. Grau-Moya, L. K. Li, J. Veness, A. Gretton, et al. (2023) Language modeling is compression. arXiv preprint arXiv:2309.10668. Cited by: §8.
- [4] C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, Y. S. Shao, K. Keutzer, and A. Gholami (2024) KVQuant: towards 10 million context length LLM inference with KV cache quantization. arXiv preprint arXiv:2401.18079. Cited by: §1, §8.
- [5] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with PagedAttention. In Proceedings of the ACM SIGOPS Symposium on Operating Systems Principles, Cited by: §1, §4.1, §8.
- [6] Y. Leviathan, M. Kalman, and Y. Matias (2023) Fast inference from transformers via speculative decoding. arXiv preprint arXiv:2211.17192. Cited by: §9.5.
- [7] Y. Li, Y. Huang, B. Yang, B. Venkitesh, A. Locatelli, H. Ye, T. Cai, P. Lewis, and D. Chen (2024) SnapKV: LLM knows what you are looking for before generation. arXiv preprint arXiv:2404.14469. Cited by: §1, §8.
- [8] Z. Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, V. Braverman, B. Chen, and X. Hu (2024) KIVI: a tuning-free asymmetric 2-bit quantization for KV cache. arXiv preprint arXiv:2402.02750. Cited by: §1, §8.
- [9] G. Magarshak (2026) Probabilistic language tries: a unified framework for compression, decision policies, and execution reuse. arXiv preprint arXiv:2604.06228. Note: arXiv:2604.06228 [cs.LG]. Submitted 29 March 2026 Cited by: item 2, §1, §2.3, §8, §8, 4th item, §9.8, Definition 1, Definition 2, Remark 1.
- [10] R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean (2023) Efficiently scaling transformer inference. In Proceedings of Machine Learning and Systems, Vol. 5. Cited by: §1.
- [11] C. E. Shannon (1951) Prediction and entropy of printed English. Bell System Technical Journal 30 (1), pp. 50–64. Cited by: §1, §8.
- [12] I. H. Witten, R. M. Neal, and J. G. Cleary (1987) Arithmetic coding for data compression. Communications of the ACM 30 (6), pp. 520–540. Cited by: Corollary 2, Remark 3.
- [13] A. Zandieh, M. Daliri, M. Hadian, V. Mirrokni, P. Kacham, L. Gottesburen, and R. Jayaram (2026) TurboQuant: online vector quantization with near-optimal distortion rate. In International Conference on Learning Representations (ICLR), Note: arXiv:2504.19874. Google Research, NYU, Google DeepMind, KAIST Cited by: §1, §2.2, §8.
- [14] Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, Z. Wang, and B. Chen (2023) H2O: heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems, Vol. 36. Cited by: §1, §8.