2509.16278

# Language Modeling with Learned Meta-Tokens **Authors**: Denotes equal contribution ## Abstract While modern Transformer-based language models (LMs) have achieved major success in multi-task generalization, they often struggle to capture long-range dependencies within their context window. This work introduces a novel approach using meta-tokens, special tokens injected during pre-training, along with a dedicated meta-attention mechanism to guide LMs to use these tokens. We pre-train a language model with a modified GPT-2 architecture equipped with meta-attention in addition to causal multi-head attention, and study the impact of these tokens on a suite of synthetic tasks. We find that data-efficient language model pre-training on fewer than 100B tokens utilizing meta-tokens and our meta-attention mechanism achieves strong performance on these tasks after fine-tuning. We suggest that these gains arise due to the meta-tokens sharpening the positional encoding. This enables them to operate as trainable, content-based landmarks, implicitly compressing preceding context and "caching" it in the meta-token. At inference-time, the meta-token points to relevant context, facilitating length generalization up to 2 $\times$ its context window, even after extension with YaRN. We provide further evidence of these behaviors by visualizing model internals to study the residual stream, and assessing the compression quality by information-theoretic analysis on the rate-distortion tradeoff. Our findings suggest that pre-training LMs with meta-tokens offers a simple, data-efficient method to enhance long-context language modeling performance, while introducing new insights into the nature of their behavior towards length generalization. ## 1 Introduction Transformer-based language models (LMs) have showcased remarkable capabilities across diverse language tasks (Brown et al., 2020b; Chowdhery et al., 2022; OpenAI, 2023). Nevertheless, such models suffer from an inability to capture dependencies spanning over their entire context window. With growing adoption and ever-expanding demands on the context over which the model can process and reason, it is vital to develop methods that facilitate long-context adaptation and length generalization. Despite numerous architectural remedies, including sparse attention (Beltagy et al., 2020; Zaheer et al., 2020), recurrent blocks (Hutchins et al., 2022), and modified positional encoding (Press et al., 2021; Su et al., 2021; Chen et al., 2023), the fundamental challenge still remains: how can models reliably access and summarize distant context in a concise, cheap, yet expressive manner? We propose a simple solution, by way of meta-tokens, learned tokens periodically injected into the input sequence during pretraining, and cleverly placed during fine-tuning. Unlike conventional dummy tokens (Goyal et al., 2024), meta-tokens are explicitly trained via a dedicated sparse attention layer, guiding the model to condense and "cache" contextual information as an in-line storage mechanism. As a result, these tokens act as adaptive landmarks (Mohtashami and Jaggi, 2023), summarizing preceding context segments into compact representations. At inference time, meta-tokens provide implicit pathways to distant information, enabling models to generalize effectively across sequences longer than those encountered during training. We demonstrate the empirical efficacy of this approach by pre-training a 152M parameter modified GPT-2 model with meta-tokens and a sparsely activated meta-attention mechanism. Our approach not only excels on recall-oriented synthetic tasks but also generalizes up to 2x the pretraining context window (via YaRN) — a rare feat for decoder-only architectures trained on 100B tokens or less. We trace these gains to a subtle mechanism: meta-tokens provably induce a sharpening effect on positional encoding, enabling the meta-token to locate its position based on the content it stores and reducing the entropy of the attention distribution. We present evidence that this sharpening is responsible for an anchoring effect on relevant distant tokens, facilitating robust length generalization. Furthermore, by analyzing internal model activations and studying the rate-distortion tradeoff, we validate that meta-tokens function as compressed representations of context. Our contributions can be summarized as follows: 1. We introduce a simple language model pre-training scheme using meta-tokens and a meta-attention mechanism to improve performance on a wide range of synthetic tasks. 1. We show that meta-tokens sharpen the positional encoding, enabling precise long-range attention; we further show that length generalization improves without positional encoding. 1. The sharpening hypothesis and implicit compression behavior are supported by visualizations of model internals and information-theoretic analysis into the rate-distortion tradeoff. ## 2 Preliminaries Causal Multi-Head Attention. Let $\mathbf{x}=\{x_{1},x_{2},\dots,x_{T}\}$ denote an input sequence of tokens of length $T$ , $\mathcal{V}$ denote the vocabulary size of V, and $E:\mathcal{V}\rightarrow\mathbb{R}^{d}$ represent the the token embedding function mapping each token to a $d$ -dimensional vector. Each $x_{t}$ is embedded into some continuous representation where $\mathbf{e}_{t}=E(x_{t})+\mathbf{p}_{t}$ , such that $\mathbf{p}_{t}$ is the positional encoding for $t$ . In decoder-only architecture, we utilize causal self-attention to ensure that predictions for a given token are only based on preceding tokens. The causal self-attention mechanism modifies the attention computation by masking future positions in the attention weights. Formally: $$ \text{Causal Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^{\top}}{\sqrt{d_{k}}}+M\right) $$ where $M$ masks future tokens, ensuring that the model can only attend to current and past tokens. If $A$ is the matrix of attentions scores, then $$ A_{ij}=\begin{cases}\text{softmax}(A_{ij})&\text{if }i\geq j\\ 0&\text{if }i<j\end{cases} $$ This masking zeros attention scores for future tokens, allowing only the relevant past tokens to influence the current token’s representation. Positional Encoding. Positional encoding was introduced in Transformer pre-training to provide models with information about the ordering of tokens. With absolute positional embeddings (APE; Vaswani et al. (2017)), each position $t$ in the sequence receives a vector $p_{t}$ , independent of its content, so tokens are distinguished in an index-by-index manner. Given learned token-embedding lookup table $E:V\rightarrow\mathbb{R}^{d}$ for vocabulary $V$ and hidden dimension $d$ , and positional embedding $p_{t}=\text{Emb}_{pos}(t)$ for $t\in[0,T-1]$ and $\text{Emb}_{pos}\in\mathbb{R}^{T\times d}$ . Each token embedding is then defined as $e_{t}=E(x_{t})+p_{t}$ ; this method was used in GPT-2 and GPT-3 (Radford et al., 2019; Brown et al., 2020a). By contrast, Rotary Position Embedding (RoPE; Su et al. (2023)) rotates each pair of embedding dimensions by an angle proportional to position, rather than adding a separate vector per position. This makes the difference in attention scores directly encode relative distance between embeddings. The hidden vector $h$ is split into $\frac{d}{2}$ contiguous 2-D slices, and the angle for a position $t$ is defined as $\theta_{t,i}=\frac{t}{10000^{2i/d}}$ . The 2-D rotation matrix is taken as $R(\theta)=\begin{pmatrix}\cos\theta&-\sin\theta\\ \sin\theta&\cos\theta\end{pmatrix}$ . Then, $\text{RoPE}(h)_{t}^{(2i:2i+1)}=R(\theta_{t,i})h^{(2i:2i+1)}$ . This has proven successful in the Llama models (Grattafiori et al., 2024). It can be observed that RoPE reflects the relative offset $i-j$ , with the dot product $\langle Q_{i},K_{j}\rangle$ introducing a new factor of $\cos{(\frac{i-j}{10000^{2i/d}})}$ . This is reflected in works using relative bias (Shaw et al., 2018), which introduces a bias term as a learned function over the $i-j$ distance. T5 (Raffel et al., 2020) then adds this bias to $\langle Q_{i},K_{j}\rangle$ . ## 3 Training Language Models with Meta-Attention We introduce a set of $M$ meta-tokens (denoted as $m$ ); given a context length or block size of the model, $n$ , we take $M=kn$ for some constant fraction $k\in[0,1]$ We take $k=0.1$ in practice; balancing next-token prediction over the standard vocabulary while injecting a non-trivial number of meta-tokens.. The aim of introducing these meta-tokens is to capture or store contextual information to enhance the model’s retrieval and reasoning capabilities; attending to a meta-token should enable implicit retrieval of the context that it stores, guiding shortcut paths over the context window. In practice, these may be treated akin to adding a filler token to the model’s vocabulary. The $M$ tokens are injected into the input sequences during pre-training uniformly at random, which was informed by two key premises. While we desire interpretability and control in applying these tokens, and as a result, prefer distinguishability at the task level, this is challenging to do without explicitly fixing a downstream task, impeding generality. The second consideration was in how they specifically they should be injected. While Zelikman et al. (2024) introduced <|startofthought|> and <|endofthought|> tokens interleaved between reasoning steps near punctuation (serving as natural break), the introduction of a rough periodicity between tokens during pre-training could result in being trapped into local minima in the optimization landscape. We instead chose to follow the random injection scheme, supported by the meta-token pre-training approach outlined in Goyal et al. (2024). We ensure that the trained model incurs no loss for predicting meta-tokens, unlike a standard token in the vocabulary – the meta-tokens’ indices are simply shifted and removed when computing the binary cross-entropy (BCE) loss. Meta-Attention Mechanism. We augment our transformer $H$ to take $P$ which contains the positions of the meta-tokens. We introduce a sparse attention mechanism, called meta-attention, which selectively modifies attention scores for the specially marked "meta-tokens" within a sequence. This allows the model to simulate selective attention, influencing the final behavior by focusing on these meta-tokens. The underlying principles of the desired behavior is influenced by dual cross-attention (Jiang et al., 2024), such that operations are performed higher on the abstraction hierarchy than the feature space alone. This induces a meta-learning-like setup over which attention on the meta-tokens is learned. Let the indices of special "meta-tokens" be denoted by $\text{positions}\in\mathbb{R}^{B\times T^{\prime}}$ , where $T^{\prime}$ is the number of meta tokens in a batch. We construct a meta mask $P\in\mathbb{R}^{B\times T\times T}$ to influence the attention mechanism. For each batch element $b$ and token positions $i$ , $j$ : $$ P[b,i,j]=\begin{cases}0&\text{if both }i\text{ and }j\text{ are meta tokens (i.e., }i,j\in\text{positions}[b,:])\\ -\infty&\text{otherwise}\end{cases} $$ The meta-attention operation is defined as: $$ \text{MetaAttention}(Q,K,V)=\text{softmax}\left(\left(\frac{QK^{\top}}{\sqrt{d_{k}}}+M\right)+P\right)V $$ Where M is the same causal mask as before. Here, the meta mask $P$ allows attention to flow only among the meta tokens in the sequence, introducing a distinct interaction compared to regular attention. This meta-attention layer selectively modifies the attention by influencing the flow of information to and from these meta tokens, distinguishing itself from the standard causal attention. In particular, if $A$ is the matrix of attentions scores then $$ A_{ij}=\begin{cases}\text{softmax}(A_{ij})&\text{if i and j are meta tokens}\\ -\infty&\text{otherwise}\end{cases} $$ To assemble the architecture used for our model, we insert the meta-attention mechanism after the causal masked self-attention computation, to specifically attend to the injected meta tokens, as defined above. We provide a complete breakdown of the architecture in Appendix A. ## 4 Results ### 4.1 Model Training and Architecture All experiments were performed with 4 NVIDIA A100 GPUs, training the meta attention transformer for 200,000 iterations or 98B tokens using Distributed Data Parallel (DDP) on the Colossal Cleaned Crawl Corpus (C4) (Raffel et al., 2020). The configuration and hyperparameters used in our pre-training are included in Appendix A and B. As a baseline, we also pre-train GPT-2 (124M) on C4, with identical hyperparameters. The primary change we make from a standard GPT-2 architecture is the addition of RoPE to enable better generalization to longer contexts and improve stability in next-token prediction tasks. We extend our transformer model’s context window from 1024 tokens to longer sequences by training two distinct models with context lengths of 4096 and 8192 tokens, respectively. This extension is implemented using the YaRN method (Peng et al., 2023), which dynamically scales Rotary Positional Embeddings (RoPE) to effectively process significantly longer sequences without compromising performance or computational efficiency. The key parameters are detailed in Appendix C ### 4.2 Experimental Setup and Tasks We design four synthetic tasks to evaluate the recall capabilities of models trained with meta-tokens. The tasks are List Recall, Segment Counting, Parity, and Copying. For each task, we define three difficulty levels by varying the maximum sequence length. In all tasks, we insert a designated _PAUSE_ meta-token at task-specific positions to indicate where the model should focus its meta-attention. We fine-tune on synthetic data that we generate for each task (binned by instance length) and report the validation score on a held-out test set. Detailed examples for each task are provided in Appendix H. - List Recall: Given $N$ named lists of length $k$ , the model is prompted to recall a specific item from a specified list. We insert a _PAUSE_ meta-token immediately following the list containing the queried item, as well as before the final question. The expected answer is the corresponding item. Task difficulty is scaled by varying the list length $k$ and number of lists $N$ . - Segment Counting: The model is presented with several named lists, with a segment in these lists wrapped by by _PAUSE_ meta-tokens. The prompt then asks how many times a specified item appears between the two meta-tokens. The task difficulty changes based on the number and size of the named lists. - Parity: In this task, the input consists of a sequence of bits, with a _PAUSE_ meta-token indicating a specific position in the sequence. The model is prompted to compute the XOR of all bits appearing before the meta-token. The task difficulty changes based on the number of bits it has to XOR. - Copying The model is given with a segment of text containing a bracketed spans marked by _PAUSE_ meta-tokens. The model is prompted to reproduce the exact content found between the meta-token-marked boundaries. The task is designed to assess the model’s ability to extract and copy arbitrary spans of text from a context, with difficulty varying according to the length and complexity of the bracketed span. We report the sequence accuracy. Within these four tasks, we investigate length generalization by fine-tuning our model in multiple phases. At each phase, we assess the model’s performance on sequence lengths exceeding those seen during that phase’s training, enabling us to evaluate its generalization to longer contexts. In addition, Appendix D reports the performance of our models on a context length of 2048 tokens, which is twice the length seen during pretraining (1024 tokens). Baselines. For a controlled comparison, we also pre-train a GPT-2 model (NanoGPT, 124M; Karpathy (2023)) on C4, with identical hyperparameters as the meta-tokens model. Additionally, we use Eleuther AI’s GPT-Neo-125M (Black et al., 2021) as another baseline. ### 4.3 Meta-Tokens Improve Performance on Synthetic Recall-Oriented Tasks. As seen in Figure 1, we find that the models trained on meta-tokens substantially outperform our pre-trained GPT-2 and GPT-Neo-125M baselines, across all tasks and all train lengths. The complete tables for these results are included in Appendix F. We observe the GPT-2 model trained with APE to generally perform poorly; however, it does achieve reasonable performance in the segment counting and parity tasks, albeit much further behind the models with meta-attention. This suggests that training on further data could improve its performance; this also highlights the data-efficiency of our meta-tokens models. The models also gain in performance much more quickly with fine-tuning when increasing the train length – a phenomenon not observed with the GPT-2 models. Our models also outperform the GPT-Neo-125M model by a substantial margin; given that GPT-Neo was pre-trained on 300B tokens, nearly three times the volume of data on which our meta-attention models were trained (albeit from a different corpus). To study the effect of positional encoding on our results, we ablate by zeroing out the positional encoding, zeroing out the text embedding, and performing both operations – all just at the meta-token indices. Curiously, we observe in Tables 11 - 14 that the score without positional encoding nearly matches or exceeds the accuracy of the model with the positional encoding as is. The lone exception is the segment counting task, where there is a gap for all settings except the model trained with APE at a length of 256, which achieves a $+4.8\$ improvement over the "Full" model. By contrast, zeroing out the token embedding hurts performance substantially in nearly every setting on List Recall, Segment Counting, and Copying; on Parity, this generally matches the performance of zeroing out the positional encoding. Thus, we find that 1. pre-training with meta-tokens and meta-attention boosts performance, and 2. zeroing out the positional encoding at just the meta-tokens can match or improve performance at inference time. <details> <summary>Figures/Accuracy_at_Eval_Length_=_512_on_List_Recall.png Details</summary>  ### Visual Description ## Bar Chart: Accuracy at Eval Length = 512 on List Recall ### Overview The chart compares the accuracy of four language models (GPT-2 APE, Meta + APE, Meta + RoPE, GPT-Neo-125M) at an evaluation length of 512 tokens, measured on list recall. Accuracy is shown for three training lengths (128, 256, 512 tokens), with distinct colors for each training length. The y-axis represents accuracy percentage, while the x-axis lists the models. ### Components/Axes - **X-axis (Models)**: - GPT-2 APE - Meta + APE - Meta + RoPE - GPT-Neo-125M - **Y-axis (Accuracy %)**: Ranges from 0 to 100. - **Legend (Top-left)**: - Red: Train Length = 128 - Orange: Train Length = 256 - Blue: Train Length = 512 ### Detailed Analysis 1. **GPT-2 APE**: - Train Length = 128: 0.0% (red bar absent). - Train Length = 256: 0.0% (orange bar absent). - Train Length = 512: 3.6% (blue bar). 2. **Meta + APE**: - Train Length = 128: 12.0% (red bar). - Train Length = 256: 42.6% (orange bar). - Train Length = 512: 98.7% (blue bar). 3. **Meta + RoPE**: - Train Length = 128: 5.9% (red bar). - Train Length = 256: 48.6% (orange bar). - Train Length = 512: 99.3% (blue bar). 4. **GPT-Neo-125M**: - Train Length = 128: 0.0% (red bar absent). - Train Length = 256: 0.0% (orange bar absent). - Train Length = 512: 81.2% (blue bar). ### Key Observations - **Trend Verification**: - For **Meta + APE** and **Meta + RoPE**, accuracy increases sharply with longer training lengths (e.g., 12.0% → 42.6% → 98.7% for Meta + APE). - **GPT-2 APE** and **GPT-Neo-125M** show no improvement at shorter training lengths (128/256), with GPT-2 APE performing poorly even at 512 tokens (3.6%). - **Meta + RoPE** achieves the highest accuracy (99.3%) at 512 tokens, slightly outperforming **Meta + APE** (98.7%). - **Notable Outliers**: - **GPT-2 APE** underperforms across all training lengths, suggesting architectural limitations. - **GPT-Neo-125M** achieves moderate accuracy (81.2%) at 512 tokens but lacks data for shorter lengths, leaving its scalability unclear. ### Interpretation The data demonstrates that: 1. **Training Length Matters**: Longer training (512 tokens) significantly improves accuracy for **Meta + APE** (+86.7% increase from 128 to 512 tokens) and **Meta + RoPE** (+93.4% increase). 2. **Model Architecture Dominates**: **Meta + RoPE** outperforms **Meta + APE** despite similar training lengths, indicating RoPE may enhance recall efficiency. 3. **Scalability Limits**: **GPT-2 APE** and **GPT-Neo-125M** show minimal or no improvement at shorter lengths, suggesting they require longer training to achieve meaningful performance gains. 4. **Efficiency Trade-offs**: **Meta + RoPE** achieves near-perfect accuracy (99.3%) with 512 tokens, implying it balances model size and training efficiency better than alternatives. This chart highlights the importance of training duration and architectural choices (e.g., RoPE vs. APE) in optimizing list recall accuracy for language models. </details> <details> <summary>Figures/Accuracy_at_Eval_Length_=_512_on_Segment_Counting.png Details</summary>  ### Visual Description ## Bar Chart: Accuracy at Eval Length = 512 on Segment Counting ### Overview The chart compares the accuracy of four language models (GPT-2 APE, Meta + APE, Meta + RoPE, GPT-Neo-125M) on a segment counting task at a fixed evaluation length of 512 tokens. Accuracy is measured across three training lengths (128, 256, 512 tokens), with results visualized as grouped bars. The highest accuracy observed is 95.3% for Meta + RoPE trained on 512 tokens. ### Components/Axes - **X-axis**: Model configurations (GPT-2 APE, Meta + APE, Meta + RoPE, GPT-Neo-125M) - **Y-axis**: Accuracy (%) at Eval Length = 512 - **Legend**: - Red = Train Length = 128 - Orange = Train Length = 256 - Blue = Train Length = 512 - **Bar Colors**: - Red bars (128 tokens) appear only for GPT-2 APE and Meta + APE/Meta + RoPE - Orange bars (256 tokens) appear for all models except GPT-Neo-125M - Blue bars (512 tokens) appear for all models ### Detailed Analysis 1. **GPT-2 APE**: - 128 tokens: 20.2% - 256 tokens: 23.7% - 512 tokens: 25.0% - *Trend*: Minimal improvement with longer training. 2. **Meta + APE**: - 128 tokens: 25.0% - 256 tokens: 53.6% - 512 tokens: 80.9% - *Trend*: Significant accuracy gains with longer training. 3. **Meta + RoPE**: - 128 tokens: 25.0% - 256 tokens: 35.7% - 512 tokens: 95.3% - *Trend*: Steep improvement, especially at 512 tokens. 4. **GPT-Neo-125M**: - Only 512 tokens shown: 24.9% - *Note*: No data for 128/256 tokens in the chart. ### Key Observations - **Meta + RoPE (512 tokens)** achieves the highest accuracy (95.3%), outperforming all other configurations. - **GPT-Neo-125M** shows the lowest accuracy (24.9%) but lacks data for shorter training lengths. - **Meta + APE** demonstrates the largest relative improvement (25.0% → 80.9%) when doubling training length from 128 to 512. - **GPT-2 APE** shows the least sensitivity to training length changes. ### Interpretation The data suggests that: 1. **Training Length Matters**: Longer training consistently improves accuracy across models, with Meta + RoPE showing the most dramatic gains. 2. **Architecture Impact**: Meta + RoPE's superior performance at 512 tokens implies architectural advantages over GPT-2 and GPT-Neo-125M. 3. **Data Gaps**: The absence of 128/256 token results for GPT-Neo-125M raises questions about whether: - The model was not trained on shorter lengths - It underperforms so severely that results were omitted 4. **Diminishing Returns**: While all models improve with longer training, the rate of improvement varies significantly, with Meta + RoPE showing near-linear scaling. This analysis highlights the interplay between model architecture and training duration in determining performance on segment counting tasks. The stark contrast between Meta + RoPE and other models at maximum training length suggests that architectural innovations (e.g., RoPE positional encoding) may be critical for high-accuracy performance. </details> <details> <summary>Figures/Accuracy_at_Eval_Length_=_512_on_Parity.png Details</summary>  ### Visual Description ## Bar Chart: Accuracy at Eval Length = 512 on Parity ### Overview The chart compares model accuracy (%) at an evaluation length of 512 across four architectures: GPT-2 APE, Meta + APE, Meta + RoPE, and GPT-Neo-125M. Accuracy is measured for three training lengths (128, 256, 512), with performance trends indicating that longer training generally improves results. Meta + APE and Meta + RoPE achieve perfect accuracy (100%) at the longest training length (512), while GPT-Neo-125M underperforms even at this scale. ### Components/Axes - **X-Axis (Categories)**: - GPT-2 APE - Meta + APE - Meta + RoPE - GPT-Neo-125M - **Y-Axis (Accuracy %)**: - Scale: 0–100 (linear) - Title: "Accuracy (%) at Eval Length = 512 on Parity" - **Legend**: - Position: Top-right - Colors: - Red = Train Length = 128 - Orange = Train Length = 256 - Blue = Train Length = 512 ### Detailed Analysis 1. **GPT-2 APE**: - Train Length = 128: 53.4% (red) - Train Length = 256: 60.7% (orange) - Train Length = 512: 60.0% (blue) - *Trend*: Minimal improvement after 256 training steps. 2. **Meta + APE**: - Train Length = 128: 67.9% (red) - Train Length = 256: 96.4% (orange) - Train Length = 512: 100.0% (blue) - *Trend*: Steep improvement from 128 to 256, plateauing at 512. 3. **Meta + RoPE**: - Train Length = 128: 76.2% (red) - Train Length = 256: 96.4% (orange) - Train Length = 512: 100.0% (blue) - *Trend*: Rapid gains from 128 to 256, full parity at 512. 4. **GPT-Neo-125M**: - Only Train Length = 512: 54.8% (blue) - *Missing Data*: No bars for 128 or 256 training lengths. ### Key Observations - **Training Length Impact**: Longer training consistently improves accuracy, with diminishing returns after 256 steps for most models. - **Meta Architecture Superiority**: Meta + APE and Meta + RoPE outperform GPT-2 APE by 13–36 percentage points at equivalent training lengths. - **GPT-Neo-125M Anomaly**: Underperforms all models even at the longest training length, suggesting architectural limitations or insufficient optimization for parity tasks. - **Perfect Parity**: Meta + APE and Meta + RoPE achieve 100% accuracy at 512 training steps, indicating robust parity handling. ### Interpretation The data demonstrates that **training duration and architectural design** are critical for parity task performance. Meta’s models (APE/RoPE) likely benefit from advanced training methodologies or architectural innovations (e.g., RoPE positional encoding) that enable near-perfect parity handling. GPT-Neo-125M’s poor performance highlights potential inefficiencies in its design for this specific task. The absence of lower-training-length data for GPT-Neo-125M may indicate either experimental omission or inherent instability at shorter training scales. These findings underscore the importance of model architecture selection and training resource allocation for fairness-critical applications. </details> <details> <summary>Figures/Accuracy_at_Eval_Length_=_512_on_Copying.png Details</summary>  ### Visual Description ## Bar Chart: Accuracy at Eval Length = 512 on Copying ### Overview The chart compares the accuracy of different language models (GPT-2 APE, Meta + APE, Meta + RoPE, GPT-Neo-125M) at an evaluation length of 512 characters for a copying task. Accuracy is measured as a percentage, with three training lengths (128, 256, 512 tokens) represented by distinct colors (red, orange, blue). The chart emphasizes performance trends across models and training scales. ### Components/Axes - **X-Axis (Categories)**: - GPT-2 APE - Meta + APE - Meta + RoPE - GPT-Neo-125M - **Y-Axis (Values)**: Accuracy (%) ranging from 0 to 100. - **Legend**: - Red = Train Length = 128 - Orange = Train Length = 256 - Blue = Train Length = 512 - **Bar Structure**: Each model has three grouped bars (one per training length), except GPT-Neo-125M, which only has a blue bar (512). ### Detailed Analysis 1. **GPT-2 APE**: - Train Length = 128: 3.0% (red) - Train Length = 256: 5.7% (orange) - Train Length = 512: 7.8% (blue) - *Trend*: Gradual improvement with longer training, but remains the lowest-performing model. 2. **Meta + APE**: - Train Length = 128: 76.2% (red) - Train Length = 256: 96.4% (orange) - Train Length = 512: 98.5% (blue) - *Trend*: Sharp improvement from 128 to 256, then marginal gains at 512. Highest accuracy among all models at 512. 3. **Meta + RoPE**: - Train Length = 128: 5.2% (red) - Train Length = 256: 23.6% (orange) - Train Length = 512: 98.9% (blue) - *Trend*: Dramatic leap from 256 to 512 training length, achieving the highest accuracy overall. 4. **GPT-Neo-125M**: - Train Length = 512: 16.9% (blue) - *Trend*: Only data point for this model; significantly lower than Meta models at the same training length. ### Key Observations - **Training Length Impact**: Longer training (512) consistently improves accuracy across all models, with the largest gains observed in Meta + RoPE (23.6% → 98.9%). - **Model Performance**: Meta models (Meta + APE, Meta + RoPE) dominate, achieving >98% accuracy at 512 training length. GPT-2 APE and GPT-Neo-125M lag far behind. - **Outlier**: GPT-Neo-125M’s 16.9% accuracy at 512 is anomalously low compared to other models at the same training length. ### Interpretation The data suggests that **training length is a critical factor** in model performance for copying tasks, with diminishing returns after a certain point (e.g., Meta + APE’s 96.4% at 256 vs. 98.5% at 512). The **Meta + RoPE** configuration demonstrates the most significant scalability, likely due to architectural advantages (e.g., RoPE positional encoding). Conversely, GPT-Neo-125M’s poor performance at 512 hints at inherent limitations in its design or training data. The stark contrast between Meta and GPT models underscores the importance of architectural choices in handling long-sequence tasks. </details> Figure 1: We study the performance of the pre-trained GPT-2 w/ APE, Meta-attention w/ APE, and Meta-attention w/ RoPE, as well as GPT-Neo-125M, all fine-tuned on synthetic data for their respective tasks at the maximum train lengths indicated in the legends. All experiments are performed on a test set of prompt lengths up to 512 tokens. Table 1: Token Accuracy (%) on List Recall and Segment Counting across long contexts. | List 4k / 4k 8k / 4k | 4k / 2k 85.0 85.0 | 19.5 88.2 95.8 | 16.0 90.2 91.2 | 13.7 20.5 97.4 | 0.9 1.8 98.2 | 0.0 1.0 96.2 | 0.0 3.5 93.9 | 0.9 4.4 31.9 | 1.1 1.1 0.0 | 0.0 2.1 2.1 | 2.1 2.1 2.1 | 1.1 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 8k / 8k | 92.9 | 98.3 | 97.1 | 100.0 | 98.2 | 100.0 | 100.0 | 89.0 | 26.1 | 10.4 | 9.6 | | | Count | 4k / 2k | 19.1 | 23.8 | 19.2 | 14.6 | 25.2 | 14.1 | 14.0 | 12.0 | 16.0 | 8.0 | 6.0 | | 4k / 4k | 17.5 | 23.8 | 31.8 | 20.3 | 30.4 | 19.3 | 19.1 | 14.0 | 26.0 | 12.0 | 16.0 | | | 8k / 4k | 19.1 | 23.8 | 14.3 | 11.1 | 20.6 | 12.7 | 12.7 | 14.0 | 16.0 | 14.0 | 12.0 | | | 8k / 8k | 27.0 | 33.3 | 15.9 | 19.1 | 27.0 | 19.1 | 23.8 | 22.0 | 18.0 | 18.0 | 18.0 | | Meta-Tokens Aid in Length Generalization. In Figure 1 and Appendix F, we find that the model trained on meta-tokens length generalizes well on the parity and copying tasks with APE, and performs somewhat well (much better than the baselines) on list recall and segment counting at a train length of 256. For instance, despite relatively similar performance at the 128 train length on the segment counting task, the performance on the test set of up to a length of 512 dramatically increases when training at the 256 length, by $+28.6\$ with APE and $+10.7\$ with RoPE, compared to $+3.5\$ for GPT-2 with APE. Table 1 exhibits a similar trend for the YaRN models, achieving strong performance across its respective context windows, and even achieves non-trivial accuracy beyond the window. Fine-tuning the 8k YaRN model on examples of up to a length of 4k can generalize very well up to 8k. These findings underscore the substantial advantages of training with meta-tokens and the nuanced role positional encoding plays in task-specific and length-generalization contexts. Moreover, when looking at the results on Meta + RoPe on test set lengths of prompts up to 1024 tokens (denoted extra-hard in Table 2), we find that zeroing out the positional encoding also plays a sizable role in improving length generalization, especially in the List Recall task. While the model originally achieves performances of $11.1\$ , $0\$ and $44.4\$ when fine-tuned on train lengths of 512 (APE), 256 and 512 (RoPE), respectively, the scores improve by $+38.9\$ , $+22.2\$ and $+11.2\$ , by simply zeroing out the positional encoding at the meta-tokens. ### 4.4 Examination of Positional Encoding through Internal Representations. Table 2: The configurations where zeroing the positional encoding at inference time results in accuracy improvements on the List Pointer task, denoted by the $\Delta$ (pp) percentage points column. | Meta + APE (medium, 128) | 77.8% | 88.9% | +11.1 | | --- | --- | --- | --- | | Meta + APE (hard, 128) | 11.1% | 22.2% | +11.1 | | Meta + APE (extra-hard, 512) | 11.1% | 50.0% | +38.9 | | Meta + RoPE (medium, 128) | 44.4% | 55.6% | +11.1 | | Meta + RoPE (hard, 256) | 33.3% | 66.7% | +33.3 | | Meta + RoPE (extra-hard, 256) | 0.0% | 22.2% | +22.2 | | Meta + RoPE (extra-hard, 512) | 44.4% | 55.6% | +11.1 | As discussed above, the results in Tables 11 - 14 suggest that the positional encoding of the meta-token can potentially be holding back the downstream performance of the meta-attention models. We posit that the model is instead relying on its content – cached context stored within the meta-token – to sharpen its sense of its position in the sequence. Next, we aim to formally define this notion of sharpness in the context of positional encoding, and its relationship to the model’s logits. Let $\alpha_{i\rightarrow k}=\text{softmax}_{k}(Q_{i}K_{j}^{T}+b_{i-j})$ be the attention distribution for query $i$ over keys $j$ , with relative bias term $b_{i-j}$ . We define the sharpness of the positional encoding by the entropy: $$ H(\alpha_{i})=-\sum_{j}\alpha_{i\rightarrow j}\log\alpha_{i\rightarrow j} $$ Intuitively, when a meta-token is present at position $t$ , the model’s attention becomes peaked around a small set of keys; this "honing in" behavior reduces $H(\alpha)$ compared to APE or RoPE without meta-tokens. In this manner, meta-tokens behave as content-driven landmarks – they serve as a low-entropy channel that serves as a pointer to relevant context. As noted prior, the data efficiency observation suggests that the meta-token helps to accelerate next-token prediction behavior while introducing a stabilizing effect in the midst of noisy positional encoding. **Theorem 4.1** *Consider a Transformer head at query position $i$ over keys $1,\dots,N$ . Let $\alpha_{i}^{\text{abs}}(j)\propto\exp(Q_{i}K_{j}^{T})$ be the attention under absolute positional encoding and let $\alpha_{i}^{\text{meta}}\propto\exp(Q_{i}K_{j}^{T}+\delta_{j,j^{*}}\Delta)$ when a meta-token at position $j^{*}$ introduces an additive logit boost of $\Delta>0$ . Then, for some function $\kappa(\Delta)>0$ : $$ \displaystyle H(\alpha_{i}^{\text{meta}})\leq H(\alpha_{i}^{\text{abs}})-\kappa(\Delta) \tag{1} $$* * Proof Sketch* Parametrize the path by $t\in[0,\Delta]$ , and define logits $\ell_{j}^{(t)}$ and their softmax $\alpha^{(t)}$ respectively. Since boosting the true index tightens the margin, the derivative of $H(\alpha^{(t)}$ is strictly negative. Therefore, over the path, $H(\alpha^{(\Delta)})<H(\alpha^{(0)})$ , so $H(\alpha^{\text{meta}})<H(\alpha^{\text{abs}})$ , where their difference must be a function of $\Delta$ . The full proof is included in Appendix G. ∎ We note that this theorem also applies to RoPE, using $\alpha_{i}^{\text{RoPE}}(j)\propto\exp{Q_{i}(\text{RoPE}(K_{j}))^{T}}$ . A natural consequence of Theorem 4.1 is that the meta-token operates as an "anchor" from a logits standpoint by creating a margin $\Delta$ that concentrates the softmax. Concretely, we can specify that for meta-token $m_{t}$ at position $t$ and embedding $e_{t}\in\mathbb{R}^{d}$ , and query at position $i$ with vector $Q_{i}$ , has contribution to the $(i,j)$ logit of $\Delta_{i,j}^{(t)}=Q_{i}\cdot We_{t}\times\textbf{1}_{j=t}$ for learned linear head $W$ . Summing over $t$ yields the bias matrix $B\in\mathcal{B}_{\text{meta}}$ , the set of all realizable bias matrices under the meta-token embeddings. Thus, any learned meta-token embedding – provided that it adds to the logits at the summary position $j^{*}$ – guarantees sharper attention by reducing that attention head’s entropy. <details> <summary>Figures/logit-boost.png Details</summary>  ### Visual Description ## Histograms: Logit-Boost and Entropy Drop Comparisons ### Overview The image contains two side-by-side histograms comparing distributions of performance metrics between "Meta" and "Non-Meta" approaches. The left histogram focuses on **Logit-Boost**, while the right examines **Entropy Drop**. Both use density as the y-axis and a delta metric on the x-axis, with vertical dashed lines indicating mean values (μ). ### Components/Axes 1. **Left Histogram (Logit-Boost: Meta vs Non-Meta)** - **X-axis**: Δ max-logit (range: 0–6) - **Y-axis**: Density (range: 0–0.8) - **Legend**: "Meta vs Non-Meta" (textual, no color coding) - **Mean (μ)**: 1.26 (dashed blue line) 2. **Right Histogram (Entropy Drop: Non-Meta - Meta)** - **X-axis**: Δ H (nats) (range: 0–6) - **Y-axis**: Density (range: 0–0.8) - **Legend**: "Non-Meta - Meta" (textual, no color coding) - **Mean (μ)**: 1.40 (dashed blue line) ### Detailed Analysis #### Logit-Boost (Meta vs Non-Meta) - **Distribution**: - Highest density (~0.8) at Δ max-logit = 0. - Density decreases sharply for Δ max-logit > 1, with minimal values beyond 2. - Mean (μ = 1.26) aligns with the peak density at 0, suggesting most data clusters near the lower end of the scale. #### Entropy Drop (Non-Meta - Meta) - **Distribution**: - Highest density (~0.8) at Δ H = 0. - Density declines gradually, with a secondary peak (~0.3) at Δ H ≈ 1. - Mean (μ = 1.40) is higher than Logit-Boost, indicating a broader spread of values. - Notable tail extending to Δ H = 6, with sparse but visible density (~0.1). ### Key Observations 1. **Concentration vs Spread**: - Logit-Boost shows a tightly concentrated distribution (peak at 0, rapid decline). - Entropy Drop exhibits a broader distribution, with a secondary peak and extended tail. 2. **Mean Differences**: - Logit-Boost mean (1.26) is lower than Entropy Drop (1.40), suggesting Non-Meta - Meta introduces larger average deviations. 3. **Performance Implications**: - Logit-Boost’s sharp decline implies more consistent performance across Meta/Non-Meta. - Entropy Drop’s spread and higher mean may indicate instability or variability in Non-Meta - Meta comparisons. ### Interpretation The histograms highlight trade-offs between the two approaches: - **Logit-Boost** prioritizes consistency, with most data near the optimal (Δ max-logit = 0). - **Entropy Drop** shows greater variability, potentially reflecting sensitivity to differences between Meta and Non-Meta. - The higher mean in Entropy Drop (1.40 vs. 1.26) suggests Non-Meta - Meta comparisons are more divergent, possibly due to model complexity or data distribution shifts. These patterns could inform model selection: Logit-Boost may be preferable for stable performance, while Entropy Drop’s spread might require further investigation into its limitations. </details> <details> <summary>Figures/Meta2_past_heatmap_padded.png Details</summary>  ### Visual Description ## Heatmap: Meta Token #2 • Past Cosine-Sim (Padded) ### Overview This heatmap visualizes cosine similarity values between Meta Token #2 and other tokens across 12 layers (0–11). The color gradient ranges from purple (-0.04) to yellow (+0.04), with values padded to ensure consistent dimensions. The x-axis represents tokens in the sequence preceding Meta Token #2, while the y-axis represents layers in a neural network or transformer architecture. ### Components/Axes - **Y-axis (Layer)**: Labeled "Layer" with integer values 0–11. - **X-axis (Token past of Meta Token #2)**: Tokens include: `iers`, `pl`, `level`, `wrench`, `hammer`, `PAUSE_`, `:`, `Tools`, `plum`, `banana`, `peach`, `orange`, `:`, `ruits`, `F`. - **Color Bar**: Labeled "cosine similarity" with values from -0.04 (dark purple) to +0.04 (bright yellow). ### Detailed Analysis - **Highest Similarity**: - **Layer 0, Token "level"**: Bright yellow (≈+0.04), indicating the strongest positive cosine similarity. - **Layer 1, Token "level"**: Yellow-green (≈+0.03), slightly lower than Layer 0. - **Lowest Similarity**: - **Layer 11, Token "plum"**: Dark purple (≈-0.04), the most negative value. - **Layer 10, Token "plum"**: Dark purple (≈-0.03), also highly negative. - **Neutral Values**: - Tokens like `PAUSE_`, `Tools`, and `:` show mid-range values (green to teal, ≈0.00–0.02). - **Vertical Gradients**: - Token "level" shows a gradient from yellow (Layer 0) to teal (Layer 11), suggesting diminishing similarity with depth. - Token "plum" shows a gradient from teal (Layer 0) to dark purple (Layer 11), indicating increasing dissimilarity. ### Key Observations 1. **Layer 0 Dominance**: Layer 0 consistently shows the highest similarity magnitudes (both positive and negative). 2. **Token "level"**: Exhibits the strongest positive similarity across early layers, dropping to neutral by Layer 11. 3. **Token "plum"**: Shows the strongest negative similarity in later layers (10–11), with minimal presence in earlier layers. 4. **Padding Artifacts**: The "PAUSE_" token and repeated colons (`:`) appear in mid-layers with moderate similarity, possibly indicating structural padding. ### Interpretation The heatmap suggests that Meta Token #2’s cosine similarity with preceding tokens varies significantly across layers. The token "level" is most closely associated with Meta Token #2 in early layers (0–1), while "plum" becomes increasingly dissimilar in deeper layers. This could reflect: - **Token Positioning**: Early layers capture semantic relationships (e.g., "level" as a key contextual token), while later layers focus on syntactic or structural patterns. - **Padding Impact**: The presence of "PAUSE_" and colons may indicate artificial separation in the sequence, affecting similarity distributions. - **Layer-Specific Dynamics**: Layer 0’s high similarity magnitudes suggest it encodes strong contextual relationships, while deeper layers may prioritize disentangling or abstracting features. The data implies that Meta Token #2’s representation is most stable in early layers, with diminishing similarity as layers progress, potentially due to hierarchical feature extraction in the model. </details> Figure 2: (Left) We analyze the change in logits at the meta-token position, and observe that the meta-tokens do indeed induce a sizable boost in logits compare to zeroing its token embedding. (Middle) We find the boost in logits to correspond with a meaningful reduction in Shannon entropy over the softmax of the logits between the zeroed meta-token sequence and the sequence with the meta-token as is. This corroborates with our assumptions and claims in Theorem 4.1. (Right) We study the implicit "caching" ability of the meta-token by studying the cosine similarity over the token embeddings. We observe high spikes (the yellow column), diminishing as we move further away. This substantiates our claims of the presence of an implicit compression and "caching" mechanism. In Figure 2, we analyze the logits, comparing two settings: (1.) the current meta-token and (2.) the meta-token with its token embedding zeroed out. We find that the former gains a sizable amount over the latter, reinforcing the assumption made in Theorem 4.1 that the meta-token introduces an additive logit boost of $\Delta>0$ . Our empirical results show that the entropy over the softmax distribution of the logits decreases (the difference between "non-meta-token" and "meta-token" is positive), thus corroborating our central claim in Theorem 4.1. ### 4.5 Inference Efficiency Meta-tokens are generated at inference. The additional computation is sparse—each attention head only considers a small number of meta positions rather than the full attention matrix. In our current PyTorch implementation, which materializes the sparse mask as a dense tensor, we observe a throughput drop from 130.82 to 117.86 tokens/sec and a TTFT increase from 7.44ms to 7.57ms, i.e., a 1.11× slowdown. We expect optimized sparse attention implementations to reduce or eliminate this overhead. Table 3: Inference speed comparison with and without meta/pause tokens. | TPS (tokens/sec) TTFT (ms) Slowdown factor | 130.82 7.44 1.00 | 117.86 7.57 1.11 | | --- | --- | --- | ## 5 Examining Context Compression with Rate-Distortion Theory Given that these results provide evidence that meta-tokens can compress context in their representation, we develop mathematical formalizations to analyze this behavior. In particular, we turn to information-theoretic tools – specifically, an information bottleneck view. For a meta-token at $x_{m}$ succeeding a sequence of tokens $X=x_{i:m-1}$ from indices $i$ to $m-1$ , we consider a compression function $\zeta(\cdot)$ which transforms the subsequence $X$ into $x_{m}$ . As such, we define $\hat{X}=\zeta(X)=\zeta(x_{i:{m-1}})$ to be the compressed representation stored in $x_{m}$ . This can be generalized to the full set of $M$ meta-tokens: $$ \hat{X}_{1:M}=[\zeta_{1}(X_{1:m_{1}-1}),\zeta_{2}(X_{m_{1}+1:m_{2}-1}),\dots\zeta_{M}({m_{M+1}:m_{n}})] $$ For practicality, we consider the variational information bottleneck (Alemi et al., 2017). This introduces an encoder $q_{\phi}(\hat{x}\mid x)$ and decoder $q_{\theta}(y\mid\hat{x})$ , along with a simple prior $r(z)$ (e.g. $N(0,1)$ ), yielding the following form to solve for these variational distributions: $$ \min_{q_{\phi},q_{\theta}}\mathop{\mathbb{E}}_{p(x,y)}[\mathop{\mathbb{E}}_{q_{\phi}(\hat{x}\mid x)}[-\log q_{\theta}(y\mid\hat{x}]]+\beta\cdot\mathop{\mathbb{E}}_{p(x)}[KL(q_{\phi}(\hat{x}\mid x)||r(x))] $$ This form admits an equivalent perspective in rate-distortion theory. Specifically, the first term measures the quality in predicting the downstream target given a lossy compression $\hat{X}$ ("distortion"). The second term measures the average number of bits required to encode $\hat{X}$ , relative to some simple reference code $r(z)$ ("rate"). As such, analyzing rate-distortion curves – sweeping over values of $\beta$ – can provide valuable insights into the quality of the "compression" behavior and its informativeness when the meta-token is attended to. **Theorem 5.1** *Let $D_{\text{abs}}(R)$ be the minimum distortion achievable at rate $R$ under the VIB objective only using absolute positional encoding (no meta-tokens), and let $D_{\text{meta}}(R)$ be the minimum distortion achievable at rate $R$ with meta-tokens. Then, for every $R\geq 0$ , $$ \displaystyle D_{\text{meta}}(R)\leq D_{\text{abs}}(R) \tag{2} $$* Intuitively, meta-tokens expand the feasible set of encoders and decoders, which will either match or lower distortion for a given rate. Thus, the quality of compression with respect to its informativeness in predicting the target can only improve. <details> <summary>Figures/attention_heatmap.png Details</summary>  ### Visual Description ## Heatmap: AbsPE: vanilla causal self-attn at PAUSE ### Overview The image is a heatmap visualizing the attention patterns of a vanilla causal self-attention mechanism across 12 layers (Layer 0 to Layer 11) at two positions: P@8 and P@32. The x-axis represents token categories (B, 0, 1, is, all, ?), and the y-axis lists layers. Colors range from dark purple (0.0) to yellow (0.4), indicating attention intensity. ### Components/Axes - **Y-Axis (Layers)**: Labeled "Layer" with values 0 to 11. Each layer is split into two sub-rows: P@8 (top) and P@32 (bottom). - **X-Axis (Token Categories)**: Categories include B, 0, 1, is, all, ?. - **Legend**: Color scale from 0.0 (dark purple) to 0.4 (yellow), with intermediate values (0.1–0.3) in red/pink. - **Title**: "AbsPE: vanilla causal self-attn at PAUSE" at the top. ### Detailed Analysis - **Layer 0**: - **P@8**: Yellow block in "B" (0.4), rest dark purple (0.0). - **P@32**: Dark purple across all categories. - **Layer 1**: - **P@8**: Yellow block in "0" (0.4), rest dark purple. - **P@32**: Dark purple across all categories. - **Layer 2**: - **P@8**: Yellow block in "1" (0.4), rest dark purple. - **P@32**: Dark purple across all categories. - **Layer 3**: - **P@8**: Yellow block in "is" (0.4), rest dark purple. - **P@32**: Red block in "?" (0.2), rest dark purple. - **Layer 4**: - **P@8**: Yellow block in "all" (0.4), rest dark purple. - **P@32**: Red block in "?" (0.2), rest dark purple. - **Layer 5**: - **P@8**: Yellow block in "?" (0.4), rest dark purple. - **P@32**: Red block in "?" (0.2), rest dark purple. - **Layer 6**: - **P@8**: Yellow block in "?" (0.4), rest dark purple. - **P@32**: Red block in "?" (0.2), rest dark purple. - **Layer 7**: - **P@8**: Yellow block in "?" (0.4), rest dark purple. - **P@32**: Red block in "?" (0.2), rest dark purple. - **Layer 8**: - **P@8**: Yellow block in "?" (0.4), rest dark purple. - **P@32**: Red block in "?" (0.2), rest dark purple. - **Layer 9**: - **P@8**: Yellow block in "?" (0.4), rest dark purple. - **P@32**: Red block in "?" (0.2), rest dark purple. - **Layer 10**: - **P@8**: Yellow block in "?" (0.4), rest dark purple. - **P@32**: Red block in "?" (0.2), rest dark purple. - **Layer 11**: - **P@8**: Yellow block in "?" (0.4), rest dark purple. - **P@32**: Red block in "?" (0.2), rest dark purple. ### Key Observations 1. **P@8 Pattern**: Each layer (0–11) has a single yellow block in a specific x-axis category (B, 0, 1, is, all, ?), indicating focused attention on one token per layer. 2. **P@32 Pattern**: Mostly dark purple, but layers 3–11 have red blocks in the "?" category (0.2), suggesting weaker but consistent attention to unknown tokens. 3. **Color Consistency**: Yellow (0.4) and red (0.2) blocks align with the legend. No discrepancies observed. ### Interpretation - **Layer-Specific Focus**: In P@8, attention is concentrated on distinct tokens (e.g., B in Layer 0, 0 in Layer 1), suggesting hierarchical processing of input tokens. - **Unknown Token Attention**: P@32 layers (3–11) show red blocks in "?", indicating the model increasingly attends to unknown tokens in deeper layers. This may reflect a mechanism to handle ambiguity or special tokens in later stages. - **Positional Encoding**: The "0" and "1" categories in P@8 might correspond to positional encodings, while "is" and "all" could represent structural or global context tokens. - **Anomalies**: The abrupt shift to "?" attention in P@32 layers suggests a potential design choice to prioritize unknown tokens in deeper layers, possibly for robustness or contextual inference. This heatmap reveals how self-attention mechanisms allocate focus across layers and token types, highlighting differences between early (P@8) and late (P@32) processing stages. </details> <details> <summary>Figures/rate-distortion.png Details</summary>  ### Visual Description ## Line Chart: Rate-Distortion: Meta-Token vs. Last-token VIB ### Overview The chart compares the relationship between **Rate (KL)** and **Distortion (Cross-Entropy Loss)** for two types of Vector Quantization (VIB) methods: **Last-token VIB** (solid blue line with circles) and **Meta-token VIB** (dashed orange line with crosses). The x-axis represents the quantization rate (KL), while the y-axis represents distortion in cross-entropy loss. Both lines show a general downward trend, indicating reduced distortion as the rate increases. --- ### Components/Axes - **Title**: "Rate-Distortion: Meta-Token vs. Last-token VIB" - **X-axis**: - Label: "Rate (KL)" - Scale: 40 to 400 (logarithmic spacing implied by axis markers) - **Y-axis**: - Label: "Distortion (Cross-Entropy Loss)" - Scale: 10.0 to 10.8 - **Legend**: - Position: Top-right corner - Entries: - **Last-token VIB**: Solid blue line with circle markers - **Meta-token VIB**: Dashed orange line with cross markers --- ### Detailed Analysis #### Last-token VIB (Blue Line) - **Data Points**: - (70 KL, 10.75) - (100 KL, 10.6) - (200 KL, 10.0) - **Trend**: - Steady linear decrease in distortion as rate increases. - Slope: Approximately -0.015 per KL (calculated from (70, 10.75) to (200, 10.0)). #### Meta-token VIB (Orange Line) - **Data Points**: - (50 KL, 10.7) - (55 KL, 10.65) - (200 KL, 10.1) - **Trend**: - Initial sharp decline (50–55 KL: -0.05 per KL), then gradual decline (-0.0045 per KL from 55–200 KL). - Converges with Last-token VIB at 200 KL (10.1 vs. 10.0). --- ### Key Observations 1. **Divergence at Low Rates**: - Meta-token VIB starts with higher distortion than Last-token VIB at lower rates (e.g., 50 KL: 10.7 vs. 10.6 at 70 KL). 2. **Convergence at High Rates**: - Both methods achieve similar distortion levels at 200 KL (10.0 vs. 10.1). 3. **Efficiency Trade-off**: - Meta-token VIB sacrifices initial performance for better scalability at higher rates. --- ### Interpretation The chart demonstrates a **rate-distortion trade-off** between the two VIB methods. **Last-token VIB** performs better at lower quantization rates, making it suitable for applications requiring high fidelity at minimal compression. Conversely, **Meta-token VIB** becomes more efficient as the rate increases, suggesting it is better suited for scenarios prioritizing compression over absolute distortion. The convergence at 200 KL implies that both methods achieve near-optimal performance at high rates, but the choice depends on the specific rate requirements of the application. The steeper initial decline of Meta-token VIB highlights its potential for rapid distortion reduction when rate flexibility is available. </details> Figure 3: (Left) This plot visualizes the residual stream after each layer, to analyze the meta-token within causal attention. The colors before the meta-token (the colored band across the layers) denote the context which the meta-token attends to and implicitly stores, and the final, rightmost colored line represents the final meta-token in the sequence, which attends to the previous one at the aforementioned band. (Right) We analyze the variational information bottleneck (VIB) objective and its decomposition into its rate and distortion components. Supporting the findings of Theorem 5.1, for a given rate $R$ , the distortion $D$ is strictly lower for the meta-token compared to the last non-meta-token element in the sequence. ### 5.1 Rate-Distortion Informs the Quality of Context Caching To obtain empirical rate–distortion curves for our meta‐token bottleneck in Figure 3, we freeze the pre-trained meta-token model and fix a small variational bottleneck head to the last meta-token hidden state. Concretely, let $h_{m}\in\mathbb{R}^{D}$ be the output of the final Transformer layer at the last meta-token position. We introduce $$ q_{\phi}(z\mid h_{m})=\mathcal{N}\bigl(\mu_{\phi}(h_{m}),\,\mathrm{diag}(\sigma^{2}_{\phi}(h_{m}))\bigr),\quad q_{\theta}(y\mid z)=\mathrm{softmax}(Wz+b), $$ with $\mu_{\phi},\sigma_{\phi}:\mathbb{R}^{D}\to\mathbb{R}^{L}$ and $W\in\mathbb{R}^{|\mathcal{V}|\times L}$ . We then optimize the ELBO: $$ \min_{\phi,\theta}\;\mathbb{E}_{h_{m},y}\bigl[-\log q_{\theta}(y\mid z)\bigr]\;+\;\beta\;\mathbb{E}_{h_{m}}\bigl[\mathrm{KL}\bigl(q_{\phi}(z\mid h_{m})\,\|\,\mathcal{N}(0,I)\bigr)\bigr]. $$ Training is performed on the small List-Pointer D.1.1 split (50 examples, batch size 1), for 5 epochs at each $\beta\in\{0.01,\,0.02,\,0.05,\,0.1,\,0.2,\,0.5,\,1.0\}$ . After each run, we record the average cross-entropy loss (“distortion”) and KL (“rate”) on the same 50 examples. Finally, we plot the resulting rate–distortion curves on a symlog x-axis (linear below 20 nats, logarithmic above) so that both the low-rate “knee” and the long tail are visible (see Figure 3). ### 5.2 Positional Encoding as a Source of Bottleneck Distortion The consistent improvements in recall fidelity and length generalization from zeroing positional encodings at meta-token positions 2 are well-explained through through our rate-distortion analysis. Prior work (e.g., NoPE; Kazemnejad et al., 2023) shows that Transformers can infer token order from the causal mask alone, often generalizing better when explicit positional embeddings (PEs) are removed. Our results extend this insight: in Theorem 5.1, we formalize meta-tokens as learned compression bottlenecks. If a meta-token retains its PE (absolute or rotary), a portion of its representational capacity is spent encoding position rather than semantic content. This index-dependent signal introduces unnecessary variance, increasing the distortion of the compressed summary. By contrast, zeroing out the PE forces the full embedding capacity to encode task-relevant information. As a result, we observe lower distortion (higher retrieval accuracy) at a given rate—both theoretically and empirically—across all four synthetic tasks. ## 6 Related Work Pause and Memory Tokens As detailed in our work, recent studies on Transformer-based models have explored the introduction of special tokens, beyond ordinary vocabulary symbols. Pause or dummy tokens as introduced in Goyal et al. (2024) enhance computational width, allowing models to perform additional internal computation by effectively delaying their outputs. This yields empirical gains on question answering and reasoning-intensive tasks. Similarly, Pfau et al. (2024) explore using filler tokens – sequences of seemingly meaningless symbols – as a stand-in for chain-of-thought. These special tokens may also delineate phases of reasoning, as in Quiet-STaR Zelikman et al. (2024). Quiet-STaR uses a begin-of-thought token and an end-of-thought token, generating a silent rationale sequence for each step before emitting the next word, showing that this helps zero-shot reasoning. Works such as Memory Transformer (Burtsev et al., 2021) and Landmark Attention (Mohtashami and Jaggi, 2023) introduce memory tokens; the former prepends them, while the latter uses them as learnable keys for retrieval over blocks of context. Our work is most closely related to the latter, while performing this retrieval in a purely implicit manner via the observed "pointer" mechanism. For vision transformers (ViTs), LeMeVit (Jiang et al., 2024) introduces a similar meta-tokens notion as our work by adding learnable sparse tokens and an attention mechanism between standard tokens and their meta tokens, improving performance and reducing spatial redundancy. Darcet et al. (2024) uses specialized "register" tokens applies to patches to denoise images by extracting the high-norm, outlier tokens, smoothening the feature and attention maps. These works suggest that special tokens, even devoid of semantic content, can influence a model’s internal reasoning and memory mechanisms. Positional Encoding We have already described absolute positional embeddings (APE), rotary positional embeddings (RoPE) and relative bias in Section 2. In addition to these methods, ALiBi (Press et al., 2022) adds a fixed linear penalty to attention scores based on the distance between query and key positions, favoring nearer tokens and generalizing to longer contexts with minimal loss in perplexity. Recent work has suggested that Transformers without any added position embeddings can still learn order information and, in some cases, generalize to longer sequences better than models with standard positional encoding. NoPE (Kazemnejad et al., 2023) showed that models trained without positional embeddings can achieve strong length extrapolation in comparison to models trained with positional encoding. They can internally represent both absolute and relative PEs without any explicit positional signal, suggesting these may emerge implicitly via training dynamics or over the data distribution. NoPos (Haviv et al., 2022) also found a similar result, suggesting that models trained without PE can infer their absolute position due to causal attention masks. These findings are highly relevant to our work, given our evidence on length generalization behavior whiling zeroing the positional encoding at the meta-tokens. ## 7 Discussion and Limitations Our findings suggest that decoder-only language models trained with meta-tokens and meta-attention achieve strong performance on recall-oriented tasks. Furthermore, they are able to length generalize, with performance improvements when removing the effect of positional encoding at the meta-tokens. Given the prior findings of NoPos, we believe the introduction of the meta-attention mechanism and a second causal mask (the "meta mask") could be responsible for this behavior, provided that this behavior is specific to the meta-tokens. We suggest that hybrid attention methods such as RNoPE (Yang et al., 2025) could be suitable for facilitating long-context modeling with meta-tokens. Given the findings that the meta-tokens operate like anchors within the context, it would be valuable to explore the impact of our proposed mechanism in pre-training larger models over longer context windows, under greater computational resources. We employ synthetic tasks that are well-aligned to recall abilities, and design experiments to test length generalization, with the aim of strong synergy with long-context modeling capabilities. Nonetheless, training larger models would indicate the viability of our approach for real-world deployment. Notably, our method requires little overhead – the addition of meta-tokens is a simple data augmentation strategy, and the meta-attention layer is added after standard causal masked self-attention, as described in Appendix A. It would also be informative to study larger-scale corpora – given the data-efficient nature of the meta-tokens approach in vastly outperforming the vanilla GPT-2 model at the $\approx 100B$ tokens scale, how rapidly does each model saturate our designed synthetic tasks? ## 8 Conclusion We introduce meta-tokens in language model pre-training, in addition to a dedicated meta-attention mechanism which learns the relationship between standard tokens and meta-tokens. We find that this improves performance on a suite of synthetic recall tasks, and enables length generalization behavior when removing the positional encoding at each meta-token. We provide evidence to suggest that the meta-tokens sharpen the positional encoding, enabling them to operate as content-based landmarks in the context; we further show that they implicitly compress preceding context, demonstrated by similar token embeddings. These interesting phenomena demonstrate the promise of long-context language modeling enabled via data-efficient pre-training using meta-tokens. ## References - Alemi et al. (2017) A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy. Deep variational information bottleneck. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=HyxQzBceg. - Beltagy et al. (2020) I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020. - Black et al. (2021) S. Black, G. Leo, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/zenodo.5297715. If you use this software, please cite it using these metadata. - Brown et al. (2020a) T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020a. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. - Brown et al. (2020b) T. B. Brown, B. Mann, N. Ryder, et al. Language models are few-shot learners. NeurIPS, 2020b. - Burtsev et al. (2021) M. S. Burtsev, Y. Kuratov, A. Peganov, and G. V. Sapunov. Memory transformer, 2021. URL https://arxiv.org/abs/2006.11527. - Chen et al. (2023) T. D. Chen et al. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023. - Chowdhery et al. (2022) A. Chowdhery, S. Narang, J. Devlin, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. - Cover and Thomas (2006) T. M. Cover and J. A. Thomas. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, USA, 2006. ISBN 0471241954. - Darcet et al. (2024) T. Darcet, M. Oquab, J. Mairal, and P. Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=2dnO3LLiJ1. - Goyal et al. (2024) S. Goyal, Z. Ji, A. S. Rawat, et al. Think before you speak: Training language models with pause tokens. In ICLR, 2024. - Grattafiori et al. (2024) A. Grattafiori et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783. - Haviv et al. (2022) A. Haviv, O. Ram, O. Press, P. Izsak, and O. Levy. Transformer language models without positional encodings still learn positional information. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1382–1390, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-emnlp.99. URL https://aclanthology.org/2022.findings-emnlp.99/. - Hutchins et al. (2022) D. Hutchins et al. Block-recurrent transformers. In NeurIPS, 2022. - Jiang et al. (2024) W. Jiang, J. Zhang, D. Wang, Q. Zhang, Z. Wang, and B. Du. Lemevit: Efficient vision transformer with learnable meta tokens for remote sensing image interpretation. In K. Larson, editor, Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, pages 929–937. International Joint Conferences on Artificial Intelligence Organization, 8 2024. doi: 10.24963/ijcai.2024/103. URL https://doi.org/10.24963/ijcai.2024/103. Main Track. - Karpathy (2023) A. Karpathy. nanoGPT. https://github.com/karpathy/nanoGPT, 2023. Accessed: 2025-05-16. - Kazemnejad et al. (2023) A. Kazemnejad, I. Padhi, K. Natesan, P. Das, and S. Reddy. The impact of positional encoding on length generalization in transformers. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=Drrl2gcjzl. - Marshall et al. (2011) A. W. Marshall, I. Olkin, and B. C. Arnold. Inequalities: Theory of Majorization and Its Applications. Springer Series in Statistics. Springer New York, New York, NY, 2 edition, 2011. ISBN 978-0-387-68276-1. doi: 10.1007/978-0-387-68276-1. - Mohtashami and Jaggi (2023) A. Mohtashami and M. Jaggi. Random-access infinite context length for transformers. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=7eHn64wOVy. - OpenAI (2023) OpenAI. Gpt-4 technical report. https://openai.com/research/gpt-4, 2023. - Peng et al. (2023) B. Peng, J. Quesnelle, H. Fan, and E. Shippole. Yarn: Efficient context window extension of large language models, 2023. URL https://arxiv.org/abs/2309.00071. - Pfau et al. (2024) J. Pfau, W. Merrill, and S. R. Bowman. Let’s think dot by dot: Hidden computation in transformer language models. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=NikbrdtYvG. - Press et al. (2021) O. Press, N. A. Smith, and O. Levy. Alibi: Trainable linear biases for transformers. arXiv preprint arXiv:2108.12409, 2021. - Press et al. (2022) O. Press, N. A. Smith, and M. Lewis. Train short, test long: Attention with linear biases enables input length extrapolation, 2022. URL https://arxiv.org/abs/2108.12409. - Radford et al. (2019) A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. OpenAI Technical Report, 2019. URL https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf. Accessed: 2025-05-15. - Raffel et al. (2020) C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1), Jan. 2020. ISSN 1532-4435. - Shaw et al. (2018) P. Shaw, J. Uszkoreit, and A. Vaswani. Self-attention with relative position representations. In M. Walker, H. Ji, and A. Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464–468, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-2074. URL https://aclanthology.org/N18-2074/. - Su et al. (2021) J. Su, Y. Lu, S. Pan, et al. Roformer: Enhanced transformer with rotary position embedding. In ACL, 2021. - Su et al. (2023) J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. URL https://arxiv.org/abs/2104.09864. - Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. - Yang et al. (2025) B. Yang, B. Venkitesh, D. Talupuru, H. Lin, D. Cairuz, P. Blunsom, and A. Locatelli. Rope to nope and back again: A new hybrid attention strategy, 2025. URL https://arxiv.org/abs/2501.18795. - Zaheer et al. (2020) M. Zaheer, G. Guruganesh, A. Dubey, et al. Big bird: Transformers for longer sequences. In NeurIPS, 2020. - Zelikman et al. (2024) E. Zelikman, G. R. Harik, Y. Shao, V. Jayasiri, N. Haber, and N. Goodman. Quiet-STar: Language models can teach themselves to think before speaking. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=oRXPiSOGH9. ## Appendix A Full Architecture Details We provide a full outline of the architecture design out method uses. Our architecture is equivalent to the NanoGPT (GPT-2) architecture, while introducing the meta-attention block after the initial causal masked attention and layer normalization computation. 1. Input Layer: Given an input sequence of tokens $\mathbf{x}=\{x_{1},x_{2},\dots,x_{T}\}$ , we first embed each token into a continuous representation. Instead of absolute positional encodings, we apply Rotary Position Embeddings (RoPE) Su et al. (2023) to inject positional information. For each token, the embedded representation is: $$ \mathbf{e}_{t}=\text{RoPE}(E(x_{t}),t), $$ where $\text{RoPE}(\cdot,t)$ denotes the rotary positional embedding applied to the $t^{\text{th}}$ position, with a base $\theta=10000.0$ . 1. Causal Masked Self-Attention: The first layer consists of the causal masked self-attention mechanism. For each head $h$ , the attention operation is computed as: $$ \text{CausalAttention}_{h}(Q,K,V)=\text{softmax}\left(\frac{QK_{h}^{\top}}{\sqrt{d_{k}}}+M\right)V_{h}, $$ where $Q,K,V$ are the query, key, and value matrices derived from the input embeddings $\mathbf{E}$ , and $M$ is the mask matrix. 1. Meta Attention Layer: After the causal masked self-attention, we integrate the meta-attention mechanism to specifically attend to the injected meta tokens. This operation is defined as: $$ \text{MetaAttention}(Q,K,V,P)=\text{softmax}\left(\frac{QK^{\top}}{\sqrt{d_{k}}}+M_{\text{causal}}+P\right)V, $$ where $P$ is the meta mask constructed from the indices of the meta tokens. 1. Feedforward Layer: Following the attention layers, we pass the output through a feedforward neural network defined by: $$ \text{FFN}(x)=\text{ReLU}(xW_{1}+b_{1})W_{2}+b_{2}, $$ where $W_{1},W_{2}$ are weight matrices, and $b_{1},b_{2}$ are bias vectors. 1. Layer Normalization: After both the causal self-attention and meta-attention operations, we apply layer normalization: $$ \text{LayerNorm}(x)=\frac{x-\mu}{\sigma+\epsilon}, $$ where $\mu$ and $\sigma$ are the mean and standard deviation of the features, and $\epsilon$ is a small constant for numerical stability. 1. Final Output Layer: The final layer projects the output of the last feedforward layer back to the vocabulary size to produce the s for the next token prediction: $$ \text{s}=\text{softmax}(xW_{\text{out}}+b_{\text{out}}), $$ where $W_{\text{out}}$ and $b_{\text{out}}$ are the output weight matrix and bias vector, respectively. ## Appendix B Pre-training Hyperparameters and Model Details Our decoder-only modified GPT-2 model was pre-trained on the C4 dataset with the following configuration and hyperparameters: Table 4: Pretraining Configuration Parameters | Batch Size Gradient Accumulation Steps Block Size | 12 40 1024 | | --- | --- | | Number of Layers | 12 | | Number of Heads | 12 | | Embedding Size | 768 | | Learning Rate | 6e-4 | | Weight Decay | 1e-1 | | Max Iterations | 600,000 | | Warmup Iterations | 2,000 | | Minimum Learning Rate | 6e-5 | | Dropout Rate | 0.0 | | RoPE Theta | 10000.0 | | Initial Model | Resume | | Optimizer | AdamW | | AdamW Beta1 | 0.90 | | AdamW Beta2 | 0.95 | | Gradient Clipping | 1.0 | | Tokenizer | tiktoken | ## Appendix C YaRN Hyperparameters | yarn_extrapolation_factor yarn_attn_factor yarn_beta_fast | 1.0 1.0 32.0 | | --- | --- | | yarn_beta_slow | 1.0 | Table 5: YaRN parameter configurations for extended context models. ## Appendix D Additional Experimental Details ### D.1 Synthetic Data Generation We generate 90,000 train examples and held-out test set of 10,000 examples for each task. #### D.1.1 List Recall We generate a suite of “list-pointer” examples by sampling random categories and list items, inserting a special meta token as a marker, and asking the model to recover the item immediately following the meta-token. Each example consists of: 1. m categories drawn without replacement from a fixed set of 20. 1. n items per category, sampled with replacement from the category’s 10–item inventory 1. One “target” category in which we inject a single meta token after the jth item ( $j\in[n]$ ) and then append the remaining items 1. A question line “Q: What is item j of <target>? _META_” This pipeline yields curriculum‐structured data that systematically probes the model’s ability to attend to and copy items in long, multi‐list contexts. | 1 2 3 | Uniform $3\text{--}8$ Uniform $8\text{--}12$ Uniform $12\text{--}19$ | Uniform $3\text{--}10$ Uniform $3\text{--}16$ (bimodal) Mixture $\{3\text{--}8,9\text{--}16,17\text{--}25\}$ | Short ( $\approx 100\text{--}200$ tokens) Mid‐range ( $\approx 200\text{--}300$ tokens) Full‐range ( $\approx 500\text{--}700$ tokens) | | --- | --- | --- | --- | | 4 | Uniform $15\text{--}20$ | Uniform $40\text{--}60$ | “Extra‐hard” $\leq 1024$ tokens | | 5 | Uniform $15\text{--}20$ | Uniform $90\text{--}110$ | “Long” $\leq 2048$ tokens | Table 6: Curriculum schedule for synthetic data. #### D.1.2 Segment Counting Similar to List-pointer, except the model must count occurrences of a target token within a meta-token bracketed list segment. Uses the schedule dictated by Table 6. Asks the question: "Q: How many times does <token> appear between the pauses around <Category>? _META_". #### D.1.3 Parity Generates examples where the model computes the XOR (parity) of a bit‐string segment up to the first L characters where L is drawn phase-dependently. the same scheduling dictated by Table 6. Asks the question: "Q: What is the XOR of all bits before this pause? _META_ " #### D.1.4 Copying Generates examples where the model must copy a bracketed span from a text. Uses schedule dictated by Table 6 and samples an additional copy length $C$ and distance length $D$ depending on the phase ## Appendix E Licenses ### nanoGPT Our implementation of the vanilla GPT-2 is based on the nanoGPT repository (https://github.com/karpathy/nanoGPT), which is licensed under the MIT License. ### EleutherAI GPT-Neo-125M We directly use the EleutherAI GPT-Neo 125M model checkpoint and weights, available via the Hugging Face Model Hub at https://huggingface.co/EleutherAI/gpt-neo-125m. This model is released under the MIT License. ### C4 Dataset Our model was trained on the C4 dataset (https://huggingface.co/datasets/allenai/c4), which is provided under the Open Data Commons Attribution License (ODC-BY). ### tiktoken We use the tiktoken library from OpenAI for tokenization (https://github.com/openai/tiktoken), which is released under the MIT License. ## Appendix F Complete Experimental Results ### F.1 Synthetic Task Accuracies Across Test Lengths We stress test RoPE models at a sequence length of 2048—twice the pretraining block size of 1024—as relative position embeddings naturally support extrapolation beyond the training context window. In contrast, absolute positional encodings (APE) cannot generalize to sequences longer than those seen during pretraining. Table 7: Accuracy (%) across evaluation lengths for each model train on List Recall Table 8: Accuracy (%) across evaluation lengths for each model trained on Segment Counting. Each model is evaluated on longer contexts than seen during training. Table 9: Accuracy (%) across evaluation lengths for each model train on Parity Table 10: Accuracy (%) across evaluation lengths for each model trained on Copying ### F.2 Ablations on Positional Encoding and Token Embedding Table 11: Accuracy (%) on the List-Recall task under different ablations: zeroing the positional encoding (No Pos), zeroing the text embeddings (No Embed), or zeroing both of the meta-tokens. Table 12: Accuracy (%) on the Segment Counting task under different ablations: zeroing the positional encoding (No Pos), text embeddings (No Embed), or both, only on the meta-token. Table 13: Accuracy (%) on the Parity task under different ablations: zeroing the positional encoding (No Pos), text embeddings (No Embed), or both, only on the meta-token. Table 14: Accuracy (%) on the Copying task under different ablations: zeroing the positional encoding (No Pos), text embeddings (No Embed), or both, only on the meta-token. ### F.3 Positional Encoding Robustness Ablations Table 15: Accuracy (%) on the List Pointer task with Gaussian noise added to positional encoding. Table 16: Accuracy (%) on the Copying task with Gaussian noise added to positional encoding. ### F.4 Length Generalization Ability under No Positional Encoding Ablation Table 17: List-Recall: “No Pos” vs. Full accuracy for Meta-attention with APE and Meta-attention with RoPE. $$ \Delta \tag{128} $$ ## Appendix G Theoretical Analysis ### G.1 Proof of Theorem 4.1 **Lemma G.1** *Let $\ell_{1},\ell_{2},\dots,\ell_{N}$ be logits and define softmax distribution $\alpha_{j}=\frac{\exp(\ell_{j})}{\sum_{k=1}^{N}\exp(\ell_{k})}$ . Suppose that for some "correct" index $j^{*}$ we have $\ell_{j^{*}}=L$ , and for all other indices $j\neq j^{*}$ , $\ell_{j}\leq L-\Delta$ for some $\Delta>0$ . Then, entropy $H(\alpha)$ is strictly decreasing in $\Delta$ .* * Proof* First, we can group the other logits (i.e. $j\neq j^{*}$ , such that $S=\sum\limits_{j\neq j^{*}}e^{\ell_{j}}$ . Then, since each $\ell_{j}$ carries the property that $e^{\ell_{j}}\leq e^{L-\Delta}$ given $\ell_{j^{*}}=L$ , we have that $S\leq(N-1)e^{L-\Delta}$ since there are $N-1$ terms. Revisiting the softmax $\alpha$ , we have that $\alpha_{j^{*}}=\frac{e^{L}}{e^{L}+S}\geq\frac{e^{L}}{e^{L}+(N-1)e^{L-\Delta}}=\frac{1}{1+(N-1)e^{-\Delta}}$ . We will denote this quantity as $p$ henceforth. Next, each other softmax $\alpha_{j}$ for $j\neq j^{*}$ must have the property that $\alpha_{j}=\frac{e^{\ell}}{e^{L}+S}\leq\frac{e^{L-\Delta}}{e^{L}(1+(N-1)e^{-\Delta})}=\frac{e^{-\Delta}}{1+(N-1)e^{-\Delta}}=\frac{1-p}{N-1}$ . As a result, we have the following entropy maximization problem: | | | $\displaystyle\underset{\alpha_{1},\dots,\alpha_{N}}{\text{maximize}}$ | | $\displaystyle-\sum_{j=1}^{N}\alpha_{j}\,\log\alpha_{j}$ | | | --- | --- | --- | --- | --- | --- | Observe that the entropy (objective) function is Schur-concave in $\alpha$ , so it is maximized when $\alpha_{j^{*}}=p$ and the remaining softmax mass is split uniformly over the $N-1$ elements, i.e. $\alpha_{j}=\frac{1-p}{N-1}\hskip 1.42262pt\forall\hskip 1.42262ptj\neq j^{*}$ . Plugging this in for $H(\alpha)$ yields: $$ \displaystyle H(\alpha)\leq-p\log p-(1-p)\log(1-p)+(1-p)\log(N-1) \tag{3} $$ Next, we aim to study the relationship between $H$ and $\Delta$ . By the chain rule, $\frac{dH}{d\Delta}=\frac{dH}{dp}\cdot\frac{dp}{d\Delta}$ . $\frac{dH}{dp}=-(1+\log p)+\log\frac{1-p}{N-1}+1=\log\frac{1-p}{(N-1)p}$ . Substituting $\frac{1-p}{p}=(N-1)e^{-\Delta}$ , we get $\frac{dH}{dp}=-\Delta$ and since $\Delta>0$ , $\frac{dH}{dp}<0$ . We then turn to $\frac{dp}{d\Delta}=\frac{(N-1)e^{-\Delta}}{[1+(N-1)e^{-\Delta}]^{2}}>0$ since both numerator and denominator must be $>0$ . Therefore, $\frac{dH}{d\Delta}=-\Delta\frac{(N-1)e^{-\Delta}}{[1+(N-1)e^{-\Delta}]^{2}}<0$ , meaning that $H(\alpha)$ is strictly decreasing in the margin $\Delta$ . ∎ We will now use Lemma G.1 to prove Theorem 4.1. * Proof of Theorem 4.1* Consider a parametrized path by variable $t\in[0,\Delta]$ ; define $\ell_{j}^{(t)}=\ell_{j}+\delta_{j,j^{*}}t$ , and $\alpha_{j}^{(t)}=\frac{e^{\ell_{j}^{(t)}}}{\sum\limits_{k=1}^{N}e^{\ell_{k}^{(t)}}}=\frac{e^{(\ell_{j}+\delta_{j,j^{*}}t)}}{\sum\limits_{k=1}^{N}e^{(\ell_{k}+\delta_{k,j^{*}}t)}}$ . Define $\ell_{j}^{{}^{\prime}(t)}=\frac{d}{dt}\ell_{j}^{(t)}$ and $\alpha_{j}^{{}^{\prime}(t)}=\frac{d}{dt}\alpha_{j}^{(t)}$ . Next, we differentiate the entropy $H(\alpha)$ with respect to $t$ : | | $\displaystyle\frac{d}{dt}H(\alpha)=-\sum_{j=1}^{N}[\alpha_{j}^{\prime}\ln\alpha_{j}+\alpha_{j}\frac{\alpha_{j}^{\prime}}{\alpha_{j}}]=-\sum\limits_{j=1}^{N}\alpha_{j}^{\prime}(1+\ln\alpha_{j})=-\sum_{j=1}^{N}\alpha_{j}^{\prime}+\alpha_{j}^{\prime}\ln\alpha_{j}$ | | | --- | --- | --- | Since $\sum{\alpha_{j}^{\prime}}=0$ due to $\sum\alpha_{j}=1$ , this simply reduces to $\frac{d}{dt}H(\alpha)=-\sum_{j=1}^{N}\alpha_{j}^{\prime}\ln\alpha_{j}$ . From Cover and Thomas (2006), we have that $\alpha_{j}^{\prime}=\alpha_{j}(\ell_{j}-\mathop{\mathbb{E}}_{\alpha}[\ell^{\prime}])$ , where $\mathop{\mathbb{E}}_{\alpha}[\ell^{\prime}]=\sum\limits_{k=1}^{N}\alpha_{k}\ell_{k}^{\prime}$ . Plugging this into the expression for the derivative of entropy with respect to $t$ : | | $\displaystyle\frac{d}{dt}H(\alpha)=-\sum_{j}\alpha_{j}(\ell_{j}^{\prime}-\mathop{\mathbb{E}_{\alpha}[\ell^{\prime}]})\ln\alpha_{j}=-(\sum_{j}a_{j}\ell_{j}^{\prime}\ln\alpha_{j}-\mathop{\mathbb{E}_{\alpha}[\ell^{\prime}]}\sum_{j}\alpha_{j}\ln\alpha_{j})$ | | | --- | --- | --- | Observe that $\sum_{j}\alpha_{j}\ln\alpha_{j}=\mathop{\mathbb{E}_{\alpha}}[\ln\alpha]$ so this simply reduces as: $$ \displaystyle\frac{d}{dt}H(\alpha)=-(\mathop{\mathbb{E}_{\alpha}}[\ell^{\prime}\ln\alpha]-\mathop{\mathbb{E}_{\alpha}}[\ell^{\prime}]\mathop{\mathbb{E}_{\alpha}}[\ln\alpha])=-\text{Cov}_{\alpha}(\ell^{\prime},\ln\alpha) \tag{4} $$ Revisiting the meta-token setup where only the "correct" logit at $j^{*}$ is boosted, this suggests that $\ell_{j}^{\prime}=\textbf{1}(j=j^{*})$ . Therefore, $\mathop{\mathbb{E}_{\alpha}}[\ell^{\prime}]=\alpha_{j^{*}}$ and $\mathop{\mathbb{E}_{\alpha}}[\ell^{\prime}\ln\alpha]=\alpha_{j^{*}}\ln\alpha_{j^{*}}$ . This can be substituted into the covariance term above: $$ \frac{d}{dt}H(\alpha)=-\text{Cov}_{\alpha}(\ell^{\prime},\ln\alpha)=-(\alpha_{j^{*}}\ln\alpha_{j^{*}}-\alpha_{j^{*}}\mathop{\mathbb{E}_{\alpha}}[\ln\alpha])=-\alpha_{j^{*}}(\ln\alpha_{j^{*}}-\mathop{\mathbb{E}_{\alpha}}[\ln\alpha]) $$ Due to the Schur-concavity of $H(\alpha)$ (Marshall et al., 2011), $\ln\alpha_{j^{*}}=\max_{j}\ln\alpha_{j}$ and $\ln\alpha_{j^{*}}>\mathop{\mathbb{E}_{\alpha}}[\ln\alpha]$ . As such, given $\alpha_{j^{*}}>0$ and $\ln\alpha_{j^{*}}-\mathop{\mathbb{E}_{\alpha}}[\ln\alpha]>0$ , this suggests that $\text{Cov}_{\alpha}(\ell^{\prime},\ln\alpha)>0$ and thus, $\frac{d}{dt}H(\alpha)<0$ . Therefore, we conclude that adding a positive logit boost at the meta-token index ("correct" logit) strictly decreases entropy, supporting the proposed "anchoring" effect notion. ∎ ### G.2 Proof of Theorem 5.1 * Proof of Theorem 5.1* The meta-tokens are simply a new (latent) channel that may be utilized to search for candidate distributions. However, this latent can be ignored, yielding the original search space; that is, any encoder $q_{\phi}(\hat{x}\mid x)$ that does not use meta-tokens can be implemented in the meta-token model by zeroing out all meta-token contributions. Therefore, $\mathcal{Q}_{\rm abs}\subseteq\mathcal{Q}_{\rm meta}$ , where $q=(q_{\phi},q_{\theta})$ over the feasible combinations of encoder and decoder. Naturally, minimizing a function over a larger feasible set cannot increase its minimum. Thus, for a fixed rate $R$ , $$ D_{\rm meta}(R)=\min_{q\in\mathcal{Q}_{\rm meta}\,\colon\,I(X;\hat{X})=R}D(q)\;\leq\;\min_{q\in\mathcal{Q}_{\rm abs}\,\colon\,I(X;\hat{X})=R}D(q)=D_{\rm abs}(R). $$ Note that the same result holds for RoPE in place of APE (i.e. $D_{\text{RoPE}}$ in place of $D_{\text{abs}}$ ), as well. ∎ ### G.3 Theorem C.2 **Theorem G.2** *Consider functions $p:\{0,\dots,T-1\}\rightarrow\mathbb{R}$ and $b:\{-(T-1),\dots,T-1\}\rightarrow\mathbb{R}$ for absolute postional biases and relative biases, respectively. Let $\mathcal{B}_{\text{abs}}$ to be the set of all fixed absolute positional bias matrices $B_{i,j}^{\text{abs}}=p(j)$ and $\mathcal{B}_{\text{rel}}$ to be the set of all fixed relative biases $B_{i,j}^{\text{rel}}=b(i-j)$ . Let $\mathcal{B}_{\text{meta}}$ be the set of bias matrices implementable by the Transformer augmented with meta-token embeddings $\{m_{t}\}$ which emit a content-dependent logit boost at their respective indices. Then, $$ \displaystyle\mathcal{B}_{\text{abs}}\cup\mathcal{B}_{\text{rel}}\subsetneq\mathcal{B}_{\text{meta}} \tag{5} $$* * Proof* We break this argument down into two parts $\rightarrow$ (i.) the forward direction, where we show that all absolute and relative biases without meta-tokens can be modeled by the meta-token model. (i) $\mathcal{B}_{\rm abs}\cup\mathcal{B}_{\rm rel}\subseteq\mathcal{B}_{\rm meta}$ . Every $B\in\mathcal{B}_{\rm meta}$ is obtained by choosing meta‐token embeddings $e_{t}\in\mathbb{R}^{d}$ at each position $t$ and a linear head $W$ , so that the total bias at $(i,j)$ is $B_{i,j}=\sum_{t}\;Q_{i}^{\top}W\,e_{t}\;\mathbf{1}_{j=t}$ . - Absolute case. Given $p(j)$ , set $W\in\mathbb{R}^{1\times d}$ and choose each $e_{j}$ so that $Q_{i}^{\top}W\,e_{j}=p(j)\hskip 2.84526pt\forall\hskip 2.84526pti.$ All other $e_{t\neq j}$ are zero. Then $B_{i,j}=p(j)$ . - Relative case. Given $b(i-j)$ , place a meta‐token at every position $j$ . Choose $W$ and embeddings $e_{j}$ so that $Q_{i}^{\top}W\,e_{j}=b(i-j)\hskip 2.84526pt\forall\hskip 2.84526pti,j.$ For instance, if we let $W=\mathrm{I}d$ and arrange that $e_{j}$ encodes the vector $\bigl(b(1-j),b(2-j),\dots,b(T-j)\bigr)$ , then $Q_{i}^{\top}e_{j}=b(i-j)$ when $Q_{i}$ is the $i$ -th standard basis vector. Therefore, every absolute or relative bias (in $\mathcal{B}_{\rm abs}$ and $\mathcal{B}_{\rm rel}$ ) lies in $\mathcal{B}_{\rm meta}$ . (ii) There exists a bias $B^{*}\in\mathcal{B}_{\rm meta}$ such that $B^{*}\notin\mathcal{B}_{\rm abs}\cup\mathcal{B}_{\rm rel}$ . Define a content‐dependent bias $B^{*}_{i,j}=f\bigl(C_{j}\bigr)$ where $C_{j}$ is the full token context preceding position $j$ and $f$ is any non‐constant function. Such a $B^{*}$ arises by setting each meta-token embedding $e_{j}=f(C_{j})$ and $W=\mathrm{I}d$ , so $B^{*}\in\mathcal{B}_{\rm meta}$ . However, if there was $B^{*}\in\mathcal{B}_{\rm abs}$ , then there is $p(j)$ with $p(j)=f(C_{j})$ for all $j$ and all possible $C_{j}$ , which is impossible since $C_{j}$ varies. Furthermore, if $B^{*}\in\mathcal{B}_{\rm rel}$ , then there is $b(i-j)$ with $b(i-j)=f(C_{j})$ independent of $i$ ; again, this condition is impossible to be satisfied. Therefore $B^{*}\notin\mathcal{B}_{\rm abs}\cup\mathcal{B}_{\rm rel}$ . As a result, we conclude that the biases represented by $\mathcal{B}_{\rm meta}$ contain the set of both absolute and relative biases without meta-tokens, and represent additional biases that cannot be represented without meta-tokens. ∎ The result of Theorem G.2 is that the introduction of meta-tokens strictly grows the expressivity of biases that may be represented, while still being entirely inclusive of the fixed realizable absolute and relative encoding biases. As a result, we do not "lose" anything representationally by introducing meta-tokens, from a positional biases standpoint. This enhanced expressive power also plays a role in enabling the model to learn to focus attention on relevant context spans, reinforcing the aforementioned sharpening effect. ## Appendix H Example Prompts for Synthetic Tasks We provide example prompts and target completions for each synthetic task used in our experiments. Each example illustrates the input format, the placement of _PAUSE_ meta-tokens, and the expected model output. List Recall Prompt: Fruits: orange peach banana plum Tools: hammer wrench level pliers Sports: boxing baseball boxing golf Spices: turmeric cardamom cumin cinnamon Animals: cat tiger wolf fox Professions: teacher nurse lawyer architect Vegetables: onion cucumber broccoli cucumber Instruments: piano clarinet violin piano Professions: teacher nurse lawyer _PAUSE_ architect Q: What is item 3 of Professions? _PAUSE_ Completion: lawyer Segment Counting Prompt: Fruits: orange peach banana plum Tools: _PAUSE_ hammer wrench level pliers _PAUSE_ Sports: boxing baseball boxing golf Spices: turmeric cardamom cumin cinnamon Q: How many times does wrench appear between the pauses around Tools? _PAUSE_ Completion: 1 Parity Prompt: Bits: 0 _PAUSE_ 1 0 0 Q: What is the XOR of all bits before this pause? _PAUSE_ Completion: 0 Copying Prompt: ...as ever Mr _PAUSE_ . Darcy had been, was much relieved by discerning such different feelings. They had not been long together before Darcy told her that Bingley was also coming to wait on her; and she had barely time to express her satisfaction, and prepare for such a visitor, when Bingleys quick step was heard on the stairs, and in a moment he entered the room. All Elizabeths anger against him had been long done away; but had she still felt any, it could hardly have stood its ground against the unaffected cordiality with which he expressed himself on seeing her again. He inquired in a friendly, though general, way, after her family, and looked and spoke with the same _PAUSE_ good-humoured ease that he had ever done. Q: Copy the bracketed text. _PAUSE_ Completion: . Darcy had been, was much relieved by discerning such different feelings. They had not been long together before Darcy told her that Bingley was also coming to wait on her; and she had barely time to express her satisfaction, and prepare for such a visitor, when Bingleys quick step was heard on the stairs, and in a moment he entered the room. All Elizabeths anger against him had been long done away; but had she still felt any, it could hardly have stood its ground against the unaffected cordiality with which he expressed himself on seeing her again. He inquired in a friendly, though general, way, after her family, and looked and spoke with the same. ## Appendix I Broader Impacts Statement Our work on learned meta-tokens and meta-attention offers a lightweight, data-efficient way to pre-train language models while demonstrating strong performance when fine-tuned for recall tasks. This suggests a path toward more capable, leaner language models that could be used to handle contexts such as like long legal or medical documents, extended multi-turn dialogues, or large codebases without resorting to prohibitively large architectures or expensive fine-tuning runs. Such models could bring real benefits to areas such as conversational agents for education or healthcare. Building off of prior literature that performs a more explicit learned retrieval from the context (Mohtashami and Jaggi, 2023), this could induce improved and efficient in-line retrieval over vast corpora. Our work relates strongly to the recent debates in the language modeling community on the impact of positional encoding, particularly around works such as NoPE (Kazemnejad et al., 2023). We provide strong evidence that zeroing the positional encoding can improve performance, providing motivation for hybrid attention mechanisms such as RNoPE (Yang et al., 2025), and other, more efficient ways to pre-train language models with long-context modeling settings in mind. We note that advances in long-context modeling could introduce risks around misuse and unintended harm. More powerful context understanding over long ranges can fuel phishing text and distracted models, especially in the phase of noisy context. However, models trained on corpora without data pre-processing a priori may be subject to harmful behavior such as profane generations. In the context of our work, which uses standard, pre-filtered corpora, this issue is avoided; we encourage users to audit the data used for pre-training first.