2601.13260v2

# Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models > Equal contribution. Abstract Tokenization underlies every large language model, yet it remains an under-theorized and inconsistently designed component. Common subword approaches such as Byte Pair Encoding (BPE) offer scalability but often misalign with linguistic structure, amplify bias, and waste capacity across languages and domains. This paper reframes tokenization as a core modeling decision rather than a preprocessing step. We argue for a context-aware framework that integrates tokenizer and model co-design, guided by linguistic, domain, and deployment considerations. Standardized evaluation and transparent reporting are essential to make tokenization choices accountable and comparable. Treating tokenization as a core design problem, not a technical afterthought, can yield language technologies that are fairer, more efficient, and more adaptable. Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models Sawsan Alqahtani ${}^{\text{\textdagger}\spadesuit}$ thanks: Equal contribution. Mir Tafseer Nayeem footnotemark: Md Tahmid Rahman Laskar Tasnim Mohiuddin M Saiful Bari ${}^{\text{\textdagger}}$ Princess Nourah Bint Abdulrahman University Saudi Data & AI Authority (SDAIA) University of Alberta Dialpad York University Qatar Computing Research Institute Amazon AGI saalqhtani@pnu.edu.sa, mnayeem@ualberta.ca 1 Introduction Large Language Models (LLMs) have achieved remarkable success across tasks ranging from fluent text generation to advanced reasoning that now pushes the boundaries of agentic behavior (Laskar et al., 2023a; OpenAI, 2024; Grattafiori et al., 2024; Anil et al., 2025; Anthropic, 2025; DeepSeek-AI et al., 2025; Langford et al., 2025). Yet, amid the focus on architectural innovations, training methods, and alignment techniques, a fundamental component of LLM development has been overlooked: tokenization, the process of segmenting raw text into discrete units that serve as the interface between text and learned representations. This choice shapes a model’s representation, efficiency, and generalization across tasks, languages, and domains (Ali et al., 2024; Goldman et al., 2024; Bommarito et al., 2025; Altıntaş et al., 2025). The NLP community continues to rely on subword tokenization methods, particularly Byte Pair Encoding (BPE) (Gage, 1994; Sennrich et al., 2016), as the de facto standard for managing vocabulary and rare words. This reliance has produced uniform designs that mask trade-offs and ignore task-, language- or domain-specific needs. This fragmentation disrupts linguistic coherence See Appendix A and undermines model performance on compositional tasks such as code generation, multi-step reasoning, and multi-hop question answering (Bostrom and Durrett, 2020; Ahia et al., 2023; Petrov et al., 2023; Dewangan et al., 2025; Ni et al., 2025). These limitations have prompted a growing body of research to reexamine traditional tokenization practices, questioning long-held assumptions about the necessity of explicit segmentation and the impact of poorly aligned boundaries on model performance (Zouhar et al., 2023a; Goldman et al., 2024; Schmidt et al., 2024). Current practices also face methodological shortcomings: obscure design processes and poor documentation hinder reproducibility and adaptation to new domains and languages (Bostrom and Durrett, 2020; Dagan et al., 2024). These issues are compounded by the absence of standardized evaluation metrics and diagnostic tools, making it difficult to systematically assess tokenizer effectiveness (Beinborn and Pinter, 2023; Ali et al., 2024). Moreover, treating tokenization as a static, detached process prevents co-adaptation between segmentation and representation learning, limiting potential gains in efficiency and linguistic fidelity (Land and Bartolo, 2024; Zheng et al., 2024). Together, these gaps highlight that tokenization remains an open and under-theorized area in LLM development. In practice, developers face consequential design decisions, including choosing token granularity, defining merge operations, and curating training corpora. As a result, many pipelines default to reusing pretrained tokenizers, a practice that often misaligns with the linguistic characteristics or practical constraints of the target setting (Chen et al., 2023; Dagan et al., 2024; Nayeem and Rafiei, 2024; Seo et al., 2025; Bommarito et al., 2025). This mismatch reinforces inefficiencies and biases, highlighting the need for explicit, principled, context-aware decision-making around tokenizer auditing, reuse, adaptation, or retraining, grounded in systematic evaluation. 1.1 Our Perspective on Tokenization In this paper, we argue that tokenization, despite its maturity, continues to be treated as an afterthought, implemented with insufficient scrutiny, contextual awareness, and methodological rigor. While subword tokenization was originally designed to balance the trade-offs between word- and character-level approaches, its reuse should not exempt it from renewed examination. We advocate for a shift from ad-hoc practices to a principled, context-aware design and evaluation process, aligned with model architecture and training objectives, elevating tokenization to an essential component of model development. We emphasize that this perspective neither assumes nor prioritizes retraining LLMs from scratch. Such retraining is often infeasible for many practitioners and language communities, particularly in low-resource settings, and even when feasible, may be unnecessary or suboptimal relative to principled reuse or adaptation. Our position is supported by Dewangan et al. (2025), who show that even with fixed vocabulary sizes, optimizing the segmentation strategy alone can yield substantial gains. Yet, despite widespread reliance on standard tokenizers, the field lacks a unified and theoretically grounded approach to its selection, adaptation, and evaluation. To bridge this gap, we advocate a principled framework that integrates tokenizer–model co-design, standardized evaluation, and transparent documentation. As illustrated in Figure 1, such a shift replaces rigid practices with a more adaptive and equitable foundation for language technology. Our paper advances this agenda through three contributions: - Identifying the hidden design choices in standard tokenization: vocabulary construction, granularity, corpus selection, and reuse. - Proposing a structured tokenizer-design framework that accounts for linguistic diversity, domain specificity, and task complexity. - Defining evaluation dimensions for fairness, representational alignment, and coverage. By challenging the view of tokenization as an afterthought, we reposition it as a foundational area of empirical inquiry, essential for building language models that are not only efficient but also adaptable and equitable across domains and languages. Current Practices Proposed Design Tokenizer Design Default to Inherited Tokenizer Used without re-evaluation Assumed compatibility Build Tokenizer from Scratch Corpus- and task-specific Critical reassessment Preprocessing Step Mindset Tokenization detached from modeling Rarely documented Treat as Design Primitive Aligned with embedding training Part of system architecture Tokenizer Evaluation Lack of Metrics and Assessment Heuristics dominate (e.g., fertility) Few diagnostics for task fit Systematic Evaluation Multi-dimensional metrics Task- and language-aware Tokenizer Reliability Ignored Risks and Biases Undertrained tokens overlooked Fragmentation harms fairness Audited and Safe Tokenization Analyze token frequency/stability Apply anomaly and bias detection Generalization & Adaptation Context-Blind Design Ignores language structure Over-fragmentation in rich morphology Context-Aware Strategy Morphology-aware segmentation Multilingual adaptability Uniform Across Domains Same tokenizer reused Naive vocabulary expansion Adapted to Use Case Informed vocabulary expansion Robust to domain shift Figure 1: Thematic comparison of current practices and proposed design principles for tokenizer development. 1.2 Alternative Perspectives on Tokenization Despite growing skepticism toward the one-size-fits-all approach, several views continue to shape current tokenization practices (Schmidt et al., 2024; Reddy et al., 2025; Wegmann et al., 2025). We summarize and respond to five common perspectives, outlined in Table 1. Table 1: Five alternative perspectives on tokenizer design and our responses to each. 2 Tokenization and Alternatives 2.1 Subword Tokenization Methods Subword tokenization addresses the drawbacks of word- and character-level models. Word-level approaches suffer from large vocabularies and out-of-vocabulary issues, while character-level models create long sequences that raise computational costs and obscure linguistic structure (Beinborn and Pinter, 2023). Subword methods (e.g., BPE, WordPiece, Unigram) balance these extremes by segmenting text into frequent, data-driven units, enabling efficient handling of rare words and manageable sequence lengths (Wolleb et al., 2023). These methods differ in segmentation strategy. BPE and WordPiece are merge-based: BPE joins frequent adjacent pairs by frequency, while WordPiece optimizes a likelihood-based objective tied to language modeling (Schmidt et al., 2024). Both yield deterministic outputs. In contrast, Unigram takes a probabilistic approach, pruning subwords with low data likelihood and allowing multiple valid segmentations. This often makes Unigram more flexible and linguistically aligned, especially for multilingual or morphologically rich languages (Kudo, 2018; Bostrom and Durrett, 2020). BPE offers simplicity and speed, WordPiece emphasizes likelihood-based merge decisions informed by local token context, and Unigram provides cross-lingual adaptability at higher computational cost. These methods underpin models like GPT (Brown et al., 2020), BERT (Devlin et al., 2019), and LLaMA (Touvron et al., 2023a, b). Yet their reliance on frequency statistics exposes key limits (Meyer and Buys, 2022; Liu et al., 2024a; Chai et al., 2024) frequent forms dominate, while rare words, low-resource languages, and domain-specific terms are poorly captured. Token boundaries often mirror statistical artifacts rather than linguistic structure, impairing generalization in typologically complex languages (see Appendix A). Despite scalability, these methods often underperform in compression efficiency and cross-lingual representational alignment (Goldman et al., 2024). 2.2 Emerging Alternatives for the De Facto Tokenization Paradigm Tokenization based on statistical frequency over large corpora has long dominated NLP workflows, but it struggles with rare languages, domain-specific terms, and structural linguistic nuances (Meyer and Buys, 2022; Liu et al., 2024a; Chai et al., 2024). In response, the research community has proposed several alternatives that challenge the subword-centric status quo. Without endorsing a particular solution, we review these approaches to underscore a broader shift toward more adaptive, linguistically informed strategies and to highlight open questions that remain. Learning Compositional Structure Explicitly: Traditional tokenization ignores linguistic composition, treating subwords as independent units (Bostrom and Durrett, 2020). Recent methods attempt to model internal structure more directly through sparse embeddings (Deiseroth et al., 2024) or morpheme-aware GRU frameworks (Singh et al., 2023), but these approaches add training complexity and remain underexplored in large-scale LLMs. From Subwords to Raw Bytes: Subword tokenizers struggle with unseen characters and typographic variation (Xue et al., 2022). Byte-level methods address this by operating directly on raw input, avoiding fixed vocabularies. The Byte-Level Transformer (BLT) (Pagnoni et al., 2024) employs entropy-based patches to reduce inference cost and remove vocabulary constraints. While promising, these methods can increase sequence length and computational overhead. Modeling Language at Semantic Level: Recent work explores higher-level abstractions by representing multi-word expressions or concepts as single units (Liu et al., 2025; Schmidt et al., 2025). Liu et al. (2024b) propose a dynamic phrase encoder with variable-length phrases to enhance generalization, while LCM team et al. (2024) introduce sentence-level embeddings that bypass token-level processing and view language as a hierarchy of meaning (Shao et al., 2025), benefiting multilingual tasks but facing data sparsity challenges. 3 Rethinking Tokenizer Design: From Pitfalls to Practice Building on prior findings that highlight tokenization’s influence on model behavior (Bostrom and Durrett, 2020; Ali et al., 2024), we identify four persistent pitfalls in current practice. Design Integration: Tokenizer design should be integrated into model training, not inherited as a fixed preprocessing artifact. Reliance on legacy vocabularies, often optimized for different data or languages, limits control and reproducibility. Effective design begins with a core choice: train a tokenizer from scratch, or reuse an existing one for efficiency and interoperability (§ 3.2.1, § 3.2.2). Adaptation and Generalization: Tokenizers are frequently reused across domains without adaptation, resulting in representational gaps and inflated sequence lengths. As discussed in § 3.2.2, domain- and language-aware tokenization is essential for fair and robust generalization across linguistic and structural variation. Evaluation Practices, Reliability, and Bias: Tokenizer evaluation often relies on surface metrics such as fertility, which correlate weakly with performance (Zouhar et al., 2023a; Ali et al., 2024). As detailed in § 3.2.4, systematic assessment across linguistic fit, task alignment, and cross-lingual robustness is needed. Current workflows overlook token fragmentation, undertrained embeddings, and biases against morphologically rich or low-resource languages. Responsible tokenization requires auditable workflows, including anomaly detection and vocabulary stability checks (§ 3.2.3). 3.1 A Structured Iterative Process An effective tokenizer should meet empirically grounded desiderata, ensuring that vocabulary units occur frequently enough during training to support robust learning and practical utility: Tokenizer Design Desiderata • Linguistically Faithful: mirrors structure, morphology, syntax, and code-switching patterns. • Usage-Aligned: matched to intended language coverage, domains, and downstream usage profiles. • Deployment-Ready: efficient under latency and memory limits. • Architecturally Compatible: fits model architecture and embeddings. To move beyond ad hoc design, we propose a structured, iterative co-design process in which tokenization is periodically re-evaluated and refined alongside model architecture and training objectives. Tokenization is informed indirectly by model behavior, using representational analyses and downstream performance as feedback. The workflow integrates linguistic analysis and empirical evaluation, with each stage refining earlier choices to maintain coherence across linguistic, operational, and deployment constraints. Table 2 summarizes the core stages and outputs. While this process offers a practical roadmap, success hinges on how language is segmented and represented. The goal is not to over-specialize tokenization to a particular benchmark, but to avoid blindly inheriting a tokenizer misaligned with the target language or domain (Nayeem et al., 2025). | Initial Assessment | (i) Define the intended LLM purpose (e.g., general or domain-specific). (ii) Identify target languages, dialects, and cultural contexts. (iii) Specify required capabilities (e.g., reasoning). (iv) Establish model constraints (architecture, memory, latency). | A clear set of design objectives and operational constraints to guide tokenizer and vocabulary development. | Designing a biomedical-domain LLM focusing on English and Spanish medical notes, with strict latency constraints for on-device clinical use. | | --- | --- | --- | --- | | Reuse Decisions | (i) Identify existing tokenizers with appropriate licenses. (ii) Evaluate against linguistic fidelity, task alignment, and compatibility. (iii) Run diagnostics (e.g., vocabulary overlap, token length distribution). (iv) Decide to reuse without changes, adapt, or create a new tokenizer. | A diagnostic report and an evidence-based decision to reuse, adapt, or develop a new tokenizer. | Comparing GPT-2 and LLaMA tokenizers for biomedical abstracts, identifying the vocabulary overlap, and deciding to train a new tokenizer for domain-specific terminology. | | Corpus Curation | (i) Build corpora aligned with the intended use case. (ii) Establish diagnostic criteria (coverage, diversity, task relevance). | A representative, task-aligned corpus. | Curating a multilingual biomedical corpus of clinical trial reports, medical guidelines, and radiology notes. | | Pretokenization & Processing | (i) Apply pretokenization rules and preprocessing steps. (ii) Normalize encoding, clean text, set boundary rules. (iii) Handle code-switching, numbers, and mixed inputs. (iv) Align with application requirements. | A documented, reproducible set of pretokenization rules applied consistently. | Normalizing biomedical text by handling hyphenated terms (e.g., “HER-2/neu”), gene names, and units of measurement. | | Tokenizer Training | (i) Select a segmentation strategy. (ii) Train tokenizer on the curated corpus to produce vocabulary. | A representative, task-aligned corpus and a trained tokenizer with an initial vocabulary. | Training a 60K-token BPE tokenizer on 25 GB of biomedical literature to capture domain-specific compounds (e.g., “immunohistochemistry”). | | Intrinsic & Extrinsic Evaluation | (i) Apply tokenizer to varied datasets. (ii) Compute metrics (e.g., token length, fragmentation). (iii) Audit vocabulary and visualize anomalies. (iv) Evaluate edge cases and small model performance. | A comprehensive evaluation report covering tokenizer-level diagnostics and model-based validation. | Measuring token length across English and Spanish biomedical texts; identifying over-segmentation in long clinical terms. | | Iteration & Model-Guided Refinement | (i) Refine tokenizer using diagnostics. (ii) Treat tokenizer as co-evolving with the model. (iii) Re-run evaluations until stable performance. | A refined, model-aligned tokenizer with improved efficiency and quality. | After observing poor tokenization of biomedical abbreviations, re-merging tokens to preserve critical acronyms like “MRI” and “ICU.” | | Final Integration & Documentation | (i) Confirm stability and finalize configuration. (ii) Document known limitations and design rationale. (iii) Archive and publish tokenizer for reproducibility. | A production-ready tokenizer with comprehensive documentation. | Publishing biomedical tokenizer specifications with open access, including examples for drug names and clinical abbreviations. | Table 2: Core Stages in Tokenizer Development Process, extended with illustrative examples. 3.2 Critical Decisions within the Workflow Each of the pitfalls identified earlier maps to design decisions that shape tokenizer quality. We unpack these decisions, showing how principled choices throughout tokenizer development can systematically address the deficiencies outlined above. Each subsection concludes with guiding questions, consolidated in Table 7 in the Appendix. 3.2.1 Training Tokenizers from Scratch Designing a tokenizer from scratch enables deliberate control over segmentation, corpus, vocabulary size, and preprocessing, aligning it with linguistic, functional, and deployment needs (Dagan et al., 2024). In contrast, reusing a pretrained model’s tokenizer (e.g., LLaMA) constrains these choices, often limiting performance and generalization. This section outlines key design factors and open questions for principled tokenizer development. Choosing the Right Tokenizer Algorithm. Selecting a tokenization algorithm is a crucial early step but often defaults to convenience over task or language needs (Ali et al., 2024). Unigram better captures morpheme-level structure in agglutinative languages (Bostrom and Durrett, 2020), while frequency-based BPE remains dominant. These choices influence representation quality, sequence length, and generalization (Ahia et al., 2023). Recent work shows that tokenizers tailored to linguistic and operational contexts outperform generic ones, especially in domain-specific and multilingual settings (Bommarito et al., 2025). Evidence also suggests that multilingual tokenizers preserve morphology better in complex languages than in simpler ones (Arnett and Bergen, 2025), though the benefits of morphologically aligned tokenization remain debated (Arnett et al., 2025). Training Data Matters for Tokenizer Effectiveness. Training corpus composition critically shapes tokenizer performance (Reddy et al., 2025). Reliance on large-scale web crawls introduces representational biases, particularly for low-resource languages and specialized domains, where vocabularies built from unrelated or noisy data degrade performance (Rust et al., 2021; Goldman et al., 2024). Yet metrics for corpus quality remain limited. Empirical studies show that compression quality, driven by corpus composition and segmentation strategy, correlates strongly with model utility, especially for rare or domain-specific terms (Goldman et al., 2024). Evidence also suggests tokenizer quality plateaus beyond roughly 150–180GB of training data, indicating diminishing returns from further scaling (Reddy et al., 2025). Pre-tokenization and Normalization Are Foundational. Pre-tokenization and normalization encode linguistic assumptions that shape token consistency, sequence length, and vocabulary fragmentation (Dagan et al., 2024). Even minor choices (e.g., Unicode normalization or diacritic removal) can markedly alter segmentation. E.g., normalizing “coöperate” to “cooperate” avoids inconsistent splits in OCR or historical texts. They also guide subword merges via boundary cues: incorporating multiword expressions or local context yields more coherent merges, especially in agglutinative languages, often requiring BPE adjustments (Schmidt et al., 2024). Removing pre-tokenization consistently degrades performance (Dagan et al., 2024; Velayuthan and Sarveswaran, 2025), sometimes more than changing vocabulary size or corpus composition (Wegmann et al., 2025), and can create indivisible units that limit gains from larger corpora (Reddy et al., 2025). Vocabulary Size Is a Strategic Design Choice. Vocabulary size should be co-optimized with model architecture and training goals. Larger vocabularies reduce fragmentation and sequence length but raise memory and embedding costs (Tao et al., 2024). These trade-offs intensify in multilingual or morphologically rich settings (Ali et al., 2024). Studies show that scaling vocabulary can improve compression and efficiency, though gains depend on corpus, task, and model capacity (Tao et al., 2024; Huang et al., 2025). Yet, Schmidt et al. (2025) warns that adding low-frequency tokens may increase redundancy with minimal benefit. Despite this, vocabulary size is often set heuristically (Salesky et al., 2020; Gowda and May, 2020). A more principled approach would treat it as a tunable parameter informed by token frequencies, learning dynamics, and compute limits, though key questions remain. Open Trade-offs. Table 3 summarizes approaches that learn tokenizers directly from raw text instead of reusing legacy vocabularies (Sennrich et al., 2016; Kudo, 2018; Kudo and Richardson, 2018; Tay et al., 2022). When data and budget are controllable, first-principles design is often preferable (Gowda and May, 2020; Zouhar et al., 2023b). Vocabulary size should be treated as a computational lever, not a sign of model “purity”: smaller vocabularies inflate sequence length and attention cost, while larger ones increase parameters and memory with diminishing returns (Gowda and May, 2020; Zouhar et al., 2023b). Classic subword methods (BPE, WordPiece, Unigram) remain strong baselines for minimizing tokens per sample (Sennrich et al., 2016; Kudo, 2018; Kudo and Richardson, 2018). Morphology-aware or lexically grounded variants are worthwhile for agglutinative or templatic languages (Libovický and Helcl, 2024; Chizhov et al., 2024; Asgari et al., 2025), while block-level schemes such as Charformer downsampling retain character-level robustness without excessive sequence inflation (Tay et al., 2022). 3.2.2 Adapting Tokenizers: Reuse Requires Verification and Documentation Reuse of mature, pretrained tokenizers is often a reasonable and practical default, particularly when compatibility, distillation from existing models, and engineering efficiency are primary concerns. Whether a pretrained tokenizer can serve as-is or requires adaptation depends on its alignment with the target language or domain. When LLMs are extended to new settings, work typically centers on weight fine-tuning, yet performance often degrades because English-optimized tokenizers fail to capture domain terms and morphology in low-resource contexts (Sha et al., 2025). Vocabulary expansion is a common but ad hoc remedy, with limited understanding of when it helps (Wang et al., 2019; Kiulian et al., 2024), often yielding short-term gains that fail to generalize. Robust adaptation instead requires treating tokenization as a core design task, guided by systematic protocols for token addition, embedding initialization, and adaptation staging. While reuse improves compatibility and cost, unexamined reuse can introduce long-term drawbacks, including loss of linguistic fidelity, coverage gaps, and fairness issues, particularly when the tokenizer’s original training data or design assumptions diverge from the new setting. Crucially, although training or adapting a tokenizer incurs a one-time engineering cost, inheriting a suboptimal tokenizer imposes a persistent inference-time penalty: inflated sequence lengths increase latency and compute per query over the lifetime of a deployed model, especially in non-English and domain-specific scenarios (see Appendix A). Base Model Choice: Tokenizer Compatibility Is Not Guaranteed. Adaptation often starts with choosing a base LLM and reusing its tokenizer by default, yet the suitability of this choice for a new domain or language is rarely explicitly tested or documented. Even when vocabularies are expanded, such as extending the LLaMA 2 tokenizer from 32K to 61K tokens for Arabic (Bari et al., 2025), the linguistic rationale behind token boundaries remains unclear. Without evaluating what the original tokenizer captures or omits, expansion risks becoming a workaround. Vocabulary Size: Incremental Growth Without Clear Guidance. In adaptation, expanding vocabulary means adding new tokens to an existing tokenizer. This can shorten sequences and improve representation for new domains or languages, but also raises issues of compatibility, memory, and convergence (Tejaswi et al., 2024). Unlike tokenizers built from scratch, expansion operates within the constraints of an inherited scheme. Evidence suggests that targeted additions (e.g., 5–10K tokens) yield better returns than aggressive scaling (Liu et al., 2024a), yet consistent methods for selecting, validating, and integrating tokens remain lacking. Without a grounded approach, expansions risk inefficiency and misalignment with the base model’s learned representations. Because expansion preserves the original token inventory, it is often preferred in settings where distillation from an existing model, checkpoint reuse, or interoperability with deployed systems is required. Initialization: A Crucial but Understudied Factor. How new token embeddings are initialized strongly affects downstream performance. Approaches include random initialization, subtoken averaging, and using external word vectors (Yao et al., 2021; Dobler and de Melo, 2023), but comparative evidence remains limited and context-dependent. Stopping criteria such as fertility scores are often borrowed from unrelated tasks and rarely validated across languages. The field needs benchmarks and reporting standards to identify which strategies generalize and under what conditions. Embedding Warm-Up: A Temporary Fix or Best Practice? Embedding warm-up (training new token embeddings in isolation before unfreezing the full model) has emerged as a practical compromise (Zhao et al., 2024). Some workflows add phased unfreezing of input or output layers (Kim et al., 2024), but these remain empirical heuristics rather than principled frameworks. Without systematic evaluation, it remains unclear whether such strategies are robust solutions or fragile stopgaps that conceal deeper misalignments. Open Trade-offs. Table 4 summarizes tokenizer adaptations across languages and domains. Blind reuse of legacy tokenizers often inflates sequence length through over-fragmentation, wasting context and compute. In contrast, retokenization, vocabulary expansion, and merge refinement are offline interventions that typically repay their cost with shorter sequences and better context use (Rust et al., 2021; Yehezkel and Pinter, 2023; Chizhov et al., 2024). Large-scale studies show that tokenizer choice materially affects both quality and training cost, and that naïve reuse can increase training steps across languages (Ali et al., 2024; Schmidt et al., 2024). Adaptations in Arabic and Southeast Asian settings demonstrate that targeted expansions or dedicated tokenizers close token-length gaps while preserving English performance (Nguyen et al., 2023; Bari et al., 2025; Fanar et al., 2025). For unseen scripts, compact in-language tokenizers with aligned embeddings reduce UNKs and avoid costly character fallback (Pfeiffer et al., 2021). Taken together, these findings suggest that reuse should be treated as a default hypothesis to be tested, rather than an unquestioned assumption. 3.2.3 Responsible Tokenization: Confronting Risks to Fairness, Stability, and Security Most tokenizer discussions emphasize performance, but robust and responsible design is equally critical. Tokenization influences fairness, stability, and security, shaping language coverage, equity, and susceptibility to attacks. Fairness refers to avoiding unequal segmentation quality, representational accuracy, or computational burden across languages, scripts, or demographic groups, ensuring parity in token lengths and downstream performance. Stability refers to how consistently and reliably a tokenizer segments text across noise, scripts, and domains, while avoiding low-quality or undertrained units. As LLMs enter real-world deployment (Laskar et al., 2023b; Fu et al., 2024; Nayeem and Rafiei, 2025), tokenization must be treated as a potential source of bias for transparent design. Undertrained Tokens: Silent Failures in the Vocabulary. Undertrained tokens, units that occur infrequently in the pretraining corpus, receive too few updates during training. They often stem from rare words, domain-specific terms, or overly fragmented vocabularies. Their effects are subtle but harmful, causing instability and poor generalization (Yang et al., 2024). The notorious SolidGoldMagikarp token, which emerged in GPT models due to skewed data distributions, illustrates how poorly trained tokens can trigger erratic behavior (Rumbelow and Watkins, 2024). Yet such tokens are seldom identified. Embedding audits, token pruning, vocabulary refinement, and frequency analysis should be standard in tokenizer evaluation (Chizhov et al., 2024; Cognetta et al., 2024a; Bauwens and Delobelle, 2024; Lian et al., 2025). Anomalous Tokens: Artifacts That Disrupt Learning. Tokenizer vocabularies often contain anomalous tokens, units introduced by data noise, encoding errors, or preprocessing inconsistencies. These may capture HTML fragments, URL encodings, or control characters that add no linguistic value and instead degrade embedding quality (Jiang et al., 2024; Chai et al., 2024). Their presence injects noise into training, causing unpredictable behavior in generation and classification. More broadly, such artifacts pose challenges for reproducibility, auditing, and verification across machine learning pipelines, where undocumented preprocessing decisions can silently affect downstream behavior. While this lack of validation is a broader issue in model reproduction and auditing, tokenizers are especially affected because vocabulary artifacts become fixed and propagate throughout training and deployment. Clearer standards for corpus preparation, token quality control, and automated anomaly detection are therefore essential not only for tokenizer quality, but for reproducible, auditable, and trustworthy model development. Tokenization Bias: Unequal Representation by Design. Tokenization may not treat all languages or identity groups equally. Morphologically rich languages often suffer excessive fragmentation, lengthening sequences and degrading performance (Toraman et al., 2023) (see Appendix A). Similarly, identity-related terms from underrepresented regions are frequently over-segmented, impairing tasks like named entity recognition (Ahia et al., 2024). These are not random artifacts but outcomes of design choices that allocate vocabulary space unequally across languages and demographics. Responsible tokenizer development must therefore ensure balanced language representation, introduce fairness constraints, and systematically analyze segmentation across diverse groups. Without such efforts, tokenization remains a structural source of bias in downstream applications. Security and Adversarial Risks: A Hidden Attack Surface. Tokenizer design can introduce overlooked security vulnerabilities. Adversarial token sequences, prompt injection, and homograph attacks exploit weaknesses in tokenization rules and preprocessing (Wang et al., 2024; Jang et al., 2025). Poor handling of visually similar characters across scripts further enables such exploits. Security should not be an afterthought: tokenization must be included in adversarial testing, with controlled preprocessing, careful treatment of special tokens, and rigorous auditing and versioning. As LLMs enter sensitive domains, securing tokenization is essential for both performance and trust. Open Trade-offs. As shown in Table 5, tokenizer design is a governance decision, not incidental plumbing. It determines sequence length, training cost, and robustness to noise and cross-language parity (Rust et al., 2021; Petrov et al., 2023; Ahia et al., 2023). Minor orthographic noise or typos can destabilize segmentation (Chai et al., 2024), while morphology-aware constraints and merge refinement improve stability and parity with minimal runtime overhead (Chizhov et al., 2024; Asgari et al., 2025). Recent studies show that targeted adaptations reduce token-length gaps in Southeast Asian and Arabic settings without harming English performance (Nguyen et al., 2023; Bari et al., 2025). 3.2.4 Evaluation Framework and Gaps Despite tokenization’s central role in model performance, evaluation remains limited and fragmented. Most work relies on surface metrics (compression rate, vocabulary size, fertility) that are easy to compute but reveal little about linguistic alignment or generalizable learning (Ali et al., 2024; Laskar et al., 2024; Uzan et al., 2024; Nayeem et al., 2025). This gap is most pronounced in multilingual and domain-specific settings, where similar fertility scores can hide large differences in morphological preservation or rare-construction handling (Bostrom and Durrett, 2020). These metrics are also sensitive to corpus composition, language typology, and preprocessing, limiting cross-domain validity (Rust et al., 2021). Without a principled evaluation framework, trade-offs among linguistic fidelity, efficiency, and robustness remain poorly understood. Tokenization choices are seldom tied to downstream performance (Dagan et al., 2024). Notably, many of these diagnostics (such as token-per-character statistics by language, fragmentation analysis, anomaly detection, and undertrained-token audits) can be applied directly to existing pretrained models without retraining, making them practical even under tight compute or data constraints. We do not argue that tokenizer evaluation requires training large models to convergence, but instead advocate low-cost intrinsic and model-based probes (e.g., token-per-character parity and information-theoretic efficiency Tsvetkov and Kipnis (2024)) that correlate with downstream multilingual performance. Toward Multi-Dimensional Evaluation. Recent work has questioned the validity of common intrinsic metrics. Early studies hypothesized that reducing corpus token counts (CTC) would improve downstream performance (Gallé, 2019; Goldman et al., 2024), but later findings show no consistent correlation between CTC, vocabulary size, and task accuracy (Schmidt et al., 2024; Ali et al., 2024). In response, researchers have proposed richer measures of token quality. Zouhar et al. (2023a) introduce Rényi efficiency, which penalizes distributions skewed toward frequent, low-content tokens and correlates more strongly with downstream results, though its generality remains debated (Cognetta et al., 2024b). Other studies note that standard pre-tokenization pipelines often over-map pretokens to high-frequency words, reducing representational depth (Reddy et al., 2025). Building on these efforts, Bommarito et al. (2025) propose a framework encompassing token-per-character efficiency, domain coverage, and token size distribution. Collectively, these approaches point to a multidimensional view of tokenizer quality that integrates linguistic structure, statistical balance, and utility. We distill this view into five core dimensions (see Appendix C for examples). Each dimension emphasizes different evaluation levels: corpus-level diagnostics for fragmentation and alignment, and model-based probes for generalization and embedding use. Efficiency often trades off with alignment or interpretability, and priorities vary across applications. Developing benchmarks to quantify and balance these trade-offs remains an essential direction for future work. Toward Practical Evaluation Methodologies. Translating these dimensions into practice requires consistent methodologies. Promising directions include unified corpus diagnostics, model-level robustness tests under distribution shift, and cross-lingual benchmarks linking tokenization quality to downstream performance. Standardized protocols would enable reproducibility and move the field beyond ad hoc, dataset-specific evaluation. Key Evaluation Dimensions for Tokenizer Quality • Coverage: Does the tokenizer represent the linguistic units relevant to the target domain or language? • Generalizability: How well does it handle unseen forms, rare constructions, and morphologically complex variants? • Linguistic Alignment: Do token boundaries align with morphemes, affixes, or syntax? • Robustness: Is it resilient to variation, noise, and distributional shift (e.g., dialects, OCR errors)? • Representation Utilization: Are embeddings meaningfully activated during inference, or is much of the vocabulary underused? Open Trade-offs. As summarized in Table 6, evaluating tokenizers by only dev-set perplexity or vocabulary size obscures the real drivers of cost and quality. Report metrics that matter like tokens per 1k characters by language, parity ratios, compression–structure balance, and estimated pretrain FLOPs at fixed data budgets (Goldman et al., 2024; Ali et al., 2024; Schmidt et al., 2024). Performance should be summarized across all data categories and shards used for training. Intrinsic metrics can triage candidates but need quick downstream probes to prevent gaming and preserve structural integrity (Zouhar et al., 2023a; Cognetta et al., 2024b; Uzan et al., 2024). Formal analyses and sizing analyses (explicit evaluations of how vocabulary size, sequence length, and token granularity affect memory use, throughput, and total training FLOPs) offer practical standards for selecting tokenizers that minimize total cost while maintaining target quality across languages (Gowda and May, 2020; Zouhar et al., 2023b; Nayeem et al., 2025). 4 Conclusion The dominance of subword tokenization has fostered a one-size-fits-all mindset. Yet evidence from multilingual, low-resource, and domain-specific settings exposes its limits. Tokenization should be reimagined as a context-aware design choice, shaped by structure, data, task, and deployment constraints. This shift is vital for models that are efficient, scalable, fair, and linguistically grounded. Tokenization is not a preprocessing afterthought but a core component of model development, designed, evaluated, and refined alongside model behavior. Limitations Our argument centers on reframing tokenization as a modeling decision, but it remains primarily conceptual. We do not provide new empirical results or large-scale ablations, and thus our claims should be viewed as hypotheses for future validation and research. Implementing our proposed co-design and evaluation framework also assumes access to multilingual data, compute, and tooling that may not be universally available. Moreover, identifying standardized proxy probes that robustly predict large-scale downstream performance remains an open empirical problem, and we view this as a key direction for the future works. Finally, while we gesture toward token-free and byte-level approaches, these alternatives evolve too quickly for comprehensive inclusion here; our discussion focuses on guiding principles rather than definitive solutions. Acknowledgements We thank all the anonymous reviewers and the meta-reviewer for their valuable feedback and constructive suggestions for improving this work. Additionally, Mir Tafseer Nayeem acknowledges support from the Huawei PhD Fellowship, and Md Tahmid Rahman Laskar acknowledges the support from the CUPE 3903 Research Grant Fund, York University. References - O. Ahia, S. Kumar, H. Gonen, V. Hofmann, T. Limisiewicz, Y. Tsvetkov, and N. A. Smith (2024) MAGNET: improving the multilingual fairness of language models with adaptive gradient-based tokenization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §3.2.3. - O. Ahia, S. Kumar, H. Gonen, J. Kasai, D. Mortensen, N. Smith, and Y. Tsvetkov (2023) Do all languages cost the same? tokenization in the era of commercial language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, pp. 9904–9923. External Links: Link, Document Cited by: Table 5, §1, §3.2.1, §3.2.3. - M. Ali, M. Fromm, K. Thellmann, R. Rutmann, M. Lübbering, J. Leveling, K. Klug, J. Ebert, N. Doll, J. Buschhoff, C. Jain, A. Weber, L. Jurkschat, H. Abdelwahab, C. John, P. Ortiz Suarez, M. Ostendorff, S. Weinbach, R. Sifa, S. Kesselheim, and N. Flores-Herr (2024) Tokenizer choice for LLM training: negligible or crucial?. In Findings of the Association for Computational Linguistics: NAACL 2024, K. Duh, H. Gomez, and S. Bethard (Eds.), Mexico City, Mexico, pp. 3907–3924. External Links: Link, Document Cited by: Appendix A, Table 4, Table 6, §1, §1, §3, §3.2.1, §3.2.1, §3.2.2, §3.2.4, §3.2.4, §3.2.4, §3. - G. S. Altıntaş, M. Ehghaghi, B. Lester, F. Liu, W. Zhao, M. Ciccone, and C. Raffel (2025) TokSuite: measuring the impact of tokenizer choice on language model behavior. arXiv preprint arXiv:2512.20757. Cited by: §1. - R. Anil, S. Borgeaud, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, D. Silver, M. Johnson, I. Antonoglou, J. Schrittwieser, A. Glaese, J. Chen, E. Pitler, T. Lillicrap, A. Lazaridou, and 1. others (2025) Gemini: a family of highly capable multimodal models. External Links: 2312.11805, Link Cited by: §B.2, §1. - Anthropic (2025) Claude 3.7 sonnet system card. Note: https://assets.anthropic.com/m/785e231869ea8b3b/original/claude-3-7-sonnet-system-card.pdf Accessed: 2025-05-22 Cited by: §1. - C. Arnett and B. Bergen (2025) Why do language models perform worse for morphologically complex languages?. In Proceedings of the 31st International Conference on Computational Linguistics, O. Rambow, L. Wanner, M. Apidianaki, H. Al-Khalifa, B. D. Eugenio, and S. Schockaert (Eds.), Abu Dhabi, UAE, pp. 6607–6623. External Links: Link Cited by: §3.2.1. - C. Arnett, M. Hudspeth, and B. O’Connor (2025) Evaluating morphological alignment of tokenizers in 70 languages. arXiv preprint arXiv:2507.06378. Cited by: §3.2.1. - E. Asgari, Y. E. Kheir, and M. A. S. Javaheri (2025) MorphBPE: a morpho-aware tokenizer bridging linguistic complexity for efficient llm training across morphologies. arXiv preprint arXiv:2502.00894. Cited by: Table 3, Table 5, §3.2.1, §3.2.3. - M. S. Bari, Y. Alnumay, N. A. Alzahrani, N. M. Alotaibi, H. A. Alyahya, S. AlRashed, F. A. Mirza, S. Z. Alsubaie, H. A. Alahmed, G. Alabduljabbar, R. Alkhathran, Y. Almushayqih, R. Alnajim, S. Alsubaihi, M. A. Mansour, S. A. Hassan, Dr. M. Alrubaian, A. Alammari, Z. Alawami, A. Al-Thubaity, A. Abdelali, J. Kuriakose, A. Abujabal, N. Al-Twairesh, A. Alowisheq, and H. Khan (2025) ALLam: large language models for arabic and english. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: Table 4, Table 5, §3.2.2, §3.2.2, §3.2.3. - T. Bauwens and P. Delobelle (2024) BPE-knockout: pruning pre-existing BPE tokenisers with backwards-compatible morphological semi-supervision. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), K. Duh, H. Gomez, and S. Bethard (Eds.), Mexico City, Mexico, pp. 5810–5832. External Links: Link, Document Cited by: §3.2.3. - L. Beinborn and Y. Pinter (2023) Analyzing cognitive plausibility of subword tokenization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, pp. 4478–4486. External Links: Link, Document Cited by: §1, §2.1. - M. J. Bommarito, D. M. Katz, and J. Bommarito (2025) KL3M tokenizers: a family of domain-specific and character-level tokenizers for legal, financial, and preprocessing applications. External Links: 2503.17247, Link Cited by: §1, §1, §3.2.1, §3.2.4. - K. Bostrom and G. Durrett (2020) Byte pair encoding is suboptimal for language model pretraining. In Findings of the Association for Computational Linguistics: EMNLP 2020, T. Cohn, Y. He, and Y. Liu (Eds.), Online, pp. 4617–4624. External Links: Link, Document Cited by: §1, §1, §2.1, §2.2, §3.2.1, §3.2.4, §3. - T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020) Language models are few-shot learners. Advances in neural information processing systems 33, pp. 1877–1901. Cited by: §2.1. - Y. Chai, Y. Fang, Q. Peng, and X. Li (2024) Tokenization falling short: on subword robustness in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, pp. 1582–1599. External Links: Link, Document Cited by: Table 5, §2.1, §2.2, §3.2.3, §3.2.3. - Z. Chen, A. Hernández-Cano, A. Romanou, A. Bonnet, K. Matoba, F. Salvi, M. Pagliardini, S. Fan, A. Köpf, A. Mohtashami, A. Sallinen, A. Sakhaeirad, V. Swamy, I. Krawczuk, D. Bayazit, A. Marmet, S. Montariol, M. Hartley, M. Jaggi, and A. Bosselut (2023) MEDITRON-70b: scaling medical pretraining for large language models. External Links: Link, 2311.16079 Cited by: §1. - P. Chizhov, C. Arnett, E. Korotkova, and I. P. Yamshchikov (2024) BPE gets picky: efficient vocabulary refinement during tokenizer training. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, pp. 16587–16604. External Links: Link, Document Cited by: Table 3, Table 4, Table 5, §3.2.1, §3.2.2, §3.2.3, §3.2.3. - A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. (2023) Palm: scaling language modeling with pathways. Journal of Machine Learning Research 24 (240), pp. 1–113. Cited by: §B.2. - M. Cognetta, T. Hiraoka, R. Sennrich, Y. Pinter, and N. Okazaki (2024a) An analysis of bpe vocabulary trimming in neural machine translation. In Proceedings of the Fifth Workshop on Insights from Negative Results in NLP, pp. 48–50. Cited by: §3.2.3. - M. Cognetta, V. Zouhar, S. Moon, and N. Okazaki (2024b) Two counterexamples to tokenization and the noiseless channel. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), N. Calzolari, M. Kan, V. Hoste, A. Lenci, S. Sakti, and N. Xue (Eds.), Torino, Italia, pp. 16897–16906. External Links: Link Cited by: Table 6, §3.2.4, §3.2.4. - G. Dagan, G. Synnaeve, and B. Rozière (2024) Getting the most out of your tokenizer for pre-training and domain adaptation. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. External Links: Link Cited by: §1, §1, §3.2.1, §3.2.1, §3.2.4. - DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, and 1. others (2025) DeepSeek-r1: incentivizing reasoning capability in llms via reinforcement learning. External Links: 2501.12948, Link Cited by: §1. - B. Deiseroth, M. Brack, P. Schramowski, K. Kersting, and S. Weinbach (2024) T-FREE: subword tokenizer-free generative LLMs via sparse representations for memory-efficient embeddings. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, pp. 21829–21851. External Links: Link, Document Cited by: §2.2. - J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio (Eds.), Minneapolis, Minnesota, pp. 4171–4186. External Links: Link, Document Cited by: §2.1. - V. Dewangan, B. R. S, G. Suri, and R. Sonavane (2025) When every token counts: optimal segmentation for low-resource language models. In Proceedings of the First Workshop on Language Models for Low-Resource Languages, H. Hettiarachchi, T. Ranasinghe, P. Rayson, R. Mitkov, M. Gaber, D. Premasiri, F. A. Tan, and L. Uyangodage (Eds.), Abu Dhabi, United Arab Emirates, pp. 294–308. External Links: Link Cited by: §1.1, §1. - K. Dobler and G. de Melo (2023) FOCUS: effective embedding initialization for monolingual specialization of multilingual models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, pp. 13440–13454. External Links: Link, Document Cited by: §3.2.2. - F. Fanar, U. Abbas, M. S. Ahmad, F. Alam, E. Altinisik, E. Asgari, Y. Boshmaf, S. Boughorbel, S. Chawla, S. Chowdhury, et al. (2025) Fanar: an arabic-centric multimodal generative ai platform. arXiv preprint arXiv:2501.13944. Cited by: Table 4, §3.2.2. - D. Feher, I. Vulić, and B. Minixhofer (2025) Retrofitting large language models with dynamic tokenization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 29866–29883. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: Table 1. - X. Fu, M. T. R. Laskar, E. Khasanova, C. Chen, and S. Tn (2024) Tiny titans: can smaller large language models punch above their weight in the real world for meeting summarization?. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track), Y. Yang, A. Davani, A. Sil, and A. Kumar (Eds.), Mexico City, Mexico, pp. 387–394. External Links: Link, Document Cited by: §3.2.3. - P. Gage (1994) A new algorithm for data compression. C Users J. 12 (2), pp. 23–38. External Links: ISSN 0898-9788, Link Cited by: §1. - M. Gallé (2019) Investigating the effectiveness of BPE: the power of shorter sequences. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), K. Inui, J. Jiang, V. Ng, and X. Wan (Eds.), Hong Kong, China, pp. 1375–1381. External Links: Link, Document Cited by: §3.2.4. - O. Goldman, A. Caciularu, M. Eyal, K. Cao, I. Szpektor, and R. Tsarfaty (2024) Unpacking tokenization: evaluating text compression and its correlation with model performance. In Findings of the Association for Computational Linguistics: ACL 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 2274–2286. External Links: Link, Document Cited by: Table 6, §1, §1, §2.1, §3.2.1, §3.2.4, §3.2.4. - T. Gowda and J. May (2020) Finding the optimal vocabulary size for neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, T. Cohn, Y. He, and Y. Liu (Eds.), Online, pp. 3955–3964. External Links: Link, Document Cited by: Table 3, Table 6, §3.2.1, §3.2.1, §3.2.4. - A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, and 5. others (2024) The llama 3 herd of models. External Links: 2407.21783, Link Cited by: §B.3, §1. - T. Hiraoka, S. Takase, K. Uchiumi, A. Keyaki, and N. Okazaki (2021) Joint optimization of tokenization and downstream model. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 244–255. Cited by: Table 1. - H. Huang, D. Zhu, B. Wu, Y. Zeng, Y. Wang, Q. Min, and X. Zhou (2025) Over-tokenized transformer: vocabulary is generally worth scaling. External Links: 2501.16975, Link Cited by: §3.2.1. - S. Hwang, B. Wang, and A. Gu (2025) Dynamic chunking for end-to-end hierarchical sequence modeling. arXiv preprint arXiv:2507.07955. Cited by: Table 1. - E. Jang, K. Lee, J. Chung, K. Park, and S. Shin (2025) Improbable bigrams expose vulnerabilities of incomplete tokens in byte-level tokenizers. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, pp. 18209–18216. External Links: Link, Document, ISBN 979-8-89176-332-6 Cited by: §3.2.3. - M. Jiang, K. Z. Liu, M. Zhong, R. Schaeffer, S. Ouyang, J. Han, and S. Koyejo (2024) Investigating data contamination for pre-training language models. arXiv preprint arXiv:2401.06059. Cited by: §3.2.3. - B. Jimenez Gutierrez, H. Sun, and Y. Su (2023) Biomedical language models are robust to sub-optimal tokenization. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, D. Demner-fushman, S. Ananiadou, and K. Cohen (Eds.), Toronto, Canada, pp. 350–362. External Links: Link, Document Cited by: Table 1. - S. Kim, S. Choi, and M. Jeong (2024) Efficient and effective vocabulary expansion towards multilingual large language models. External Links: 2402.14714, Link Cited by: §3.2.2. - A. Kiulian, A. Polishko, M. Khandoga, Y. Kostiuk, G. Gabrielli, Ł. Gagała, F. Zaraket, Q. A. Obaida, H. Garud, W. W. Y. Mak, D. Chaplynskyi, S. B. Amor, and G. Peradze (2024) From english-centric to effective bilingual: llms with custom tokenizers for underrepresented languages. External Links: 2410.18836, Link Cited by: §3.2.2. - T. Kudo and J. Richardson (2018) SentencePiece: a simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, E. Blanco and W. Lu (Eds.), Brussels, Belgium, pp. 66–71. External Links: Link, Document Cited by: Table 3, §3.2.1. - T. Kudo (2018) Subword regularization: improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), I. Gurevych and Y. Miyao (Eds.), Melbourne, Australia, pp. 66–75. External Links: Link, Document Cited by: Table 3, §2.1, §3.2.1. - S. Land and M. Bartolo (2024) Fishing for magikarp: automatically detecting under-trained tokens in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, pp. 11631–11646. External Links: Link, Document Cited by: §1. - A. Langford, A. Shah, A. Gupta, A. Bhatter, A. Goyal, A. Mathur, A. Mohanty, A. Kumar, A. Sethi, A. Komma, et al. (2025) The amazon nova family of models: technical report and model card. arXiv preprint arXiv:2506.12103. Cited by: §1. - M. T. R. Laskar, S. Alqahtani, M. S. Bari, M. Rahman, M. A. M. Khan, H. Khan, I. Jahan, A. Bhuiyan, C. W. Tan, M. R. Parvez, et al. (2024) A systematic survey and critical review on evaluating large language models: challenges, limitations, and recommendations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 13785–13816. Cited by: §3.2.4. - M. T. R. Laskar, M. S. Bari, M. Rahman, M. A. H. Bhuiyan, S. Joty, and J. Huang (2023a) A systematic study and comprehensive evaluation of chatgpt on benchmark datasets. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 431–469. Cited by: §1. - M. T. R. Laskar, X. Fu, C. Chen, and S. B. Tn (2023b) Building real-world meeting summarization systems using large language models: a practical perspective. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 343–352. Cited by: §3.2.3. - LCM team, L. Barrault, P. Duquenne, M. Elbayad, A. Kozhevnikov, B. Alastruey, P. Andrews, M. Coria, G. Couairon, M. R. Costa-jussà, D. Dale, H. Elsahar, K. Heffernan, J. M. Janeiro, T. Tran, C. Ropers, E. Sánchez, R. S. Roman, A. Mourachko, S. Saleem, and H. Schwenk (2024) Large concept models: language modeling in a sentence representation space. External Links: 2412.08821, Link Cited by: §2.2. - H. Lian, Y. Xiong, J. Niu, S. Mo, Z. Su, Z. Lin, H. Chen, J. Han, and G. Ding (2025) Scaffold-bpe: enhancing byte pair encoding for large language models with simple and effective scaffold token removal. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, pp. 24539–24548. Cited by: §3.2.3. - J. Libovický and J. Helcl (2024) Lexically grounded subword segmentation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, pp. 7403–7420. External Links: Link, Document Cited by: Table 3, §3.2.1. - A. Liu, J. Hayase, V. Hofmann, S. Oh, N. A. Smith, and Y. Choi (2025) Superbpe: space travel for language models. arXiv preprint arXiv:2503.13423. Cited by: §2.2. - C. Liu, S. Wang, L. Qing, K. Kuang, Y. Kang, C. Sun, and F. Wu (2024a) Gold panning in vocabulary: an adaptive method for vocabulary expansion of domain-specific LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, pp. 7442–7459. External Links: Link, Document Cited by: §2.1, §2.2, §3.2.2. - Y. Liu, T. Ji, C. Sun, Y. Wu, and X. Wang (2024b) Generation with dynamic vocabulary. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, pp. 18931–18948. External Links: Link, Document Cited by: §2.2. - F. Meyer and J. Buys (2022) Subword segmental language modelling for nguni languages. In Findings of the Association for Computational Linguistics: EMNLP 2022, Y. Goldberg, Z. Kozareva, and Y. Zhang (Eds.), Abu Dhabi, United Arab Emirates, pp. 6636–6649. External Links: Link, Document Cited by: §2.1, §2.2. - S. Moon, T. Hiraoka, and N. Okazaki (2025) Bit-level bpe: below the byte boundary. External Links: 2506.07541, Link Cited by: Table 1. - M. T. Nayeem, S. Alqahtani, M. T. R. Laskar, T. Mohiuddin, and M. S. Bari (2025) Beyond fertility: analyzing strr as a metric for multilingual tokenization evaluation. External Links: 2510.09947, Link Cited by: §3.1, §3.2.4, §3.2.4. - M. T. Nayeem and D. Rafiei (2024) KidLM: advancing language models for children – early insights and future directions. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, pp. 4813–4836. External Links: Link, Document Cited by: §1. - M. T. Nayeem and D. Rafiei (2025) OpinioRAG: towards generating user-centric opinion highlights from large-scale online reviews. In Second Conference on Language Modeling, External Links: Link Cited by: §3.2.3. - P. Neitemeier, B. Deiseroth, C. Eichenberg, and L. Balles (2025) Hierarchical autoregressive transformers: combining byte-and word-level processing for robust, adaptable language models. arXiv preprint arXiv:2501.10322. Cited by: Table 1. - X. Nguyen, W. Zhang, X. Li, M. Aljunied, Z. Hu, C. Shen, Y. K. Chia, X. Li, J. Wang, Q. Tan, et al. (2023) SeaLLMs–large language models for southeast asia. arXiv preprint arXiv:2312.00738. Cited by: Table 4, Table 5, §3.2.2, §3.2.3. - R. Ni, D. Xiao, Q. Meng, X. Li, S. Zheng, and H. Liang (2025) Benchmarking and understanding compositional relational reasoning of llms. Proceedings of the AAAI Conference on Artificial Intelligence 39 (18), pp. 19703–19711. External Links: Link, Document Cited by: §1. - OpenAI (2024) GPT-4 technical report. External Links: 2303.08774, Link Cited by: §1. - A. Pagnoni, R. Pasunuru, P. Rodriguez, J. Nguyen, B. Muller, M. Li, C. Zhou, L. Yu, J. Weston, L. Zettlemoyer, et al. (2024) Byte latent transformer: patches scale better than tokens. arXiv preprint arXiv:2412.09871. External Links: Link Cited by: Table 1, Table 1, §2.2. - A. Petrov, E. L. Malfa, P. Torr, and A. Bibi (2023) Language model tokenizers introduce unfairness between languages. In Thirty-seventh Conference on Neural Information Processing Systems, External Links: Link Cited by: Table 5, §1, §3.2.3. - J. Pfeiffer, I. Vulić, I. Gurevych, and S. Ruder (2021) UNKs everywhere: Adapting multilingual language models to new scripts. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, M. Moens, X. Huang, L. Specia, and S. W. Yih (Eds.), Online and Punta Cana, Dominican Republic, pp. 10186–10203. External Links: Link, Document Cited by: Table 4, §3.2.2. - A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever (2019) Language models are unsupervised multitask learners. OpenAI. Note: Accessed: 2024-11-15 External Links: Link Cited by: §B.1. - V. Reddy, C. W. Schmidt, Y. Pinter, and C. Tanner (2025) How much is enough? the diminishing returns of tokenization training data. External Links: 2502.20273, Link Cited by: §1.2, §3.2.1, §3.2.1, §3.2.4. - J. Rumbelow and M. Watkins (2024) SolidGoldMagikarp (plus, prompt generation). lesswrong.com. External Links: Link Cited by: §3.2.3. - P. Rust, J. Pfeiffer, I. Vulić, S. Ruder, and I. Gurevych (2021) How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), C. Zong, F. Xia, W. Li, and R. Navigli (Eds.), Online, pp. 3118–3135. External Links: Link, Document Cited by: Table 4, Table 5, §3.2.1, §3.2.2, §3.2.3, §3.2.4. - E. Salesky, A. Runge, A. Coda, J. Niehues, and G. Neubig (2020) Optimizing segmentation granularity for neural machine translation. Machine Translation 34 (1), pp. 41–59. Cited by: §3.2.1. - C. W. Schmidt, V. Reddy, H. Zhang, A. Alameddine, O. Uzan, Y. Pinter, and C. Tanner (2024) Tokenization is more than compression. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, pp. 678–702. External Links: Link, Document Cited by: Table 4, Table 6, §1.2, Table 1, Table 1, §1, §2.1, §3.2.1, §3.2.2, §3.2.4, §3.2.4. - C. W. Schmidt, V. Reddy, C. Tanner, and Y. Pinter (2025) Boundless byte pair encoding: breaking the pre-tokenization barrier. External Links: 2504.00178, Link Cited by: Table 1, §2.2, §3.2.1. - R. Sennrich, B. Haddow, and A. Birch (2016) Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), K. Erk and N. A. Smith (Eds.), Berlin, Germany, pp. 1715–1725. External Links: Link, Document Cited by: Table 3, §1, §3.2.1. - J. Seo, J. Kim, S. Byun, and H. Shin (2025) How does a language-specific tokenizer affect llms?. External Links: 2502.12560, Link Cited by: §1. - J. Sha, M. Zhu, C. Feng, and Y. Shang (2025) VEEF-multi-LLM: effective vocabulary expansion and parameter efficient finetuning towards multilingual large language models. In Proceedings of the 31st International Conference on Computational Linguistics, O. Rambow, L. Wanner, M. Apidianaki, H. Al-Khalifa, B. D. Eugenio, and S. Schockaert (Eds.), Abu Dhabi, UAE, pp. 7963–7981. External Links: Link Cited by: §3.2.2. - C. Shao, F. Meng, and J. Zhou (2025) Beyond next token prediction: patch-level training for large language models. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §2.2. - F. T. Shohan, M. T. Nayeem, S. Islam, A. U. Akash, and S. Joty (2024) XL-HeadTags: leveraging multimodal retrieval augmentation for the multilingual generation of news headlines and tags. In Findings of the Association for Computational Linguistics: ACL 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 12991–13024. External Links: Link, Document Cited by: Appendix C. - T. J. Singh, S. R. Singh, and P. Sarmah (2023) Subwords to word back composition for morphologically rich languages in neural machine translation. In Proceedings of the 37th Pacific Asia Conference on Language, Information and Computation, C. Huang, Y. Harada, J. Kim, S. Chen, Y. Hsu, E. Chersoni, P. A, W. H. Zeng, B. Peng, Y. Li, and J. Li (Eds.), Hong Kong, China, pp. 691–700. External Links: Link Cited by: §2.2. - C. Tao, Q. Liu, L. Dou, N. Muennighoff, Z. Wan, P. Luo, M. Lin, and N. Wong (2024) Scaling laws with vocabulary: larger models deserve larger vocabularies. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §3.2.1. - Y. Tay, V. Q. Tran, S. Ruder, J. Gupta, H. W. Chung, D. Bahri, Z. Qin, S. Baumgartner, C. Yu, and D. Metzler (2022) Charformer: fast character transformers via gradient-based subword tokenization. In International Conference on Learning Representations, External Links: Link Cited by: Table 3, §3.2.1. - G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, et al. (2024) Gemma: open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. Cited by: §B.2. - A. Tejaswi, N. Gupta, and E. Choi (2024) Exploring design choices for building language-specific LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, pp. 10485–10500. External Links: Link, Document Cited by: §3.2.2. - C. Toraman, E. H. Yilmaz, F. Şahinuç, and O. Ozcelik (2023) Impact of tokenization on language models: an analysis for turkish. ACM Transactions on Asian and Low-Resource Language Information Processing 22 (4), pp. 1–21. External Links: Link Cited by: Appendix A, §3.2.3. - H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. (2023a) Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §B.3, §2.1. - H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. (2023b) Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. External Links: Link Cited by: §B.3, §2.1. - A. Tsvetkov and A. Kipnis (2024) Information parity: measuring and predicting the multilingual capabilities of language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, pp. 7971–7989. External Links: Link, Document Cited by: §3.2.4. - O. Uzan, C. W. Schmidt, C. Tanner, and Y. Pinter (2024) Greed is all you need: an evaluation of tokenizer inference methods. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 813–822. External Links: Link, Document Cited by: Table 4, Table 6, §3.2.4, §3.2.4. - M. Velayuthan and K. Sarveswaran (2025) Egalitarian language representation in language models: it all begins with tokenizers. In Proceedings of the 31st International Conference on Computational Linguistics, O. Rambow, L. Wanner, M. Apidianaki, H. Al-Khalifa, B. D. Eugenio, and S. Schockaert (Eds.), Abu Dhabi, UAE, pp. 5987–5996. External Links: Link Cited by: §3.2.1. - D. Wang, Y. Li, J. Jiang, Z. Ding, G. Jiang, J. Liang, and D. Yang (2024) Tokenization matters! degrading large language models through challenging their tokenization. arXiv preprint arXiv:2405.17067. External Links: Link Cited by: §3.2.3. - H. Wang, D. Yu, K. Sun, J. Chen, and D. Yu (2019) Improving pre-trained multilingual model with vocabulary expansion. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), M. Bansal and A. Villavicencio (Eds.), Hong Kong, China, pp. 316–327. External Links: Link, Document Cited by: §3.2.2. - A. Wegmann, D. Nguyen, and D. Jurgens (2025) Tokenization is sensitive to language variation. In Findings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 10958–10983. External Links: Link, Document, ISBN 979-8-89176-256-5 Cited by: §1.2, §3.2.1. - B. Wolleb, R. Silvestri, G. Vernikos, L. Dolamic, and A. Popescu-Belis (2023) Assessing the importance of frequency versus compositionality for subword-based tokenization in NMT. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, M. Nurminen, J. Brenner, M. Koponen, S. Latomaa, M. Mikhailov, F. Schierl, T. Ranasinghe, E. Vanmassenhove, S. A. Vidal, N. Aranberri, M. Nunziatini, C. P. Escartín, M. Forcada, M. Popovic, C. Scarton, and H. Moniz (Eds.), Tampere, Finland, pp. 137–146. External Links: Link Cited by: §2.1. - L. Xue, A. Barua, N. Constant, R. Al-Rfou, S. Narang, M. Kale, A. Roberts, and C. Raffel (2022) ByT5: towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics 10, pp. 291–306. External Links: Link, Document Cited by: §B.2, §2.2. - J. Yang, Z. Wang, Y. Lin, and Z. Zhao (2024) Problematic tokens: tokenizer bias in large language models. 2024 IEEE International Conference on Big Data (BigData), pp. 6387–6393. External Links: Link Cited by: §B.1, §3.2.3. - Y. Yao, S. Huang, W. Wang, L. Dong, and F. Wei (2021) Adapt-and-distill: developing small, fast and effective pretrained language models for domains. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, C. Zong, F. Xia, W. Li, and R. Navigli (Eds.), Online, pp. 460–470. External Links: Link, Document Cited by: §3.2.2. - S. Yehezkel and Y. Pinter (2023) Incorporating context into subword vocabularies. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, A. Vlachos and I. Augenstein (Eds.), Dubrovnik, Croatia, pp. 623–635. External Links: Link, Document Cited by: Table 4, §3.2.2. - J. Zhao, Z. Zhang, L. Gao, Q. Zhang, T. Gui, and X. Huang (2024) LLaMA beyond english: an empirical study on language capability transfer. External Links: 2401.01055, Link Cited by: §3.2.2. - M. Zheng, H. Chen, T. Guo, C. Zhu, B. Zheng, C. Xu, and Y. Wang (2024) Enhancing large language models through adaptive tokenizers. Advances in Neural Information Processing Systems 37, pp. 113545–113568. Cited by: §1. - V. Zouhar, C. Meister, J. Gastaldi, L. Du, M. Sachan, and R. Cotterell (2023a) Tokenization and the noiseless channel. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.), Toronto, Canada, pp. 5184–5207. External Links: Link, Document Cited by: Table 6, §1, §3, §3.2.4, §3.2.4. - V. Zouhar, C. Meister, J. Gastaldi, L. Du, T. Vieira, M. Sachan, and R. Cotterell (2023b) A formal perspective on byte-pair encoding. In Findings of the Association for Computational Linguistics: ACL 2023, A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.), Toronto, Canada, pp. 598–614. External Links: Link, Document Cited by: Table 3, Table 6, §3.2.1, §3.2.4. Supplementary Material: Appendices Appendix A Illustrative Example for Token Fragmentation A simple illustration of inefficient token fragmentation can be seen when applying a standard English-trained BPE tokenizer to morphologically rich languages such as Arabic. We emphasize that such tokenizers were not designed to support Arabic or other morphologically rich non-European languages, and are used here solely to illustrate the consequences of language–tokenizer misalignment rather than to imply a design failure. Consider the Arabic word in Figure 2 (“and they will recognize it”). This word combines a conjunction, a verb root, a subject marker, and an object pronoun, all within a single surface form. An English-centric tokenizer typically fails to capture this structure and instead fragments the word into arbitrary subword units. <details> <summary>figures/arabic-tokenization.png Details</summary>  ### Visual Description ## Table: Tokenizer Performance on Arabic Text "وصف و وصف و وصف و وصف و وصف و وصف و وصف و وصف و وصف و وصف و وصف" ### Overview This table compares the tokenization performance of five different NLP tokenizers when processing the Arabic text "وصف و وصف و وصف و وصف و وصف و وصف و وصف و وصف و وصف و وصف و وصف و وصف". The comparison focuses on the number of tokens produced and the sequence length for each tokenizer. ### Components/Axes - **Columns**: 1. **Tokenizer**: Names of the tokenizers used. 2. **Tokens Produced for "وصف"**: Tokenized output for the input text. 3. **Sequence Length**: Number of tokens generated. ### Detailed Analysis | Tokenizer | Tokens Produced for "وصف" | Sequence Length | |------------------------------------|------------------------------------------------------------------------------------------|-----------------| | GPT-2 (English BPE) | ["وصف", "و", "وصف", "و", "وصف", "و", "وصف", "و", "وصف", "و", "وصف"] | 11 | | XLM-R (Multilingual Unigram) | ["وصف", "وصف", "وصف"] | 3 | | LLaMA 1 | ["وصف", "و", "وصف", "و", "وصف", "و", "وصف", "و", "وصف", "و", "وصف"] | 11 | | LLaMA 3.1 / LLaMA 3.2 | ["وصف", "وصف", "وصف"] | 3 | | Morpheme-Aware Tokenizer | ["وصف", "وصف", "وصف", "وصف", "وصف"] | 5 | ### Key Observations 1. **GPT-2 (English BPE)** and **LLaMA 1** produce identical token sequences with a sequence length of 11, matching the word count of the input text. This suggests a 1:1 token-to-word ratio, likely due to subword tokenization strategies. 2. **XLM-R (Multilingual Unigram)** and **LLaMA 3.1/3.2** achieve the shortest sequence length (3 tokens), indicating aggressive merging of subwords or morphemes. 3. **Morpheme-Aware Tokenizer** produces 5 tokens, balancing between raw word-level tokenization and subword merging. Its tokens are all "وصف", suggesting a focus on root morphemes. ### Interpretation - **Efficiency Trade-offs**: - GPT-2 and LLaMA 1 prioritize granularity, which may be beneficial for tasks requiring precise word-level analysis but increases computational load. - XLM-R and LLaMA 3.1/3.2 optimize for brevity, reducing sequence length at the cost of losing some morphological detail. - The Morpheme-Aware Tokenizer strikes a middle ground, potentially useful for tasks requiring morphological awareness without excessive tokenization. - **Tokenizer Behavior**: - The repetition of "وصف" (description) in all tokenized outputs confirms the input text's repetitive structure. - The use of "و" (and) as a separate token in GPT-2 and LLaMA 1 highlights their tendency to split conjunctions, whereas XLM-R and LLaMA 3.1/3.2 merge them into the preceding word. - **Practical Implications**: - Shorter sequence lengths (XLM-R, LLaMA 3.1/3.2) may improve model efficiency but could obscure nuanced linguistic patterns. - The Morpheme-Aware Tokenizer’s approach might enhance performance in low-resource languages with rich morphology, though its 5-token output for 11 words suggests potential over-merging. This analysis underscores the importance of tokenizer selection based on task requirements, balancing between granularity and efficiency. </details> Figure 2: Tokenization of the Arabic word using different tokenization strategies. In this example, the English-trained BPE tokenizer (GPT-2) produces eleven tokens, heavily fragmenting the Arabic word into individual characters and ignoring its internal morphological structure. The multilingual XLM-R tokenizer and the more recent LLaMA 3.1 model perform significantly better, segmenting the word into larger, semantically meaningful units. Interestingly, the morpheme-aware tokenizer results in more tokens than LLaMA 3.1, raising the question of whether fewer tokens always indicate better linguistic alignment or modeling efficiency. In contrast, the earlier LLaMA 1 tokenizer exhibits even greater over-fragmentation, treating spaces and letters as separate tokens and producing eleven fragments. Such over-fragmentation increases sequence length, reduces modeling efficiency, and disrupts semantic representation, effects that become especially pronounced in morphologically rich languages like Arabic, where single surface forms often encode multiple grammatical and semantic elements (Toraman et al., 2023; Ali et al., 2024). This over-fragmentation leads to higher tokenization costs and reduced model efficiency, particularly for non-English users who encounter faster context length limits and increased API costs. Appendix B Evolution of Tokenization in Leading Models As discussed, the tokenizer is a fundamental component of model training, not merely an implementation detail. As LLMs evolve, future models must prioritize tokenization strategies to strengthen linguistic and logical capabilities. The following examples demonstrate that improving a language model often begins with revisiting and refining its vocabulary. B.1 GPT Series Tokenizer Advancements OpenAI’s early GPT models, including GPT-2, used BPE tokenizer with a vocabulary of approximately 50k tokens (Radford et al., 2019). This tokenizer worked well for English but led to over-fragmentation of text in many other languages, causing significantly longer token sequences for non-English content (See § A). To address these issues, OpenAI improved the tokenizer in later models like ChatGPT (GPT-3.5) and GPT-4, introducing the cl100k_base tokenizer with around 100k tokens, doubling the vocabulary size of GPT-2 (Yang et al., 2024). This reduction in token count led to significant improvements. OpenAI continued refining the tokenizer with the 2024 "GPT-4o" update, which expanded the vocabulary to roughly 200k tokens. This change further reduced token usage, especially for non-English languages; however, this expansion came with a trade-off: the inclusion of rare tokens that were under-trained, leading to occasional issues like hallucinations or unexpected behavior, particularly with complex words in languages such as Chinese (See § 3.2.3). B.2 Tokenization Strategies in Google’s Multilingual Models: PaLM, Gemma, and ByT5 Google’s PaLM (Pathways Language Model) (Chowdhery et al., 2023) employs a SentencePiece tokenizer with a 256k vocabulary to handle a broad range of languages, reducing token fragmentation in languages like Vietnamese and Tamil. This strategy enhanced PaLM’s multilingual performance, particularly in tasks like translation. While the larger vocabulary increased memory usage, it provided improved language coverage. In 2024, Google released Gemma (Team et al., 2024), an open-access LLM with a redesigned tokenizer featuring a 262k vocabulary, also used in the Gemini models (Anil et al., 2025). This improved tokenizer mitigated over-tokenization and mis-splitting issues, enabling Gemma to manage over 100 languages and perform well in multilingual benchmarks, underscoring the key role of the tokenizer in its success, albeit with increased memory usage. Google’s ByT5 (Xue et al., 2022) takes a different approach by eliminating tokenization altogether, working directly on raw bytes to address challenges like out-of-vocabulary words, misspellings, and non-Latin scripts that subword tokenizers struggle with. ByT5 uses a 256-byte "vocabulary" (1 byte = 1 token), simplifying the process and ensuring comprehensive text coverage at the expense of longer sequences. While this increases sequence length, ByT5 reduces the model size for embeddings and improves robustness, particularly for noisy inputs. It outperformed tokenized models on tasks such as GLUE and QA, especially for smaller models, demonstrating the potential benefits of eliminating tokenization in specific scenarios. However, byte-level models, such as ByT5, face the trade-off of slower training and inference times due to longer sequence lengths. Despite this, Transformers remain computationally efficient, though ByT5 is slower during training and significantly slower during inference. B.3 LLaMA Series Tokenizer Advancements Meta’s LLaMA (Touvron et al., 2023a) and LLaMA-2 (Touvron et al., 2023b) used a 32k BPE vocabulary primarily trained on English and European languages, leading to longer sequences, wasted context space, and poor performance on underrepresented languages (See § A). Meta did not substantially redesign the tokenizer between LLaMA 1 and LLaMA 2, but improving multilingual performance in LLaMA 3 required changes beyond simple reuse, underscoring how tokenizer choices can necessitate retraining or substantial adaptation. With the release of LLaMA 3 and LLaMA 3.1 (Grattafiori et al., 2024), Meta addressed these issues by introducing a new tokenizer with a larger vocabulary and broader language coverage. The improved tokenizer reduces over-fragmentation in underrepresented languages and enhances performance across a wider range of languages. This update allows LLaMA 3 models to handle multilingual inputs more effectively, providing a more balanced representation of diverse languages and increasing model efficiency. However, these improvements come with the trade-off of higher memory usage and computation, which Meta optimized through architectural adjustments in LLaMA 3.1. Appendix C Core Evaluation Dimensions (i) Coverage: Evaluation must include coverage of rare words, domain-specific terminology, and named entities. Excessive fragmentation often indicates poor vocabulary alignment and leads to inefficiencies during modeling. For example, in the sentence "The patient was diagnosed with pheochromocytoma and referred to Dr. Al-Rahmani at the Hospital," a poorly designed tokenizer may over-segment the rare medical term pheochromocytoma or unnaturally split named entities like Al-Rahmani. High-coverage tokenizers should preserve such expressions with minimal degradation. As another example, for morphologically rich languages, tokenizer coverage can also be evaluated by comparing its output to that of a morphological analyzer. Measuring the divergence between tokenized segments and known morphemes can reveal how well the tokenizer captures meaningful subword units, contributing to both the depth (internal structure) and breadth (lexical variety) of vocabulary representation. (ii) Generalizability: For instance, in the sentence “She re-nationalized the industry after years of privatization,” a well-generalizing tokenizer should segment re-nationalized in a way that reflects its compositional structure, even if the full form was absent from training. Tokenizers that treat related forms inconsistently can harm representation learning and impair downstream generalization. (iii) Linguistic Alignment: Poor alignment obscures grammatical structure, reduces interpretability, and degrades performance in syntax-sensitive tasks. For example, Arabic verbs like yaktubūn (“they write”) and yarsimūn (“they draw”) share morphological patterns that should be preserved to support better generalization and cross-linguistic representation learning. (iv) Robustness: Robust tokenizers should manage spelling errors, informal usage, dialectal forms, and code-switching without excessive fragmentation. In the mixed-language sentence "We discussed the results en la reunión this morning," a resilient tokenizer should treat the embedded Spanish phrase as a cohesive unit rather than fragmenting it unnaturally. Robustness is especially important in multilingual, real-world deployments Shohan et al. (2024). (v) Representation Utilization: Evaluation must also consider which parts of the vocabulary and embedding space are actually used during inference. If a benchmark activates only 30% of token embeddings, the majority of the learned vocabulary remains untested. Tokenizer evaluation should include coverage diagnostics that quantify how well a benchmark exercises the full representational capacity of the model. Appendix D Usage of Large Language Models We disclose that large language models were used in limited, assistive roles. Specifically, they supported text polishing, including improvements to grammar, spelling, phrasing, and word choice, with all suggestions reviewed by the authors. All outputs were manually verified, and the authors remain fully responsible for the research content and conclusions. Table 3: Training Tokenizers from Scratch: core algorithms, learned tokenizers, and theory that shape efficiency, fidelity, and compute from the ground up. Table 4: Adapting Tokenizers: rethinking reuse through language expansion, refinement, and context-aware construction to cut tokens and cost in new domains. Table 5: Responsible tokenization addressing parity, robustness, and stability to reduce unequal costs and failures across languages and noisy inputs. Table 6: Evaluation Framework and Missing Standards of metrics, analyses, and protocols that make tokenizer selection measurable, comparable, and compute-aware. | Training from Scratch | (i) Do target languages over-fragment under English-centric vocabularies? (ii) Which vocabulary size minimizes end-to-end FLOPs rather than dev-set perplexity? (iii) Can morphology-aware constraints reduce token counts in dominant languages? (iv) Could a character/byte backbone with learned downsampling add robustness without higher compute cost? | | --- | --- | | Adapting Existing Tokenizers | (i) Where is token inflation worst, and what parity ratio is acceptable in production? (ii) Is a small vocabulary expansion sufficient, or is a dedicated tokenizer needed? (iii) Can checkpoint compatibility be preserved when modifying the tokenizer (e.g., mapped or appended embeddings)? (iv) What intrinsic or downstream probes will you run before a full retrain? (v) How will you attribute compute savings to shorter sequences versus parameter changes? | | Responsible Tokenization | (i) What token-length disparity across top languages is acceptable, and how will you enforce it? (ii) Do evaluations include noisy, code-switched, or adversarial text to test segmentation stability? (iii) Which mitigation (dropout, merge refinement, morphology-aware rules, or language-specific expansion) yields the largest token reduction for the weakest languages? (iv) How will you track pricing and latency equity linked to tokenizer-induced length differences? (v) What rollout and monitoring plan will detect regressions in parity or robustness after deployment? | | Evaluation Framework and Missing Standards | (i) Which metrics act as pass/fail gates for selection—tokens per 1k characters, parity ratios, compression–structure measures, or estimated FLOPs deltas? (ii) What small, repeatable downstream tasks will validate candidates beyond intrinsic metrics? (iii) How will you disentangle and report the effects of sequence length versus embedding size on step time and memory? (iv) What decision rule determines when to stop iterating and lock the tokenizer for production? (v) How will you document tokenizer choices to ensure cost and quality claims are reproducible? | Table 7: Consolidated guiding questions for principled tokenizer development and adaptation. Each stage highlights key diagnostic prompts for evaluating design trade-offs, resource efficiency, and cross-lingual robustness.