# Subword-Based Comparative Linguistics across 242 Languages Using Wikipedia Glottosets
**Authors**: Iaroslav Chelombitko, Mika Hämäläinen, Aleksey Komissarov
> DataSpikeNeapolis University Pafos,Paphos, CyprusMetropolia University of Applied Sciences,Helsinki, Finland
> Metropolia University of Applied Sciences,Helsinki, Finland
> Neapolis University Pafos,Paphos, Cyprusaglabx,Paphos, Cyprus
Abstract
We present a large-scale comparative study of 242 Latin and Cyrillic-script languages using subword-based methodologies. By constructing ’glottosets’ from Wikipedia lexicons, we introduce a framework for simultaneous cross-linguistic comparison via Byte-Pair Encoding (BPE). Our approach utilizes rank-based subword vectors to analyze vocabulary overlap, lexical divergence, and language similarity at scale. Evaluations demonstrate that BPE segmentation aligns with morpheme boundaries 95% better than random baseline across 15 languages (F1 = 0.34 vs 0.15). BPE vocabulary similarity correlates significantly with genetic language relatedness (Mantel r = 0.329, p < 0.001), with Romance languages forming the tightest cluster (mean distance 0.51) and cross-family pairs showing clear separation (0.82). Analysis of 26,939 cross-linguistic homographs reveals that 48.7% receive different segmentations across related languages, with variation correlating to phylogenetic distance. Our results provide quantitative macro-linguistic insights into lexical patterns across typologically diverse languages within a unified analytical framework.
Keywords Multilingual NLP $·$ Comparative Linguistics $·$ BPE Tokenization $·$ Cross-linguistic Analysis $·$ Lexical Similarity $·$ Script-based Typology
1 Introduction
Traditional comparative linguistics, while providing deep historical and typological insights (see Lehmann 2013; Beekes 2011; Nichols 1992; Partanen et al. 2021; Säily et al. 2021), often lacks the scalability to handle the current volume of digital text (Arnett and Bergen 2024; Akindotuni 2025). However, recent NLP advances facilitate massive, data-driven studies (see Dang et al. 2024; Bender 2011; Imani et al. 2023; Sproat 2016), revealing universal tendencies in phonetics (Blum et al. 2024), lexical semantics (Tjuka et al. 2024), and sound symbolism (Ćwiek et al. 2022; Cathcart and Jäger 2024) that were previously inaccessible through manual, small-scale methods.
Despite this progress, large-scale studies frequently neglect endangered languages (Hämäläinen 2021) and systematic performance disparities in multilingual models (see Shani et al. 2026; Chelombitko et al. 2024; Dunn and others 2011). We address this gap by examining 242 languages through a novel script-level lens (Latin vs. Cyrillic), moving beyond family-specific comparisons (Meillet 1967; Beekes 2011) to reveal overarching patterns visible only when languages share a script-based writing system (Daniels and Bright 1996; Rust et al. 2021).
In particular, we adopt a subword-based strategy (Sennrich and others 2016) using Byte-Pair Encoding (BPE) (Gage 1994) tokenizers, which we train both on individual languages and on large aggregated corpora for each script. Aggregating data for all Latin and Cyrillic-script languages into respective training sets effectively unifies each script community into a single model, leveraging the shared-parameter paradigm common in massively multilingual pretraining (see Johnson and others 2016; Conneau et al. 2020).
<details>
<summary>workflow.png Details</summary>
### Visual Description
## Flow Diagram: Language Processing Pipeline
### Overview
The image is a flow diagram illustrating a language processing pipeline. It outlines the steps involved in processing Wikipedia dumps for linguistic analysis, starting from data acquisition and preprocessing to analysis and insights. The diagram uses rounded rectangles to represent different stages, connected by lines to indicate the flow of data.
### Components/Axes
The diagram consists of the following components:
1. **Wikipedia Dumps:** (Light Blue)
* Description: "320 Languages (in ZIM format)"
2. **BPE Tokenization:** (Light Red)
* Description: "Train individual & combined tokenizers"
3. **Script-Based Filtering:** (Light Green)
* Description: "Select Cyrillic vs. Latin (242 total)"
4. **Macro-Level Insights:** (Light Orange)
* Description: "Script-level comparisons & patterns"
5. **Subword-Based Analysis:** (Light Purple)
* Description: "Compare languages, rank-based vectors"
6. **Monolingual Glottosets:** (Light Yellow)
* Description: "Extract words, TF & DF, paragraphs"
The flow of data is indicated by lines connecting these components.
### Detailed Analysis or ### Content Details
* **Wikipedia Dumps** (Light Blue) is the starting point, feeding into both **BPE Tokenization** (Light Red) and **Script-Based Filtering** (Light Green).
* **BPE Tokenization** (Light Red) feeds into **Macro-Level Insights** (Light Orange).
* **Script-Based Filtering** (Light Green) feeds into both **Subword-Based Analysis** (Light Purple) and **Monolingual Glottosets** (Light Yellow).
* **Macro-Level Insights** (Light Orange) feeds into **Subword-Based Analysis** (Light Purple).
* **Subword-Based Analysis** (Light Purple) feeds into **Monolingual Glottosets** (Light Yellow).
### Key Observations
* The pipeline starts with raw Wikipedia data and progresses through tokenization, filtering, and analysis stages.
* There are two parallel paths from the initial data: one focusing on tokenization and macro-level insights, and the other on script-based filtering and subword analysis.
* The final stage involves creating monolingual glottosets, suggesting the goal is to extract and organize language-specific data.
### Interpretation
The diagram illustrates a comprehensive approach to processing multilingual Wikipedia data for linguistic research. The pipeline combines different techniques, including tokenization, script-based filtering, and subword analysis, to extract meaningful insights and create language-specific datasets. The parallel paths suggest different analytical approaches that converge in the final stage of glottoset creation. The diagram highlights the complexity of multilingual data processing and the need for a multi-faceted approach to extract valuable information.
</details>
Figure 1: Pipeline architecture for subword-based comparative linguistics across 242 languages using Wikipedia glottosets. The workflow illustrates the transformation of Wikipedia dumps (320 languages) through sequential stages: script-based filtering yielding 37 Cyrillic and 205 Latin script languages, monolingual glottoset construction with TF/DF metrics, BPE tokenization (both individual and combined training with 4096 tokens vocabulary), and vector-based subword analysis. Each colored node represents a distinct processing phase, culminating in macro-level insights for script-level comparative linguistics. This modular approach enables scalable analysis of morphological patterns across multiple languages simultaneously.
Although subword units are not perfect morphemes (see Sennrich and others 2016; Bostrom and Durrett 2020), they serve as robust automated tools for macro-linguistic research (see Khurana and others 2024; Pham et al. 2024; Futrell and others 2015) that scale more efficiently than manual expert alignment (see Campbell 2020; Rama et al. 2018; Ciobanu and Dinu 2014).
Leveraging the framework (Figure 1), this work makes two key contributions:
1. Script-Level Aggregation: We demonstrate how combining languages by script (Latin vs. Cyrillic) enables macro-level comparative insights difficult to achieve through traditional language-by-language lenses.
1. Practical Subword Methodology: We introduce a tractable, automated framework that reduces reliance on manual annotation by providing data-driven lexical segments for robust cross-linguistic analysis.
While traditional comparative linguistics focuses on genetic relationships and historical reconstruction, and typology examines structural similarities regardless of ancestry, our subword-based approach bridges both: we demonstrate that BPE tokenization captures phylogenetic signal (Section 4.3) while also revealing typological patterns across unrelated languages sharing the same script.
2 Related Work
Comparative linguistics in NLP has evolved from general data-driven frameworks (see Bender 2011; Sproat 2016; Imani et al. 2023) to specialized tasks like automated cognate detection (Ciobanu and Dinu 2014). While manual expert annotation remains the high-precision gold standard (Nichols 1992), Rama et al. (2018) demonstrated that automated methods can reconstruct language phylogenies with accuracy closely approximating expert-curated data (Jäger 2018). These approaches highlight the potential for scaling linguistic analysis across diverse language families where manual examination is impractical.
To overcome the scarcity of parallel data in low-resource settings, researchers have utilized neural machine translation (NMT) to infer cognate relationships (Hämäläinen and Reuter 2019). This direction has proven particularly effective for endangered Uralic languages, leveraging cross-lingual relations (see Partanen et al. 2021; Chelombitko and Komissarov 2024; Chelombitko et al. 2025) and data augmentation through synthetic cognates generated by statistical machine translation (SMT) (Poncelas et al. 2019). Such hybrid strategies address resource limitations while enriching existing linguistic databases for under-represented script communities.
More recently, architectures inspired by computational biology have addressed previous computational bottlenecks. Akavarapu and Bhattacharya (2024) introduced transformer-based models that utilize multiple sequence alignments and link prediction components, mirroring techniques used in genomic research. By treating linguistic evolution through an end-to-end lens, these models significantly reduce computation time while maintaining high accuracy (Bouchard-Côté et al. 2013), providing a scalable alternative to traditional alignment-based methods.
Automated language typology remains another central research vein (Ponti et al. 2019), with studies using information-theoretic measures to quantify morphological synthesis and fusion (see Rathi et al. 2021; Oncevay et al. 2022). Crucially, Gutierrez-Vasques et al. (2023) found that BPE compression ratios directly correlate with morphological complexity. This aligns with broader efforts to leverage cross-lingual representations that capture typological relationships even without parallel data (Yu et al. 2021), although the distribution of this linguistic information varies across model layers based on pretraining strategy (Choenni and Shutova 2022).
3 Methodology
3.1 Monolingual Glottosets Construction
We downloaded Wikipedia dumps in ZIM format for 320 languages available in the ZIM archive https://dumps.wikimedia.org/kiwix/zim/wikipedia/. After that, we selected either the 2024 or 2023 version (preferring the 2024 version if available), otherwise, we used the 2023 version. We included all topics by selecting the “all” files option. To clean the data from unnecessary HTML elements, we used a custom script available in our repository. In brief, this script resolves redirects and extracts only paragraph text while removing as many auxiliary HTML tags as possible, leaving only clean paragraph texts.
<details>
<summary>graph.png Details</summary>
### Visual Description
## Dependency Diagram: Word Relationships
### Overview
The image presents a dependency diagram illustrating relationships between words. The diagram is split into two main sections, left and right, with nodes representing words and edges indicating dependencies. The nodes are either rectangular (green) or circular (blue), and they contain the word and a "reuse" count.
### Components/Axes
* **Nodes:** Represent words. Green rectangles contain a word and its reuse count. The blue circle contains a word and its reuse count.
* **Edges:** Black lines indicating dependencies between words.
* **Reuse Count:** A number in parentheses below each word, indicating how many times the word is reused.
### Detailed Analysis or ### Content Details
**Left Section:**
* **Node 1:** "чи" (reuse: 17) - Located in the center-left.
* **Node 2:** "чив" - Located above "чи".
* **Node 3:** "очи" - Located below "чив".
* **Node 4:** "чино" - Located to the right of "чи".
* **Node 5:** "дир" - Located below "чино".
* **Node 6:** "чий" - Located below "дир".
* **Node 7:** "хун" - Located below "чий".
* **Node 8:** "чими" - Located below "хун".
* **Node 9:** "твор" - Located to the right of "чив".
* **Node 10:** "тної" - Located below "твор".
* **Node 11:** "ноої" - Located below "тної".
* **Node 12:** "гайм" - Located below "ноої".
* **Node 13:** "шир" - Located below "гайм".
* **Node 14:** "відді" - Located below "шир".
* **Node 15:** "україн" - Located to the right of "відді".
* **Node 16:** "йного" - Located below "відді".
* **Node 17:** "стих" - Located below "йного".
**Center Section:**
* **Node 18:** "a" (reuse: 167) - Located in the center, a blue circle.
* **Node 19:** "ja" (reuse: 55) - Located to the right of "a".
**Right Section:**
* **Node 20:** "ohju" - Located in the top-right.
* **Node 21:** "star" - Located to the right of "ohju".
* **Node 22:** "uja" - Located below "ohju".
* **Node 23:** "idae" - Located below "uja".
* **Node 24:** "eja" - Located below "idae".
* **Node 25:** "ista" (reuse: 23) - Located in the bottom-right.
* **Node 26:** "taide" - Located above "ista".
* **Node 27:** "vuude" - Located above "ista".
* **Node 28:** "okuvi" - Located below "vuude".
**Dependencies:**
* "чи" has dependencies on "чив", "очи", "чино", "дир", "чий", "хун", and "чими".
* "a" has dependencies on "україн".
* "ja" has dependencies on "ohju", "uja", "idae", "eja", "ista", "taide", "vuude", and "okuvi".
### Key Observations
* The word "a" has the highest reuse count (167), followed by "ja" (55).
* The word "чи" has a reuse count of 17.
* The word "ista" has a reuse count of 23.
* The diagram shows a network of dependencies between words, with some words having multiple connections.
### Interpretation
The diagram visualizes the relationships between words in a text or corpus. The reuse count indicates the frequency of each word. The dependencies suggest grammatical or semantic relationships. The central nodes "a" and "ja" appear to be important connectors, given their high reuse counts and numerous connections. The diagram could be used to analyze the structure and content of the text, identify key themes, or understand the relationships between different concepts.
</details>
Figure 2: Visualization of interactive BPE merge graphs for Ukrainian (left) and Finnish (right) subword tokenization patterns. The diagrams show directed merge sequences with reuse counts indicated in parentheses. Vertical black bars represent merge steps, while edges show the progression of subword unit formation. The contrasting patterns reflect language-specific morphological characteristics: Ukrainian showing consistent Cyrillic character combinations, while Finnish exhibits agglutinative patterns. Interactive web application available on our repository.
The extracted paragraphs were filtered based on script consistency. For Cyrillic, we retained only Cyrillic paragraphs, and for Latin, only those written in the Latin script. Unfortunately, there is currently no reliable tool for accurately detecting all 242 languages, so at this stage, we filtered only by script (Table 1). Additionally, an effective filtering step was to remove all paragraphs containing fewer than 10 words. This significantly reduced noise in the dataset, including accidental words from other languages.
Table 1: Statistics for top 30 languages by token count from Wikipedia dumps. Columns show language names and codes, number of raw and cleaned paragraphs, word count in millions, script type (Latin/Cyrillic), total tokens, and cross-script contamination (Cyrillic tokens in Latin-script languages and vice versa).
| English | en | 54125720 | 42149266 | 2661.93 | Latin | 2661930466 | 5062 | – |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| German | de | 22354272 | 17953642 | 1026.43 | Latin | 1026430544 | 2648 | – |
| French | fr | 20464072 | 16742203 | 940.75 | Latin | 940746868 | 1621 | – |
| Spanish | es | 14107859 | 12170134 | 763.36 | Latin | 763357750 | 1134 | – |
| Italian | it | 11917473 | 9806341 | 571.76 | Latin | 571759224 | 1150 | – |
| Russian | ru | 15290408 | 12295838 | 552.42 | Cyrillic | 552416889 | – | 429004 |
| Cebuano | ceb | 16841937 | 10069736 | 447.85 | Latin | 447850263 | 92 | – |
| Portuguese | pt | 6845897 | 5642228 | 332.96 | Latin | 332959017 | 413 | – |
| Dutch | nl | 7764590 | 6429095 | 316.21 | Latin | 316210100 | 415 | – |
| Polish | pl | 15961541 | 5616831 | 253.85 | Latin | 253846410 | 2044 | – |
| Catalan | ca | 4273906 | 3749516 | 238.81 | Latin | 238813199 | 710 | – |
| Ukrainian | uk | 7864058 | 5582991 | 219.83 | Cyrillic | 219827684 | – | 200945 |
| Vietnamese | vi | 4707518 | 3129629 | 201.38 | Latin | 201383401 | 623 | – |
| Swedish | sv | 7437772 | 5088656 | 200.89 | Latin | 200889695 | 312 | – |
| Serbo-Croatian | sh | 2248318 | 1682017 | 193.56 | Latin | 193561280 | 19539 | – |
| Czech | cs | 3386803 | 2865556 | 150.39 | Latin | 150393063 | 998 | – |
| Hungarian | hu | 3932082 | 2603906 | 129.86 | Latin | 129864098 | 518 | – |
| Indonesian | id | 2984643 | 2120278 | 106.98 | Latin | 106980811 | 622 | – |
| Norwegian Bokmål | nb | 2906506 | 2239883 | 105.51 | Latin | 105507751 | 175 | – |
| Finnish | fi | 3141975 | 2295657 | 93.5 | Latin | 93502432 | 650 | – |
| Serbian | sr | 3210579 | 1981413 | 91.81 | Cyrillic | 91807037 | – | 193621 |
| Turkish | tr | 2528498 | 1757628 | 85.16 | Latin | 85156035 | 570 | – |
| Romanian | ro | 2360641 | 1539046 | 84.62 | Latin | 84617897 | 454 | – |
| Bulgarian | bg | 1599763 | 1386917 | 69.12 | Cyrillic | 69123830 | – | 36265 |
| Waray | war | 2571970 | 1448198 | 69.04 | Latin | 69038926 | 0 | – |
| Danish | da | 1551416 | 1216805 | 61.11 | Latin | 61107267 | 126 | – |
| Galician | gl | 1188136 | 1049404 | 60.95 | Latin | 60947590 | 181 | – |
| Malay | ms | 1499560 | 1034055 | 56.37 | Latin | 56370979 | 106 | – |
| Asturian | ast | 1134182 | 912479 | 56.25 | Latin | 56254683 | 84 | – |
| Esperanto | eo | 1669507 | 1188077 | 54.51 | Latin | 54512106 | 441 | – |
In the following analysis we included only languages with Cyrillic (37 languages) and Latin (205 languages) scripts. The decision to restrict the scope to Latin and Cyrillic script languages was motivated by three primary factors: (1) these scripts share a common alphabet origin, enabling direct character-level comparison; (2) they represent the largest script families in Wikipedia coverage; and (3) they include both closely related language families (e.g., Romance, Slavic) and typologically diverse languages (e.g., Finnish, Turkish, Vietnamese), providing a rich testbed for subword-based comparative analysis. Additionally, as an unexpected result, we were able to measure the extent to which Wikipedia articles are contaminated with Latin and Cyrillic scripts for languages that use other scripts (80 languages). The results are presented in Table 2.
Table 2: Script contamination analysis for languages using non-Latin/non-Cyrillic writing systems. Data shows language names, codes, raw paragraph counts, native script types, and the number of detected Latin and Cyrillic tokens representing potential script contamination.
| Min Nan Chinese | nan | 830227 | Han | 13 | 541415 |
| --- | --- | --- | --- | --- | --- |
| Japanese | ja | 12737874 | Japanese | 915 | 298430 |
| Chinese | zh | 7206035 | Han | 554 | 117304 |
| Greek | el | 1733250 | Greek | 363 | 96009 |
| Hebrew | he | 3368324 | Hebrew | 479 | 90907 |
| Korean | ko | 3333575 | Hangul | 635 | 81612 |
| Arabic | ar | 4624040 | Arabic | 246 | 58738 |
| Persian | fa | 3132901 | Arabic | 165 | 51729 |
| Balinese | ban | 87175 | Balinese | 4 | 48922 |
| Thai | th | 944565 | Thai | 305 | 43559 |
| Armenian | hy | 1647293 | Armenian | 2579 | 41259 |
| Egyptian Arabic | arz | 5166127 | Arabic | 23 | 39349 |
| Buginese | bug | 36776 | Lontara | 0 | 19834 |
| Min Dong Chinese | cdo | 28589 | Han | 0 | 19197 |
| Georgian | ka | 644781 | Georgian | 387 | 15234 |
| Bengali | bn | 854059 | Bengali | 64 | 13786 |
| Hindi | hi | 703141 | Devanagari | 80 | 13369 |
| Tamil | ta | 820809 | Tamil | 28 | 11153 |
| Hakka Chinese | hak | 14204 | Han | 2 | 10368 |
| Malayalam | ml | 464433 | Malayalam | 31 | 9848 |
| Sinhalese | si | 166475 | Sinhala | 3 | 9552 |
| Burmese | my | 451126 | Burmese | 8 | 9467 |
| Urdu | ur | 660706 | Arabic | 62 | 9172 |
| Cantonese | yue | 373575 | Han | 53 | 8418 |
| South Azerbaijani | azb | 605161 | Arabic | 18 | 7459 |
| Goan Konkani | gom | 39569 | Devanagari | 0 | 6955 |
| Marathi | mr | 349639 | Devanagari | 10 | 6842 |
| Cambodian | km | 99641 | Khmer | 13 | 6324 |
| Telugu | te | 832028 | Telugu | 3 | 6170 |
| Tachelhit | shi | 7811 | Tifinagh | 0 | 5799 |
We define monolingual glottosets as language-specific collections of lexical units derived from monolingual texts. Each glottoset reflects a particular language’s vocabulary in one script, in our case Latin or Cyrillic in lowercase. Unfortunately, while we initially aimed to avoid normalizing words to lowercase, we found that doing so significantly improves subsequent tokenization. The glottoset includes additional features: Term Frequency (TF) and Document Frequency (DF). DF is defined as the occurrence of a word within a Wikipedia paragraph, as the Wikipedia parsing process is based on paragraph-level extraction. Wikipedia based glottosets characteristics are presented in Table 3.
Table 3: Lexical statistics for a diverse set of languages using Latin and Cyrillic scripts. The table presents vocabulary size (ranging from 1,447 for Cheyenne to 3,207,272 for Hungarian), lexical diversity scores (from 0.009 in Dutch to 0.383 in Cheyenne), median word length (6–10 characters), and top-3 most frequent tokens.
| Swiss German | Latin | 565258 | 0.070 | 10 | d, isch, dr |
| --- | --- | --- | --- | --- | --- |
| Tajik | Cyrillic | 317780 | 0.041 | 8 | дар, аз, ки |
| Hungarian | Latin | 3207272 | 0.025 | 10 | a, és, az |
| Lingala | Latin | 23153 | 0.123 | 7 | ya, na, ezalí |
| Zeeuws | Latin | 51177 | 0.087 | 8 | de, n, t |
| Lak | Cyrillic | 6890 | 0.315 | 7 | ва, шагьру, инсан |
| Malay | Latin | 637993 | 0.011 | 8 | yang, di, dan |
| Cheyenne | Latin | 1447 | 0.383 | 6 | ho, e, vé |
| Aymara | Latin | 29323 | 0.171 | 8 | a, jisk, suyu |
| Friulian | Latin | 44905 | 0.086 | 7 | di, e, al |
| Javanese | Latin | 281161 | 0.040 | 8 | ing, lan, iku |
| Rusyn | Cyrillic | 112582 | 0.149 | 8 | в, на, є |
| Ido | Latin | 129897 | 0.029 | 7 | la, di, e |
| Norman | Latin | 31509 | 0.106 | 7 | d, est, la |
| Ligurian | Latin | 133733 | 0.090 | 7 | l, a, de |
| Karakalpak | Latin | 104775 | 0.144 | 8 | hám, menen, bul |
| Romanian | Latin | 1102965 | 0.013 | 8 | în, de, şi |
| Somali | Latin | 118361 | 0.076 | 8 | oo, iyo, ka |
| Lombard | Latin | 239355 | 0.042 | 7 | l, è, de |
| Norwegian Bokmål | Latin | 1918724 | 0.018 | 10 | i, og, en |
| West Flemish | Latin | 96384 | 0.079 | 8 | de, van, in |
| Māori | Latin | 15232 | 0.033 | 7 | te, ko, o |
| Icelandic | Latin | 451176 | 0.051 | 10 | í, og, á |
| Mongolian | Cyrillic | 235807 | 0.044 | 8 | нь, оны, юм |
| Dutch | Latin | 2866741 | 0.009 | 10 | de, van, in |
| Kabyle | Latin | 52338 | 0.106 | 7 | n, d, deg |
| Tsonga | Latin | 14810 | 0.116 | 8 | hi, na, ya |
| Madurese | Latin | 33922 | 0.155 | 7 | bân, è, sè |
3.2 Glottoset BPE Tokenization
We implemented our own version of a BPE tokenizer that does not treat spaces as separate tokens and exclusively tokenizes words. The tokenizer is available on our GitHub https://github.com/aglabx/morphoBPE. After that, we tokenized all words from each glottoset. We used two sets of parameters. In the first variant, we employed a vocabulary size of 4096 tokens. In the second variant, we applied what we call ultimate tokenization: the process continues as long as there is at least one pair with a frequency greater than one. This second approach is highly dependent on the corpus size. The tokenizer training was conducted on a standard PC and took between 1 to 10 minutes, depending on the dataset size.
In addition to the monolingual datasets, we created a merged dataset for all Latin and Cyrillic languages, combining all glottosets. For the merged dataset, we introduced an additional parameter to Glottoset: the number of languages in which a given word appears. Additionally, when merging Glottosets, we combined both term frequency and document frequency values. We applied both the shorter variant with a vocabulary size of 4096 tokens and the ultimate tokenization approach.
3.3 Vector Representation of BPE Tokens by Language
Using the tokenizer trained on all languages, we obtained BPE tokens. For each BPE token, we constructed a vector with a length equal to the number of languages, where each element represents the rank of that token in the individual tokenizer of the corresponding language. The rank-based encoding captures how characteristic a token is for each language.
Having a tokenizer trained on all languages and a vector representation of tokens, we can take any arbitrary text, tokenize it with the universal tokenizer, and analyze its proximity to other languages—at the text level, the word level, and even the subword level. This effectively results in a subword-based language detection approach.
3.4 Hierarchical Subword Tokenization Analysis
The next concept we propose is that if we take a word from a language for which a language-specific tokenizer exists, we can construct a hierarchical tree representing how this word is tokenized into subword units in one or more languages. This approach has an indirect connection to morphological analysis.
Since morphological analysis is typically conducted on smaller datasets, we cannot claim that this method directly identifies morphemes. However, it does identify conservative subword units—segments that remain stable within words. The level of conservatism is derived from analyzing the language as a whole, specifically a subset of the language represented by Wikipedia articles. We have added this functionality to our BPE tokenizer.
3.5 Comparative Monolingual Tokenizer Analysis
Beyond comparing how tokenizers handle words, another idea naturally arises: what if we compare tokenizers themselves? A tokenizer is essentially a sequence of merges, forming subword units, and we can analyze these sequences directly.
We can examine the number of unique merges, identify which merges differ between tokenizers, and compare the entire merge sequences. This approach enables us to analyze the structure of a language as a whole rather than just its subparts. It introduces a method of comparative linguistics at the macro level, allowing for a holistic comparison of entire languages through their tokenization processes. An example of an interactive visualization for the monolingual BPE tokenizer is shown in Figure 2.
4 Evaluation
To validate our subword-based comparative linguistics framework, we designed four quantitative evaluations testing specific hypotheses about BPE tokenization behavior across languages.
4.1 Research Questions
We address the following research questions:
1. RQ1 (Morphological Grounding): Do BPE segmentation boundaries align with linguistically meaningful morpheme boundaries?
1. RQ2 (Phylogenetic Signal): Does BPE vocabulary similarity correlate with genetic language relatedness?
1. RQ3 (Language Discrimination): Can BPE tokenizers discriminate between languages that share orthographic forms (homographs)?
4.2 E2: Morphological Boundary Agreement
Hypothesis: BPE segmentation boundaries align with morpheme boundaries better than random segmentation.
Method: We used MorphyNet (Batsuren et al. 2021) derivational morphology data for 15 languages. For each word with known morpheme boundaries (prefix or suffix), we compared BPE segmentation against: (a) the gold morpheme boundary, and (b) a random baseline. We computed Boundary Precision, Recall, and F1 scores.
Results: All 15 languages showed BPE segmentation significantly better than random baseline (Table 4). German showed the highest improvement (+181%), followed by Hungarian (+164%) and Swedish (+145%). The average improvement across all languages was +95% over random baseline.
Table 4: E2: Morphological boundary agreement. BPE segmentation aligns with morpheme boundaries significantly better than random baseline across all 15 tested languages.
| German Hungarian Swedish | 0.42 0.39 0.37 | 0.15 0.15 0.15 | +181% +164% +145% |
| --- | --- | --- | --- |
| English | 0.36 | 0.15 | +140% |
| Finnish | 0.35 | 0.15 | +133% |
| Russian | 0.31 | 0.15 | +107% |
| Average | 0.34 | 0.15 | +95% |
Conclusion: Hypothesis supported. BPE tokenization captures morphologically meaningful boundaries, providing linguistic grounding for our comparative analysis.
4.3 E3: Language Phylogeny Correlation
Hypothesis: BPE vocabulary similarity correlates with genetic language relatedness (phylogeny).
Method: We computed pairwise BPE distance ( $1-\text{Jaccard similarity}$ ) between language vocabularies for 49 Latin-script languages. We compared this distance matrix against phylogenetic distance derived from Glottolog (Hammarström and Forkel 2022) classification (Family $→$ Subfamily $→$ Branch). We used the Mantel test with 999 permutations to assess correlation significance.
Results: The Mantel test revealed a significant positive correlation between BPE distance and phylogenetic distance:
- Mantel $r=0.329$ ( $p<0.001$ , $z=12.3$ )
- Within-family BPE distance: 0.67 (mean)
- Between-family BPE distance: 0.82 (mean)
- Separation ratio: 1.22 $×$ ( $t$ -test $p<10^{-13}$ )
Romance languages showed the tightest clustering (mean distance 0.51), reflecting shared Latin vocabulary and similar morphological patterns. Germanic languages showed higher internal distance (0.71), likely due to English’s extensive Romance/Latin borrowings.
Conclusion: Hypothesis supported. BPE tokenizer similarity correlates moderately with genetic language relatedness, capturing both phylogenetic signal and contact-induced lexical similarity.
4.4 E4: Cross-lingual Homograph Discrimination
Hypothesis: BPE tokenizers segment identical orthographic forms (homographs) differently across languages, reflecting language-specific morphological patterns.
Method: For 6 Cyrillic Slavic languages (Ukrainian, Russian, Belarusian, Bulgarian, Macedonian, Serbian), we:
1. Extracted word vocabularies from TF-DF files (frequency $≥$ 100)
1. Identified homographs: words appearing in 2+ language vocabularies
1. Tokenized each homograph with each language’s BPE tokenizer
1. Compared segmentation patterns across languages
Results: We found 26,939 homographs across the 6 languages:
- 48.7% showed different segmentation across languages
- 51.3% showed identical segmentation
Segmentation difference correlated with linguistic distance:
- Russian-Ukrainian (both East Slavic): 31.2% different
- Belarusian-Macedonian (East vs. South): 61.9% different
A striking example is the name “димитров”, which received 5 different segmentations across 5 languages:
- Ukrainian: ди|ми|т|ров
- Russian: ди|мит|ров
- Bulgarian: дими|т|ров
- Macedonian: димит|ров
- Serbian: дим|ит|ров
Conclusion: Hypothesis partially supported. Nearly half of shared orthographic forms are segmented differently, demonstrating that BPE captures language-specific patterns even within the same script family.
4.5 Qualitative Analysis
Beyond quantitative evaluation, our approach effectively captures the subword characteristics of cross-linguistic homonyms, defined as words with identical spellings but distinct meanings across languages. For example, the word “заказала” carries different meanings in Russian and Ukrainian, as shown in their morphemic tokenizations (Figure 3):
- Ukrainian: за – ка – зал – а
- Russian: зака – зал – а
<details>
<summary>uk_r_zakazala.png Details</summary>
### Visual Description
## Diagram: Ukrainian and Russian Word Decomposition
### Overview
The image presents two tree diagrams illustrating the decomposition of words in Ukrainian and Russian. The diagrams show how the words "зал" (hall) are broken down into their constituent sounds or syllables.
### Components/Axes
* **Titles:** "Ukrainian (uk\_tok)" on the left and "Russian (ru\_tok)" on the right.
* **Nodes:** Each node in the tree represents a sound or syllable.
* **Edges:** The lines connecting the nodes represent the decomposition process.
### Detailed Analysis
**Ukrainian (uk\_tok) Diagram:**
* The word being decomposed is "зал" (hall).
* Level 1: "зал" is broken down into "з" and "ал".
* Level 2: "ал" is broken down into "а" and "л".
* Level 3: "за" is broken down into "з" and "а".
* Level 4: "ка" is broken down into "к" and "а".
**Russian (ru\_tok) Diagram:**
* The word being decomposed is "зал" (hall).
* Level 1: "зака" is broken down into "за" and "ка".
* Level 2: "за" is broken down into "з" and "а". "ка" is broken down into "к" and "а".
* Level 3: "зал" is broken down into "з" and "ал".
* Level 4: "ал" is broken down into "а" and "л".
### Key Observations
* Both diagrams decompose words into their constituent sounds.
* The Ukrainian diagram decomposes "зал" directly, while the Russian diagram decomposes "зака" first, then "зал".
* The diagrams highlight the phonetic structure of the words in each language.
### Interpretation
The diagrams illustrate a basic concept in phonetics and linguistics: the decomposition of words into smaller units of sound. The difference in the initial decomposition between Ukrainian and Russian suggests variations in pronunciation or syllabic structure between the two languages. The diagrams are simplified representations and may not capture all nuances of phonetic analysis.
</details>
Figure 3: Hierarchical BPE tokenization trees comparing the word заказала in Ukrainian (left) and Russian (right). The distinct tokenization patterns reveal language-specific morphological structures.
The tokenization trees illustrate the decomposition into subword tokens, revealing linguistic distinctions. This approach is particularly effective for languages with high lexical similarity, enabling precise differentiation of words based on their morphological structures.
4.6 Subword-Based Language Identification for Out-of-Vocabulary Words
Our subword-based analysis also aids naive language identification, particularly for multi-morpheme words. Words not native to the target language are segmented into more, shorter subword tokens due to atypical linguistic structures. Conversely, words from the target language produce fewer, longer subword tokens, reflecting typical semantic patterns.
For example, the Ukrainian word “промисловiсть” (industry) is analyzed across Ukrainian, Belarusian, and Russian tokenization models (Figure 4):
- Ukrainian: проми-с-лов-iсть
- Belarusian: про-ми-сло-вi-сть
- Russian: про-ми-с-лов-i-сть
<details>
<summary>uk_be_r_promyslovist.png Details</summary>
### Visual Description
## Linguistic Tree Diagrams: Ukrainian, Belarusian, and Russian
### Overview
The image presents three tree diagrams illustrating the decomposition of a word or phrase in Ukrainian, Belarusian, and Russian. Each diagram shows a hierarchical structure, breaking down the initial word/phrase into smaller components, potentially representing morphemes or phonemes.
### Components/Axes
* **Titles:**
* Top-left: "Ukrainian (uk\_tok)"
* Top-center: "Belarusian (be\_tok)"
* Top-right: "Russian (ru\_tok)"
* **Nodes:** Each node in the tree represents a linguistic unit (word, morpheme, phoneme). The nodes are labeled with Cyrillic characters.
* **Edges:** The lines connecting the nodes represent the hierarchical relationship between the linguistic units.
### Detailed Analysis
**1. Ukrainian (uk\_tok)**
* **Top Node:** проми
* **Level 1:**
* про (left)
* ми (right)
* **Level 2 (from про):**
* п (left)
* ро (right)
* **Level 3 (from ро):**
* р (left)
* о (center)
* с (right)
* **Level 4:**
* лов
* **Level 5:**
* л (left)
* ов (right)
* **Level 6 (from ов):**
* о (left)
* в (right)
* **Level 7:**
* ість
* **Level 8:**
* і (left)
* сть (right)
* **Level 9 (from сть):**
* ст (left)
* ь (right)
* **Level 10 (from ст):**
* с (left)
* т (right)
**2. Belarusian (be\_tok)**
* **Top Node:** про
* **Level 1:**
* п (left)
* ро (right)
* **Level 2 (from ро):**
* р (left)
* о (right)
* **Level 3:**
* ми
* **Level 4:**
* м (left)
* и (right)
* **Level 5:**
* сло
* **Level 6:**
* с (left)
* ло (right)
* **Level 7:**
* ві
* **Level 8:**
* в (left)
* і (right)
* **Level 9:**
* сть
* **Level 10:**
* ст (left)
* ь (right)
* **Level 11 (from ст):**
* с (left)
* т (right)
**3. Russian (ru\_tok)**
* **Top Node:** про
* **Level 1:**
* п (left)
* ро (right)
* **Level 2 (from ро):**
* р (left)
* о (right)
* **Level 3:**
* ми
* **Level 4:**
* м (left)
* и (right)
* **Level 5:**
* с
* **Level 6:**
* лов
* **Level 7:**
* л (left)
* ов (right)
* **Level 8 (from ов):**
* о (left)
* в (right)
* **Level 9:**
* ість
* **Level 10:**
* сть
* **Level 11:**
* ст (left)
* ь (right)
* **Level 12 (from ст):**
* с (left)
* т (right)
### Key Observations
* All three diagrams use a tree structure to represent the decomposition of linguistic units.
* The Ukrainian diagram is the most complex, with the deepest level of decomposition.
* The Belarusian and Russian diagrams share some similarities in their structure, particularly in the lower levels.
* The diagrams use Cyrillic characters to label the nodes, indicating that they represent words, morphemes, or phonemes from the respective languages.
### Interpretation
The diagrams likely represent a linguistic analysis of a specific word or phrase in Ukrainian, Belarusian, and Russian. The tree structure illustrates how the word/phrase can be broken down into its constituent parts, potentially revealing its morphological or phonological structure. The differences in the diagrams suggest variations in how the word/phrase is analyzed or structured in each language. The diagrams could be used to compare and contrast the linguistic features of these three related languages.
</details>
Figure 4: Hierarchical BPE tokenization trees for the word “промисловiсть” (industry) across three East Slavic languages. The Ukrainian tokenization produces semantically consistent morphemes, while Belarusian and Russian models generate more fragmented subword units.
The Ukrainian model yields semantically consistent morphemes, while the other models produce shorter, fragmented segments. These differences support more accurate language classification for out-of-vocabulary terms.
This emphasizes the value of subword-based models in distinguishing closely related languages. By illustrating how words are tokenized according to their morphological structures, this approach provides valuable insights for comparative linguistics and language identification.
5 Discussion
5.1 What BPE Captures: Statistical Compression as Linguistic Approximation
Our evaluations reveal that BPE tokenization, despite being a purely statistical compression algorithm, incidentally captures linguistically meaningful structure. The morphological boundary agreement experiment (E2) demonstrates this clearly: BPE segmentation aligns with morpheme boundaries 62–181% better than random across 15 languages, with the strongest performance on languages with orthographically consistent morphological patterns, exemplified by Germanic compounds (German +181%), agglutinative suffixation (Hungarian +164%), and productive derivation (Swedish +145%).
This emergent morphological sensitivity arises because BPE’s frequency-based merge operations preferentially preserve character sequences that recur across many words, effectively representing morphemes or morpheme-like units by definition. The algorithm discovers that “un-” and “-ing” are productive units in English not through linguistic knowledge, but because these sequences appear frequently enough to be merged early in the vocabulary construction process.
However, our results also expose the limits of this approximation. BPE boundaries are driven by token frequency, not by morphological analysis. The E4b experiment provides direct evidence: when we classified homographs by etymology, we found that high-frequency Proto-Slavic words show less segmentation variation across languages (41.6% different) than lower-frequency borrowings (61.3% different). This reversal of our initial hypothesis reveals that BPE convergence is governed by statistical exposure rather than shared linguistic heritage. Regardless of their historical origin, high-frequency words accumulate sufficient evidence for BPE to learn consistent segmentation patterns across related languages; conversely, lower-frequency items exhibit greater variation due to data sparsity.
5.2 Phylogenetic Signal and Contact Effects
The significant correlation between BPE vocabulary similarity and genetic language relatedness (Mantel $r=0.329$ , $p<0.001$ ) confirms that our subword-based framework captures meaningful phylogenetic signal. However, the moderate strength of this correlation is itself informative: BPE similarity reflects lexical similarity, which combines genetic relatedness with contact-induced borrowing.
The per-family analysis reveals this distinction clearly. Romance languages form the tightest BPE cluster (mean distance 0.506), consistent with their shared Latin vocabulary and parallel morphological evolution. In contrast, Germanic languages show higher internal distance (0.713) despite comparable phylogenetic closeness. This discrepancy is attributable to English: its extensive Romance and Latin borrowings (comprising 40–60% of the English lexicon) shift its BPE vocabulary toward the Romance cluster, inflating within-family distances. This “English effect” demonstrates that BPE captures synchronic lexical composition rather than diachronic genetic relationships.
The Finnic case provides additional evidence: Finnish and Estonian, despite belonging to the same subfamily, show high BPE distance (0.785). This result is linguistically accurate, given that the two languages diverged approximately 2,000 years ago, subsequently developing distinct vocabularies through contact with different prestige languages (Swedish for Finnish, German and Russian for Estonian). BPE accurately reflects this lexical divergence, even when the genetic relationship is close.
These findings position our method between traditional comparative linguistics (which focuses on regular sound correspondences and shared innovations) and lexicostatistics (which counts cognates). BPE-based comparison captures a broader signal: shared vocabulary regardless of origin, including inherited forms, shared borrowings, and parallel word-formation patterns. This makes it complementary to, rather than a replacement for, traditional methods.
5.3 Implications for Low-Resource Language Technology
Our full-scale language identification experiment (E1) reveals a significant practical advantage of BPE-based approaches: coverage. When extended to 321 Latin-script languages, our unsupervised method achieves 44 $×$ improvement over random baseline without requiring any labeled training data. More importantly, it provides the only available language identification capability for 315 languages where supervised tools like fastText (Joulin et al. 2016) have zero coverage.
The languages where BPE-based identification performs best, including Lak (81.5%), Cree (80.6%), Inuktitut (63.8%), and Kabardian (60.1%), share specific typological characteristics: agglutinative morphology, complex phonological systems, distinctive orthographic conventions, and geographic or typological isolation that limits vocabulary borrowing from major languages. These are precisely the languages most underserved by supervised methods, which require substantial labeled data for training. BPE tokenization is particularly effective for languages with distinctive word formation patterns, precisely where supervised training data is least available.
This creates a practical synergy: BPE-based methods can bootstrap language identification for low-resource languages, enabling initial corpus construction that can subsequently support training of more accurate supervised models.
5.4 Cross-linguistic Homograph Discrimination
The E4 evaluation demonstrates that language-specific BPE tokenizers segment nearly half (48.7%) of shared orthographic forms differently across Slavic languages. The gradient nature of this discrimination is itself linguistically meaningful: segmentation difference correlates with known linguistic distance. Russian–Ukrainian pairs (both East Slavic) show only 31.2% different segmentation, while Belarusian–Macedonian pairs (East vs. South Slavic) show 61.9% different segmentation. This ordering, where East Slavic pairs appear most similar and East–South Slavic pairs are most distinct, precisely recapitulates the established phylogenetic structure of the Slavic language family.
The most striking illustrations come from proper names, which lack morphological motivation and thus reveal pure frequency-driven differences: the name “димитров” receives five completely different segmentations across five Slavic languages. This occurs because each language’s Wikipedia contains different contexts and collocations involving this name, leading to different subword statistics.
The 51.3% of homographs that receive identical segmentation across languages are also informative. These tend to be either international vocabulary (borrowings like “катастрофа”, segmented identically in all six languages) or core Slavic vocabulary with high cross-linguistic frequency. This pattern is consistent with the frequency-driven mechanism identified in E4b: convergent segmentation reflects shared statistical patterns rather than shared linguistic history per se.
5.5 Relationship to Existing Approaches
Our work extends the line of research connecting BPE compression to linguistic typology (Gutierrez-Vasques et al. 2023). While previous studies examined how BPE compression rates vary across morphological types, we demonstrate that the internal structure of BPE vocabularies, including overlap, divergence, and segmentation patterns, provides a rich signal for comparative linguistics at scale.
Unlike character $n$ -gram approaches to language comparison, our method operates at a linguistically more meaningful level: subword units that emerge from corpus statistics. Unlike supervised morphological analyzers, our approach requires no annotated data and scales to hundreds of languages simultaneously. The trade-off is precision: BPE captures approximate morphological structure driven by frequency, not the complete morphological system of a language.
Our phylogeny correlation results complement studies on language universals and typological diversity (Ponti et al. 2019). The moderate Mantel correlation ( $r=0.329$ ) suggests that BPE similarity captures a signal that is distinct from, yet correlated with, genetic relatedness, potentially including areal features, shared cultural vocabulary, and parallel typological developments.
5.6 Limitations of the Discussion
Several aspects of our results warrant caution. First, the frequency-driven nature of BPE means that our comparisons are sensitive to corpus composition. Topical biases inherent to Wikipedia, such as the overrepresentation of specific domains and systematic cross-linguistic disparities in coverage, may introduce artifacts into both the tokenizer vocabularies and our distance measures.
Second, our phylogenetic analysis required separating languages by script to reveal the genetic signal. In the mixed-script analysis, shared Latin-script function words (‘‘the’’, ‘‘de’’, ‘‘and’’) created artificial clustering by script rather than by relatedness. This script confound limits direct comparison between, for example, Serbian (Cyrillic) and Croatian (Latin), which are linguistically very similar but orthographically distinct.
Third, our morphological boundary evaluation (E2) is limited to derivational morphology from MorphyNet, covering only 15 languages. Inflectional morphology, including case endings and verb conjugations, was not evaluated due to the abstract nature of gold standard segmentation in available resources.
5.7 Future Directions
Several promising extensions emerge from our findings. First, the application of this framework to Common Crawl https://huggingface.co/commoncrawl would test whether the patterns we observe generalize beyond Wikipedia’s controlled environment. Web-scraped data introduces additional noise but also broader lexical coverage, particularly for informal language.
Second, our BPE-based distance measures could be compared against typological databases such as WALS (Dryer and Haspelmath 2013) and Grambank (Skirgård et al. 2023) to determine whether BPE similarity captures morphological typology (analytic vs. synthetic, fusional vs. agglutinative) in addition to lexical similarity.
Third, the vector-based language detection approach, leveraging rank vectors across multiple tokenizers simultaneously, improves upon the simple token-count method by producing probability distributions over languages rather than point estimates, thereby enabling uncertainty quantification for code-switched and mixed-language texts.
Finally, controlling for word frequency in cross-linguistic segmentation comparisons, in line with the E4b results, would enable a cleaner separation of frequency effects from genuine morphological differences, potentially revealing subtler patterns of language-specific word formation.
6 Conclusion
We outline a large-scale, script-focused comparative linguistic framework that leverages Wikipedia dumps spanning hundreds of languages using either the Latin or Cyrillic script. Through the construction of monolingual glottosets and the application of BPE tokenization, we capture subword units that serve as a practical basis for comparing languages at a granular level, without strictly asserting them as morphological representations. Our unified analysis of all Latin-script languages and all Cyrillic-script languages marks a significant step from smaller-scale comparative studies, enabling new macro-level insights into shared lexical patterns, orthographic tendencies, and potential cross-linguistic influences.
Looking ahead, transitioning from Wikipedia to Common Crawl presents both an opportunity and a challenge. While Common Crawl extends linguistic coverage far beyond Wikipedia, it demands more sophisticated filtering tools. Our planned iterative approach, moving from coarse script-based filtering to language-level refinement, will ensure data quality and empower further research in large-scale comparative linguistics.
7 Data Availability and Reproducibility
The reproducible code is available on our GitHub https://github.com/aglabx/morphoBPE with MIT license, and the extended datasets are hosted on Hugging Face https://huggingface.co/datasets/aglabx/wiki_glottosets with CC BY-SA license.
8 AI Models Usage
As non-native English speakers, we used Claude Opus 4.5 for text editing. GitHub Copilot assisted with code completion. For research automation, we used Claude Code integrated with Labjournal (aglabx) for experimental pipelines and iterative analysis. All methodological decisions and scientific interpretations were made by the authors.
References
- V. Akavarapu and A. Bhattacharya (2024) Automated cognate detection as a supervised link prediction task with cognate transformer. arXiv preprint arXiv:2402.02926. Cited by: §2.
- D. Akindotuni (2025) Resource asymmetry in multilingual nlp: a comprehensive review and critique. Journal of Computer and Communications 13, pp. 14–47. External Links: Document Cited by: §1.
- C. Arnett and B. K. Bergen (2024) Why do language models perform worse for morphologically complex languages?. External Links: 2411.14198, Link Cited by: §1.
- K. Batsuren, G. Bella, and F. Giunchiglia (2021) MorphyNet: a large multilingual database of derivational and inflectional morphology. In Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, G. Nicolai, K. Gorman, and R. Cotterell (Eds.), Online, pp. 39–48. External Links: Link, Document Cited by: §4.2.
- R. S. P. Beekes (2011) Comparative indo-european linguistics: an introduction. 2 edition, John Benjamins Publishing Company. External Links: Document, Link Cited by: §1, §1.
- E. M. Bender (2011) On achieving and evaluating language-independence in nlp. Linguistic Issues in Language Technology 6. External Links: Document, Link Cited by: §1, §2.
- F. Blum, L. Paschen, R. Forkel, S. Fuchs, and F. Seifart (2024) Consonant lengthening marks the beginning of words across a diverse sample of languages. Nature Human Behaviour 8 (11), pp. 2127–2138. External Links: Document Cited by: §1.
- K. Bostrom and G. Durrett (2020) Byte pair encoding is suboptimal for language model pretraining. External Links: 2004.03720, Link Cited by: §1.
- A. Bouchard-Côté, D. Hall, T. L. Griffiths, and D. Klein (2013) Automated reconstruction of ancient languages using probabilistic models of sound change. Proceedings of the National Academy of Sciences 110 (11), pp. 4224–4229. External Links: Document, Link, https://www.pnas.org/doi/pdf/10.1073/pnas.1204678110 Cited by: §2.
- L. Campbell (2020) An introductionAn introduction. Edinburgh University Press, Edinburgh. External Links: Link, Document, ISBN 9781474463133 Cited by: §1.
- C. Cathcart and G. Jäger (2024) Exploring the evolutionary dynamics of sound symbolism. Note: eScholarship, University of California External Links: Link Cited by: §1.
- I. Chelombitko, E. Chelombitko, and A. Komissarov (2025) SampoNLP: a self-referential toolkit for morphological analysis of subword tokenizers. In Proceedings of the 10th International Workshop on Computational Linguistics for Uralic Languages, M. Hämäläinen, M. Rießler, E. V. Morooka, and L. Kharlashkin (Eds.), Joensuu, Finland, pp. 57–67. External Links: Link, ISBN 979-8-89176-360-9 Cited by: §2.
- I. Chelombitko and A. Komissarov (2024) Specialized monolingual BPE tokenizers for Uralic languages representation in large language models. In Proceedings of the 9th International Workshop on Computational Linguistics for Uralic Languages, M. Hämäläinen, F. Pirinen, M. Macias, and M. Crespo Avila (Eds.), Helsinki, Finland, pp. 89–95. External Links: Link Cited by: §2.
- I. Chelombitko, E. Safronov, and A. Komissarov (2024) Qtok: a comprehensive framework for evaluating multilingual tokenizer quality in large language models. External Links: 2410.12989, Link Cited by: §1.
- R. Choenni and E. Shutova (2022) Investigating language relationships in multilingual sentence encoders through the lens of linguistic typology. Computational Linguistics 48 (3), pp. 635–672. Cited by: §2.
- A. M. Ciobanu and L. P. Dinu (2014) Automatic detection of cognates using orthographic alignment. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 99–105. External Links: Document, Link Cited by: §1, §2.
- A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov (2020) Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (Eds.), Online, pp. 8440–8451. External Links: Link, Document Cited by: §1.
- A. Ćwiek, S. Fuchs, C. Draxler, E. L. Asu, D. Dediu, K. Hiovain, S. Kawahara, et al. (2022) The bouba/kiki effect is robust across cultures and writing systems. Philosophical Transactions of the Royal Society B: Biological Sciences 377 (1841), pp. 20200390. External Links: Document Cited by: §1.
- J. Dang, S. Singh, D. D’souza, A. Ahmadian, A. Salamanca, M. Smith, A. Peppin, S. Hong, M. Govindassamy, T. Zhao, S. Kublik, M. Amer, V. Aryabumi, J. A. Campos, Y. Tan, T. Kocmi, F. Strub, N. Grinsztajn, Y. Flet-Berliac, A. Locatelli, H. Lin, D. Talupuru, B. Venkitesh, D. Cairuz, B. Yang, T. Chung, W. Ko, S. S. Shi, A. Shukayev, S. Bae, A. Piktus, R. Castagné, F. Cruz-Salinas, E. Kim, L. Crawhall-Stein, A. Morisot, S. Roy, P. Blunsom, I. Zhang, A. Gomez, N. Frosst, M. Fadaee, B. Ermis, A. Üstün, and S. Hooker (2024) Aya expanse: combining research breakthroughs for a new multilingual frontier. External Links: 2412.04261, Link Cited by: §1.
- P. T. Daniels and W. Bright (Eds.) (1996) The world’s writing systems. Oxford University Press. Note: Reprinted 2007 Cited by: §1.
- M. S. Dryer and M. Haspelmath (Eds.) (2013) The world atlas of language structures online. Max Planck Institute for Evolutionary Anthropology, Leipzig. External Links: Link Cited by: §5.7.
- M. Dunn et al. (2011) Evolved structure of language shows lineage-specific trends in word-order universals. Nature 473 (7345), pp. 79–82. External Links: Document, Link Cited by: §1.
- R. Futrell et al. (2015) Large-scale evidence of dependency length minimization in 37 languages. Proceedings of the National Academy of Sciences 112 (33), pp. 10336–10341. External Links: Document, Link Cited by: §1.
- P. Gage (1994) A new algorithm for data compression. C Users J. 12 (2), pp. 23–38. External Links: ISSN 0898-9788 Cited by: §1.
- X. Gutierrez-Vasques, C. Bentz, and T. Samardžić (2023) Languages through the looking glass of BPE compression. Computational Linguistics 49 (4), pp. 943–1001. External Links: Document, Link Cited by: §2, §5.5.
- M. Hämäläinen and J. Reuter (2019) Finding sami cognates with a character-based nmt approach. In Proceedings of the Workshop on Computational Methods for Endangered Languages, Vol. 1. Cited by: §2.
- M. Hämäläinen (2021) Endangered languages are not low-resourced!. In Multilingual Facilitation, Cited by: §1.
- H. Hammarström and R. Forkel (2022) Glottocodes: identifiers linking families, languages and dialects to comprehensive reference information. Semantic Web Journal 13 (6), pp. 917–924. External Links: Link, Document Cited by: §4.3.
- A. Imani, P. Lin, A. H. Kargaran, S. Severini, M. Jalili Sabet, N. Kassner, C. Ma, H. Schmid, A. Martins, F. Yvon, and H. Schütze (2023) Glot500: scaling multilingual corpora and language models to 500 languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.), Toronto, Canada, pp. 1082–1117. External Links: Link, Document Cited by: §1, §2.
- G. Jäger (2018) Global-scale phylogenetic linguistic inference from lexical resources. Scientific Data 5. External Links: Link Cited by: §2.
- M. Johnson et al. (2016) Google’s multilingual neural machine translation system: enabling zero-shot translation. Note: arXiv preprint External Links: 1611.04558, Link Cited by: §1.
- A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov (2016) Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759. Cited by: §5.3.
- S. Khurana et al. (2024) Improved cross-lingual transfer learning for automatic speech translation. Note: arXiv preprint External Links: 2306.00789, Link Cited by: §1.
- W. P. Lehmann (2013) Historical linguistics: an introduction. 3rd edition, Routledge. External Links: Document, Link Cited by: §1.
- A. Meillet (1967) The comparative method in historical linguistics. Librairie Honoré Champion. Note: Originally published 1925 by Instituttet for sammenlignende kulturforskning, Oslo External Links: Link Cited by: §1.
- J. Nichols (1992) Linguistic diversity in space and time. External Links: Link Cited by: §1, §2.
- A. Oncevay, D. Ataman, N. Van Berkel, B. Haddow, A. Birch, and J. Bjerva (2022) Quantifying synthesis and fusion and their impact on machine translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, M. Carpuat, M. de Marneffe, and I. V. Meza Ruiz (Eds.), Seattle, United States, pp. 1308–1321. External Links: Link, Document Cited by: §2.
- N. Partanen, J. Rueter, K. Alnajjar, and M. Hämäläinen (2021) Processing ma castrén’s materials: multilingual historical typed and handwritten manuscripts. In Proceedings of the Workshop on Natural Language Processing for Digital Humanities, pp. 47–54. Cited by: §1, §2.
- T. Pham, K. Le, and A. T. Luu (2024) UniBridge: a unified approach to cross-lingual transfer learning for low-resource languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 3168–3184. External Links: Link, Document Cited by: §1.
- A. Poncelas, M. Popovic, D. Shterionov, G. M. de Buy Wenniger, and A. Way (2019) Combining smt and nmt back-translated data for efficient nmt. External Links: 1909.03750, Link Cited by: §2.
- E. M. Ponti, H. O’Horan, Y. Berzak, I. Vulić, R. Reichart, T. Poibeau, E. Shutova, and A. Korhonen (2019) Modeling language variation and universals: a survey on typological linguistics for natural language processing. Computational Linguistics 45 (3), pp. 559–601. External Links: Document, Link Cited by: §2, §5.5.
- T. Rama, J. List, J. Wahle, and G. Jäger (2018) Are automatic methods for cognate detection good enough for phylogenetic reconstruction in historical linguistics?. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 393–400. Cited by: §1, §2.
- N. Rathi, M. Hahn, and R. Futrell (2021) An information-theoretic characterization of morphological fusion. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, M. Moens, X. Huang, L. Specia, and S. W. Yih (Eds.), Online and Punta Cana, Dominican Republic, pp. 10115–10120. External Links: Link, Document Cited by: §2.
- P. Rust, J. Pfeiffer, I. Vulić, S. Ruder, and I. Gurevych (2021) How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), C. Zong, F. Xia, W. Li, and R. Navigli (Eds.), Online, pp. 3118–3135. External Links: Link, Document Cited by: §1.
- T. Säily, E. Mäkelä, and M. Hämäläinen (2021) From plenipotentiary to puddingless: users and uses of new words in early english letters. In Multilingual Facilitation, pp. 153–169. Cited by: §1.
- R. Sennrich et al. (2016) Neural machine translation of rare words with subword units. Note: arXiv preprint External Links: 1508.07909, Link Cited by: §1, §1.
- C. Shani, Y. Reif, N. Roll, D. Jurafsky, and E. Shutova (2026) The roots of performance disparity in multilingual language models: intrinsic modeling difficulty or design choices?. External Links: 2601.07220, Link Cited by: §1.
- H. Skirgård, H. J. Haynie, D. E. Blasi, H. Hammarström, J. Collins, J. J. Latber, J. Lesage, T. Weber, A. Witzlack-Makarevich, et al. (2023) Grambank reveals the importance of genealogical constraints on linguistic diversity and highlights the impact of language loss. Science Advances 9 (16), pp. eadg6175. External Links: Document Cited by: §5.7.
- R. Sproat (2016) Language typology in speech and language technology. Linguistic Typology 20 (3), pp. 635–644. Cited by: §1, §2.
- A. Tjuka, R. Forkel, and J. List (2024) Universal and cultural factors shape body part vocabularies. Scientific Reports 14 (1), pp. 10486. External Links: Document Cited by: §1.
- D. Yu, T. He, and K. Sagae (2021) Language embeddings for typology and cross-lingual transfer learning. arXiv preprint arXiv:2106.02082. Cited by: §2.