# Subword-Based Comparative Linguistics across 242 Languages Using Wikipedia Glottosets
**Authors**: Iaroslav Chelombitko, Mika Hämäläinen, Aleksey Komissarov
> DataSpike Neapolis University Pafos, Paphos, Cyprus Metropolia University of Applied Sciences, Helsinki, Finland
> Metropolia University of Applied Sciences, Helsinki, Finland
> Neapolis University Pafos, Paphos, Cyprus aglabx, Paphos, Cyprus
## Abstract
We present a large-scale comparative study of 242 Latin and Cyrillic-script languages using subword-based methodologies. By constructing ’glottosets’ from Wikipedia lexicons, we introduce a framework for simultaneous cross-linguistic comparison via Byte-Pair Encoding (BPE). Our approach utilizes rank-based subword vectors to analyze vocabulary overlap, lexical divergence, and language similarity at scale. Evaluations demonstrate that BPE segmentation aligns with morpheme boundaries 95% better than random baseline across 15 languages (F1 = 0.34 vs 0.15). BPE vocabulary similarity correlates significantly with genetic language relatedness (Mantel r = 0.329, p < 0.001), with Romance languages forming the tightest cluster (mean distance 0.51) and cross-family pairs showing clear separation (0.82). Analysis of 26,939 cross-linguistic homographs reveals that 48.7% receive different segmentations across related languages, with variation correlating to phylogenetic distance. Our results provide quantitative macro-linguistic insights into lexical patterns across typologically diverse languages within a unified analytical framework.
Keywords Multilingual NLP $\cdot$ Comparative Linguistics $\cdot$ BPE Tokenization $\cdot$ Cross-linguistic Analysis $\cdot$ Lexical Similarity $\cdot$ Script-based Typology
## 1 Introduction
Traditional comparative linguistics, while providing deep historical and typological insights (see Lehmann 2013; Beekes 2011; Nichols 1992; Partanen et al. 2021; Säily et al. 2021), often lacks the scalability to handle the current volume of digital text (Arnett and Bergen 2024; Akindotuni 2025). However, recent NLP advances facilitate massive, data-driven studies (see Dang et al. 2024; Bender 2011; Imani et al. 2023; Sproat 2016), revealing universal tendencies in phonetics (Blum et al. 2024), lexical semantics (Tjuka et al. 2024), and sound symbolism (Ćwiek et al. 2022; Cathcart and Jäger 2024) that were previously inaccessible through manual, small-scale methods.
Despite this progress, large-scale studies frequently neglect endangered languages (Hämäläinen 2021) and systematic performance disparities in multilingual models (see Shani et al. 2026; Chelombitko et al. 2024; Dunn and others 2011). We address this gap by examining 242 languages through a novel script-level lens (Latin vs. Cyrillic), moving beyond family-specific comparisons (Meillet 1967; Beekes 2011) to reveal overarching patterns visible only when languages share a script-based writing system (Daniels and Bright 1996; Rust et al. 2021).
In particular, we adopt a subword-based strategy (Sennrich and others 2016) using Byte-Pair Encoding (BPE) (Gage 1994) tokenizers, which we train both on individual languages and on large aggregated corpora for each script. Aggregating data for all Latin and Cyrillic-script languages into respective training sets effectively unifies each script community into a single model, leveraging the shared-parameter paradigm common in massively multilingual pretraining (see Johnson and others 2016; Conneau et al. 2020).
<details>
<summary>workflow.png Details</summary>
### Visual Description
## Flowchart: Multilingual Language Processing Pipeline
### Overview
The diagram illustrates a multistage pipeline for processing linguistic data across 320 languages, starting with raw Wikipedia dumps and culminating in monolingual glottosets. The flowchart uses color-coded nodes connected by bidirectional arrows to represent data flow and interdependencies between processing stages.
### Components/Axes
1. **Nodes** (color-coded):
- **Wikipedia Dumps** (blue): 320 languages in ZIM format
- **BPE Tokenization** (pink): Training individual & combined tokenizers
- **Script-Based Filtering** (green): Selecting Cyrillic vs. Latin scripts (242 total)
- **Monolingual Glottosets** (yellow): Extracting words, TF/DF, paragraphs
- **Subword-Based Analysis** (purple): Language comparisons using rank-based vectors
- **Macro-Level Insights** (orange): Script-level comparisons & patterns
2. **Connections**:
- Bidirectional arrows indicate data flow between stages
- Primary flow direction: Left-to-right (top-left to bottom-right)
- Feedback loops between tokenization and analysis stages
### Detailed Analysis
1. **Data Flow**:
- **Wikipedia Dumps** → **BPE Tokenization** (direct input)
- **Wikipedia Dumps** → **Script-Based Filtering** (parallel processing)
- **BPE Tokenization** → **Macro-Level Insights** (script comparisons)
- **BPE Tokenization** → **Subword-Based Analysis** (language comparisons)
- **Script-Based Filtering** → **Monolingual Glottosets** (filtered output)
- **Subword-Based Analysis** → **Monolingual Glottosets** (vector-based inputs)
2. **Key Data Points**:
- 320 languages processed from initial dumps
- 242 languages survive script filtering (Cyrillic vs. Latin)
- Dual-path processing: Tokenization and script filtering both feed into glottoset creation
- Subword analysis provides comparative vectors for language relationships
### Key Observations
1. **Bidirectional Flow**: Arrows between BPE Tokenization and Subword Analysis suggest iterative refinement
2. **Script Filtering Bottleneck**: 242 languages (75.6% of original) survive filtering, indicating significant script-based reduction
3. **Convergent Output**: All paths ultimately feed into Monolingual Glottosets, emphasizing its role as final processing stage
4. **Color Coding**: Distinct colors for each node type enhance visual separation of processing stages
### Interpretation
This pipeline demonstrates a hierarchical approach to multilingual NLP processing:
1. **Data Preparation**: Raw Wikipedia dumps (320 languages) require preprocessing through BPE tokenization and script filtering
2. **Parallel Processing**: Tokenization and script filtering operate concurrently but feed into different analysis paths
3. **Comparative Analysis**: Subword-based methods enable language comparisons through rank-based vectors
4. **Script-Level Insights**: Macro-level analysis focuses on script-specific patterns, suggesting potential for cross-linguistic pattern discovery
5. **Final Output**: Monolingual glottosets represent the distilled output containing extracted linguistic features (words, TF/DF statistics, paragraphs)
The bidirectional arrows between tokenization and analysis stages imply an iterative refinement process, where analysis results may inform tokenizer improvements. The script filtering stage acts as a critical quality control checkpoint, reducing the dataset size while maintaining linguistic diversity through Cyrillic/Latin selection.
</details>
Figure 1: Pipeline architecture for subword-based comparative linguistics across 242 languages using Wikipedia glottosets. The workflow illustrates the transformation of Wikipedia dumps (320 languages) through sequential stages: script-based filtering yielding 37 Cyrillic and 205 Latin script languages, monolingual glottoset construction with TF/DF metrics, BPE tokenization (both individual and combined training with 4096 tokens vocabulary), and vector-based subword analysis. Each colored node represents a distinct processing phase, culminating in macro-level insights for script-level comparative linguistics. This modular approach enables scalable analysis of morphological patterns across multiple languages simultaneously.
Although subword units are not perfect morphemes (see Sennrich and others 2016; Bostrom and Durrett 2020), they serve as robust automated tools for macro-linguistic research (see Khurana and others 2024; Pham et al. 2024; Futrell and others 2015) that scale more efficiently than manual expert alignment (see Campbell 2020; Rama et al. 2018; Ciobanu and Dinu 2014).
Leveraging the framework (Figure 1), this work makes two key contributions:
1. Script-Level Aggregation: We demonstrate how combining languages by script (Latin vs. Cyrillic) enables macro-level comparative insights difficult to achieve through traditional language-by-language lenses.
1. Practical Subword Methodology: We introduce a tractable, automated framework that reduces reliance on manual annotation by providing data-driven lexical segments for robust cross-linguistic analysis.
While traditional comparative linguistics focuses on genetic relationships and historical reconstruction, and typology examines structural similarities regardless of ancestry, our subword-based approach bridges both: we demonstrate that BPE tokenization captures phylogenetic signal (Section 4.3) while also revealing typological patterns across unrelated languages sharing the same script.
## 2 Related Work
Comparative linguistics in NLP has evolved from general data-driven frameworks (see Bender 2011; Sproat 2016; Imani et al. 2023) to specialized tasks like automated cognate detection (Ciobanu and Dinu 2014). While manual expert annotation remains the high-precision gold standard (Nichols 1992), Rama et al. (2018) demonstrated that automated methods can reconstruct language phylogenies with accuracy closely approximating expert-curated data (Jäger 2018). These approaches highlight the potential for scaling linguistic analysis across diverse language families where manual examination is impractical.
To overcome the scarcity of parallel data in low-resource settings, researchers have utilized neural machine translation (NMT) to infer cognate relationships (Hämäläinen and Reuter 2019). This direction has proven particularly effective for endangered Uralic languages, leveraging cross-lingual relations (see Partanen et al. 2021; Chelombitko and Komissarov 2024; Chelombitko et al. 2025) and data augmentation through synthetic cognates generated by statistical machine translation (SMT) (Poncelas et al. 2019). Such hybrid strategies address resource limitations while enriching existing linguistic databases for under-represented script communities.
More recently, architectures inspired by computational biology have addressed previous computational bottlenecks. Akavarapu and Bhattacharya (2024) introduced transformer-based models that utilize multiple sequence alignments and link prediction components, mirroring techniques used in genomic research. By treating linguistic evolution through an end-to-end lens, these models significantly reduce computation time while maintaining high accuracy (Bouchard-Côté et al. 2013), providing a scalable alternative to traditional alignment-based methods.
Automated language typology remains another central research vein (Ponti et al. 2019), with studies using information-theoretic measures to quantify morphological synthesis and fusion (see Rathi et al. 2021; Oncevay et al. 2022). Crucially, Gutierrez-Vasques et al. (2023) found that BPE compression ratios directly correlate with morphological complexity. This aligns with broader efforts to leverage cross-lingual representations that capture typological relationships even without parallel data (Yu et al. 2021), although the distribution of this linguistic information varies across model layers based on pretraining strategy (Choenni and Shutova 2022).
## 3 Methodology
### 3.1 Monolingual Glottosets Construction
We downloaded Wikipedia dumps in ZIM format for 320 languages available in the ZIM archive https://dumps.wikimedia.org/kiwix/zim/wikipedia/. After that, we selected either the 2024 or 2023 version (preferring the 2024 version if available), otherwise, we used the 2023 version. We included all topics by selecting the “all” files option. To clean the data from unnecessary HTML elements, we used a custom script available in our repository. In brief, this script resolves redirects and extracts only paragraph text while removing as many auxiliary HTML tags as possible, leaving only clean paragraph texts.
<details>
<summary>graph.png Details</summary>
### Visual Description
## Network Diagram: Multilingual Word Reuse Relationships
### Overview
The image contains two interconnected network diagrams showing word reuse relationships in Ukrainian and English. Nodes represent words with reuse counts, while arrows indicate connections between them. The diagrams use color coding (blue circle for central node "a") and include both Cyrillic and Latin script labels.
### Components/Axes
**Left Diagram (Ukrainian):**
- Nodes:
- чi (reuse: 17)
- щiв
- очi
- твор
- тнoй
- нoой
- гайм
- шiр
- видцi
- йного
- стих
- україн
- Connections:
- чi → щiв, очi, твор, тнoй, нoой, гайм, шiр, видцi, йного, стих, україн
- щiв → твор, тнoй, нoой, гайм, шiр, видцi, йного, стих
- очi → твор, тнoй, нoой, гайм, шiр, видцi, йного, стих
- твор → тнoй, нoой, гайм, шiр, видцi, йного, стих
- тнoй → нoой, гайм, шiр, видцi, йного, стих
- нoой → гайм, шiр, видцi, йного, стих
- гайм → шiр, видцi, йного, стих
- шiр → видцi, йного, стих
- видцi → йного, стих
- йного → стих
- стих → україн
**Right Diagram (English):**
- Nodes:
- a (reuse: 167)
- ja (reuse: 55)
- ohju
- uja
- idae
- eja
- star
- taide
- vuude
- okuvi
- ista (reuse: 23)
- Connections:
- a → ja
- ja → ohju, uja, idae, eja, star
- star → taide, vuude, okuvi, ista
- taide → vuude
- vuude → okuvi
- okuvi → ista
### Detailed Analysis
**Left Diagram (Ukrainian):**
- Highest reuse: чi (17)
- Central node: твор (connected to 7 other nodes)
- Terminal node: україн (connected only to стих)
- Reuse pattern: Decreases from central nodes to peripheral nodes
**Right Diagram (English):**
- Highest reuse: a (167)
- Secondary reuse: ja (55), ista (23)
- Central node: star (connected to 4 nodes)
- Terminal node: ista (connected only to okuvi)
- Reuse pattern: Concentrated around central nodes with decreasing reuse
### Key Observations
1. **Multilingual Connection**: The word "star" connects Ukrainian and English networks
2. **Reuse Concentration**:
- Ukrainian: 68% of reuse in top 3 nodes (чi, твор, очi)
- English: 85% of reuse in top 3 nodes (a, ja, ista)
3. **Connection Density**:
- Ukrainian network has 21 connections
- English network has 12 connections
4. **Color Coding**: Blue circle highlights "a" as central node in English network
### Interpretation
The diagrams suggest a comparative analysis of word reuse patterns between Ukrainian and English texts. The Ukrainian network shows more distributed reuse with multiple central nodes, while the English network has a single dominant node ("a"). The connection between "star" in both networks indicates potential cross-linguistic relationships. The reuse counts suggest "a" is the most frequently reused word in the English corpus analyzed, while "чi" holds similar prominence in Ukrainian texts. The network structures imply different semantic clustering patterns between the languages, with Ukrainian showing more interconnected relationships and English demonstrating a more hierarchical structure.
</details>
Figure 2: Visualization of interactive BPE merge graphs for Ukrainian (left) and Finnish (right) subword tokenization patterns. The diagrams show directed merge sequences with reuse counts indicated in parentheses. Vertical black bars represent merge steps, while edges show the progression of subword unit formation. The contrasting patterns reflect language-specific morphological characteristics: Ukrainian showing consistent Cyrillic character combinations, while Finnish exhibits agglutinative patterns. Interactive web application available on our repository.
The extracted paragraphs were filtered based on script consistency. For Cyrillic, we retained only Cyrillic paragraphs, and for Latin, only those written in the Latin script. Unfortunately, there is currently no reliable tool for accurately detecting all 242 languages, so at this stage, we filtered only by script (Table 1). Additionally, an effective filtering step was to remove all paragraphs containing fewer than 10 words. This significantly reduced noise in the dataset, including accidental words from other languages.
Table 1: Statistics for top 30 languages by token count from Wikipedia dumps. Columns show language names and codes, number of raw and cleaned paragraphs, word count in millions, script type (Latin/Cyrillic), total tokens, and cross-script contamination (Cyrillic tokens in Latin-script languages and vice versa).
| English | en | 54125720 | 42149266 | 2661.93 | Latin | 2661930466 | 5062 | – |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| German | de | 22354272 | 17953642 | 1026.43 | Latin | 1026430544 | 2648 | – |
| French | fr | 20464072 | 16742203 | 940.75 | Latin | 940746868 | 1621 | – |
| Spanish | es | 14107859 | 12170134 | 763.36 | Latin | 763357750 | 1134 | – |
| Italian | it | 11917473 | 9806341 | 571.76 | Latin | 571759224 | 1150 | – |
| Russian | ru | 15290408 | 12295838 | 552.42 | Cyrillic | 552416889 | – | 429004 |
| Cebuano | ceb | 16841937 | 10069736 | 447.85 | Latin | 447850263 | 92 | – |
| Portuguese | pt | 6845897 | 5642228 | 332.96 | Latin | 332959017 | 413 | – |
| Dutch | nl | 7764590 | 6429095 | 316.21 | Latin | 316210100 | 415 | – |
| Polish | pl | 15961541 | 5616831 | 253.85 | Latin | 253846410 | 2044 | – |
| Catalan | ca | 4273906 | 3749516 | 238.81 | Latin | 238813199 | 710 | – |
| Ukrainian | uk | 7864058 | 5582991 | 219.83 | Cyrillic | 219827684 | – | 200945 |
| Vietnamese | vi | 4707518 | 3129629 | 201.38 | Latin | 201383401 | 623 | – |
| Swedish | sv | 7437772 | 5088656 | 200.89 | Latin | 200889695 | 312 | – |
| Serbo-Croatian | sh | 2248318 | 1682017 | 193.56 | Latin | 193561280 | 19539 | – |
| Czech | cs | 3386803 | 2865556 | 150.39 | Latin | 150393063 | 998 | – |
| Hungarian | hu | 3932082 | 2603906 | 129.86 | Latin | 129864098 | 518 | – |
| Indonesian | id | 2984643 | 2120278 | 106.98 | Latin | 106980811 | 622 | – |
| Norwegian Bokmål | nb | 2906506 | 2239883 | 105.51 | Latin | 105507751 | 175 | – |
| Finnish | fi | 3141975 | 2295657 | 93.5 | Latin | 93502432 | 650 | – |
| Serbian | sr | 3210579 | 1981413 | 91.81 | Cyrillic | 91807037 | – | 193621 |
| Turkish | tr | 2528498 | 1757628 | 85.16 | Latin | 85156035 | 570 | – |
| Romanian | ro | 2360641 | 1539046 | 84.62 | Latin | 84617897 | 454 | – |
| Bulgarian | bg | 1599763 | 1386917 | 69.12 | Cyrillic | 69123830 | – | 36265 |
| Waray | war | 2571970 | 1448198 | 69.04 | Latin | 69038926 | 0 | – |
| Danish | da | 1551416 | 1216805 | 61.11 | Latin | 61107267 | 126 | – |
| Galician | gl | 1188136 | 1049404 | 60.95 | Latin | 60947590 | 181 | – |
| Malay | ms | 1499560 | 1034055 | 56.37 | Latin | 56370979 | 106 | – |
| Asturian | ast | 1134182 | 912479 | 56.25 | Latin | 56254683 | 84 | – |
| Esperanto | eo | 1669507 | 1188077 | 54.51 | Latin | 54512106 | 441 | – |
In the following analysis we included only languages with Cyrillic (37 languages) and Latin (205 languages) scripts. The decision to restrict the scope to Latin and Cyrillic script languages was motivated by three primary factors: (1) these scripts share a common alphabet origin, enabling direct character-level comparison; (2) they represent the largest script families in Wikipedia coverage; and (3) they include both closely related language families (e.g., Romance, Slavic) and typologically diverse languages (e.g., Finnish, Turkish, Vietnamese), providing a rich testbed for subword-based comparative analysis. Additionally, as an unexpected result, we were able to measure the extent to which Wikipedia articles are contaminated with Latin and Cyrillic scripts for languages that use other scripts (80 languages). The results are presented in Table 2.
Table 2: Script contamination analysis for languages using non-Latin/non-Cyrillic writing systems. Data shows language names, codes, raw paragraph counts, native script types, and the number of detected Latin and Cyrillic tokens representing potential script contamination.
| Min Nan Chinese | nan | 830227 | Han | 13 | 541415 |
| --- | --- | --- | --- | --- | --- |
| Japanese | ja | 12737874 | Japanese | 915 | 298430 |
| Chinese | zh | 7206035 | Han | 554 | 117304 |
| Greek | el | 1733250 | Greek | 363 | 96009 |
| Hebrew | he | 3368324 | Hebrew | 479 | 90907 |
| Korean | ko | 3333575 | Hangul | 635 | 81612 |
| Arabic | ar | 4624040 | Arabic | 246 | 58738 |
| Persian | fa | 3132901 | Arabic | 165 | 51729 |
| Balinese | ban | 87175 | Balinese | 4 | 48922 |
| Thai | th | 944565 | Thai | 305 | 43559 |
| Armenian | hy | 1647293 | Armenian | 2579 | 41259 |
| Egyptian Arabic | arz | 5166127 | Arabic | 23 | 39349 |
| Buginese | bug | 36776 | Lontara | 0 | 19834 |
| Min Dong Chinese | cdo | 28589 | Han | 0 | 19197 |
| Georgian | ka | 644781 | Georgian | 387 | 15234 |
| Bengali | bn | 854059 | Bengali | 64 | 13786 |
| Hindi | hi | 703141 | Devanagari | 80 | 13369 |
| Tamil | ta | 820809 | Tamil | 28 | 11153 |
| Hakka Chinese | hak | 14204 | Han | 2 | 10368 |
| Malayalam | ml | 464433 | Malayalam | 31 | 9848 |
| Sinhalese | si | 166475 | Sinhala | 3 | 9552 |
| Burmese | my | 451126 | Burmese | 8 | 9467 |
| Urdu | ur | 660706 | Arabic | 62 | 9172 |
| Cantonese | yue | 373575 | Han | 53 | 8418 |
| South Azerbaijani | azb | 605161 | Arabic | 18 | 7459 |
| Goan Konkani | gom | 39569 | Devanagari | 0 | 6955 |
| Marathi | mr | 349639 | Devanagari | 10 | 6842 |
| Cambodian | km | 99641 | Khmer | 13 | 6324 |
| Telugu | te | 832028 | Telugu | 3 | 6170 |
| Tachelhit | shi | 7811 | Tifinagh | 0 | 5799 |
We define monolingual glottosets as language-specific collections of lexical units derived from monolingual texts. Each glottoset reflects a particular language’s vocabulary in one script, in our case Latin or Cyrillic in lowercase. Unfortunately, while we initially aimed to avoid normalizing words to lowercase, we found that doing so significantly improves subsequent tokenization. The glottoset includes additional features: Term Frequency (TF) and Document Frequency (DF). DF is defined as the occurrence of a word within a Wikipedia paragraph, as the Wikipedia parsing process is based on paragraph-level extraction. Wikipedia based glottosets characteristics are presented in Table 3.
Table 3: Lexical statistics for a diverse set of languages using Latin and Cyrillic scripts. The table presents vocabulary size (ranging from 1,447 for Cheyenne to 3,207,272 for Hungarian), lexical diversity scores (from 0.009 in Dutch to 0.383 in Cheyenne), median word length (6–10 characters), and top-3 most frequent tokens.
| Swiss German | Latin | 565258 | 0.070 | 10 | d, isch, dr |
| --- | --- | --- | --- | --- | --- |
| Tajik | Cyrillic | 317780 | 0.041 | 8 | дар, аз, ки |
| Hungarian | Latin | 3207272 | 0.025 | 10 | a, és, az |
| Lingala | Latin | 23153 | 0.123 | 7 | ya, na, ezalí |
| Zeeuws | Latin | 51177 | 0.087 | 8 | de, n, t |
| Lak | Cyrillic | 6890 | 0.315 | 7 | ва, шагьру, инсан |
| Malay | Latin | 637993 | 0.011 | 8 | yang, di, dan |
| Cheyenne | Latin | 1447 | 0.383 | 6 | ho, e, vé |
| Aymara | Latin | 29323 | 0.171 | 8 | a, jisk, suyu |
| Friulian | Latin | 44905 | 0.086 | 7 | di, e, al |
| Javanese | Latin | 281161 | 0.040 | 8 | ing, lan, iku |
| Rusyn | Cyrillic | 112582 | 0.149 | 8 | в, на, є |
| Ido | Latin | 129897 | 0.029 | 7 | la, di, e |
| Norman | Latin | 31509 | 0.106 | 7 | d, est, la |
| Ligurian | Latin | 133733 | 0.090 | 7 | l, a, de |
| Karakalpak | Latin | 104775 | 0.144 | 8 | hám, menen, bul |
| Romanian | Latin | 1102965 | 0.013 | 8 | în, de, şi |
| Somali | Latin | 118361 | 0.076 | 8 | oo, iyo, ka |
| Lombard | Latin | 239355 | 0.042 | 7 | l, è, de |
| Norwegian Bokmål | Latin | 1918724 | 0.018 | 10 | i, og, en |
| West Flemish | Latin | 96384 | 0.079 | 8 | de, van, in |
| Māori | Latin | 15232 | 0.033 | 7 | te, ko, o |
| Icelandic | Latin | 451176 | 0.051 | 10 | í, og, á |
| Mongolian | Cyrillic | 235807 | 0.044 | 8 | нь, оны, юм |
| Dutch | Latin | 2866741 | 0.009 | 10 | de, van, in |
| Kabyle | Latin | 52338 | 0.106 | 7 | n, d, deg |
| Tsonga | Latin | 14810 | 0.116 | 8 | hi, na, ya |
| Madurese | Latin | 33922 | 0.155 | 7 | bân, è, sè |
### 3.2 Glottoset BPE Tokenization
We implemented our own version of a BPE tokenizer that does not treat spaces as separate tokens and exclusively tokenizes words. The tokenizer is available on our GitHub https://github.com/aglabx/morphoBPE. After that, we tokenized all words from each glottoset. We used two sets of parameters. In the first variant, we employed a vocabulary size of 4096 tokens. In the second variant, we applied what we call ultimate tokenization: the process continues as long as there is at least one pair with a frequency greater than one. This second approach is highly dependent on the corpus size. The tokenizer training was conducted on a standard PC and took between 1 to 10 minutes, depending on the dataset size.
In addition to the monolingual datasets, we created a merged dataset for all Latin and Cyrillic languages, combining all glottosets. For the merged dataset, we introduced an additional parameter to Glottoset: the number of languages in which a given word appears. Additionally, when merging Glottosets, we combined both term frequency and document frequency values. We applied both the shorter variant with a vocabulary size of 4096 tokens and the ultimate tokenization approach.
### 3.3 Vector Representation of BPE Tokens by Language
Using the tokenizer trained on all languages, we obtained BPE tokens. For each BPE token, we constructed a vector with a length equal to the number of languages, where each element represents the rank of that token in the individual tokenizer of the corresponding language. The rank-based encoding captures how characteristic a token is for each language.
Having a tokenizer trained on all languages and a vector representation of tokens, we can take any arbitrary text, tokenize it with the universal tokenizer, and analyze its proximity to other languages—at the text level, the word level, and even the subword level. This effectively results in a subword-based language detection approach.
### 3.4 Hierarchical Subword Tokenization Analysis
The next concept we propose is that if we take a word from a language for which a language-specific tokenizer exists, we can construct a hierarchical tree representing how this word is tokenized into subword units in one or more languages. This approach has an indirect connection to morphological analysis.
Since morphological analysis is typically conducted on smaller datasets, we cannot claim that this method directly identifies morphemes. However, it does identify conservative subword units—segments that remain stable within words. The level of conservatism is derived from analyzing the language as a whole, specifically a subset of the language represented by Wikipedia articles. We have added this functionality to our BPE tokenizer.
### 3.5 Comparative Monolingual Tokenizer Analysis
Beyond comparing how tokenizers handle words, another idea naturally arises: what if we compare tokenizers themselves? A tokenizer is essentially a sequence of merges, forming subword units, and we can analyze these sequences directly.
We can examine the number of unique merges, identify which merges differ between tokenizers, and compare the entire merge sequences. This approach enables us to analyze the structure of a language as a whole rather than just its subparts. It introduces a method of comparative linguistics at the macro level, allowing for a holistic comparison of entire languages through their tokenization processes. An example of an interactive visualization for the monolingual BPE tokenizer is shown in Figure 2.
## 4 Evaluation
To validate our subword-based comparative linguistics framework, we designed four quantitative evaluations testing specific hypotheses about BPE tokenization behavior across languages.
### 4.1 Research Questions
We address the following research questions:
1. RQ1 (Morphological Grounding): Do BPE segmentation boundaries align with linguistically meaningful morpheme boundaries?
1. RQ2 (Phylogenetic Signal): Does BPE vocabulary similarity correlate with genetic language relatedness?
1. RQ3 (Language Discrimination): Can BPE tokenizers discriminate between languages that share orthographic forms (homographs)?
### 4.2 E2: Morphological Boundary Agreement
Hypothesis: BPE segmentation boundaries align with morpheme boundaries better than random segmentation.
Method: We used MorphyNet (Batsuren et al. 2021) derivational morphology data for 15 languages. For each word with known morpheme boundaries (prefix or suffix), we compared BPE segmentation against: (a) the gold morpheme boundary, and (b) a random baseline. We computed Boundary Precision, Recall, and F1 scores.
Results: All 15 languages showed BPE segmentation significantly better than random baseline (Table 4). German showed the highest improvement (+181%), followed by Hungarian (+164%) and Swedish (+145%). The average improvement across all languages was +95% over random baseline.
Table 4: E2: Morphological boundary agreement. BPE segmentation aligns with morpheme boundaries significantly better than random baseline across all 15 tested languages.
| German Hungarian Swedish | 0.42 0.39 0.37 | 0.15 0.15 0.15 | +181% +164% +145% |
| --- | --- | --- | --- |
| English | 0.36 | 0.15 | +140% |
| Finnish | 0.35 | 0.15 | +133% |
| Russian | 0.31 | 0.15 | +107% |
| Average | 0.34 | 0.15 | +95% |
Conclusion: Hypothesis supported. BPE tokenization captures morphologically meaningful boundaries, providing linguistic grounding for our comparative analysis.
### 4.3 E3: Language Phylogeny Correlation
Hypothesis: BPE vocabulary similarity correlates with genetic language relatedness (phylogeny).
Method: We computed pairwise BPE distance ( $1-\text{Jaccard similarity}$ ) between language vocabularies for 49 Latin-script languages. We compared this distance matrix against phylogenetic distance derived from Glottolog (Hammarström and Forkel 2022) classification (Family $\to$ Subfamily $\to$ Branch). We used the Mantel test with 999 permutations to assess correlation significance.
Results: The Mantel test revealed a significant positive correlation between BPE distance and phylogenetic distance:
- Mantel $r=0.329$ ( $p<0.001$ , $z=12.3$ )
- Within-family BPE distance: 0.67 (mean)
- Between-family BPE distance: 0.82 (mean)
- Separation ratio: 1.22 $\times$ ( $t$ -test $p<10^{-13}$ )
Romance languages showed the tightest clustering (mean distance 0.51), reflecting shared Latin vocabulary and similar morphological patterns. Germanic languages showed higher internal distance (0.71), likely due to English’s extensive Romance/Latin borrowings.
Conclusion: Hypothesis supported. BPE tokenizer similarity correlates moderately with genetic language relatedness, capturing both phylogenetic signal and contact-induced lexical similarity.
### 4.4 E4: Cross-lingual Homograph Discrimination
Hypothesis: BPE tokenizers segment identical orthographic forms (homographs) differently across languages, reflecting language-specific morphological patterns.
Method: For 6 Cyrillic Slavic languages (Ukrainian, Russian, Belarusian, Bulgarian, Macedonian, Serbian), we:
1. Extracted word vocabularies from TF-DF files (frequency $\geq$ 100)
1. Identified homographs: words appearing in 2+ language vocabularies
1. Tokenized each homograph with each language’s BPE tokenizer
1. Compared segmentation patterns across languages
Results: We found 26,939 homographs across the 6 languages:
- 48.7% showed different segmentation across languages
- 51.3% showed identical segmentation
Segmentation difference correlated with linguistic distance:
- Russian-Ukrainian (both East Slavic): 31.2% different
- Belarusian-Macedonian (East vs. South): 61.9% different
A striking example is the name “димитров”, which received 5 different segmentations across 5 languages:
- Ukrainian: ди|ми|т|ров
- Russian: ди|мит|ров
- Bulgarian: дими|т|ров
- Macedonian: димит|ров
- Serbian: дим|ит|ров
Conclusion: Hypothesis partially supported. Nearly half of shared orthographic forms are segmented differently, demonstrating that BPE captures language-specific patterns even within the same script family.
### 4.5 Qualitative Analysis
Beyond quantitative evaluation, our approach effectively captures the subword characteristics of cross-linguistic homonyms, defined as words with identical spellings but distinct meanings across languages. For example, the word “заказала” carries different meanings in Russian and Ukrainian, as shown in their morphemic tokenizations (Figure 3):
- Ukrainian: за – ка – зал – а
- Russian: зака – зал – а
<details>
<summary>uk_r_zakazala.png Details</summary>
### Visual Description
## Tree Diagram: Ukrainian vs. Russian Word Decomposition
### Overview
The image compares hierarchical decomposition structures for the word "зак" (Ukrainian: "tok," Russian: "tok") across three levels. Each language uses distinct alphabetic systems (Latin for Ukrainian, Cyrillic for Russian) and numerical annotations to represent phonetic or morphological breakdowns.
### Components/Axes
- **Titles**:
- Left: "Ukrainian (uk_tok)"
- Right: "Russian (ru_tok)"
- **Nodes**:
- **Ukrainian (Latin)**:
- Level 1: `3a` (root)
- Level 2: `ка` (ka)
- Level 3: `зал` (zal)
- **Russian (Cyrillic)**:
- Level 1: `зака` (3a)
- Level 2: `за` (za) and `ка` (ka)
- Level 3: `зал` (zal)
- **Connectors**: Lines represent hierarchical relationships between nodes.
### Detailed Analysis
#### Ukrainian (uk_tok)
1. **Root**: `3a` (3 + a)
2. **Second Level**: `ка` (ka)
3. **Third Level**: `зал` (zal)
- Subcomponents: `a` and `л` (l)
#### Russian (ru_tok)
1. **Root**: `зака` (3a)
2. **Second Level**:
- Left: `за` (za)
- Right: `ка` (ka)
3. **Third Level**: `зал` (zal)
- Subcomponents: `a` and `л` (l)
### Key Observations
1. **Structural Differences**:
- Ukrainian decomposition is linear (single branch per level).
- Russian decomposition splits into two branches at Level 2.
2. **Numerical Annotations**:
- `3` appears in both languages, possibly indicating syllable count or stress patterns.
3. **Alphabetic Systems**:
- Ukrainian uses Latin letters with diacritics (e.g., `3a`, `зал`).
- Russian uses Cyrillic letters (e.g., `зака`, `зал`).
### Interpretation
The diagrams likely illustrate phonetic or morphological segmentation of the word "зак" (token/word) in Ukrainian and Russian. The numerical `3` may denote syllable count (e.g., "за-ка" in Russian, "зак" in Ukrainian). The Cyrillic structure for Russian shows a bifurcation at Level 2, suggesting a split into consonant-vowel components (`за` + `ка`), while Ukrainian maintains a linear progression. This reflects differences in linguistic analysis frameworks between the two languages.
### Notable Patterns
- Both languages converge on `зал` (zal) at Level 3, indicating shared phonetic elements.
- Russian decomposition emphasizes consonant-vowel separation, whereas Ukrainian prioritizes linear progression.
- The use of `3` in both systems suggests a standardized metric (e.g., syllable count) for decomposition.
</details>
Figure 3: Hierarchical BPE tokenization trees comparing the word заказала in Ukrainian (left) and Russian (right). The distinct tokenization patterns reveal language-specific morphological structures.
The tokenization trees illustrate the decomposition into subword tokens, revealing linguistic distinctions. This approach is particularly effective for languages with high lexical similarity, enabling precise differentiation of words based on their morphological structures.
### 4.6 Subword-Based Language Identification for Out-of-Vocabulary Words
Our subword-based analysis also aids naive language identification, particularly for multi-morpheme words. Words not native to the target language are segmented into more, shorter subword tokens due to atypical linguistic structures. Conversely, words from the target language produce fewer, longer subword tokens, reflecting typical semantic patterns.
For example, the Ukrainian word “промисловiсть” (industry) is analyzed across Ukrainian, Belarusian, and Russian tokenization models (Figure 4):
- Ukrainian: проми-с-лов-iсть
- Belarusian: про-ми-сло-вi-сть
- Russian: про-ми-с-лов-i-сть
<details>
<summary>uk_be_r_promyslovist.png Details</summary>
### Visual Description
## Diagram: Comparative Linguistic Tree Structures of Ukrainian, Belarusian, and Russian
### Overview
The image presents three comparative linguistic tree diagrams representing the morphological or syntactic structures of Ukrainian (uk_tok), Belarusian (be_tok), and Russian (ru_tok). Each tree uses Cyrillic script to label nodes, with hierarchical branching patterns illustrating linguistic relationships. The diagrams are positioned side-by-side, with Ukrainian on the left, Belarusian in the center, and Russian on the right.
### Components/Axes
- **Root Nodes**:
- Ukrainian: "промі" (promi)
- Belarusian: "про" (pro)
- Russian: "про" (pro)
- **Branching Structure**:
- Each tree splits into sub-branches labeled with Cyrillic letters (e.g., "п", "р", "о", "М", "И", "С", "Т", "B", "В").
- Terminal nodes include combinations like "CT", "CTb", "CTb", and "CTb" (with variations in subscripts).
- **Positioning**:
- Trees are vertically aligned, with roots at the top and terminal nodes at the bottom.
- No explicit legend or color coding; all elements use black lines and nodes.
### Detailed Analysis
#### Ukrainian (uk_tok)
- **Root**: "промі" (promi) → branches into "про" (pro) and "МИ" (mi).
- **Sub-branches**:
- "про" → "п" (p) and "ро" (ro).
- "МИ" → "р" (r) and "о" (o).
- Further splits include "с" (s) under "ро" and "CTb" (C-T-b) under "і" (i).
#### Belarusian (be_tok)
- **Root**: "про" (pro) → branches into "п" (p) and "ро" (ro).
- **Sub-branches**:
- "п" → "М" (M) and "и" (i).
- "ро" → "C" (C) and "ПО" (po).
- Terminal nodes include "CT" (C-T) and "B" (b).
#### Russian (ru_tok)
- **Root**: "про" (pro) → branches into "п" (p) and "ро" (ro).
- **Sub-branches**:
- "п" → "М" (M) and "и" (i).
- "ро" → "C" (C) and "С" (s).
- Terminal nodes include "CTb" (C-T-b) and "B" (b).
### Key Observations
1. **Shared Roots**: Belarusian and Russian share the root "про" (pro), while Ukrainian uses "промі" (promi), suggesting divergent lexical origins.
2. **Branching Patterns**:
- Ukrainian trees exhibit more complex sub-branching (e.g., "CTb" with subscripts).
- Belarusian and Russian trees show simpler, more linear splits.
3. **Terminal Nodes**: All trees converge on terminal nodes labeled with combinations of Cyrillic letters (e.g., "CT", "B", "V"), indicating shared morphological components.
### Interpretation
The diagrams highlight structural similarities and differences in the three Slavic languages:
- **Linguistic Relationships**: Shared roots ("про") and terminal nodes ("CT", "B") suggest common etymological roots, while divergent root structures ("промі" vs. "про") reflect historical linguistic evolution.
- **Morphological Complexity**: Ukrainian's more intricate branching may indicate richer inflectional morphology compared to Belarusian and Russian.
- **Phonetic Patterns**: Repeated use of "C" (ts) and "T" (t) in terminal nodes points to shared consonant clusters in Slavic languages.
The trees likely represent computational models of word segmentation or morphological analysis, illustrating how words decompose into phonetic or syntactic units across languages. The absence of numerical data suggests a focus on structural relationships rather than quantitative analysis.
</details>
Figure 4: Hierarchical BPE tokenization trees for the word “промисловiсть” (industry) across three East Slavic languages. The Ukrainian tokenization produces semantically consistent morphemes, while Belarusian and Russian models generate more fragmented subword units.
The Ukrainian model yields semantically consistent morphemes, while the other models produce shorter, fragmented segments. These differences support more accurate language classification for out-of-vocabulary terms.
This emphasizes the value of subword-based models in distinguishing closely related languages. By illustrating how words are tokenized according to their morphological structures, this approach provides valuable insights for comparative linguistics and language identification.
## 5 Discussion
### 5.1 What BPE Captures: Statistical Compression as Linguistic Approximation
Our evaluations reveal that BPE tokenization, despite being a purely statistical compression algorithm, incidentally captures linguistically meaningful structure. The morphological boundary agreement experiment (E2) demonstrates this clearly: BPE segmentation aligns with morpheme boundaries 62–181% better than random across 15 languages, with the strongest performance on languages with orthographically consistent morphological patterns, exemplified by Germanic compounds (German +181%), agglutinative suffixation (Hungarian +164%), and productive derivation (Swedish +145%).
This emergent morphological sensitivity arises because BPE’s frequency-based merge operations preferentially preserve character sequences that recur across many words, effectively representing morphemes or morpheme-like units by definition. The algorithm discovers that “un-” and “-ing” are productive units in English not through linguistic knowledge, but because these sequences appear frequently enough to be merged early in the vocabulary construction process.
However, our results also expose the limits of this approximation. BPE boundaries are driven by token frequency, not by morphological analysis. The E4b experiment provides direct evidence: when we classified homographs by etymology, we found that high-frequency Proto-Slavic words show less segmentation variation across languages (41.6% different) than lower-frequency borrowings (61.3% different). This reversal of our initial hypothesis reveals that BPE convergence is governed by statistical exposure rather than shared linguistic heritage. Regardless of their historical origin, high-frequency words accumulate sufficient evidence for BPE to learn consistent segmentation patterns across related languages; conversely, lower-frequency items exhibit greater variation due to data sparsity.
### 5.2 Phylogenetic Signal and Contact Effects
The significant correlation between BPE vocabulary similarity and genetic language relatedness (Mantel $r=0.329$ , $p<0.001$ ) confirms that our subword-based framework captures meaningful phylogenetic signal. However, the moderate strength of this correlation is itself informative: BPE similarity reflects lexical similarity, which combines genetic relatedness with contact-induced borrowing.
The per-family analysis reveals this distinction clearly. Romance languages form the tightest BPE cluster (mean distance 0.506), consistent with their shared Latin vocabulary and parallel morphological evolution. In contrast, Germanic languages show higher internal distance (0.713) despite comparable phylogenetic closeness. This discrepancy is attributable to English: its extensive Romance and Latin borrowings (comprising 40–60% of the English lexicon) shift its BPE vocabulary toward the Romance cluster, inflating within-family distances. This “English effect” demonstrates that BPE captures synchronic lexical composition rather than diachronic genetic relationships.
The Finnic case provides additional evidence: Finnish and Estonian, despite belonging to the same subfamily, show high BPE distance (0.785). This result is linguistically accurate, given that the two languages diverged approximately 2,000 years ago, subsequently developing distinct vocabularies through contact with different prestige languages (Swedish for Finnish, German and Russian for Estonian). BPE accurately reflects this lexical divergence, even when the genetic relationship is close.
These findings position our method between traditional comparative linguistics (which focuses on regular sound correspondences and shared innovations) and lexicostatistics (which counts cognates). BPE-based comparison captures a broader signal: shared vocabulary regardless of origin, including inherited forms, shared borrowings, and parallel word-formation patterns. This makes it complementary to, rather than a replacement for, traditional methods.
### 5.3 Implications for Low-Resource Language Technology
Our full-scale language identification experiment (E1) reveals a significant practical advantage of BPE-based approaches: coverage. When extended to 321 Latin-script languages, our unsupervised method achieves 44 $\times$ improvement over random baseline without requiring any labeled training data. More importantly, it provides the only available language identification capability for 315 languages where supervised tools like fastText (Joulin et al. 2016) have zero coverage.
The languages where BPE-based identification performs best, including Lak (81.5%), Cree (80.6%), Inuktitut (63.8%), and Kabardian (60.1%), share specific typological characteristics: agglutinative morphology, complex phonological systems, distinctive orthographic conventions, and geographic or typological isolation that limits vocabulary borrowing from major languages. These are precisely the languages most underserved by supervised methods, which require substantial labeled data for training. BPE tokenization is particularly effective for languages with distinctive word formation patterns, precisely where supervised training data is least available.
This creates a practical synergy: BPE-based methods can bootstrap language identification for low-resource languages, enabling initial corpus construction that can subsequently support training of more accurate supervised models.
### 5.4 Cross-linguistic Homograph Discrimination
The E4 evaluation demonstrates that language-specific BPE tokenizers segment nearly half (48.7%) of shared orthographic forms differently across Slavic languages. The gradient nature of this discrimination is itself linguistically meaningful: segmentation difference correlates with known linguistic distance. Russian–Ukrainian pairs (both East Slavic) show only 31.2% different segmentation, while Belarusian–Macedonian pairs (East vs. South Slavic) show 61.9% different segmentation. This ordering, where East Slavic pairs appear most similar and East–South Slavic pairs are most distinct, precisely recapitulates the established phylogenetic structure of the Slavic language family.
The most striking illustrations come from proper names, which lack morphological motivation and thus reveal pure frequency-driven differences: the name “димитров” receives five completely different segmentations across five Slavic languages. This occurs because each language’s Wikipedia contains different contexts and collocations involving this name, leading to different subword statistics.
The 51.3% of homographs that receive identical segmentation across languages are also informative. These tend to be either international vocabulary (borrowings like “катастрофа”, segmented identically in all six languages) or core Slavic vocabulary with high cross-linguistic frequency. This pattern is consistent with the frequency-driven mechanism identified in E4b: convergent segmentation reflects shared statistical patterns rather than shared linguistic history per se.
### 5.5 Relationship to Existing Approaches
Our work extends the line of research connecting BPE compression to linguistic typology (Gutierrez-Vasques et al. 2023). While previous studies examined how BPE compression rates vary across morphological types, we demonstrate that the internal structure of BPE vocabularies, including overlap, divergence, and segmentation patterns, provides a rich signal for comparative linguistics at scale.
Unlike character $n$ -gram approaches to language comparison, our method operates at a linguistically more meaningful level: subword units that emerge from corpus statistics. Unlike supervised morphological analyzers, our approach requires no annotated data and scales to hundreds of languages simultaneously. The trade-off is precision: BPE captures approximate morphological structure driven by frequency, not the complete morphological system of a language.
Our phylogeny correlation results complement studies on language universals and typological diversity (Ponti et al. 2019). The moderate Mantel correlation ( $r=0.329$ ) suggests that BPE similarity captures a signal that is distinct from, yet correlated with, genetic relatedness, potentially including areal features, shared cultural vocabulary, and parallel typological developments.
### 5.6 Limitations of the Discussion
Several aspects of our results warrant caution. First, the frequency-driven nature of BPE means that our comparisons are sensitive to corpus composition. Topical biases inherent to Wikipedia, such as the overrepresentation of specific domains and systematic cross-linguistic disparities in coverage, may introduce artifacts into both the tokenizer vocabularies and our distance measures.
Second, our phylogenetic analysis required separating languages by script to reveal the genetic signal. In the mixed-script analysis, shared Latin-script function words (‘‘the’’, ‘‘de’’, ‘‘and’’) created artificial clustering by script rather than by relatedness. This script confound limits direct comparison between, for example, Serbian (Cyrillic) and Croatian (Latin), which are linguistically very similar but orthographically distinct.
Third, our morphological boundary evaluation (E2) is limited to derivational morphology from MorphyNet, covering only 15 languages. Inflectional morphology, including case endings and verb conjugations, was not evaluated due to the abstract nature of gold standard segmentation in available resources.
### 5.7 Future Directions
Several promising extensions emerge from our findings. First, the application of this framework to Common Crawl https://huggingface.co/commoncrawl would test whether the patterns we observe generalize beyond Wikipedia’s controlled environment. Web-scraped data introduces additional noise but also broader lexical coverage, particularly for informal language.
Second, our BPE-based distance measures could be compared against typological databases such as WALS (Dryer and Haspelmath 2013) and Grambank (Skirgård et al. 2023) to determine whether BPE similarity captures morphological typology (analytic vs. synthetic, fusional vs. agglutinative) in addition to lexical similarity.
Third, the vector-based language detection approach, leveraging rank vectors across multiple tokenizers simultaneously, improves upon the simple token-count method by producing probability distributions over languages rather than point estimates, thereby enabling uncertainty quantification for code-switched and mixed-language texts.
Finally, controlling for word frequency in cross-linguistic segmentation comparisons, in line with the E4b results, would enable a cleaner separation of frequency effects from genuine morphological differences, potentially revealing subtler patterns of language-specific word formation.
## 6 Conclusion
We outline a large-scale, script-focused comparative linguistic framework that leverages Wikipedia dumps spanning hundreds of languages using either the Latin or Cyrillic script. Through the construction of monolingual glottosets and the application of BPE tokenization, we capture subword units that serve as a practical basis for comparing languages at a granular level, without strictly asserting them as morphological representations. Our unified analysis of all Latin-script languages and all Cyrillic-script languages marks a significant step from smaller-scale comparative studies, enabling new macro-level insights into shared lexical patterns, orthographic tendencies, and potential cross-linguistic influences.
Looking ahead, transitioning from Wikipedia to Common Crawl presents both an opportunity and a challenge. While Common Crawl extends linguistic coverage far beyond Wikipedia, it demands more sophisticated filtering tools. Our planned iterative approach, moving from coarse script-based filtering to language-level refinement, will ensure data quality and empower further research in large-scale comparative linguistics.
## 7 Data Availability and Reproducibility
The reproducible code is available on our GitHub https://github.com/aglabx/morphoBPE with MIT license, and the extended datasets are hosted on Hugging Face https://huggingface.co/datasets/aglabx/wiki_glottosets with CC BY-SA license.
## 8 AI Models Usage
As non-native English speakers, we used Claude Opus 4.5 for text editing. GitHub Copilot assisted with code completion. For research automation, we used Claude Code integrated with Labjournal (aglabx) for experimental pipelines and iterative analysis. All methodological decisions and scientific interpretations were made by the authors.
## References
- V. Akavarapu and A. Bhattacharya (2024) Automated cognate detection as a supervised link prediction task with cognate transformer. arXiv preprint arXiv:2402.02926. Cited by: §2.
- D. Akindotuni (2025) Resource asymmetry in multilingual nlp: a comprehensive review and critique. Journal of Computer and Communications 13, pp. 14–47. External Links: Document Cited by: §1.
- C. Arnett and B. K. Bergen (2024) Why do language models perform worse for morphologically complex languages?. External Links: 2411.14198, Link Cited by: §1.
- K. Batsuren, G. Bella, and F. Giunchiglia (2021) MorphyNet: a large multilingual database of derivational and inflectional morphology. In Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, G. Nicolai, K. Gorman, and R. Cotterell (Eds.), Online, pp. 39–48. External Links: Link, Document Cited by: §4.2.
- R. S. P. Beekes (2011) Comparative indo-european linguistics: an introduction. 2 edition, John Benjamins Publishing Company. External Links: Document, Link Cited by: §1, §1.
- E. M. Bender (2011) On achieving and evaluating language-independence in nlp. Linguistic Issues in Language Technology 6. External Links: Document, Link Cited by: §1, §2.
- F. Blum, L. Paschen, R. Forkel, S. Fuchs, and F. Seifart (2024) Consonant lengthening marks the beginning of words across a diverse sample of languages. Nature Human Behaviour 8 (11), pp. 2127–2138. External Links: Document Cited by: §1.
- K. Bostrom and G. Durrett (2020) Byte pair encoding is suboptimal for language model pretraining. External Links: 2004.03720, Link Cited by: §1.
- A. Bouchard-Côté, D. Hall, T. L. Griffiths, and D. Klein (2013) Automated reconstruction of ancient languages using probabilistic models of sound change. Proceedings of the National Academy of Sciences 110 (11), pp. 4224–4229. External Links: Document, Link, https://www.pnas.org/doi/pdf/10.1073/pnas.1204678110 Cited by: §2.
- L. Campbell (2020) An introductionAn introduction. Edinburgh University Press, Edinburgh. External Links: Link, Document, ISBN 9781474463133 Cited by: §1.
- C. Cathcart and G. Jäger (2024) Exploring the evolutionary dynamics of sound symbolism. Note: eScholarship, University of California External Links: Link Cited by: §1.
- I. Chelombitko, E. Chelombitko, and A. Komissarov (2025) SampoNLP: a self-referential toolkit for morphological analysis of subword tokenizers. In Proceedings of the 10th International Workshop on Computational Linguistics for Uralic Languages, M. Hämäläinen, M. Rießler, E. V. Morooka, and L. Kharlashkin (Eds.), Joensuu, Finland, pp. 57–67. External Links: Link, ISBN 979-8-89176-360-9 Cited by: §2.
- I. Chelombitko and A. Komissarov (2024) Specialized monolingual BPE tokenizers for Uralic languages representation in large language models. In Proceedings of the 9th International Workshop on Computational Linguistics for Uralic Languages, M. Hämäläinen, F. Pirinen, M. Macias, and M. Crespo Avila (Eds.), Helsinki, Finland, pp. 89–95. External Links: Link Cited by: §2.
- I. Chelombitko, E. Safronov, and A. Komissarov (2024) Qtok: a comprehensive framework for evaluating multilingual tokenizer quality in large language models. External Links: 2410.12989, Link Cited by: §1.
- R. Choenni and E. Shutova (2022) Investigating language relationships in multilingual sentence encoders through the lens of linguistic typology. Computational Linguistics 48 (3), pp. 635–672. Cited by: §2.
- A. M. Ciobanu and L. P. Dinu (2014) Automatic detection of cognates using orthographic alignment. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 99–105. External Links: Document, Link Cited by: §1, §2.
- A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov (2020) Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (Eds.), Online, pp. 8440–8451. External Links: Link, Document Cited by: §1.
- A. Ćwiek, S. Fuchs, C. Draxler, E. L. Asu, D. Dediu, K. Hiovain, S. Kawahara, et al. (2022) The bouba/kiki effect is robust across cultures and writing systems. Philosophical Transactions of the Royal Society B: Biological Sciences 377 (1841), pp. 20200390. External Links: Document Cited by: §1.
- J. Dang, S. Singh, D. D’souza, A. Ahmadian, A. Salamanca, M. Smith, A. Peppin, S. Hong, M. Govindassamy, T. Zhao, S. Kublik, M. Amer, V. Aryabumi, J. A. Campos, Y. Tan, T. Kocmi, F. Strub, N. Grinsztajn, Y. Flet-Berliac, A. Locatelli, H. Lin, D. Talupuru, B. Venkitesh, D. Cairuz, B. Yang, T. Chung, W. Ko, S. S. Shi, A. Shukayev, S. Bae, A. Piktus, R. Castagné, F. Cruz-Salinas, E. Kim, L. Crawhall-Stein, A. Morisot, S. Roy, P. Blunsom, I. Zhang, A. Gomez, N. Frosst, M. Fadaee, B. Ermis, A. Üstün, and S. Hooker (2024) Aya expanse: combining research breakthroughs for a new multilingual frontier. External Links: 2412.04261, Link Cited by: §1.
- P. T. Daniels and W. Bright (Eds.) (1996) The world’s writing systems. Oxford University Press. Note: Reprinted 2007 Cited by: §1.
- M. S. Dryer and M. Haspelmath (Eds.) (2013) The world atlas of language structures online. Max Planck Institute for Evolutionary Anthropology, Leipzig. External Links: Link Cited by: §5.7.
- M. Dunn et al. (2011) Evolved structure of language shows lineage-specific trends in word-order universals. Nature 473 (7345), pp. 79–82. External Links: Document, Link Cited by: §1.
- R. Futrell et al. (2015) Large-scale evidence of dependency length minimization in 37 languages. Proceedings of the National Academy of Sciences 112 (33), pp. 10336–10341. External Links: Document, Link Cited by: §1.
- P. Gage (1994) A new algorithm for data compression. C Users J. 12 (2), pp. 23–38. External Links: ISSN 0898-9788 Cited by: §1.
- X. Gutierrez-Vasques, C. Bentz, and T. Samardžić (2023) Languages through the looking glass of BPE compression. Computational Linguistics 49 (4), pp. 943–1001. External Links: Document, Link Cited by: §2, §5.5.
- M. Hämäläinen and J. Reuter (2019) Finding sami cognates with a character-based nmt approach. In Proceedings of the Workshop on Computational Methods for Endangered Languages, Vol. 1. Cited by: §2.
- M. Hämäläinen (2021) Endangered languages are not low-resourced!. In Multilingual Facilitation, Cited by: §1.
- H. Hammarström and R. Forkel (2022) Glottocodes: identifiers linking families, languages and dialects to comprehensive reference information. Semantic Web Journal 13 (6), pp. 917–924. External Links: Link, Document Cited by: §4.3.
- A. Imani, P. Lin, A. H. Kargaran, S. Severini, M. Jalili Sabet, N. Kassner, C. Ma, H. Schmid, A. Martins, F. Yvon, and H. Schütze (2023) Glot500: scaling multilingual corpora and language models to 500 languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.), Toronto, Canada, pp. 1082–1117. External Links: Link, Document Cited by: §1, §2.
- G. Jäger (2018) Global-scale phylogenetic linguistic inference from lexical resources. Scientific Data 5. External Links: Link Cited by: §2.
- M. Johnson et al. (2016) Google’s multilingual neural machine translation system: enabling zero-shot translation. Note: arXiv preprint External Links: 1611.04558, Link Cited by: §1.
- A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov (2016) Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759. Cited by: §5.3.
- S. Khurana et al. (2024) Improved cross-lingual transfer learning for automatic speech translation. Note: arXiv preprint External Links: 2306.00789, Link Cited by: §1.
- W. P. Lehmann (2013) Historical linguistics: an introduction. 3rd edition, Routledge. External Links: Document, Link Cited by: §1.
- A. Meillet (1967) The comparative method in historical linguistics. Librairie Honoré Champion. Note: Originally published 1925 by Instituttet for sammenlignende kulturforskning, Oslo External Links: Link Cited by: §1.
- J. Nichols (1992) Linguistic diversity in space and time. External Links: Link Cited by: §1, §2.
- A. Oncevay, D. Ataman, N. Van Berkel, B. Haddow, A. Birch, and J. Bjerva (2022) Quantifying synthesis and fusion and their impact on machine translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, M. Carpuat, M. de Marneffe, and I. V. Meza Ruiz (Eds.), Seattle, United States, pp. 1308–1321. External Links: Link, Document Cited by: §2.
- N. Partanen, J. Rueter, K. Alnajjar, and M. Hämäläinen (2021) Processing ma castrén’s materials: multilingual historical typed and handwritten manuscripts. In Proceedings of the Workshop on Natural Language Processing for Digital Humanities, pp. 47–54. Cited by: §1, §2.
- T. Pham, K. Le, and A. T. Luu (2024) UniBridge: a unified approach to cross-lingual transfer learning for low-resource languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 3168–3184. External Links: Link, Document Cited by: §1.
- A. Poncelas, M. Popovic, D. Shterionov, G. M. de Buy Wenniger, and A. Way (2019) Combining smt and nmt back-translated data for efficient nmt. External Links: 1909.03750, Link Cited by: §2.
- E. M. Ponti, H. O’Horan, Y. Berzak, I. Vulić, R. Reichart, T. Poibeau, E. Shutova, and A. Korhonen (2019) Modeling language variation and universals: a survey on typological linguistics for natural language processing. Computational Linguistics 45 (3), pp. 559–601. External Links: Document, Link Cited by: §2, §5.5.
- T. Rama, J. List, J. Wahle, and G. Jäger (2018) Are automatic methods for cognate detection good enough for phylogenetic reconstruction in historical linguistics?. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 393–400. Cited by: §1, §2.
- N. Rathi, M. Hahn, and R. Futrell (2021) An information-theoretic characterization of morphological fusion. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, M. Moens, X. Huang, L. Specia, and S. W. Yih (Eds.), Online and Punta Cana, Dominican Republic, pp. 10115–10120. External Links: Link, Document Cited by: §2.
- P. Rust, J. Pfeiffer, I. Vulić, S. Ruder, and I. Gurevych (2021) How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), C. Zong, F. Xia, W. Li, and R. Navigli (Eds.), Online, pp. 3118–3135. External Links: Link, Document Cited by: §1.
- T. Säily, E. Mäkelä, and M. Hämäläinen (2021) From plenipotentiary to puddingless: users and uses of new words in early english letters. In Multilingual Facilitation, pp. 153–169. Cited by: §1.
- R. Sennrich et al. (2016) Neural machine translation of rare words with subword units. Note: arXiv preprint External Links: 1508.07909, Link Cited by: §1, §1.
- C. Shani, Y. Reif, N. Roll, D. Jurafsky, and E. Shutova (2026) The roots of performance disparity in multilingual language models: intrinsic modeling difficulty or design choices?. External Links: 2601.07220, Link Cited by: §1.
- H. Skirgård, H. J. Haynie, D. E. Blasi, H. Hammarström, J. Collins, J. J. Latber, J. Lesage, T. Weber, A. Witzlack-Makarevich, et al. (2023) Grambank reveals the importance of genealogical constraints on linguistic diversity and highlights the impact of language loss. Science Advances 9 (16), pp. eadg6175. External Links: Document Cited by: §5.7.
- R. Sproat (2016) Language typology in speech and language technology. Linguistic Typology 20 (3), pp. 635–644. Cited by: §1, §2.
- A. Tjuka, R. Forkel, and J. List (2024) Universal and cultural factors shape body part vocabularies. Scientific Reports 14 (1), pp. 10486. External Links: Document Cited by: §1.
- D. Yu, T. He, and K. Sagae (2021) Language embeddings for typology and cross-lingual transfer learning. arXiv preprint arXiv:2106.02082. Cited by: §2.