Subword Tokenization for Low- and Medium-Resource Languages: A Systematic Evaluation

1 Introduction

Subword tokenization addresses the challenge of handling words that a model has never seen before. Unlike wordlevel tokenizers, which treat each word as an indivisible unit, subword tokenization breaks unfamiliar words into smaller, known components. This approach is particularly beneficial for morphologically rich languages, where corpora often con- A key advantage of subword tokenization is its ability to maintain a fixed-size vocabulary while effectively representing an unlimited number of possible words. By decomposing rare or unknown words into familiar subwords, this method significantly mitigates the data sparsity issue inherent in traditional word-level tokenization. Models learn more efficiently because they encounter each subword more frequently, reducing, if not eliminating, out-of-vocabulary (OOV) words. These benefits have made subword tokenization a standard technique in pre-trained language models (LMs) [Sennrich et al., 2016; Kudo, 2018; Devlin et al., 2019].

The vocabulary size, specified during training, directly influences how the tokenizer segments words. A smaller vocabulary reduces memory requirements, but results in more aggressive word splitting and longer token sequences. Conversely, a larger vocabulary preserves more complete words, but requires more memory. Table 1 illustrates this trade-off using an Icelandic news headline processed with WordPiece tokenizers [Wu et al., 2016] trained on the Icelandic Gigaword Corpus (IGC) [Steingrímsson et al., 2018] at different vocabulary sizes.

Although subword tokenizers aim to minimize the number of splits, they may still encounter OOV words that must be decomposed into the smallest available subwords. This could involve breaking unknown words into individual characters. However, since vocabulary size is limited, not all Unicode characters can be included. When encountering sequences of unseen characters that cannot be broken down into smaller subwords, some tokenizers might replace them with a special unknown token, significantly limiting the model's ability to capture semantic information. This is especially common for scripts that are absent or underrepresented in the training data. For example, Unicode defines over 100,000 Chinese characters, several thousand of which see regular use. Including all of them in a tokenizer's vocabulary would be extremely inefficient for models not primarily designed to process Chinese text. As a result, many such characters are replaced by the unknown token.

The need for the unknown token can be avoided altogether by representing the minimal vocabulary using individual bytes rather than characters. Any Unicode character can be encoded as a sequence of one to four UTF-8 bytes, each taking one of 256 possible values, forming the smallest possible vocabulary. This approach, popularized by RoBERTa [Liu et al., 2019], has since been widely adopted by models such as GPT-2 and its successors [Radford et al., 2019; Brown et al., 2020; OpenAI, 2024].

In resource-constrained settings, tokenization choices have a more pronounced impact, as there is less data to compensate for suboptimal configurations. This makes tokenization particularly critical for low- to medium-resource languages. Additionally, it is often argued that subword tokenizers should ideally segment words along morphological boundaries, especially in morphologically rich languages. The reasoning is that encoding meaningful morphological units, rather than arbitrary substrings, could help models capture semantic information more effectively [Bostrom and Durrett, 2020]. However, empirical findings on this hypothesis remain inconclusive. Some studies report improvements in specific downstream tasks [Hofmann et al., 2021], while others find that morphology-aware tokenization underperforms compared to traditional subword methods [Toraman et al., 2023; Kaya and Tantu˘g, 2024].

In this paper, we pre-train encoder-only TEAMS models [Shen et al., 2021] on the IGC using WordPiece, Byte-Pair Encoding (BPE) [Sennrich et al., 2016], and Unigram [Kudo, 2018] tokenizers with varying vocabulary sizes. Each of these algorithms (see Section 2) can use either characters or bytes as its base units. We evaluate character-level WordPiece, BPE, and Unigram, and additionally compare character-level vs. byte-level BPE.

Finally, we take the tokenizer configuration that achieved the best overall performance on the IGC and use it to train tokenizers for Estonian, Basque, Galician, Nepali, and Tajik. We then pre-train TEAMS-Small models for these languages and compare their downstream performance against models trained with a WordPiece tokenizer using a 32k vocabulary. Our goal is to determine whether the trends observed for Icelandic hold for other low- and medium-resource languages and to determine how different subword tokenization algorithms and vocabulary sizes impact downstream performance.

While previous studies have examined the impact of subword tokenization algorithms and vocabulary sizes, most have focused on individual high-resource languages or have been limited in scope. Various studies have evaluated different tokenization strategies at a fixed vocabulary size or tested varying vocabulary sizes within a single algorithm (see Section 3). In contrast, our work systematically explores the effects of different tokenization strategies on downstream performance across six morphologically rich, low- and mediumresource languages. Our findings provide practical guidance for tokenizer selection in resource-constrained settings.

2 Subword Tokenization Algorithms

This section provides a brief overview of the subword tokenization algorithms used in our experiments.

2.1 Byte-Pair Encoding (BPE)

The BPE algorithm is widely used in models such as RoBERTa, OpenAI's GPT series [Radford et al., 2019; Brown et al., 2020; OpenAI, 2024], and ModernBERT [Warner et al., 2024]. It starts with a vocabulary consisting of the smallest elements in the training corpus, typically either individual characters or bytes. The algorithm iteratively expands the vocabulary by merging the most frequent adjacent subword pair at each step. The process continues until the vocabulary reaches a predefined size.

2.2 WordPiece

WordPiece is used in models such as BERT [Devlin et al., 2019] and ELECTRA [Clark et al., 2020]. While it also builds a vocabulary by iteratively merging subword units, it differs from BPE in its selection criterion. Instead of choosing the most frequent pair, WordPiece selects the pair that maximizes the increase in log-likelihood across the training corpus.

2.3 Unigram

The Unigram algorithm is used in models such as XLNet [Yang et al., 2019], and T5 [Raffel et al., 2020]. Unlike BPE and WordPiece, which progressively build their vocabularies through merging, Unigram follows a subtractive approach. It starts with a large vocabulary containing words and frequent substrings and gradually reduces it to a target size. The tokenizer assigns probabilities to subword sequences using a unigram language model. During training, it evaluates the impact of each subword's removal on the overall corpus likelihood. A percentage of the least impactful subwords is iteratively pruned until the vocabulary reaches the desired size.

3 RelatedWork

3.1 Tokenization Algorithms

Several studies have compared subword tokenization methods in terms of their impact on downstream performance.

Bostrom and Durrett [2020] examined RoBERTa-Base models pre-trained on English and Japanese corpora using BPE and Unigram tokenizers, both with a vocabulary size of 20k. The Unigram tokenizer consistently outperformed BPE across four downstream tasks. In English question answering (QA), the Unigram model achieved an F1 score 1.1 points higher than the BPE model, while in Japanese QA, the gap widened to 12.3 points in F1. The authors attribute this difference to the Unigram tokenizer's ability to better handle languages without explicit word boundaries, such as Japanese, and its greater tendency to align subwords with linguistic morphology. Similarly, Zhang et al. [2020] found that for English abstractive summarization, Unigram yielded comparable or slightly better ROUGE scores than BPE across four datasets. Ali et al. [2024] compared BPE and Unigram using a 2.6B parameter decoder-only model. In an English monolingual setting, Unigram outperformed BPE on a benchmark of NLP tasks by 0.24 percentage points. For a multilingual model trained on German, French, Italian, Spanish, and English text, BPE achieved an average score 0.17 percentage points higher than Unigram. However, performance varied by language, with Unigram performing better for French and Spanish, and BPE for the other languages.

Toraman et al. [2023] investigated BPE and WordPiece tokenizers for Turkish RoBERTa models. Across six downstream tasks, WordPiece performed similarly to or slightly better than BPE, suggesting that while these methods differ algorithmically, their practical impact on downstream performance may be minimal in some cases.

Beyond these comparisons, several studies have evaluated byte-level tokenization. Qarah and Alsanoosy [2024] pretrained BERT-Base models on an Arabic corpus using WordPiece, Unigram, and byte-level BPE tokenizers, each with a 50k vocabulary. They evaluated the models on seven tasks across 29 datasets, for a total of 36 different dataset-task combinations (with some datasets annotated for multiple tasks). The Unigram model achieved the best performance in 21 out of 36 evaluations, surpassing WordPiece by 0.77 percentage points on average. Despite the fact that byte-level BPE obtained the highest compression ratio (i.e., characters per token), it underperformed against the other two algorithms on most tasks, obtaining an average score 0.95 percentage points lower than WordPiece. Liu et al. [2019] similarly found that RoBERTa models pre-trained with byte-level BPE performed slightly worse on English NLP tasks compared to characterlevel BPE. Zhang et al. [2022] investigated byte-fallback in a Unigram tokenizer and observed that excessive segmentation into byte units resulted in longer token sequences and, in many cases, degraded translation performance. These findings suggest that while byte-based tokenization ensures full character coverage, this benefit does not necessarily translate into improved downstream performance.

A separate line of work has explored morphological tokenization, which aims to segment words based on their internal structure. Toraman et al. [2023] pre-trained a Turkish RoBERTa model with a morphological tokenizer that split words into morphemes using a morphological analyzer. While this tokenizer performed competitively on some tasks, it generally underperformed compared to BPE and WordPiece. The authors noted that errors introduced by the analyzer may have contributed to these results. Given that accurate analyzers are often unavailable or difficult to develop in low-resource settings, this raises concerns about the feasibility of morphological tokenization in such cases. Kaya and Tantu˘g [2024] proposed an alternative morphological approach that extracts word roots and appends suffix tags as subwords. However, for a Turkish BERT-Base model, this method performed similarly or worse than a standard WordPiece tokenizer.

Overall, these studies indicate that Unigram tokenizers often outperform BPE or WordPiece, particularly in languages with complex morphology or ambiguous word boundaries. In contrast, byte-level tokenization tends to produce longer token sequences, which can negatively impact performance. Meanwhile, morphological tokenization remains an open research question, with mixed results suggesting that potential linguistic benefits of morpheme-based segmentation may be offset by practical challenges in implementation and the robustness of existing subword methods.

3.2 Vocabulary Size

Beyond the choice of tokenization algorithm, vocabulary size plays a crucial role in model performance by influencing the balance between vocabulary coverage and memory usage.

Zhang et al. [2020] explored the effect of vocabulary size in English abstractive summarization, comparing Unigram tokenizers with vocabulary sizes ranging from 32k to 256k. They found that ROUGE scores improved up to a vocabulary size of 96k, after which performance began to degrade, suggesting diminishing returns beyond a certain threshold.

For morphologically rich languages, larger vocabularies have shown a similar trend of initial performance gains followed by diminishing improvement. Toraman et al. [2023] evaluated vocabulary sizes for BPE, WordPiece, morphological, and word-level tokenizers in a Turkish RoBERTa model, finding that increasing vocabulary size generally improved downstream performance. However, for BPE and WordPiece, both of which outperformed the other tokenizers, the gains were minimal beyond 66k. The authors attributed this to increased vocabulary coverage and a reduction in OOV tokens. Ali et al. [2024] examined the impact of different vocabulary sizes (33k, 50k, 82k, and 100k) for BPE and Unigram tokenizers on downstream performance. For an English monolingual model, a vocabulary size of 33k yielded the highest average score, while larger vocabularies led to slightly lower performance. In contrast, for a multilingual model covering German, French, Italian, Spanish, and English, the best results were achieved with a vocabulary size of 100k, roughly three times larger than the optimal size for the monolingual model. However, the optimal vocabulary size differed between languages in the multilingual setting, with 33k performing best for German and 82k for English.

A more fine-grained analysis was conducted by Kaya and Tantu˘g [2024], who varied vocabulary size in WordPiece tokenization for Turkish NLP tasks using a BERT-Base model. Increasing the vocabulary size from 32k to 64k led to substantial improvements, with named entity recognition (NER) scores improving by 1.6 percentage points and QA scores by up to 6.2 percentage points. However, increasing the vocabulary further to 128k and 256k provided only marginal additional gains. The authors suggested that reducing OOV words improves performance by limiting the overall number of token splits, which in turn reduces the likelihood of splits occurring at arbitrary positions rather than at morphological boundaries, potentially improving the model's ability to capture semantic information effectively.

To summarize, these studies suggest that increasing vocabulary size improves model performance up to a certain point, after which returns diminish. While larger vocabularies reduce OOV words, excessively large vocabularies may introduce inefficiencies without significant downstream benefits.

4 Experimental Setup

4.1 Tokenization

We trained WordPiece and BPE tokenizers using the tokenizers library for Python [Moi and Patry, 2023]. For characterlevel tokenizers, we limited single-character subwords to 1,000 to prevent rare characters, such as emojis and mixedscript Unicode characters, from taking up too much of the vocabulary. We normalized the text by removing all Unicode control characters and split on whitespace characters and punctuation marks.

We trained Unigram tokenizers using the SentencePiece library [Kudo and Richardson, 2018] with settings that roughly matched the configuration we used for the WordPiece and BPE tokenizers. During training, we adjusted the character coverage of the tokenizer to restrict the number of singlecharacter subwords to approximately 1,000, and replaced the default NFKC Unicode normalization with the less aggressive NFC normalization, converting all Unicode characters to their canonical forms. Due to the higher memory usage when training with SentencePiece, we limited the training corpus for Icelandic to approximately 2M documents or 654M tokens, and for Nepali to 800k documents or 348M tokens (owing to the greater number of bytes required to represent characters in the Devanagari script).

For all tokenizers, we maintained original casing and retained all accents, and only used tokens that appeared at least twice for training.

4.2 Pre-Training Settings

TEAMS is a bidirectional, encoder-only version of the ELECTRA architecture [Clark et al., 2020]. We pre-trained TEAMS-Small models using TPU v4-8 accelerators and the TensorFlow Model Garden repository [Yu et al., 2020], following the same settings as in [Shen et al., 2021]. The models consist of 14M parameters each and were pre-trained for 500,000 steps with a maximum sequence length of 512 and a batch size of 256. Pre-training took approximately 18 hours per model to complete. Default pre-training parameters were used for all experiments unless otherwise specified.

4.3 Fine-Tuning Settings

We evaluated the models on six tasks: part-of-speech (POS) tagging, NER, dependency parsing (DP), automatic text summarization (ATS), QA, and topic classification (TC). For fine-tuning, we used the example scripts provided with the Hugging Face Transformers library [Wolf et al., 2020]: run_ner.py for POS and NER, run_qa.py for QA, and run_classification.py for TC. Models were finetuned for DP using DiaParser [Attardi et al., 2021], a biaffine dependency parser. For ATS, we used TransformerSum, 1 a Python library implementing the extractive summarization approach described by Liu and Lapata [2019].

To ensure a controlled comparison, we applied the same fine-tuning configuration to all models within each task.

1 https://github.com/HHousen/TransformerSum

Although performing hyperparameter optimization for each model individually would likely yield improved results, we prioritized consistency to enable fair evaluation across models and languages. For POS, NER, DP, and ATS, we adopted the settings used by Daðason and Loftsson [2022], which we found to be robust across all languages. For QA and TC, we performed a hyperparameter search over learning rates in [3e-5, 5e-5, 8e-5, 1e-4], batch sizes in [8, 16, 32], and number of epochs in [5, 10, 20], selecting a single configuration that performed well across all languages. The final hyperparameters used in our experiments are summarized in Table 2. Note that for DP, the batch size is token-based and corresponds to the number of tokens processed per optimization step.

For datasets with predefined training, validation, and test splits, we report results averaged over 10 runs with different random seeds. Datasets with k-fold splits were evaluated using k-fold cross-validation. For datasets without predefined splits, we applied stratified 10-fold cross-validation, except for datasets containing fewer than 10,000 examples, where we used stratified 5-fold cross-validation with five repetitions to reduce variance and increase statistical power for significance testing.

4.4 Pre-Training Corpora

For pre-training, we used the IGC for Icelandic, the Estonian National Corpus (ENC) [Koppel and Kallas, 2022] for Estonian, and EusCrawl [Artetxe et al., 2022] for Basque. For Galician, Nepali, and Tajik, we used the filtered versions of their subsets in the web-crawled mC4 corpus [Xue et al., 2021], using a Gaussian Mixture Model (GMM) classifier as described in [Daðason and Loftsson, 2024] to filter out lowquality documents. Table 3 provides information on the corpora used in our work.

4.5 Evaluation Datasets

We fine-tuned and evaluated each pre-trained LM on a diverse benchmark of downstream tasks. For Icelandic, we used MIM-GOLD [Loftsson et al., 2010] for POS, MIMGOLD-NER [Ingólfsdóttir et al., 2020] for NER, IcePaHCUD [Arnardóttir et al., 2020] for DP, IceSum [Daðason et al., 2021] for ATS, and RUQuAD [Skarphéðinsson et al., 2023] for QA. For Estonian, we used EDT-UD [Muischnek et al., 2016] for POS and DP, and EstNER-New [Sirts, 2023] for NER. For Basque, we used BDT-UD [Aranzabe et al., 2015] for POS and DP, and EIEC [Alegria et al., 2004] for NER. For Galician, we used CTG [Agerri et al., 2018] for POS, NERC [Agerri et al., 2018] for NER, and GalicianTreeGal-UD [Garcia et al., 2018] for DP. For Nepali, we used NNC-CS [Yadava et al., 2008] for POS, EverestNER [Niraula and Chapagain, 2022] for NER, and SIB-200 [Adelani et al., 2024] for TC. Finally, for Tajik, we used WikiANN [Pan et al., 2017] for NER and SIB-200 for TC. Table 4 provides an overview of the datasets, their associated tasks, and the number of labeled examples.

5 Results

In this section, we present the results of our experiments with different tokenization strategies across several low- and medium-resource languages.

5.1 Tokenization Configurations for Icelandic

To assess the impact of different tokenization strategies, we pre-trained TEAMS-Small models on the IGC using WordPiece, BPE, and Unigram tokenizers, each with vocabulary sizes of 16k, 32k, and 64k. Table 5 provides statistics for each tokenizer, including the number of tokens generated from the corpus, and the number of pre-training examples.

Larger vocabularies lead to fewer tokens and a higher compression ratio, which in turn affects the number of input sequences extracted from the corpus for pre-training. For example, a WordPiece tokenizer with a 64k vocabulary can represent the IGC with 4.13M examples (with each example consisting of 512 tokens), which is 13.6% fewer than required with a 16k vocabulary. With a fixed number of training steps, this effectively means that models using tokenizers with larger vocabularies are trained for more epochs. In 500k steps, a TEAMS-Small model will learn from each example 31 times with a WordPiece vocabulary of 64k, but only 26.8 times with a 16k vocabulary. Increasing the vocabulary size may therefore allow the model to learn more effectively during pre-training.

Table 6 presents the downstream performance of each model across five NLP tasks: POS tagging, NER, DP, ATS, and QA. The results show that Unigram tokenization with a 64k vocabulary achieved the highest average score, slightly outperforming WordPiece and BPE at the same vocabulary size. This aligns with prior research suggesting that the Unigram algorithm has an advantage on some tasks [Bostrom and Durrett, 2020; Zhang et al., 2020;

Qarah and Alsanoosy, 2024]. Additionally, WordPiece performs similarly or slightly better than BPE, consistent with previous findings by Toraman et al. [2023].

Across all tokenization algorithms, larger vocabularies tend to yield better downstream performance, consistent with findings by Toraman et al. [2023] and Kaya and Tantu˘g [2024]. The effect is particularly pronounced for NER and QA, where increasing the vocabulary from 32k to 64k improved scores by an average of 0.50 and 1.11 percentage points, respectively. A greater compression ratio allows more characters to fit within each 512-token sequence, effectively increasing the amount of textual context available to the model. Tasks that require a broader context window, such as QA, likely benefit from this effect, as they rely on longerrange dependencies.

Overall, these results suggest that while the Unigram algorithm offers a modest advantage, vocabulary size has a greater impact on downstream performance.

5.2 Byte-Level Tokenization for Icelandic

To evaluate the impact of byte-level tokenization on downstream performance, we pre-trained a TEAMS-Small model on the IGC using a byte-level BPE tokenizer with a 64k vocabulary, comparing it against a character-level tokenizer with the same vocabulary size. Table 7 provides statistics for each tokenizer.

There were minimal differences between the two tokenizers in terms of compression ratio and the number of examples generated. This is likely due to the fact that we used a tokenizer with a large vocabulary to process a curated, highquality monolingual corpus. The character-level tokenizer generated 2.11 billion tokens when processing the IGC, of which only 5,475 were unknown tokens (i.e., sequences of characters that did not exist in its vocabulary). As such, there were negligible benefits from representing unknown characters as bytes. Since the tokenizer has a large vocabulary, a considerable portion of its capacity is allocated to byte-level merges that provide no practical advantage in this setting.

Table 8 presents the downstream performance of the character-level and byte-level tokenizers. The two tokenizers performed nearly identically across all tasks, with no statistically significant differences in downstream performance.

For high-quality monolingual corpora, byte-level tokenization does not appear to offer any advantages. Even in noisier corpora containing a variety of Unicode characters or mixed scripts, representing such noise as bytes rather than unknown tokens may not impact downstream performance, unless the downstream datasets contain similar noise distributions. The primary advantages of byte-level tokenization are more apparent in massively multilingual models, where vocabulary constraints make character-level representation impractical, and in generative models intended to be capable of outputting any Unicode character.

5.3 Cross-Linguistic Evaluation

For Icelandic, the Unigram tokenizer with a vocabulary size of 64k obtained the highest overall performance. To determine whether the same trends extend to other languages, we pre-trained TEAMS-Small models using a tokenizer with the same configuration on Estonian, Basque, Galician, Nepali, and Tajik corpora. Table 9 provides statistics for each tokenizer.

We observe an improvement in compression ratio across all languages, with the largest gain in Estonian, where the pre-training corpus could be encoded with 10% fewer tokens. For Icelandic, the Unigram tokenizer achieved a compression ratio (i.e., characters per token) of 4.18, substantially higher than the 3.88 obtained by the WordPiece tokenizer. A WordPiece tokenizer with a 64k vocabulary achieved a ratio of 4.12 on the IGC (see Table 5), indicating that much of the observed improvement in compression ratio is likely due to the larger vocabulary. Nevertheless, it seems that the Unigram algorithm may be somewhat more effective at capturing subword structure, particularly in morphologically rich languages. Table 10 presents the downstream performance of each model, comparing the Unigram tokenizer to a WordPiece tokenizer with a 32k vocabulary.

Overall, the results aligned with our findings for Icelandic. The Unigram tokenizer significantly outperformed WordPiece in eight out of 19 tasks, particularly in NER and TC, where average scores improved by 0.85 and 0.97 percentage points, respectively. The only case where WordPiece significantly outperformed Unigram was DP for Galician, where it achieved a 0.75 percentage point higher score.

In contrast, a larger vocabulary appeared to have minimal or slightly negative impact on DP and POS, indicating that these tasks may not benefit from reduced subword fragmentation or increased context to the same extent as NER and TC. This suggests that while Unigram tokenization is generally advantageous, its benefits are task-dependent.

6 Conclusions

In this paper, we have addressed the question of how different subword tokenization algorithms and vocabulary sizes impact downstream performance for low- and medium-resource languages. Our experiments across six morphologically rich languages have demonstrated that a Unigram tokenizer with a 64k vocabulary consistently outperformed other configurations, particularly for tasks like NER and TC, where scores improved by an average of 0.85 and 0.97 percentage points, respectively. These results align with prior research suggesting that Unigram tokenization better preserves linguistic structure, leading to improved downstream performance [Bostrom and Durrett, 2020], though the magnitude of improvement varied by task and language.

Vocabulary size had a greater impact on downstream performance than the choice of tokenization algorithm, with larger vocabularies consistently yielding better results. However, the benefits were task-dependent, with NER and TC showing substantial improvements, while POS tagging and DP saw minimal gains from increased vocabulary size. Our evaluation of byte-level tokenization found no significant advantages for high-quality monolingual corpora, indicating that such methods may be more beneficial in multilingual settings, where broader character coverage is essential.

Overall, these findings provide practical guidance for pretraining LMs in resource-constrained settings, highlighting that optimizing vocabulary size can yield greater performance gains than switching between tokenization algorithms.

Acknowledgments

This research was supported with Cloud TPUs from Google's TPU Research Cloud (TRC).