---
title: "Subword Tokenization for Low- and Medium-Resource Languages: A Systematic Evaluation"
authors:
  - "Jón Daðason"
  - "Hrafn Loftsson"
doi: "10.24963/ijcai.2026/633"
url: "https://ijcai.org/proceedings/2026/633"
pdf: "https://ijcai.org/proceedings/2026/0633.pdf"
venue: "Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence"
year: 2026
pages: "5684-5692"
track: "Main Track"
keywords:
  - "Natural Language Processing: Resources and evaluation"
  - "Natural Language Processing: Phonology, morphology, and word segmentation"
  - "Natural Language Processing: Language models"
---

# Subword Tokenization for Low- and Medium-Resource Languages: A Systematic Evaluation

Jón Daðason, Hrafn Loftsson

*Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence*, pages 5684-5692. DOI: [10.24963/ijcai.2026/633](https://doi.org/10.24963/ijcai.2026/633)

## Abstract

Subword tokenization is a standard technique for pre-trained language models, mapping text into sequences of tokens from a fixed-size vocabulary. Despite its widespread use, the impact of tokenization algorithms and vocabulary sizes on downstream performance remains underexplored, particularly for low- and medium-resource languages, where suboptimal configurations are difficult to compensate with additional data. Prior studies have mainly focused on high-resource languages, individual tokenization algorithms, or fixed vocabulary sizes, limiting their scope. In this paper, we present a systematic evaluation of subword tokenization strategies in six diverse low- and medium-resource languages: Icelandic, Estonian, Basque, Galician, Nepali, and Tajik. We pre-trained monolingual TEAMS-Small models using WordPiece, BPE, and Unigram tokenizers with vocabulary sizes of 16k, 32k, and 64k, and additionally compared byte-level and character-level tokenization for Icelandic. Evaluation on a benchmark of NLP tasks revealed that Unigram tokenization with a 64k vocabulary consistently outperformed other configurations, with vocabulary size having a greater impact on downstream performance than algorithm choice. These gains were task-dependent, with statistically significant improvements for part-of-speech tagging, named entity recognition, and question answering, but not for dependency parsing and summarization. Finally, byte-level tokenization provided no measurable advantage in a monolingual setting, suggesting that its benefits are primarily relevant for multilingual models.