{"type":"ScholarlyArticle","title":"Subword Tokenization for Low- and Medium-Resource Languages: A Systematic Evaluation","authors":[{"name":"Jón Daðason","slug":"jon-da-ason","given":"Jón","family":"Daðason","affiliation":""},{"name":"Hrafn Loftsson","slug":"hrafn-loftsson","given":"Hrafn","family":"Loftsson","affiliation":""}],"abstract":"Subword tokenization is a standard technique for pre-trained language models, mapping text into sequences of tokens from a fixed-size vocabulary. Despite its widespread use, the impact of tokenization algorithms and vocabulary sizes on downstream performance remains underexplored, particularly for low- and medium-resource languages, where suboptimal configurations are difficult to compensate with additional data. Prior studies have mainly focused on high-resource languages, individual tokenization algorithms, or fixed vocabulary sizes, limiting their scope. In this paper, we present a systematic evaluation of subword tokenization strategies in six diverse low- and medium-resource languages: Icelandic, Estonian, Basque, Galician, Nepali, and Tajik. We pre-trained monolingual TEAMS-Small models using WordPiece, BPE, and Unigram tokenizers with vocabulary sizes of 16k, 32k, and 64k, and additionally compared byte-level and character-level tokenization for Icelandic. Evaluation on a benchmark of NLP tasks revealed that Unigram tokenization with a 64k vocabulary consistently outperformed other configurations, with vocabulary size having a greater impact on downstream performance than algorithm choice. These gains were task-dependent, with statistically significant improvements for part-of-speech tagging, named entity recognition, and question answering, but not for dependency parsing and summarization. Finally, byte-level tokenization provided no measurable advantage in a monolingual setting, suggesting that its benefits are primarily relevant for multilingual models.","doi":"10.24963/ijcai.2026/633","url":"https://ijcai.org/proceedings/2026/633","pdf":"https://ijcai.org/proceedings/2026/0633.pdf","markdown":"https://ijcai.org/proceedings/2026/633.md","bibtex":"https://ijcai.org/proceedings/2026/bibtex/633","volume":{"slug":"2026","title":"Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2026","url":"https://ijcai.org/proceedings/2026/","year":2026,"location":"Bremen, Germany","dates":"15-21 August 2026","editors":"Diego Calvanese","isbn":"978-1-956792-09-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2026"},"section":"Natural Language Processing","track":"Main Track","number":633,"pages":{"first":5684,"last":5692},"keywords":["Natural Language Processing: Resources and evaluation","Natural Language Processing: Phonology, morphology, and word segmentation","Natural Language Processing: Language models"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}