TreasureX Researchcs.AIUpgrading Tokenizers for Multilingual LLMs Without Retraining From Scratch
Jimmy T. H. Smith, Tarek Dakhran, Alberto Cabrera, Simon S. Lee, Paul Pak, Aditya Tadimeti, Tim Seyde, Maxime Labonne, Alexander Amini, Mathias Lechner · July 16, 2026
Large language models (LLMs) are typically trained with a fixed tokenizer that splits text into tokens. If the tokenizer was designed for English, later-added languages like Thai or Hindi get split into many more tokens per word, increasing latency, compute, and energy costs for those users. This paper introduces a method called tokenizer expansion that lets model producers upgrade the tokenizer of a pre-trained LLM without retraining from scratch. The key idea: continue the existing tokenizer's BPE merges on a multilingual corpus, so most original tokens remain unchanged, and new tokens are decomposed into existing sub-tokens. Embedding rows for new tokens are initialized as the mean of their source sub-token embeddings. Then a two-stage adaptation (embedding-only training followed by full-model continued pre-training) recovers original quality. Applied to an 8B-parameter MoE model, the expanded 128K tokenizer reduced token count for Hindi and Vietnamese by ~2.4x and 2.6x (up to 4x for Thai), leading to an estimated 2.2–3.7x per-character decode speedup on reference devices. This is especially important for on-device models where vocabulary size directly impacts memory and bandwidth. The method is practical for any model producer controlling the tokenizer design, and the authors release the expanded model and tokenizer.
Read the full paper on arXiv: https://arxiv.org/abs/2607.15232
Educational content only, generated with AI assistance from public research abstracts. Not investment advice. Thank you to arXiv for use of its open access interoperability.