Burmese People Pay More Than Their Neighbours

Meta Description: Learn how 17 GPT, Claude and Gemini tokenizers price Burmese text, with costs up to 11.7 times English, and what developers should check before building.

Tags: Burmese tokenization, AI tokenizers, GPT, Claude, Gemini

Featured image: Abstract friendly illustration of small text fragments turning into coin-like shapes, with no real people, no logos, and no readable text.

A benchmark from September 2026 measured how 17 GPT, Claude and Gemini tokenizers count Burmese text. It gives developers, editors and API users a concrete reason to check Burmese tokenization before setting prices, context limits or retrieval sizes: the same meaning can consume up to 11.7 times more tokens than English.

Why tokens matter

Large language models do not read letters or words. They read tokens, which are small pieces of text. The count matters because API providers charge by token, models generate one token at a time, and a context window such as 128K tokens limits how much text the model can see at once.

A simple example from the test made the cost visible. The Burmese spelling of Myanmar entered into GPT-4 returned 12 tokens, while the English word Myanmar was one token. The gap comes from tokenizer vocabularies. Most tokenizers learn from text that is heavy in English, code and a few large languages. Scripts with little training data get few dedicated pieces, so their text can be cut into very small fragments, sometimes bytes. Burmese characters take 3 bytes in UTF-8, so byte-level fallback hits Burmese especially hard.

What the benchmark measured

To compare languages fairly, the test needed the same meaning in each language. It used FLORES-200, a public dataset from Meta AI with 1,012 sentences from Wikinews, Wikijunior and Wikivoyage, professionally translated into 200 languages.

The developer counted tokens for English, Burmese, Thai, Khmer, Lao, Hindi, Vietnamese and Chinese across 17 models. Open tokenizers counted locally were OpenAI cl100k for GPT-4, OpenAI o200k for GPT-4o, Llama 3, Gemma 3, Qwen 3, DeepSeek V3, Mistral, Phi-4 and NLLB-200. Commercial models were counted through official token-counting APIs: GPT-5, GPT-5.5, GPT-6 Sol, Claude Opus 5, Sonnet 5 and Haiku 4.5, Gemini 2.5 Flash and Gemini 3.8 Flash.

The main ratio was the Burmese tax: tokens needed for Burmese divided by tokens needed for English for the same sentences and the same tokenizer. A value of 1x means Burmese is as cheap as English.

How Burmese compares with neighbours

Across the tested tokenizers, Burmese required between 1.6x and 11.7x as many tokens as English. The effect is not only a billing issue. With Llama 3, a 128K-token context window held about 4,766 English sentences, but only 408 Burmese sentences, meaning the model had more than ten times less effective memory in Burmese.

Two structural findings stood out. GPT-5, GPT-5.5 and GPT-6 Sol produced exactly the same token counts as GPT-4o's o200k tokenizer in every language, showing OpenAI had not changed that tokenizer since GPT-4o. Gemini and Gemma share one tokenizer: Gemini 2.5 Flash, Gemini 3.8 Flash and the open Gemma 3 model gave identical counts, and this tokenizer was the best for Burmese among the chat models tested.

Using the median cost across 10 distinct tokenizers, Burmese needed 4.7x the tokens of English. That was twice Thai and Hindi, each at 2.3x, three times Vietnamese at 1.5x, and four times Chinese at 1.2x. Burmese cost more than Thai, Hindi, Vietnamese and Chinese with every tokenizer in the test. With 5 of the 10 tokenizers, Burmese was the most expensive of all eight languages.

Thai was the closest comparison because it is also a Southeast Asian language with its own script and no spaces between words. Even then, Burmese cost 2.3x as much as Thai on average, and the gap reached 5.3x with Llama 3. Lao and Khmer showed the same pattern: their median costs were 6.8x and 6.1x. The problem appears to be scripts with little training data, not Burmese alone.

Absolute token counts and bills

The Burmese tax compares one tokenizer with itself. When choosing a provider, the absolute number of tokens for the same Burmese text matters more because that is the number used for billing. For the same 1,012 Burmese sentences, Gemini needed about 66,000 tokens, GPT-5 and GPT-6 Sol needed about 85,000 tokens, and Claude needed about 173,000 tokens. Claude used roughly twice as many tokens as GPT and 2.6 times as many as Gemini for Burmese.

A ratio can also hide a worse baseline. Claude Opus 5 had a Burmese tax of 4.0x, which looked better than Claude Haiku 4.5 at 5.8x. That was not because Burmese became cheaper. Opus 5 used a newer tokenizer that spent 44% more tokens on English, while Burmese stayed at about 172,000 tokens with both models. The ratio improved because the English baseline got worse. The same warning applies to any comparison: check absolute token counts first, then multiply by the provider's price per token to compare real bills.

Who should use it / who should skip it

Burmese developers and teams building products for Myanmar users should use the benchmark because the tokenizer is a real business decision. The gap between 2.4x and 11.7x affects money, speed and context. A 128K model does not provide 128K tokens' worth of English-sized content when the text is Burmese, so context budgets should be measured in Burmese tokens. For search and retrieval systems, chunking Burmese text by syllables rather than characters can make each piece carry more stable meaning. The developer built Burmese retrieval systems around this idea.

The benchmark can also run on a team's own documents with one command, because news sentences are not the same as chat messages. It is open source at github.com/aungthuhein2005/myanmar-tokenizer-bench. Users can clone it with git clone https://github.com/aungthuhein2005/myanmar-tokenizer-bench, install it with cd myanmar-tokenizer-bench followed by pip install -e ., then run mtb run. The project also asks for more test texts, Zawgyi samples and new tokenizers to compare, and says a GitHub star helps other Burmese developers find it.

The findings also point to a design choice for model companies. Meta's NLLB-200, built for 200 languages, handled Burmese at 1.6x, almost as efficiently as Thai, while several chat models used far more tokens. Fair tokenization is possible, but it requires training vocabularies that reflect the scripts they are meant to serve.

The benchmark has limits. It used FLORES news and wiki text, so chat, code or informal Burmese may tokenize differently. It measured Unicode Burmese only, and Zawgyi text was not included. Fewer tokens does not mean better answers, and providers can change tokenizers with new models. These numbers are from September 2026. The original article was published in Towards AI on Medium.

Teams that only handle English, or that do not need Burmese token estimates, can skip the detailed benchmark. Teams that ship Burmese text should not treat tokenizer choice as a footnote: before pricing, context limits or retrieval chunks are set, they should measure absolute Burmese token counts across the models they are considering.