Two freelancers sign up for the same AI writing tool on the same plan. One works in English from Lisbon. The other works in Malayalam from Kochi. They do the same job, send prompts of the same length, and one of them burns through her quota several times faster. Nobody told her that was part of the deal.
A study published in September by the Coalition for Language Equity in AI Research calls this the tokenizer tax. AI models don’t read words. They read tokens, chunks of text cut up by a tokenizer that was trained mostly on English and a few other big web languages. English words usually become one token each. Words in less common scripts get chopped into many small pieces. More pieces means more tokens, and tokens are what you pay for.
Among the 20 most used languages online, the study found the priciest language needed 1.5 to 5 times as many tokens as the cheapest for the same content. Widen the set to 200 languages and the gap in one model grew to 17 times. The worst off were languages with scripts the tokenizers rarely saw: Burmese, Tibetan, Tifinagh, Khmer, Lao and Ol Chiki. Speaker numbers aren’t the whole story. Asturian has few speakers, but it uses the Latin alphabet and borrows from Italian and Spanish, so it gets off lightly.
What it costs a remote worker
If you pay per token through an API, the cost is direct. A developer building a client tool for Bengaluru pays a surcharge on every prompt and every answer that a developer in Austin doesn’t. If you use a flat monthly app, you pay in other ways. Usage caps run out sooner. Responses come back slower, because more tokens take longer to generate. And the context window, the amount of text the model can hold in mind at once, fills up faster. That last one hurts the most for real work. Paste in a long contract or a week of client emails in Khmer and the model starts forgetting the beginning well before it would in English.
Pratyush Choudhury of Activate AI gave the example of a tutoring app where a student’s history fills the context window faster in some languages, so the service quietly gets worse for exactly the people it was built for. The same thing happens to a translator, a support agent or a content writer working across languages from a laptop in a co-working space.
What you can do about it now
Check the numbers yourself. Most providers have a free tokenizer page. Paste a typical paragraph in your working language and in English and compare the counts. That ratio is your real price multiplier.
Pick the model by language, not by brand. The study found the script penalty across nearly all models, but the size of it varies. A model that is cheap per token can be the expensive one for your language.
Draft long instructions in English where the client doesn’t need to see them. System prompts, style rules and reference notes can often stay in English while the output is produced in the target language. That saves context for the material that matters.
Watch for regional models. Startups in India and elsewhere are building tokenizers and datasets for their own languages. Maya Research’s CEO argued that better data alone won’t fix it, and that tokenizers have to be designed for these scripts from the start. Token prices are also falling fast, which softens the direct cost even if the ratio stays.
The gap is a design choice, not a law of nature. Choudhury’s suggestion is that AI labs publish a language-by-language cost scorecard and commit to improving the worst-served languages. Until they do, remote workers outside English carry a cost they can’t see on the pricing page. Now at least you know where to look.