Which language costs the most in AI tokens?
The short answer
Amharic. Saying the same three sentences in Amharic takes 146 tokens against English's 28, a 5.2x penalty for identical meaning. Every token is billed, so that is a 421% surcharge on the same conversation.
But the honest headline is quieter than the one usually circulated. Across 20 languages, most now land between 1.0x and 1.6x English, not the 2x to 5x you see quoted. The inflated figures come from measuring one sentence.
Method: 3 sentences, professionally comparable in meaning, tokenized with the same byte-pair vocabulary a current model uses. Every number on this page is computed when the page renders. Tokenize your own text in the token calculator.
Large language models bill per token, and a token is not a word. It is a fragment produced by a vocabulary that was learned by counting which character sequences appeared most often in the training data. That data was overwhelmingly English. The consequence is that English got the whole words, and everyone else got the leftovers, assembled fresh out of fragments every time they type.
This is usually framed as a fairness scandal or dismissed as a technical detail. It is neither, and the interesting questions start once you have actual numbers. So here they are, followed by the questions I think are genuinely hard.
The leaderboard
The same 3 sentences in 20 languages. Switch the vocabulary to see how much a tokenizer generation changed things, and switch the measure to see how differently the same data reads depending on what you normalise by.
| Language | Tokens | Cost vs English |
|---|
Tokens per word is a trap
Switch the measure above to tokens per word and the table scrambles. Japanese leaps to the top and Bengali sinks to the bottom, which tells you nothing true about either. Japanese does not put spaces between words, so counting them is meaningless. Bengali marks a lot of grammar inside single orthographic words, so it has few of them.
Tokens per character is fairer across scripts, but it flatters languages whose characters carry more meaning. Chinese needs only 37 characters to say what English needs 153 characters for, so a low tokens-per-character figure is partly just density.
That is why the first measure is the one that matters for a bill. The three sentences mean the same thing in every row, so comparing total tokens compares the price of the same idea. Everything else is a normalisation that happens to flatter somebody.
The numbers you have read are too high
This part surprised me, and it is the reason this post exists. The multipliers circulating online, including in things I wrote before running this, mostly come from tokenizing one sentence. Here is the same set measured on one sentence versus 3:
| Language | One sentence | 3 sentences | Overstated by |
|---|---|---|---|
| Amharic | 6.88x | 5.21x | +1.66 |
| Korean | 2.25x | 1.61x | +0.64 |
| Bengali | 2.00x | 1.39x | +0.61 |
| Swahili | 1.88x | 1.32x | +0.55 |
| Hindi | 1.88x | 1.25x | +0.62 |
| Turkish | 1.88x | 1.21x | +0.66 |
| Arabic | 1.38x | 1.00x | +0.38 |
Every single language dropped. That is not noise, it is a systematic artifact: the test sentence everyone reaches for tends to contain words like "yesterday" and "inspected" that happen to be single English tokens and rare everywhere else. Pick a different sentence and the penalty shrinks. Pick three and it shrinks further.
The honest position is that the gap is real, smaller than advertised, and wildly uneven. Arabic has essentially closed it. Amharic has not moved into the same universe. Treating "non-English" as one category is the first thing to stop doing.
What the gap actually costs
Pick a price tier and a monthly volume, and see what the same product costs to serve in each language. This is the number that decides whether any of this matters.
| Language | Monthly cost | Extra vs English |
|---|
Assumes a message the length of our sample sentences, billed as input. Real products differ; the ratio between rows is the part that holds.
Run it at a realistic volume and the shape of the problem appears. For most languages the surcharge is a rounding error on a small product and a real line item on a large one. For Amharic it is neither: it is a structural multiplier that makes some products simply not viable in that market at that price.
It was never only about money
Focusing on the bill undersells the problem, because three other things degrade at exactly the same rate and none of them can be fixed by paying more:
- The context window shrinks. A 128K window holds 25K worth of English-equivalent content in Amharic. Your document does not fit, your conversation truncates sooner, your retrieved passages get cut.
- Output limits arrive earlier. A cap of 4,096 output tokens is a much shorter answer in a fragmenting language. The model does not get to finish the thought.
- Generation is slower. Decoding is sequential and per token, so more tokens for the same meaning means a visibly slower reply. The user experience is worse, not just pricier.
A speaker of a fragmenting language therefore pays more, waits longer, gets cut off sooner, and can feed the model less context. Any one of those is tolerable. Together they add up to a materially worse product for the same money.
Now the hard questions
I do not think any of these have clean answers. I have put my own position at the end of each, but the arguments against are not strawmen and I have tried not to write them as such.
1. Should providers bill per character instead of per token?
It is the obvious fix and it removes the tokenizer from the invoice entirely. It also breaks the link between price and cost. Compute really is consumed per token: a model doing a forward pass over 146 tokens does more work than one doing 28, regardless of what the characters say. Per-character billing does not make that work cheaper, it just moves who pays for it.
It would also produce new unfairness immediately. Chinese says the same thing in far fewer characters, so a per-character price would hand Chinese an enormous discount and charge Vietnamese, which needs many more characters for the same meaning, a large premium. You would have swapped one arbitrary proxy for another.
Where I land: per character is not fairer, it is differently unfair, and it hides the subsidy instead of naming it. If you want a subsidy, name it.
2. Should tokenization be parity-based?
You could build a vocabulary that deliberately allocates slots by language rather than by frequency: reserve a fixed share for each of the top hundred languages, so the same meaning costs roughly the same everywhere. This is technically straightforward. Nobody does it.
The reason is that a vocabulary is a fixed budget, and every slot given to Amharic is a slot taken from something else. Frequency-based allocation is not arbitrary, it is an optimisation: it minimises total sequence length over the expected workload, which reduces cost and latency for everyone in aggregate. Parity allocation would make the median request slightly worse to make the tail request much better.
That is a real tradeoff, but notice it is the same tradeoff we make constantly elsewhere and usually resolve in favour of the tail: kerb cuts, screen readers, minority language broadcasting. "The majority pays a small cost so a minority can participate at all" is not an exotic principle. It is most of public infrastructure.
Where I land: full parity is the wrong target, because it would waste vocabulary on languages with almost no digital text. A floor is the right target. No language above some population threshold should sit above, say, 2x English. That is achievable, it is measurable, and it would have fixed Amharic years ago.
3. Is this discrimination, or is it just physics?
Nobody sat in a room and decided Amharic speakers should pay five times more. The vocabulary fell out of a corpus, the corpus fell out of what is on the web, and what is on the web fell out of four centuries of unequal development. There is no villain in the loop.
But "no intent" is not the same as "no responsibility". The training corpus is a choice. Its composition is a choice. The decision to ship a 100K vocabulary rather than a 250K one is a choice, and we can see from the o200k numbers exactly what that choice was worth: Bengali went from 5.6x to 1.4x when someone decided multilingual coverage mattered enough to spend vocabulary on it.
Where I land: calling it discrimination misdescribes the mechanism and lets the actual fix get lost in an argument about motives. Calling it physics is worse: it is a design outcome, it responded immediately to design effort, and the people who could change it can see these numbers as easily as I can.
4. Who would actually pay for parity?
Suppose a provider announced tomorrow that all languages are billed at the English rate for equivalent meaning. The cost does not vanish; it lands somewhere. Either English-language customers subsidise it through higher prices, or the provider absorbs it, or the languages that are already cheap to serve get quietly repriced upward.
And here is the uncomfortable part: the customers who would fund that subsidy are mostly not the beneficiaries, and the beneficiaries are mostly in markets with low willingness to pay. There is no commercial mechanism that makes this happen on its own. It happens because someone decides it should, or it does not happen.
Where I land: the fix is not pricing, it is the tokenizer. Fix the vocabulary and the price fixes itself, permanently, with no subsidy and no ongoing commitment from anybody. Every pricing remedy is a workaround for a technical problem that has a technical solution.
5. Would a fairer tokenizer make models worse?
This is the strongest objection and the one most often waved away. Vocabulary slots spent on a language with little training data may not earn their keep: the model sees those tokens rarely, learns weak representations for them, and you have spent capacity to produce tokens the model handles badly. Fragmentation is sometimes doing useful work, forcing rare words through well-learned subword pieces.
There is real evidence in both directions here, and anyone claiming the answer is obvious is not being careful. What we can say from the numbers above is narrower: the move from the older vocabulary to the newer one cut several languages' costs dramatically, and those models did not get worse. That is one data point in favour of "there was slack", not proof that slack is infinite.
Where I land: the honest answer is that we do not know where the ceiling is, and that is an argument for measuring it rather than for assuming we have already hit it.
6. Does any of this survive contact with the actual market?
Here is the argument I find hardest to dismiss. For most languages the gap is now 10 to 60 percent. On a product spending fifty dollars a month, that is fifteen dollars. Nobody has ever chosen a different vendor over fifteen dollars. The gap that is large enough to matter, Amharic's, affects a language whose speakers are largely not yet buying LLM API access at scale, so fixing it changes almost no revenue and almost no user's day tomorrow.
If that reasoning holds, this is a genuine but minor inefficiency that markets will grind down on their own as multilingual data grows, and the moral framing is doing more work than the arithmetic supports.
Where I land: the money argument is weak and I would not push it. The context window and truncation arguments are the strong ones, because they are not inconveniences you can pay your way out of. They are a smaller product, and no amount of willingness to pay buys the missing window back.
If you are building something today
- Measure your own text. Do not budget from a blog post, including this one. The multiplier for your actual content is the only one that matters, and you can get it in one call to the tokenize endpoint.
- Budget context, not just cost. If your users write in a fragmenting language, your effective context window is smaller than the number on the box. Size your history and retrieval accordingly.
- Check the newer vocabularies. Model families differ, and for some languages the difference between two tokenizers is larger than the difference between two providers' prices.
- Watch output caps. Truncation in the middle of an answer reads as the product being broken, not as a billing setting.
The question I keep coming back to
The gap closed for Bengali, Hindi, Thai and Urdu between two tokenizer generations, and it closed because someone spent vocabulary on them. It did not close for Amharic. Nothing about Ge'ez script makes it harder than Devanagari; there is simply less Amharic text on the internet, and so less reason for a frequency count to notice it.
Which means the languages that get fixed are the ones with enough digital presence to show up in a corpus, and the languages that most need fixing are the ones that do not. Frequency-based allocation gives most to those who already have most. That is not a bug in the algorithm. It is the algorithm working exactly as specified, and it is worth being clear-eyed that this is what we chose, even if nobody remembers choosing it.
Tokenize your own language in the token calculator, or read why parameter count is a bad predictor of VRAM for the other place where an intuitive number turns out to be the wrong one.