We peak at 4% of our rate limit and hit it every day

The number that actually binds

Every response from that provider carries its own explanation, in headers most clients never look at. Here is what ours says on an idle key:

x-ratelimit-limit-requests:      1000
x-ratelimit-remaining-requests:   999
x-ratelimit-limit-tokens:        8000     <- this one
x-ratelimit-remaining-tokens:    7923
x-ratelimit-reset-tokens:        577ms

Two budgets, refilling independently. The request budget is generous and we barely touch it. The token budget is 8,000 per minute for the whole key, shared by every caller we route, and that is the one we spend all day.

This is the normal shape of a free or entry tier, and it is almost never the number in the marketing copy. Requests per minute is an easy number to publish and a hard number to hit. Tokens per minute is the real meter. If you are building against any LLM API and you have not read your own rate limit headers, the number you are planning around is probably not the number that will stop you.

The same refusal, two status codes

Of those 813 refusals, 490 arrived as 429 and 323 as 413. Same key, same provider, same exhausted token budget. The code depends roughly on whether you tripped the limit by accumulation or by sending one large thing.

That split matters more than it looks, because the two codes mean opposite things to a router. A 429 says "you are going too fast", which argues for waiting. A 413 says "this payload is too big", which argues for not sending it again. Getting a 413 that is really a rate limit will make a well-written client do the wrong thing: it will permanently give up on a request that would have worked thirty seconds later. Getting a 429 that is really a size problem does the reverse, and the client retries forever.

We now classify on the body rather than the status line. The body is unambiguous where the code is not:

{
  "error": {
    "message": "Request too large for model `openai/gpt-oss-120b` ... on
                tokens per minute (TPM): Limit 8000, Requested 8478,
                please reduce your message size and try again",
    "type": "tokens",
    "code": "rate_limit_exceeded"
  }
}

Some requests can never succeed

Read that message again: Limit 8000, Requested 8478, please reduce your message size. The per-minute budget is 8,000 tokens and this single request wanted 8,478. There is no quiet moment that fixes it. An empty bucket still holds 8,000 tokens, and the request still needs more than that. It will fail at 3am on a bank holiday exactly as it fails at peak.

We had a caller sending prompts of around 11,000 tokens on a loop. Every single one of their requests bought a guaranteed refusal from the fastest provider in the chain before failing over to a slower one. Not most. Every one, deterministically, forever.

Once you can predict a failure, paying for it is a choice. So we stopped sending them.

Sizing the skip without guessing

To skip a request before sending it, you have to estimate its token count from the only thing you have cheaply, which is the character count. The folk constant for this is four characters per token. We had 30 days of our own traffic with both numbers recorded, so rather than trust the folklore we measured it across 11,618 requests:

The median landed on 3.99, which is the folk constant to two decimal places. That is a satisfying result and it is not the point. The point is the spread: the same 32,000 characters is 8,000 tokens of prose, 13,000 tokens of minified JSON, or 1,600 tokens of a repeated string. Any single threshold is going to be wrong for somebody, so the question is not "what is the right ratio" but "which way should it be wrong".

We picked the median, giving a cap of 32,000 characters, then went back to the same 30 days and asked what that rule would have done:

One false skip against a class of guaranteed failures is an easy trade. It is also a trade we could only price because the answer was already sitting in the request log. Setting this number from intuition would have taken two minutes and we would still not know whether it was right.

Try it on your own prompt

Paste anything in and see where it lands against an 8,000 token per minute budget. The estimate uses the 3.99 median above, and the range uses the p5 and p95, which is roughly the honest uncertainty on a character-count estimate.

Character counting is an estimate, not a tokenizer. It is good enough to decide whether to send something and not good enough to bill on.

What the fix did not fix

This is the part that would be easy to leave out. Of the 813 refusals over 30 days:

So the change removes 55 doomed round trips out of 813 refusals. The other 758 are a provider correctly telling us its bucket is empty, and failing over to the next provider is exactly the right answer to that. We are not going to engineer our way out of a shared free tier being finite.

The reason it is still worth doing is the shape of the trend rather than the size of the number. All 55 landed in the last eight days. There were none at all before that. Long contexts are not a rare event any more, they are what people are starting to send, and a class of failure that grows with the population is worth closing while it is still small.

If you are the one sending 11,000 token prompts

You are not doing anything wrong. A long context is what the context window is for, and our shared free-tier allowance is simply not where that request belongs.

Two things help. The first is that our own routing now moves you straight to a provider whose budget can hold your payload instead of spending a round trip proving that one cannot. That happens automatically and there is nothing to configure.

The second is that if you already have an account with any of the providers we route to, you can add your own key and get your own token budget rather than a slice of a shared one. Requests on your key do not count against our daily quota, and the per-minute ceiling becomes whatever your account with that provider allows, which on a paid tier is usually a great deal more than 8,000.

Our gateway is free and OpenAI-compatible, and the routing layer that does all of the above is open source as openai-shim. The rate limit handling described here is in health.py, and it is there because of the numbers on this page rather than the other way round.