Blog · 2026-08-26 · Open source

Every LLM API says it is OpenAI-compatible. None of them agree.

We run a free API that owns no models. Every request is forwarded to somebody else's free tier, and if that one refuses, to the next one. Twelve providers, one shape going in, one shape coming out.

Almost all of them advertise an OpenAI-compatible endpoint. It took about four months to learn that the phrase means seven different things, and that each difference is worth exactly one production incident.

The part of our gateway that absorbs all of it is now a package: openai-shim. MIT, no dependencies, 109 tests.
pip install git+https://github.com/waracci/openai-shim

The same request, four dialects

Here is one ordinary chat request with a system prompt, a tool and an image, as it leaves our gateway and as it has to arrive at four different upstreams. Click through them. The differences are not cosmetic.

What actually goes on the wire

What comes back

Every one of those returns an identical chat.completion object to the caller. That is the whole job.

Seven disagreements, and what each one cost

These are not hypotheticals. Each row is a thing that was broken in production, found in the request log, and fixed.

1. A 200 is not an answer

A safety filter with no message, a reasoning model that spent its entire token budget thinking, a stream that opens and closes in silence. All three arrive as a well-formed success carrying nothing, and a plain relay turns that into an empty box on somebody's screen. We had 49 of those in the log before we started reading the body instead of the status line.

2. A 413 is sometimes a rate limit

Groq answers a tokens-per-minute limit with 413 Request too large. Read literally that means the payload was too big, so it cools nothing, so a caller sending a long conversation buys a failed round trip on every single turn. 226 of them in three days, 194 from one application. Measured from our box: 20,000 characters go through on a quiet minute and fail on a busy one, so the size was never the problem.

3. "That model does not exist" is not about the provider

Groq decommissioned the two models that happened to sit at positions 0 and 1 of our list, which made them the default for every automatic request. A 404 correctly does not take a provider out of the chain, so every request bought a guaranteed 404 on the way past: 219 in 24 hours, and it would have been forever. Fixing the list fixes today. Remembering the answer is what makes the next retirement free.

4. "We do not serve your country" is about the provider

Gemini answers our server User location is not supported for the API use on every model, from every worker, on every request. That is a geographic block, not weather, and we were retrying it every fifteen minutes: 210 wasted round trips in three days, each one sitting in front of a real user's request, and not one of them could ever have succeeded.

5. A small max_tokens can silence a reasoning model

"Reply with exactly: OK" against gpt-oss-120b returned 24 completion tokens and an empty string. All 24 went to thinking. On a needle-in-a-haystack sweep we ran last week the same trap produced an empty answer 61 times out of 62, and the identical call with a low reasoning effort answered correctly in 66 tokens. We had briefly concluded the model was broken.

6. Nobody sets a charset on an event stream

The HTTP specification says to decode a text/* response with no charset as Latin-1, and Python's requests obeys it. Server-Sent Events are UTF-8 only, so every streamed non-ASCII character in 84 logged answers arrived mangled. One em dash became three characters. Swahili, Arabic and Japanese fared worse.

7. Being blind is invisible, being mute is not

A provider that cannot make tool calls returns prose where a tool call was expected, and you notice within seconds. A provider that cannot see images returns a fluent, confident description of an image it never received, with a 200 on it, and you may never notice. So the two defaults are deliberately asymmetric: tool support is assumed, vision has to be declared.

The one rule underneath all of it

Every fix above is the same fix wearing a different hat. An error code tells you what happened. Only the body tells you what it was about.

And the thing an error is about decides what to do with it. A rate limit is about capacity, so the provider steps out of the chain for a minute. A retired model is about the model, so the model steps out and the provider keeps its place: otherwise one caller pinning a dead name takes a provider away from everybody else. A country block is about the host, so it steps out for six hours. Any other 4xx is about the request that was just sent, so nothing happens at all, because cooling on it would punish every other caller for one malformed message.

Four sentences. Getting them wrong in either direction is a class of outage that looks, in a dashboard, exactly like normal traffic.

So we pulled it out

None of this is specific to us. Anyone routing across more than one model provider hits every item on that list eventually, usually in the same order. So the routing layer is now a package with no DreamPrompting in it.

from openai_shim import Client, presets

client = Client([
    presets.groq(),        # OpenAI-compatible already
    presets.anthropic(),   # a completely different API
    presets.ollama(),      # running on your laptop
])

print(client.chat("Name a bird that cannot fly.").content)

If Groq is rate limited, Anthropic answers. The Messages API has no system role, no tool role, requires max_tokens and streams a state machine instead of deltas. None of that reaches your code.

The part we are most pleased with is the smallest one. An adapter answers exactly one question: what does this upstream want, and what does its answer mean in OpenAI terms. It knows nothing about routing, and the router knows nothing about any upstream's dialect. Which means an endpoint that is not really an API at all, a query string in and a paragraph of prose out, becomes a first-class model in six lines:

from openai_shim import Provider, TextAdapter

Provider(
    id="chat", models=["ch.at"], keyless=True,
    adapter=TextAdapter(url="https://ch.at/", query_param="q"),
)

That now streams, reports usage, takes a message history and sits at the end of a failover chain as the provider that still answers on the day every API key in it has expired. Which is not a hypothetical either: it is why our gateway returned something useful before it had a single provider key configured.

What we deliberately left behind

A good extraction is defined by what stays. These are ours and they are not protocol translation, so they are not in the package:

The identity note. A model's weights do not encode which endpoint is serving them, so asked "which model are you", a Llama on Groq very often says ChatGPT. We attach a line telling it what it actually is. That is a product decision about honesty, not a compatibility problem, and a library that edited your system prompt without being asked would be a bad library.
Quotas, keys and rate limiting. Ours are tied to accounts, tokens and a database.
Logging and media storage. Same reason.
Provider lists as truth. The package ships presets, and the docs say plainly that they go stale, because the whole retired-model machinery exists precisely because no list survives contact with an upstream's release calendar.

One bug it found on its way out

The package has no dependencies, so its default transport is the standard library. The first time we pointed it at a real provider, Groq answered 403 error code: 1010 with a page of HTML. Nothing to do with the API: urllib introduces itself as Python-urllib/3.14, and a Cloudflare rule in front of the API rejects that string before the request gets anywhere near a model. Four months of running this in production had never surfaced it, because production uses a library that sets a User-Agent for you. Extracting something is its own kind of test.

Take it

pip install git+https://github.com/waracci/openai-shim · source · MIT

Or skip the library and use the gateway it came out of. Same failover, none of the setup: the API is OpenAI-compatible and free, and the playground needs no account.