The future of AI, read from 35,000 API requests

Seven weeks in four numbers

"Typical" is the median. September runs to the 14th, and the log begins on 23 July, so "late July" is nine days.

The fourth tile is the one that makes the other three worth reading. On a service this size, a single enthusiastic user can move every average on the page, and in late July one caller accounted for nearly half of all tokens. By September the heaviest caller was down to 7.6%. The trends below got broader as they got bigger, which is the opposite of what one person's side project looks like.

Bet one: long context stops being a feature and becomes the load

Here is the single clearest movement in the log: the share of requests carrying more than 8,000 tokens of prompt.

Roughly forty times over in seven weeks, and it is not one person: the number of callers who sent at least one prompt over 8,000 tokens went from 4 of 14 to 42 of 88. The typical request grew much more gently, from 194 to 543 tokens. So the shape is not "every prompt got bigger". It is that the long tail stopped being unusual.

Our reading is that people have stopped treating a model as something you ask a question and started treating it as something you hand a document, a codebase or a whole conversation history to. Context windows of a million tokens have existed for a while. They even work. What changed is that ordinary callers started filling them.

The consequence nobody plans for is that limits and prices are about to be felt in tokens rather than requests. We found this the hard way: one provider allows us a thousand requests a minute and our busiest minute in the last month used 37 of them, yet it refuses us every day, because the budget that actually binds is 8,000 tokens a minute. As prompts grow, that is the wall more people will walk into.

Bet two: models become disposable, and the interface is what lasts

Between 20 and 28 August, five models we were routing traffic to stopped answering. Groq decommissioned two, NVIDIA ended two, and a stealth preview model closed its free window and, on the way out, revealed which lab had built it. None of that was a crisis for anyone using the gateway, because none of them had written a model name into their code that only one provider could serve.

It also does not end when a model dies. Cloudflare still advertises a Llama model that answers every call with 410 deprecated, and lists three more that a normal account is not allowed to call. We wrote that up in Cloudflare Workers AI lists 31 text models, we could call 28. Catalogues are drifting away from reality faster than anyone is keeping them in step.

So the durable thing is not the model. It is the shape of the request. Almost every provider now speaks some version of the same chat completions format, even if they disagree about seven details of it, and that shared shape is what lets an application outlive the model it launched on. Our bet is that within a couple of years, hardcoding a single model name will look the way hardcoding a server's IP address looks now: fine for a demo, a liability in production.

Bet three: the caller becomes a program

In August, the models serving 7 of our callers made at least one tool call, which is a model deciding to run a function rather than answer in prose. In the first two weeks of September that was 24 callers, and tool calls were 7.8% of all successful responses, up from 2.3%. A tool call only happens when there is software on the other end prepared to act on it, so every one is evidence of an agent loop rather than a person typing.

We want to be careful not to oversell this one, because the same log contains a counterweight. Requests sent from web browsers went from almost nothing to a fifth of traffic over the same period, from 47 different accounts, nearly all of them web apps calling the API straight from a browser tab rather than our own chat page. A browser tab usually means a person is on the other side of it. People have not left. What is changing is who does the typing: increasingly a person sets a program going, and the program holds the conversation with the model.

Bet four: free AI stops being a price and becomes a routing problem

Everything we route to is somebody's free allowance, and every one of those allowances has moved in the last month. Hugging Face's inference router lists 327 model and provider pairs and flags none of them as free. The stealth model's free week ended on schedule. Cloudflare's free tier is a daily allowance on a paid product. Google's free tier refuses our server outright because of the country it sits in.

No single free tier is stable enough to build on. What is stable is the sum of them: at any moment some provider has headroom, and a router that can find it is the only version of free that lasts. We think "free AI" in future means exactly that, an aggregate that keeps working while every individual component comes and goes, plus a way for people who outgrow it to bring their own key instead of paying a middleman.

Bet five: small models answer most of the questions

The typical request in our log is 543 tokens of prompt. That is a paragraph or two, and most of those requests do not need the largest model on the market. They need a fast one that is cheap enough to call without thinking about it.

The prices already point that way. On Cloudflare, IBM's Granite 4.0 micro costs $0.017 per million input tokens, seventeen times less than the 70 billion parameter Llama next to it, and answered our test prompt 20 milliseconds slower. A 17 billion parameter model on the same platform read a solid-colour image correctly in 0.27 seconds. Our bet is a split: large models for the long tail that bet one describes, and small ones for everything else, chosen per request rather than per application.

What would prove us wrong

These are bets from a small window, and it is worth being exact about how small.

• The sample is modest. 88 distinct callers in September. Trends at that size can reverse with a handful of people.
• The window is short. Seven weeks, beginning 23 July. We do not have a year of history to show these are trends rather than a season.
• A free service attracts experiments. People try long prompts and agent loops on something that costs them nothing, so our traffic probably overstates both compared with a paid service.
• Tool calls were not recorded before August. July's number is unknown, not zero, which is why the tool call comparison starts in August.

Concretely: if the share of long prompts falls back under a few percent by the end of the year, bet one was a burst of experimentation. If the number of callers making tool calls stops growing while total callers keep rising, bet three was a fashion. We will run the same queries again and publish whichever way they come out, which is more than most predictions about AI commit to.

curl https://dreamprompting.com/api/v1/chat/completions \
  -H "Authorization: Bearer $DP_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"auto","messages":[{"role":"user","content":"hello"}]}'