Jev was 12.7x faster and 12 points less accurate

What Jev is, in plain terms

A language model answers by producing one token, feeding it back in, producing the next, and so on. That loop is where the time goes.

Before the loop starts, the model makes a single pass over your prompt. This is called the prefill. At the end of it the model already holds a score for every token in its vocabulary: its opinion about what should come next. Generation is just the act of picking from those scores and going round again.

Jev reads the scores and stops. If the answer can only be A, B, C or D, there is no need for the model to write anything down. You look at the four scores belonging to those four tokens, turn them into probabilities, and take the largest.

What it is for: anything with a fixed set of answers. Routing a request to a handler, classifying a ticket, a yes or no gate inside an agent loop, choosing which tool to call. What it is not for is anything whose answer is a sentence, and its authors say so plainly: it is "lame at answering long-answer type questions".

One property is easy to miss and is arguably the real prize. You get a probability rather than a pick. "87% sure this is billing" lets you route the confident cases automatically and send the rest to a person. A generated letter gives you no such number, and we will come back to whether the probability can be trusted.

How to test it

Hold three things constant: the same model, the same prompts, the same machine. Then run the question two ways.

With Jev. One forward pass, read the logits at the final position, keep only the token ids of the answer letters, softmax over those, take the argmax. Nothing is generated:

enc = tok(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    logits = model(**enc).logits[0, -1, :]        # the prefill, and stop

scores = [logits[letter_id[k]].item() for k in "ABCD"]
probs  = torch.softmax(torch.tensor(scores), dim=0)
answer = "ABCD"[int(torch.argmax(probs))]        # plus a probability, free

Without Jev. Generate, then read the answer out of the text. This arm needs to be run twice, because how much Jev saves depends entirely on how long the answer would otherwise have been:

• Constrained: "Reply with the single letter only." The model emits about two tokens.
• Unconstrained: "Which category is it?" This is what people actually write, and the model answers in sentences.

Sixty support requests, four categories, fifteen each. Qwen2.5-1.5B-Instruct in float16 on an M-series laptop. Both methods see identical items in the same order, alternating, so neither gets a warmer machine than the other.

What we measured

Read the first column and Jev looks extraordinary: one tenth of a second against one and a half. Read the second and it looks like a bad trade: it got twelve to thirteen more questions wrong out of sixty.

Both readings are correct, and the useful conclusion is the one in the middle. Most of the 12.7x is not Jev's doing. It is the cost of letting a model ramble. Simply writing "reply with the single letter only" takes 1,401ms down to 142ms, which is 90% of the available saving for the price of six words. Jev collects the remaining 1.29x on top.

Why it loses accuracy at all

This surprised us, because greedy generation picks the highest-scoring token from the very same numbers Jev reads. The two ought to agree almost always. They agreed on 63% of items.

The gap is that generation is not obliged to put the letter first. Left to itself the model may open with "The", or a space, or "Category", and reach the letter a token later, having by then conditioned on its own opening. Jev cannot do that: it forces the answer to be one of four tokens at the one position it looks at. When the model wanted a run-up, Jev takes the answer before the run-up has happened.

The probability is the weakest part

The free confidence score is the feature we most wanted. On this model it barely works:

Six points apart. A threshold that keeps most of the correct answers keeps nearly all of the wrong ones with it, so "route the confident ones automatically" does not survive contact with this model. Autoloops found the same thing and named the fix: their smaller Gemma-4B was "overconfident" at 99% where 70% was right, and needed a LoRA calibration pass to be usable. Our result is an argument for that calibration step, not against the technique.

Can you do this through a hosted API?

Mostly not, and this is the part that matters if you do not run your own weights. Jev needs the logits. The nearest thing an OpenAI-compatible API offers is max_tokens: 1 with logprobs, which still costs one decode step but gives you the distribution. We asked every provider our gateway routes to, one request each:

One provider out of nine, and even there the default model is a reasoning model whose first token was "We" with probability 0.866, not an answer. Point the same request at google/gemma-4-31b-it, the model family Autoloops adapted, and it does answer "A" first and return 20 alternatives, but that single token took 98 seconds to come back, which tells you about queueing rather than about Jev.

There is a second obstacle that is easy to miss and harder to route around: a reasoning model cannot be Jevved at all. Its first token is a thought, not an answer. We asked NVIDIA's default for a classification and the top of the distribution was "We" at p=0.866, with no answer letter anywhere in the top twenty. Jev reads the position before the model has started reasoning, and for a model whose whole method is to reason first, there is nothing there to read. The technique needs a model that answers immediately, which is a narrowing constraint in a year where reasoning is the default.

So: Jev is a thing you do to a model you control. If you are calling someone else's endpoint, the honest version of this article is the sentence "write a better prompt", which buys you 90% of the speedup and costs nothing.

What would change our answer

• A bigger model. 1.5B parameters is small, and the accuracy gap may be a small model's habit of needing a run-up rather than a property of the method. We tried a 3B and it thrashed on a 16GB laptop, which is its own kind of result but not a measurement.
• Calibration. The confidence number is the reason to want Jev, and untrained it is close to useless here. The LoRA pass Autoloops describe is the obvious next experiment.
• Our labels. Four of the sixty items are arguably ambiguous, since a login failure can be read as a bug. That caps every arm equally, so the comparison holds, but it is why no method here scores near 100%.
• Batching. Autoloops attribute much of their speedup to caching the prefill and answering questions in parallel. We measured one question at a time, which is the pessimistic case for Jev.

Jev's own materials are at autoloops.ai/jev, with the package at github.com/kushalpatil07/jevify and a LoRA adapter on Hugging Face. If you want to try the fixed-choice idea against a free endpoint first, our OpenAI-compatible API is free and the "single letter only" prompt is the part that works everywhere. Related: why your rate limit is measured in tokens, where a shorter answer also turns out to be the whole game.