Why parameter count is a bad predictor of VRAM

Here are two models at a 128K context, both quantized to Q4_K_M:

The smaller model needs nearly twice the memory. It has one seventh the parameters. If you sized your hardware by parameter count you would have bought the wrong card.

This is not an edge case. It is what happens as soon as context length matters, because the weights are only one of three things competing for your VRAM, and the other two do not scale with parameter count at all.

What actually occupies the memory

Three components, and only the first is about model size:

weights  = parameters x bits_per_weight / 8
kv_cache = 2 x layers x kv_heads x head_dim x context x batch x bytes
overhead = ~0.65 GB + ~8% of the above

The 2 in the KV formula is for keys and values, stored separately. Notice what is missing from that line: parameter count. The cache is governed by layer count, key-value head count, head dimension and how many tokens you are holding. A model can be small and still be expensive to attend over.

The weights are the easy part

This is the part the rule of thumb gets roughly right. Llama 3.3 70B, weights only:

One detail people miss: Q4_K_M is 4.85 bits per weight, not 4. The K-quant formats keep some tensors at higher precision, so a nominal 4-bit quantization is about 21 percent larger than the name implies. Q8_0 is 8.5 bits, not 8, for the same reason. If you have been computing params / 2 for Q4, you have been underestimating by a fifth.

The KV cache is where the rule of thumb dies

Every token in your context has its keys and values cached so the model does not recompute them on each new token. That cache grows linearly with context, and its per-token cost varies by more than an order of magnitude between architectures.

Here is the KV cache alone, at a 128K context, fp16, single request:

A 671B model with an 8.6 GB cache. A 3.8B model with a 48 GB cache. The ordering has almost nothing to do with size. Four mechanisms explain the whole table.

Multi-head attention: the expensive default

Every query head gets its own key and value head. Command R 35B has 64 of each, so at 128K its cache reaches 160 GB, four times the size of its Q4 weights. Phi-3.5-mini is only 3.8B but caches 32 full heads, which is why it lands at 48 GB. MHA models look cheap at a 2K context and become unrunnable at 128K.

Grouped-query attention: the standard fix

Several query heads share one key-value head. Llama 3.3 70B has 64 query heads and 8 key-value heads, so its cache is one eighth of what MHA would cost. This is why a 70B model has a smaller cache than a 35B one: not size, just head sharing. Almost everything released since 2024 uses GQA.

Multi-head latent attention: the outlier

DeepSeek compresses keys and values into a shared low-rank latent vector and caches that instead. It stores 576 elements per token per layer rather than 2 x 128 heads x 128 dims, a 57x reduction. That is how a 671B model ends up with a smaller cache than an 8B Llama. Kimi K2 uses the same approach at 1T parameters.

Sliding-window attention: cheap long contexts

Some layers only attend to a recent window, so they never cache more than that window regardless of your context. Gemma 3 gives five of every six layers a 1024-token window, which cuts its 128K cache from a nominal 62 GB to 10.4 GB. gpt-oss alternates every other layer with a 128-token window and lands at 3 GB.

One caveat worth knowing: sliding windows do nothing until you exceed the window. At a 1K context Gemma 3 27B caches as much per token as a normal GQA model. The saving is entirely a long-context effect, which is exactly where it matters.

Same size class, very different requirements

If parameter count were a good predictor, models of the same size would need the same hardware. At a 32K context, Q4_K_M:

• Qwen3 32B: 29.3 GB. Fits a single 32GB card.
• Qwen2.5 32B: 29.3 GB. Same.
• Yi 1.5 34B: 29.7 GB. Same.
• Command R 35B: 65.2 GB. Needs two 48GB cards.

Four models within 3B parameters of each other, and one of them needs more than twice the hardware. The difference is 64 key-value heads instead of 8.

Context length is a hardware decision

The same model, Llama 3.1 8B at Q4_K_M, across context lengths:

One model, a 3.9x range, and a jump from an 8GB card to a 24GB card. Nobody in that comparison changed models. Allocating a 128K context because the model supports it is a decision to buy a bigger GPU.

Mixture of experts breaks the intuition the other way

MoE models are expensive to hold and cheap to run. Qwen3 30B-A3B needs 20.1 GB at an 8K context, so you budget for 30B of weights, but only about 3.3B parameters are active per token, so it generates at roughly the speed of a 3B model. Compare Qwen3 32B at 22.8 GB, similar memory, several times slower.

The practical rule: total parameters set your VRAM, active parameters set your speed. Anyone quoting a single number for an MoE model is answering only one of those questions.

The overhead nobody budgets for

Roughly 0.65 GB goes to the CUDA or Metal context, compiled kernels and the allocator before a single weight is loaded, and activation buffers and fragmentation add a further few percent. It is small, and it is exactly why an 8 GB model will not load on an 8 GB card, a result that surprises people who sized their download against their spec sheet.

What to do instead

Allocate the context you need, not the context the model supports. This is the single biggest lever, it is free, and it is linear. Dropping 128K to 32K on that 8B model returns 12 GB.

Check the key-value head count before you commit to a model. If query heads equal key-value heads, it is MHA and long contexts will hurt. A ratio of 8:1 is normal and healthy.

Quantize the cache when the weights are already as small as you want them. Llama 3.3 70B at 128K goes from 86.9 GB to 66.6 GB with a Q8 cache and 55.8 GB with Q4, which is the difference between two 48GB cards and one. Long-context recall degrades, so test it on your workload.

Count concurrency. Weights are shared but every simultaneous request carries its own cache. That 8B at 8K costs 6.6 GB for one user, 14.2 GB for eight, and 40.1 GB for thirty-two. Serving is a different sizing problem from running.

Reference numbers

The figures people usually want, at Q4_K_M with an 8K context:

• 8B: 6.6 GB. An 8GB card, tightly. A 12GB card comfortably.
• 70B: 46.4 GB. One 48GB card, or two 24GB cards.
• 70B at 128K: 86.9 GB. An 80GB accelerator will not quite do it at fp16 cache.
• 671B (DeepSeek R1): 419 GB. A multi-GPU node, though the cache is the least of your problems.

Every number in this post came out of our VRAM calculator, which carries the architecture data (layers, key-value heads, head dimension, latent size, window size and ratio) for 63 open-weight models rather than a parameter count and a multiplier. Drag the budget rail to your card's capacity and it lists what fits. If you would rather start from the model, the GPU finder ranks every card that can hold it, and the model finder filters by real memory footprint.

If it does not fit

You can offload layers to system RAM, but every offloaded layer is read over PCIe on each token and decode speed collapses, often to single-digit tokens per second. Usually the better answers are a smaller model, a shorter context, or not running it yourself.

Most of the models in this post are also callable over our free, OpenAI-compatible API: Llama, Qwen, DeepSeek, Gemma, Mistral and gpt-oss through one endpoint with a free key and no credit card. Worth confirming a model is right for your task before you spend on hardware to run it.