VRAM calculator

Speed is a bandwidth-bound estimate for single-stream decoding, not a benchmark. Prompt processing is compute bound and behaves differently.

How to read the three-part breakdown

Almost every "will it fit" argument on the internet is really an argument about which of three numbers someone forgot. The download size on a model card is one of them. The other two only exist once the model is loaded and answering, which is exactly when running out of memory hurts.

Take Llama 3.1 8B at Q4_K_M with an 8K context, one request at a time, F16 cache. That is the most ordinary local setup there is, and here is every byte of it:

weights  = 8.03e9 params x 4.85 bits / 8      = 4.87e9 bytes = 4.53 GB
kv       = 2 x 32 layers x 8 kv-heads x 128 dim
           x 8192 tokens x 2 bytes             = 1.07e9 bytes = 1.00 GB
overhead = 0.65 + 0.08 x (4.53 + 1.00)                        = 1.09 GB
                                                                -------
total                                                            6.63 GB

The weights are the only part you can look up. They are pure arithmetic: parameter count times bits per weight, divided by eight. Nothing about your workload changes them. If you never touched the other two numbers you would conclude that an 8B model at 4 bits needs 4.5 GB and fits comfortably on a 6 GB card. It does not.

The KV cache is the part that moves. Every token in the context has a key and a value vector stored for every layer, so the model can attend to the past without recomputing it. That is what makes generation fast, and it is why the cache is not optional. It scales linearly with context, linearly with concurrent requests, and it is decided by the attention layout rather than the parameter count. Here it costs exactly 0.125 GB per 1024 tokens, which is small because Llama 3.1 uses eight key-value heads for thirty-two query heads.

Runtime overhead is the part nobody budgets for and everybody hits. Compiled kernels, CUDA or Metal context, activation and logit buffers, the allocator that cannot pack blocks perfectly, and the sampler working set. This calculator models it as a fixed 0.65 GB plus 8 percent of everything else. It is a deliberate approximation, but the shape is right: overhead does not vanish on small models and it does not stay constant on large ones.

One unit note, because it causes real confusion. Every GB on this page is computed by dividing by 1024 to the third power, so it is technically a gibibyte. That is the same unit your GPU driver reports, which is why a card sold as 24 GB shows around 24564 MiB. Model files on Hugging Face are usually listed in decimal gigabytes, so a 5.0 GB download shows up here as roughly 4.65 GB. The gap is about 7 percent and it goes in your favour, not against you.

Choosing a quantization level

Quantization stores each weight in fewer bits. A model released in BF16 spends two bytes per parameter; a 4-bit quantization spends a bit over half a byte. Nothing else about the model changes, which is why quantization is the single biggest lever you have over whether something fits.

Here is one model, Llama 3.3 70B, at every level this calculator offers. Weights are the file on disk; the other two columns are the full loaded footprint including cache and overhead at a single stream with an F16 cache.

Three things in that table are worth staring at. First, the nominal bit count is a lie in your favour and against it. Q4_K_M is not 4 bits per weight, it is 4.85, because the K-quant formats keep attention and embedding tensors at higher precision where damage is worst. Q2_K averages 3.35 bits for the same reason. If you size a model as parameters times four over eight you will underestimate by roughly a fifth.

Second, the returns are steeply diminishing on the way down. Going from BF16 to Q8_0 halves the model and costs you nothing measurable. Going from Q8_0 to Q4_K_M nearly halves it again for a quality cost most people cannot detect in ordinary use. Going from Q4_K_M to Q3_K_M saves 7.8 GB and starts to show up as worse instruction following, weaker code and more arithmetic slips. Below that you are trading capability for capacity in a way you will notice.

Third, the damage from aggressive quantization is not distributed evenly across model sizes. A 70B at Q2_K is still a usable model, because there is enormous redundancy to destroy. A 3B at Q2_K frequently produces broken output. The rule of thumb that survives contact with practice: prefer a larger model at a lower quantization down to about 4 bits, and prefer a smaller model at 4 bits or better below that. A 70B at Q4_K_M beats a 32B at Q8_0 for most tasks; a 70B at IQ2_XXS usually does not.

One special case worth knowing. Some models are published natively in a low-bit format rather than quantized down afterwards. gpt-oss ships in MXFP4 at 4.25 bits per weight, and because the training accounted for it, the quality is not comparable to a 4-bit squeeze of a BF16 release. That is why the calculator only offers MXFP4 for models that actually ship in it, and why gpt-oss 120B lands at 63.4 GB rather than the 250 GB a 117B model would need at full precision.

Why context length is a hardware decision

People treat context length as a feature flag: the model card says 128K, so set it to 128K. It is not a feature flag, it is a memory allocation, and on some architectures it is the largest single line in the budget. Here are four models at Q4_K_M, single stream, F16 cache, with the total loaded footprint at four context lengths.

Llama 3.1 8B quadruples in size between 2K and 128K, from 5.8 GB to 22.8 GB, which is the difference between running on almost anything and not fitting a 24 GB card. Command R 35B is the extreme case: it starts smaller than Qwen3 32B at 2K and ends up nearly four times larger at 128K, because it uses full multi-head attention with 64 key-value heads and pays 1.25 GB of cache per 1024 tokens. Gemma 3 27B barely moves, because five of every six of its layers only attend to a 1024-token window and therefore stop growing almost immediately.

The practical consequence is that the context length you configure should be the context length you actually use, not the maximum the model supports. If your prompts are 3000 tokens and your answers are 800, setting 32K wastes memory you could have spent on a better quantization. Llama 3.1 8B at Q4_K_M on a 24 GB card can hold a 64K context, but not 128K, because 128K needs 22.8 GB and the card only leaves about 22.3 GB after the display and driver take their share. Dropping the cache to Q8 gets you there. Nothing else has to change.

It matters which runtime you use, too. llama.cpp allocates the cache for the context size you pass and nothing more, so a small context genuinely gives the memory back. vLLM instead claims a fraction of the card up front and manages a paged pool inside it, so nvidia-smi will show the card nearly full no matter what you set. The number this calculator gives you is the working set you need, which maps directly onto the llama.cpp behaviour and onto the pool vLLM has to be able to allocate.

The cache is decided by architecture, not by size

The single most useful fact about KV cache is that parameter count barely predicts it. What predicts it is how many key-value heads a model keeps, how wide each one is, how many layers it has, and whether it does anything clever. Here is the cache cost per 1024 tokens for models that span three orders of magnitude in parameter count.

Read the top and bottom rows together. A 671B model has a smaller cache than a 35B one, and a 3.8B model has a cache four and a half times larger than a 27B one. Phi-3.5 mini at a full 128K context needs 54.8 GB in total, of which only 2.2 GB is weights. That is a model people describe as tiny needing more memory than a 32B.

Four mechanisms explain the whole table. Grouped-query attention shares one key-value head across several query heads, which is the industry default now and typically cuts cache by four to eight times. Sliding-window attention gives most layers a fixed window so their cache stops growing past it, which is why Gemma 3 and gpt-oss flatten out. Multi-head latent attention, used by DeepSeek, compresses keys and values into a single 576-element vector per token per layer instead of storing them in full. And plain multi-head attention, still used by older releases such as Command R and Phi-3.5, does none of this and pays for every head.

If you want the long version of this argument with the derivations, it is written up separately in why parameter count is a bad predictor of VRAM.

When KV cache quantization is worth it

The cache can be quantized just like the weights. F16 spends 2 bytes per stored element; the Q8 option here averages 1.06 bytes and Q4 averages 0.56, the overhead coming from per-block scales. Every serious runtime supports it: llama.cpp exposes it as cache-type-k and cache-type-v, vLLM as a kv-cache-dtype flag.

The question is never whether it saves memory. It is whether the saving is large enough to matter, and that depends entirely on how much of your footprint is cache in the first place. At a 32K context with Q4_K_M weights:

Command R gains 20.3 GB from a single flag, which is the difference between needing two 48 GB cards and needing one. Llama 3.3 70B gains 5.1 GB from the same flag, because grouped-query attention already made its cache small. The pattern holds generally: KV quantization is transformative on multi-head models at long context and marginal on grouped-query models at short context. Check the KV cache segment in the breakdown above before you reach for it. If that bar is a thin sliver, the flag will not save you.

The cost is real but narrow. Q8 cache is close to free in quality terms and worth enabling by default on any long-context workload. Q4 cache degrades exactly the thing long context is for: precise recall of specific facts buried deep in the input. Summarization and open-ended chat tolerate it well; retrieval over a long document, needle-in-haystack lookups and multi-file code reasoning do not. There is also an asymmetry worth knowing, which is that keys are more sensitive to quantization than values, so if your runtime lets you set them separately, a Q8 key cache with a Q4 value cache is usually better than Q4 on both.

A rule that will not steer you wrong: quantize weights first, cache second. Q4_K_M weights with an F16 cache beats Q6_K weights with a Q4 cache at the same total size, in almost every case, because weight quantization has been studied and tuned far more thoroughly than cache quantization has.

Two 24 GB cards are not one 48 GB card

Splitting a model across GPUs works, and for large models it is the only affordable route. But the arithmetic is not addition, and the difference shows up in two places: usable capacity and decode speed.

On capacity, a single card loses a slice to the driver, the display output and allocator fragmentation. This calculator assumes about 93 percent is usable on a consumer card and 95 percent on a workstation or datacenter card with no display attached. A multi-card split loses more, around 90 percent, because each card carries its own runtime context and its own copy of the activation buffers, and because the layer split almost never divides evenly. Two RTX 4090s therefore give you about 43.2 GB of working space where a single RTX 6000 Ada with the same nominal 48 GB gives 45.6 GB.

That 2.4 GB gap decides real cases. Llama 3.3 70B at Q4_K_M with an 8K context needs 46.4 GB, so it misses both. Switching the cache to Q8 brings it to 45.1 GB, which fits the single 48 GB card and still misses the pair. To land it on two 4090s you have to drop to Q4_0 weights as well, at 42.5 GB. Same nominal memory, two extra compromises.

On speed, the usual expectation is that two cards are twice as fast. For single-stream decoding they are not, and often they are slower than one card would be if the model fit. The default split in llama.cpp and in most single-node setups is pipeline parallelism: layers 0 to 39 live on card one, 40 to 79 on card two. Generating a token walks through all eighty layers in order, so exactly one GPU is doing useful work at any moment while the other waits, plus a hidden-state transfer over PCIe at the boundary. You have bought capacity, not throughput. This calculator models that as a drop in bandwidth efficiency from 0.75 on a single card to 0.6 across a split, which puts two RTX 4090s at roughly 13 tokens per second on a 70B and a single RTX 6000 Ada at roughly 16, despite the pair having the higher raw bandwidth per card.

Tensor parallelism, which vLLM and TensorRT-LLM use, is the alternative: every layer is sliced across all cards and they work simultaneously. That genuinely scales decode speed, but it needs an all-reduce between cards on every layer, so it wants NVLink or at least full x16 PCIe lanes to each card. On a consumer motherboard running the second card at x4, tensor parallelism can be slower than pipeline parallelism. It also requires the number of attention heads to be divisible by the number of cards, which is why three-card setups are awkward.

The practical summary: add cards to fit a model you otherwise could not run, not to make a model you can already run go faster. And if you are choosing between one big card and two small ones at the same price, the single card wins on simplicity, power draw, usable capacity and single-stream latency. Multiple cards win when you are serving concurrent requests, where the batching keeps every card busy.

What happens when you offload to system RAM

If a model does not fit, llama.cpp and Ollama will happily keep some layers in system RAM and run those on the CPU. It works. It is also the point where people conclude their hardware is broken, because the speed does not degrade gently.

The reason is that decoding is bound by memory bandwidth, not by compute. To produce one token the runtime reads every active weight and the whole cache once. For Llama 3.3 70B at Q4_K_M with an 8K context that is about 45.5 GB of reads per token. Divide that by the bandwidth of whatever memory holds it and you have your ceiling before any compute has happened.

An RTX 4090 reads at 1008 GB/s. Dual-channel DDR5 in a typical desktop reads at roughly 90 GB/s. That is an eleven-fold difference, and it is the whole story. If the 70B fitted entirely in the 4090 it would decode at roughly 17 tokens per second. It does not fit: 22.3 GB of the 46.4 GB is resident, so 52 percent of every token read comes from system RAM at a ninth of the speed. The result is about 2.6 tokens per second. You kept 48 percent of the model on the fast device and lost 84 percent of the speed.

The same shape applies at every scale. Llama 3.1 8B reads 5.9 GB per token; on a 4090 that is roughly 127 tokens per second, and entirely on CPU with 90 GB/s of RAM it is roughly 11. Qwen3 32B at Q4_K_M needs 22.8 GB, so on a 16 GB RTX 5080 about 65 percent stays resident and the estimate falls from roughly 33 tokens per second to roughly 8.

Two details make the real experience worse than these numbers suggest. Prompt processing on CPU layers is compute bound rather than bandwidth bound, so a long prompt can take minutes before the first token appears even when the streaming rate afterwards looks tolerable. And offloading is not free on the GPU side either, since the runtime still needs working buffers for the layers it does host.

When offloading is still the right call: batch jobs you are not waiting on, models where a couple of tokens per second is genuinely enough, and the case where only a handful of layers spill over. Ten percent offloaded costs you far less than fifty percent. When it is the wrong call: anything interactive. A smaller model at a good quantization will almost always serve you better than a larger one crawling through your DIMMs, and if you need the capability of the larger model, calling it over an API is faster and cheaper than making your desktop pretend.

Apple Silicon and unified memory

Apple Silicon has no separate VRAM. The CPU and GPU share one pool of LPDDR on the same package, which means a Mac with 128 GB of memory can load a model that no consumer GPU on the market can hold. That is a genuine advantage and it is why Macs are common in local-inference circles. It comes with two qualifications people discover late.

The first is that you do not get all of it. macOS reserves memory for everything else the machine is doing and caps what the GPU may wire down; this calculator assumes about 75 percent is available to a model, so 128 GB of unified memory becomes roughly 96 GB of budget and a 16 GB Mac mini becomes roughly 12 GB. The limit can be raised with the iogpu wired limit setting, but pushing it near the physical total risks the system swapping, and swapping a model is worse than not loading it.

The second is bandwidth. Unified memory is wide but not GDDR wide. An M4 Max reads at about 546 GB/s against 1008 GB/s on an RTX 4090 and 1792 GB/s on an RTX 5090, and the base M4 in a Mac mini reads at about 120 GB/s, which is a third of what a five-year-old RTX 3060 manages. On Llama 3.1 8B at Q4_K_M that works out to roughly 60 tokens per second on an M4 Max, roughly 13 on a base M4 Mac mini, and roughly 45 on the RTX 3060. Capacity and speed are separate purchases, and Apple sells a lot of the first per dollar and relatively little of the second.

Prompt processing is the sharper edge. Decoding is bandwidth bound, where Apple is merely behind; prefill is compute bound, where the gap against a modern NVIDIA card with dedicated tensor cores is much wider. A 30000-token document pasted into a chat on a Mac can sit for a noticeable stretch before anything streams, even though the streaming itself then looks fine. If your workload is long prompts and short answers, that is the number to test before you buy.

Where a Mac wins outright is the middle of the range that discrete cards cannot reach at any sane price. Llama 3.3 70B at Q4_K_M fits comfortably in a 128 GB M4 Max at roughly 8 tokens per second, and a 192 GB M2 Ultra runs it at roughly 11 while also being able to hold models that need well over 100 GB. Getting there with NVIDIA hardware means several cards, a motherboard with the lanes to feed them, and a power supply to match. Use MLX rather than a generic backend when you can, since it is written for this memory model and consistently beats the alternatives on the same machine.

Serving more than one person at a time

Everything above assumes one request at a time. The moment you put the model behind an endpoint that other people use, the weights stay shared but every concurrent request carries its own KV cache. That turns the cache from a side item into the dominant cost. Llama 3.1 8B at Q4_K_M with an 8K context, at increasing concurrency:

At sixteen concurrent users the cache is larger than the model. At thirty-two, an 8B model needs more memory than a 70B does for a single user. This is the number that surprises teams who prototyped on a laptop and then wondered why the server fell over with eight people on it.

The upside is that concurrency is where GPUs stop being wasted. Single-stream decoding leaves most of the compute idle because it is waiting on memory; batching amortizes one weight read across many sequences, so total throughput climbs far faster than memory use. This is also the case where a paged cache earns its keep, since vLLM allocates cache in blocks as sequences actually grow rather than reserving the worst case for every slot. If you are sizing a server, size for the cache at your real concurrency and your real median prompt length, not for the maximum context multiplied by the maximum users.

Common mistakes when sizing a GPU

Questions people actually ask

How much VRAM do I need to run a 7B or 8B model?

About 6.6 GB at Q4_K_M with an 8K context, which fits an 8 GB card with a little room. At BF16 the same model needs 17.9 GB and wants a 24 GB card. If you want the full 128K context it climbs to 22.8 GB, so context matters as much as precision here.

Why does my 8 GB model refuse to load on an 8 GB card?

Three reasons stack. The driver and desktop have already taken about 7 percent of the card, the KV cache is allocated on top of the weights, and the runtime needs roughly 0.65 GB of context and kernel space before it does any work. The file size is the one number that does not include any of that.

Can a 24 GB card run a 70B model?

Not in VRAM alone. Llama 3.3 70B needs 46.4 GB at Q4_K_M and 8K, and even at IQ2_XXS with a 2K context it still needs 22.5 GB against roughly 22.3 GB available. You can offload to system RAM and get a couple of tokens per second, or run a 32B instead and get thirty.

Is Q4_K_M actually good enough?

For nearly everything, yes, and it is the default for good reason. It stores 4.85 bits per weight rather than a flat 4, keeping attention and embedding tensors at higher precision. The measurable gap to Q8_0 is small and the memory saving is close to half. Drop below it only when the alternative is not running the model at all.

Is a bigger model at low precision better than a smaller one at high precision?

Down to about 4 bits, yes. A 70B at Q4_K_M beats a 32B at Q8_0 on most tasks at a similar footprint. Below 3 bits the ordering flips for small and mid-size models, because there is less redundancy to give away. Very large models tolerate 2-bit formats far better than small ones do.

Why is the KV cache on a 3.8B model bigger than on a 27B one?

Because Phi-3.5 mini uses full multi-head attention with 32 key-value heads, at 0.375 GB per 1024 tokens, while Gemma 3 27B gives five of every six layers a 1024-token sliding window. At 128K that is 48 GB against 10.4 GB. Cache size follows the attention layout, not the parameter count.

Should I quantize the KV cache?

Enable Q8 whenever your context is long; it is close to free. Reach for Q4 only when the cache is a large share of your footprint and your workload is not recall-heavy. On Command R 35B at 32K, Q8 saves 20.3 GB. On Llama 3.3 70B at the same context it saves 5.1 GB, because grouped-query attention already did the work.

Do two 24 GB cards equal one 48 GB card?

No. You get roughly 43.2 GB usable instead of 45.6, because each card carries its own runtime context and the layer split is uneven. Single-stream speed does not improve either, since pipeline parallelism means one card computes while the other waits. Two cards buy capacity and, under concurrency, throughput.

How fast will it actually run?

Divide memory bandwidth by the bytes read per token. Llama 3.1 8B at Q4_K_M reads 5.9 GB per token, so an RTX 4090 at 1008 GB/s lands near 127 tokens per second after efficiency losses. That estimate holds for decoding only. Prompt processing is compute bound and follows a different curve.

What actually happens if I offload to system RAM?

Speed falls off a cliff rather than sloping down. Llama 3.3 70B with 48 percent of its layers on an RTX 4090 and the rest in DDR5 decodes at roughly 2.6 tokens per second instead of the 17 it would manage fully resident. A small spill is survivable; half the model is not.

Is a Mac a good machine for local inference?

For capacity per dollar, yes: a 128 GB M4 Max offers roughly 96 GB of usable budget and runs a 70B at around 8 tokens per second. For raw speed, no: 546 GB/s of bandwidth against 1792 on an RTX 5090, and prompt processing lags further still. Buy one for large models and quiet operation, not for latency.

Do mixture-of-experts models save memory?

They save time, not space. Qwen3 30B-A3B still needs 20.1 GB because every expert must be resident, but it only reads about 3.3B parameters per token, so it decodes several times faster than a dense 32B in the same footprint. Judge an MoE model by its total for fitting and by its active parameters for speed.

Does flash attention reduce VRAM?

It removes the large intermediate attention matrix, which is a real saving during prefill on long prompts, and it makes prompt processing faster. It does not shrink the KV cache, which is the part that grows with your context. Turn it on regardless; just do not expect the totals on this page to move.

Why does nvidia-smi show the card full when I set a small context?

You are probably running vLLM, which claims a fixed fraction of the card at startup and manages a paged cache pool inside it. llama.cpp allocates only what the context you asked for needs. The figures here describe the working set, which is what vLLM has to be able to allocate and what llama.cpp reports directly.

How much VRAM do I need to fine-tune rather than run a model?

Far more, and this calculator does not model it. A full fine-tune holds weights, gradients and optimizer state at roughly 16 bytes per parameter, so an 8B model lands near 130 GB before activations. QLoRA on a 4-bit base is much closer to the inference footprint plus activation memory, which is why it is the only realistic option on a single consumer card.

Do vision models change the math?

The vision encoder is included in the parameter count of the multimodal models listed here, so the weights figure is right. What the totals do not capture is that images are expanded into a large number of tokens before they reach the language model, so a few high-resolution images can consume more context, and therefore more cache, than a page of text.

What if nothing fits?

Reduce context first, since it is usually the cheapest concession, then quantize the cache to Q8, then step the weights down one tier. If you are still short, a smaller model at a sane quantization beats a larger one crawling out of system RAM. Or call the same open-weight models over the free API and let somebody else own the hardware.