GPU finder

The reasoning behind the table

Why memory bandwidth, not compute, sets your tokens per second

Generating one token means reading every active weight in the model plus the whole key-value cache out of memory, running a small amount of arithmetic on each value, and writing back almost nothing. The arithmetic finishes long before the next batch of weights arrives. That is why a card with enormous compute and modest memory bandwidth feels sluggish on a single chat stream, and why the speed column in the table above is derived from bandwidth rather than from teraflops.

The arithmetic is simple enough to do in your head. Divide the memory bandwidth by the number of bytes the model reads per token and you have an upper bound on decode speed. Llama 3.1 8B at Q4_K_M holds about 4.5 GB of weights and roughly 1 GB of cache at an 8K context. An RTX 5080 moves 960 GB per second, so in a perfect world it would produce a couple of hundred tokens a second; in practice a runtime reaches somewhere around three quarters of peak, which is where the estimate of 121 tokens per second comes from.

The clearest way to see this is to hold capacity constant. Every card below has exactly 16GB, so every one of them holds the same 8B model at the same quantization with room to spare. The only thing that changes is how fast memory feeds the chip.

The spread is more than three to one on identical capacity, and price does not track it cleanly. The RTX 5070 Ti is cheaper than the RTX 4080 Super and roughly a fifth faster. The RTX 4060 Ti costs more than an Arc A770 and generates half as many tokens per second. If you take one habit away from this page, make it this: read the bandwidth column before the price column.

One honest caveat. Bandwidth governs decoding, which is the part you watch stream onto the screen. Reading your prompt is a different operation: the model processes all input tokens in parallel, which turns into a dense matrix multiply that is limited by arithmetic throughput instead. A card with weak compute and healthy bandwidth will start answering slowly on a very long document and then stream quickly once it starts. If your workload is mostly long prompts and short answers, the picture shifts back toward raw compute.

A worked comparison: one model, six cards

Qwen3 14B at Q4_K_M with an 8K context needs about 11.0 GB once weights, cache and runtime overhead are counted. That is a demanding but realistic target for a mid-range build, and it separates the cards in an instructive way.

Three things fall out of that table. The RTX 3060 technically holds the model but leaves 0.2 GB of slack, which means the first long conversation will push it over and force a reload at a shorter context. The RTX 4060 Ti fits comfortably and is the slowest card in the group, because it pairs generous capacity with the narrowest memory bus here. And a used RTX 3090 at roughly $700 matches a new RTX 5070 Ti on speed while giving you 7.4 GB more usable memory, which is the difference between running a 14B model and running a 24B one.

The 24GB cliff, and how to plan around it

A 24GB card advertises 24GB and gives a runtime about 22.3 GB after the driver, the CUDA context and allocator fragmentation take their cut. Qwen3 32B at Q4_K_M with an 8K context needs 22.8 GB: 18.5 GB of weights, 2.0 GB of cache and the rest overhead. It misses by half a gigabyte. This is the single most common disappointment in local inference, and it is worth understanding the three ways out of it.

You can cut the context. At 4K the same model needs 21.7 GB, which the tool calls tight rather than impossible. You can cut the precision. Q3_K_M brings it down to 18.9 GB and lifts the estimate on an RTX 4090 from impossible to about 42 tokens per second, at a quality cost you will notice on hard reasoning and barely notice on summarization. Or you can buy the next capacity tier. An RTX 5090 offers 29.8 GB usable, holds the model at Q4 with room for a 32K context, and runs it at roughly 61 tokens per second.

The general lesson is that capacity tiers are cliffs, not slopes. Between 22.3 GB and 29.8 GB of usable memory there is nothing, so a model that misses by half a gigabyte misses by the same amount as one that misses by five. When you are shopping, work out the requirement at the context length you actually intend to use, add roughly twenty percent, and only then look at cards.

Mixture-of-experts models rewrite the shopping list

A dense model reads all of its weights for every token. A mixture-of-experts model stores many experts and routes each token through a small subset of them, so the memory it reads per token is far smaller than the memory it occupies. That decouples the two things you are shopping for: capacity is set by total parameters, speed is set by active parameters.

gpt-oss 120B is the clearest example. In its native MXFP4 format it needs about 63.4 GB with an 8K context, well beyond any single consumer card, but only 5.1B parameters are active per token. On an RTX PRO 6000 Blackwell with 1792 GB/s it is estimated at roughly 446 tokens per second. Llama 3.3 70B at Q4 is smaller on disk at 46.4 GB and runs on the same card at about 30 tokens per second, because all 70.6B parameters are read every time. Smaller footprint, fifteen times slower.

The same trick works further down the range. Qwen3 30B-A3B needs 20.1 GB at Q4 with an 8K context, which fits an RTX 4090, and only 3.3B parameters are active, so the estimate is about 269 tokens per second. Dense Qwen3 32B on the same card does not fit at all. If you are buying capacity on a budget, a mixture-of-experts model is the cheapest way to get large-model behaviour out of moderate hardware, and it is the one case where a slow but roomy machine such as a Mac makes a lot of sense.

New, used, or rented

The used market is where most of the value in this list sits. An RTX 3090 at around $700 delivers 936 GB/s and 24GB, which works out to 1.34 GB/s per dollar. A new RTX 4090 at around $1,600 delivers 1008 GB/s, or 0.63 GB/s per dollar. You are paying two and a half times as much per unit of the thing that determines your speed, in exchange for a warranty, lower power draw and better support for newer numeric formats. That is a real trade, but it is a trade rather than an upgrade.

Renting is the honest baseline you should measure a purchase against, and the break-even is easy to compute: divide the purchase price by the hourly rate you are quoted, and that is how many hours of use it takes before owning is cheaper. A $700 used RTX 3090 pays for itself against a card rented at fifty cents an hour after 1,400 hours. If you use the machine two hours a day, that is close to two years, by which time the card is older and the model you wanted to run has been replaced twice.

Electricity belongs in the calculation too. A card drawing 450 watts under load for eight hours a day consumes 3.6 kilowatt hours; at twenty cents per kilowatt hour that is about $263 a year, and rather more if the room needs cooling. Substitute your own tariff, but do not leave the line out.

There is also a genuinely free tier worth knowing about. The T4 that free Colab notebooks hand out has 16GB and 320 GB/s, good for roughly 40 tokens per second on an 8B model. The P100 on free Kaggle sessions has the same 16GB but 732 GB/s, which works out to about 92 tokens per second, comfortably ahead of a $449 RTX 4060 Ti. Neither is a workstation, both come with session limits, and both are excellent for finding out whether a model is worth buying hardware for.

Consumer, workstation, or datacenter

The price premium on professional cards buys four things: more memory on one board, ECC protection, cooling designed for a dense rack, and a licence that permits datacenter deployment. It does not buy proportionally more bandwidth, which is what actually decides your token rate.

Read the last two columns together. The RTX PRO 6000 Blackwell matches an RTX 5090 on bandwidth and gives you three times the memory in one slot, at $89 per gigabyte. That is expensive next to four used RTX 3090s at $29 per gigabyte, but the single card runs Llama 3.3 70B at roughly 30 tokens per second against about 12 for the four-card rig, needs one PCIe slot instead of four, and does not require a dedicated circuit. An H100 costs $338 per gigabyte and is worth it only when many users are hitting the machine at once, because its bandwidth advantage is best spent on batching rather than on one conversation.

Be suspicious of the datacenter badge on its own. The L4 costs $2,500 for 24GB at 300 GB/s and is designed for low power and high density, not for fast single-stream decoding. On an 8B model it is estimated at about 38 tokens per second, roughly a third of what a $700 used RTX 3090 does. It is an excellent card in a rack that needs seventy-five watt slots and a poor one on a desk.

What multiple GPUs really give you

The default way to split a model across cards is by layer: the first card holds layers one through twenty, the second holds the rest, and the hidden state is handed over the bus once per card boundary per token. That hand-off is a few hundred kilobytes, so a slot running at four lanes is generally enough and PCIe bandwidth is rarely the bottleneck people expect it to be. Capacity adds up cleanly.

Speed does not. With a layer split, only one card is computing at any instant while the others wait their turn, so a single conversation sees roughly the bandwidth of one card minus the synchronization cost. The estimates on this page reflect that: an 8B model on one RTX 4090 lands near 127 tokens per second, and the same model on two RTX 4090s lands near 102. Adding a card made one user slower. It is worth it only when the extra memory lets you run something you could not run before, or when you have enough simultaneous users to keep both cards busy.

Tensor parallelism is the alternative: every layer is sliced across all cards so they compute together, which does raise single-stream speed. It also exchanges activations at every layer instead of every boundary, so it wants fast interconnect, and it usually wants a card count that divides the attention heads evenly. On consumer boards without NVLink it is often disappointing. Watch the capacity arithmetic too: the 48GB in a two-card RTX 4090 setup becomes about 43.2 GB usable, which is not enough for Llama 3.3 70B at Q4 and an 8K context, since that wants 46.4 GB.

Apple silicon: capacity per dollar, at a price in bandwidth

Unified memory means the CPU and GPU share one pool, so a Mac can dedicate far more memory to a model than any consumer graphics card. A Mac Studio with 512GB works out to about $19 per gigabyte, the cheapest capacity in this entire list, and a 192GB Mac Studio is about $29 per gigabyte. Against that, the default allocation limit hands a runtime around three quarters of installed memory, so plan on 384 GB and 144 GB respectively rather than the sticker figure.

The catch is bandwidth, and it varies enormously across the range. A Mac mini M4 has 120 GB/s, which is roughly 13 tokens per second on an 8B model. A MacBook Pro with an M4 Pro has 273 GB/s and gives about 10 tokens per second on Gemma 3 27B, where an RTX 4090 gives about 43. An M4 Max reaches 546 GB/s and an M2 Ultra reaches 800 GB/s, which is finally in the neighbourhood of a good desktop card. So the shape of the trade is: buy a Mac for models nothing else in your budget can hold, not for speed on models that already fit a GPU.

Two practical notes. Prompt processing is the weak spot: the compute available for the parallel prefill pass lags well behind a discrete GPU, so long documents take noticeably longer to ingest even when the streaming afterwards looks respectable. And capacity is still finite: DeepSeek R1 at Q4 needs roughly 410 GB, which overflows even the 384 GB a 512GB Mac Studio will actually hand out. Mixture-of-experts models are where this hardware shines, because they ask for capacity rather than bandwidth.

AMD and Intel, honestly

On paper the non-NVIDIA options look strong. A Radeon RX 7900 XTX has 24GB at 960 GB/s for around $900, more bandwidth than a used RTX 3090 for slightly more money. An Arc A770 offers 560 GB/s and 16GB for around $280, which is 2.00 GB/s per dollar, the best ratio of any card in this dataset. An Instinct MI300X carries 192GB at 5300 GB/s for around $15,000, more memory and more bandwidth than an H100 at roughly half the price.

The gap is software, and it has narrowed a great deal without closing. Anything built on llama.cpp runs well on both vendors through ROCm, Vulkan or SYCL, so ordinary chat and completion workloads are fine. The friction shows up at the edges: a new quantization format usually ships with a CUDA kernel first, some attention implementations and speculative decoding paths are CUDA-only for months, training and fine-tuning stacks assume NVIDIA far more than inference stacks do, and the reference answer on a forum will assume CUDA. Budget time rather than money for that.

The reasonable rule: if you want to run released models with a mainstream runtime and you enjoy tinkering, AMD and Intel are good value. If your work depends on being able to adopt a research technique the week it is published, or you need to fine-tune as well as serve, the CUDA tax is buying you something real.

Power, heat, noise and the rest of the machine

Board power for the cards people actually buy runs from about 350 watts for an RTX 3090 to 450 for an RTX 4090 and 575 for an RTX 5090. Add a hundred and fifty watts or so for the rest of a desktop and size the power supply with headroom, because these cards draw brief transient spikes well above their rated figure and a supply running at its limit will trip rather than throttle. One RTX 4090 is comfortable on a 850 watt unit; two RTX 3090s want 1000 watts or more.

Multi-card builds run into the wall socket surprisingly fast. Four RTX 3090s at full tilt is 1400 watts before the rest of the system, which exceeds what a standard 15 amp 120 volt circuit will carry continuously. This is the practical reason large home rigs end up power limited, and the good news is that power limiting costs less than you would expect: decoding is bandwidth bound, memory keeps running at full speed, and capping the core to seventy percent of stock typically costs single-digit percentages of token rate while dropping heat and noise sharply.

Cooling deserves a thought before you order. Consumer cards are open-air designs that dump heat into the case, so two of them stacked in adjacent slots will cook each other unless the board spaces them out or you use riser cables. Datacenter parts such as the A100, H100, L40S and L4 are passively cooled: they have no fans at all and depend on the violent front-to-back airflow of a rack chassis. Dropping one into an ordinary desktop is a well-known way to discover thermal throttling. Blower-style workstation cards are the compromise that actually works in a tower.

Buying for a model you do not own yet

The weights are the predictable part of the bill. The cache is the part that surprises people, because it grows linearly with context and does not care how well you quantized the model. Llama 3.1 8B at Q4 needs 6.6 GB at an 8K context and 22.8 GB at its full 128K, where 16 GB of that is cache alone. Qwen3 32B goes from 22.8 GB at 8K to 55.2 GB at 128K. If you intend to feed a model whole codebases, size the card for the cache, not the weights.

Architecture changes this a lot. Gemma 3 27B uses sliding-window attention on most of its layers, so a 131K context costs it about 10.4 GB of cache where a conventional 32B model pays 32 GB. Models using multi-head latent attention, such as the DeepSeek family, compress the cache further still. Two models with the same parameter count can have wildly different appetites at long context, so check the real figure rather than assuming.

You have levers if you miss. Quantizing the cache itself is the most underused one: Llama 3.3 70B at Q4 with a 32K context needs 54.5 GB with a normal 16-bit cache, 49.4 GB with an 8-bit cache and 46.7 GB at 4-bit, with the quality cost landing on long-range recall rather than on fluency. Serving several people changes the arithmetic again, because the weights are read once per step for the whole batch: eight concurrent streams of an 8B model need 14.2 GB rather than 6.6, and each stream sees about 56 tokens per second instead of 127 while the machine as a whole produces far more.

So the buying checklist is short. Decide the largest model you seriously expect to run, at the quantization you can live with, at the context you will really use. Add twenty percent for the runtime and for a model release that is slightly larger than the last one. Compare bandwidth per dollar among the cards that clear that bar. Then, before any money moves, run the model through a hosted endpoint for a week and find out whether you actually want it.