Placement, not size
A model’s parameter count is a fact about the file. Whether it answers on your machine is a fact about where its weights sit while it runs. Most of the "you need a big GPU" argument comes from collapsing those two questions into one.
For a dense model every token touches every weight, so the whole model wants to be resident in fast memory at once. Fast means VRAM: the small, expensive tier bolted to the card. Size the model to the card, buy a bigger card, repeat forever.
Three tiers, one stream
There are exactly three places to put a weight: VRAM, system DDR, and NVMe. Local inference at this size treats them as a stream instead of a wall. Weights are read from ordinary system RAM and from NVMe, promoted through a small working set in VRAM, and evicted again. DDR is the RAM already in your machine; NVMe is the drive it already boots from.
So the file is large and the resident working set is not. A box with a hundred-odd gigabytes of system memory holds a model that would never fit on a single consumer card, because "hold" means "stream from", not "pin in VRAM".
Where the number comes from
This is not a claim that any box runs any model well. It is a claim about a specific rig and engine, measured on that box. Read throughput is the number that matters here, because streaming weights is a memory-bandwidth problem before it is a compute problem: two machines with the same card behave differently depending on how many DDR channels they have and how much of the hot set the engine keeps resident. That is a placement decision, not a silicon one. The benchmark index records those figures with the engine version beside them.
The upgrade you already own
When a model stops fitting, the answer is more of the memory you already buy by the stick and a drive you already have, not a larger accelerator on a purchase order. The machine in front of you stretches further than the card alone suggests, because the card was never the only place the weights could live.
We measure your box first, then recommend the engine and the placement, before anything is bought. Ask about an on-site assessment at privatelocalai@proton.me.
Figures for this build are recorded in the benchmark index: https://localai.hex7.com/benchmark.html.
Leave a Reply