The parameter count is not the memory footprint
A Mixture of Experts is the reason a very large model can answer on an ordinary machine, and people keep reading it wrong. The file lists hundreds of billions of parameters. The number that decides whether your box runs it is how many of those are resident for a given token – and for an MoE that number is a small fraction.
A conventional network puts one large block of weights between every pair of layers, and every token passes through all of it. An MoE replaces that single block with a fleet of small, specialised experts and a router. The router reads the token and sends it to a handful of experts. The others are not consulted for that token and, in a well-run deployment, are not even loaded into memory.
Only the used experts stay resident
That distinction is the whole trick. The parameters all exist in the file; most of them sit on the drive. A query activates the experts it needs, the engine keeps the recently-hot experts close, and the rest stream in from NVMe only when some later token happens to route to them.
So the working set is not "the model". It is "the experts this kind of query keeps hitting". A help desk asking the same class of questions repeatedly stays warm: the same experts light up, they stay resident, and the machine behaves as though the model were far smaller than its file. A wide, unfocused workload is the harder case, and we measure it as the harder case rather than hiding it.
Why it matters for a support inbox
This is where the shape of real work helps. A support queue over one product keeps returning to the same few topics, which keeps routing to the same few experts. The hot set stays small and stays put, so throughput holds. That is not a benchmark curiosity; it is why a single machine can front a live inbox without a datacentre behind it.
The counterweight: expert placement is a scheduling problem, and the engine’s resident budget decides how much of the hot set survives between tokens. Set it too low and experts thrash in from the drive on every call; too high and you have spent memory you needed elsewhere. Getting that balance is engineering, and it is why the engine version belongs next to the speed figure rather than underneath it.
Measured, on your box
We do not quote tokens-per-second we have not earned. The same prompts, the same read and write timing, on the named machine, across engines, versions recorded – that is what the benchmark index holds, and it is what we run before recommending anything to you.
The payoff is ordinary hardware running a model that its own file would disqualify it from, answering from inside your building with a person on site to keep it warm. Ask about an on-site assessment at privatelocalai@proton.me.
The routing and residency numbers are recorded in the benchmark index: https://localai.hex7.com/benchmark.html.
Leave a Reply