Each token lights only the experts it needs – the rest stay asleep

The model is dormant until a token moves through it

Each token lights only the experts it needs; the rest stay asleep. That single line is the whole behaviour of a Mixture of Experts, but the interesting part is what it puts on the machine: a hot set of experts that stays resident, and a cold set that does not. Deciding that split, and defending it token after token, is an engine job, not a model job.

That is where Strata earns its place. It forks llama.cpp to keep the hot expert set resident and park the rest, so the memory budget goes to the experts a workload keeps routing to instead of to weights that are dormant most of the time. Avalon, in turn, forks Strata to tune that hot set tighter and stretch context on the same box. Each layer keeps the mechanism honest and trades a little more of the machine’s memory for it.

What "resident" actually buys

A token routed to an expert already in memory costs read bandwidth you already paid for. The same token routed to an expert parked on NVMe costs a drive read before it can answer. Keep the frequently-used experts resident and the machine stops paying that toll on most tokens; let the hot set thrash and every call re-buys a read. So throughput is mostly a statement about which experts survived in memory between the last token and this one.

A focused workload makes that tractable: a support queue over one product keeps routing to the same experts, and a well-tuned resident budget simply holds them. A broad, unfocused workload is the hard case, and the honest answer is that the same box serves it worse. We measure both and show both.

Read before you believe a speed number

Because residency drives the result, the same model file and the same card can produce different numbers depending on the engine and its resident budget. That is exactly why we refuse to quote a figure without the rig and version beside it, and why the benchmark index lists engines as columns and machines as rows over one model file with weights and prompt length held identical, read and write timed apart. When a project’s own documentation hedges a feature – GPU support landing later on some paths, for instance – we repeat the hedge rather than promising past it.

The point for you is a machine that runs a large model like a focused one, on hardware you already own, inside your building, tuned by someone who will be there. Ask about an on-site assessment at privatelocalai@proton.me.

Engine-by-engine residency and throughput are recorded in the benchmark index: https://localai.hex7.com/benchmark.html.