The claim, stated plainly
One gaming PC, one model with 552 billion parameters, served as a local coding agent behind an OpenAI-compatible API. That is what Xolotl does on this stack. The sentence sounds like a benchmark trick, so it is worth unpacking: the trick is not that the model is small. The trick is that a Mixture of Experts does not load most of itself for most queries.
What an MoE actually loads
A Mixture of Experts replaces one giant network with a fleet of small, specialised experts plus a gate. Each query activates only the experts it needs. The rest never load into memory at all. That is where the efficiency comes from, and it is the reason a file that would embarrass a server room can answer a question on a desk machine: the parameters exist, but they are not all resident, and they do not all have to be.
The three tiers
Weight placement is the whole problem, and there are only three places to put them.
- VRAM is the fast tier and the small one. Whatever lives here is read at full speed on every
token.
- Pinned system RAM is the middle tier: slower than VRAM, but far larger, and the engine can
keep a page of experts there and move it up without the page-fault roulette of ordinary memory.
- NVMe is the cold tier. It is the largest one, and it is not slow enough to be a problem for
weights that are read once and reused across a whole conversation.
The engine keeps the MoE experts tiered across those three – VRAM, pinned RAM and NVMe – and reads weights from disk straight into the GPU when an expert is called that is not already resident. The skill is in deciding what stays hot.
What "hot" means, and why it works
Strata, forked from llama.cpp, is built around exactly this: it keeps resident only the experts that are hot – the ones actually in use – and leaves the rest parked. That is how big MoE models run on small RAM. Avalon forks Strata and tunes the same idea further: a tighter hot set, which buys longer usable context and larger models on the same hardware.
The reason this is survivable rather than a bottleneck is locality. Expert selection is not random. A conversation about one codebase keeps calling on the same handful of experts, so the hot set stabilises, the page-in traffic drops, and the cold weights on NVMe are read a handful of times rather than on every token. A model that "does not fit" is a model that does not fit all at once.
The size lever: non-uniform quantisation
The other lever is the file itself. Uniform quantisation spends the same number of bits on every weight, whether or not that weight deserves them. GSQ and RCO do not: Gumbel-Softmax Quantization learns a precise low-bit scalar grid for each weight tensor, and Riemannian Constrained Optimization finds the best per-tensor allocation under a fixed size budget. The result is a non-uniform GGUF – more bits for sensitive weights, fewer for tolerant ones – same task quality at a smaller file.
The cheaper hardware answer: DDR over VRAM
For small and medium private systems the tier that matters is often the second one. The engine streams model weights through ordinary system memory – fast, vast DDR instead of expensive GPU VRAM – so one machine with 256 GB of DDR and an RTX 5080 runs small and medium private AI comfortably. Buying memory has never been the expensive part of a computer; buying it in the form of VRAM has.
What it looks like at the desk
None of this is exotic at the point of use. The result is an OpenAI-compatible API on localhost, so everything already written against one works against it: coding agents, IDE bridges, MCP servers, RAG pipelines. PolyStrata packages the same idea the other way round – two engines in one app, the Strata engine for Qwen3.8-Flash-Next plus a second engine written here for eight more MoE models – with a one-click install on Windows and Linux.
Measured, not promised
Two honest caveats. Expert offload is not free: the first token of a cold path costs more than a fully-resident model, and a machine with no fast NVMe will feel it. And engines differ – exo, which clusters heterogeneous machines and splits a model along the links using measured latency and bandwidth, says in its own README that on Linux it currently runs on CPU while GPU support is being developed. We quote projects about themselves rather than rounding them up.
So we benchmark: one model file, weights and prompt length held identical across every engine that can load it on that box, reading speed and writing speed measured apart, versions recorded with the number. The published index – engines as columns, machines as rows – is at localai.hex7.com/benchmark.html.
Where the service starts
If the question is which engine, on which box, for which model – that is an assessment, not a blog post. Small and medium organisations in the Perpinyà – Girona – Barcelona corridor run their own private AI on their own hardware, at a discount, because efficiency is built in: smaller models run well, memory works smarter, and inference stops being billed per query. Ask about an on-site assessment: [privatelocalai@proton.me](mailto:privatelocalai@proton.me).
Leave a Reply