Xolotl – running a 552-billion MoE on one gaming PC, and what it asks of a constrained machine
Xolotl (github.com/Indras-Mirror/Xolotl, MIT) is a descendant of Strata. The parent project’s engine, server and core kernels are still inside it – merged at 0.1.44.1 – but the author is candid that more than 70,000 lines have gone on top: its own DeepSeek-V4.1 / V4-Flash and GLM-5.3-Flash engines, a Qwen3.8-27B dense engine grown from Mirko Perrone’s port, an LFM3-Omni multimodal engine, a “fast mode”, and the quantisation, expert-tiering and serving work in between. It serves one OpenAI- and Anthropic-compatible endpoint across all of them.
The name is the project’s own: Xolotl is Quetzalcoatl’s dark twin in Aztec myth, the one who carries the dead up from the underworld. The engine carries experts up from RAM and NVMe. That is the whole design in one sentence, and it is also why this piece goes on the list of resource-constrained inference engines rather than the “big GPU” list.
What it costs to run, stated plainly
Most engines tell you what model they support. Xolotl’s README has a separate document for the machine budget, and the numbers are unusual for being honest about which part of the system does the work.
The measured setup is one desktop: RTX 4090 (24 GB), Ryzen 7 5700X, 110 GB DDR4, one NVMe drive, Linux. The document spells out what each of those budget items means:
- GPU: any NVIDIA card at compute capability 7.5 or newer (RTX 20/30/40/50). A Volta V100 path exists as an
- RAM is the real constraint. With 110 GB, the engine keeps an 80 GB expert cache in RAM; on a 90 GB machine that
- Disk: roughly 152 GiB for the quantised model, about 200 GB more for the Engram tables, and ~40 GB more if you
experimental flag, not as a promise.
cache is 64 GB. Less RAM means a smaller arena, so more experts are read from NVMe per token and decode drops. Around 256 GB is quoted as the point where every expert simply lives in RAM.
want fast mode. It asks for a fast NVMe on ext4 or NTFS – and it says why: not exFAT, because Linux’s exFAT driver serialises parallel reads and made expert reads five times slower there; and not a hard drive, because the Engram lookups are around 48 small random reads per token.
That is a very different thing to “needs 24 GB of VRAM”. The card is the least important number on that list, and the authorship says so: community runs on 12 GB cards (an RTX 3060 with 46 GB of RAM reaches 149 tok/s prefill at 8K and 8-10 tok/s decode; an RTX 5070 gets 4.7 tok/s), while the same class of card runs Qwen3.8-Flash-Next at 39-48 tok/s. Same model family, same engine, four times the speed with a different quant – which is a reminder that in this work the quant you choose is the memory plan.
The mechanism: three tiers and an explicit trade
A mixture-of-experts model only touches a few experts per token, so Xolotl keeps every expert somewhere: the hottest in VRAM, tens of gigabytes in a pinned RAM arena that the GPU reads by DMA, the rest on NVMe read at the drive’s ceiling – and moves whichever experts the current token’s router asks for, a layer ahead of time. That “a layer ahead” is the part that separates it from a general runtime that pages weights in lazily and hopes the OS co-operates: an engine that owns its own memory policy is the lesson my Avalon and PolyStrata notes made, and here it is again.
Per-model work on top of that: calibrated mixed-precision quantisation (the DeepSeek V4.1 build uses the GSQ family, which I covered in GSQ and RCO), speculative decoding with exact sampling, a bit-exact 4-bit KV cache for 512K context, an arena that adapts to the conversation, MTP speculative decoding for the Qwen models, GLM’s abliteration LoRA on every path plus soft pruning.
Two features are worth reading for the engineering rather than the marketing:
- Xolotl Direct (Linux, experimental): the NVMe drive writes expert blocks straight into the card’s VRAM instead of
- Fast mode (opt-in): a ~40 GB sidecar holding a 2-bit copy of the two largest expert formats, so the tokens that
routing them through host memory – 3.1 ms per expert where the normal path costs about 11 ms.
matter read from a smaller file. It changes decode on agent tasks from 32.6 to 41.1 tok/s (+26%) and keeps the same benchmark score, while next-token agreement against normal mode drops from 87.0% to 86.2%. The README is unusually direct about that: it is lossy, so it is opt-in, and “keep normal mode where every token counts”. Turning it off is one unset environment variable.
That last paragraph is the one I’d point at if someone asked me what to learn from this project. A speed claim without an agreement metric beside it is a claim I cannot use; here the trade is quantified in the same table as the speed.
The measurement discipline, which is the real reason to read it
Every number in that README comes from one machine, through the real server, while the model works its benchmark: six hard agentic coding tasks graded by hidden tests, ten turns each, thinking enabled with a 32K budget. Decode speed is measured during those agent tasks rather than in an idle generation loop – which is the only honest way to quote a number for anything used by an agent harness.
It also scores its own models against the same models’ hosted APIs on those tasks, and reports both directions: a local DeepSeek-V4.1 build at 4.99/6 against that model’s API score of 4.49; Qwen3.8-27B at 5.50 against an API 4.47. Where it has not measured, it says “to come” – GLM-5.3-Flash is quoted as decode speed without a benchmark score, because there is no run yet. And it retired its own HumanEval table for a reason: every model scored 19-21/21 on it, so the table no longer separated them.
It is worth being clear about what this is not: research code from one developer, measured on one PC, with the invitation “reports from other hardware welcome” in its own README. The Windows path exists only for DeepSeek-V4.1 and is native MSVC + CUDA; everything quoted is Linux.
Where it fits in this family
- Strata (my notes) is
- PolyStrata took the breadth path too, but as an app with a dozen backends; Xolotl’s breadth is narrower and much
- Avalon (my notes)
- llama.cpp stays the general runtime in the picture, and there is a small engineering detail in how Xolotl uses it:
the parent: one flagship model, well served. Xolotl kept that core and grew engines outward.
deeper per model – its own kernels for DeepSeek, GLM, Qwen dense, LFM3-Omni.
bets on old AMD cards and one model; Xolotl’s bet is a large NVIDIA card plus lots of RAM and NVMe.
it fetches llama.cpp at a pinned commit and applies its own ggml patch on top, rather than carrying a fork of the whole runtime. Patch plus pinned commit is a cheaper maintenance contract than a fork – which is worth stealing if you run llama.cpp in production and need one custom kernel.
What I would take into our own stack
- Budget RAM as a first-class resource, not an after-sale. The tiering decision is the deployment plan: 80 GB of
- Never run model weights off exFAT or a spinning disk. The two specific reasons here – serialised parallel reads
- Measure inside the agent loop. A benchmark whose prompts are hidden tests, run ten turns deep with thinking on,
- Ship lossy speed modes as opt-in, with an agreement metric in the same table. Fast mode’s 26% is quoted together
expert arena on a 110 GB machine, 64 GB on a 90 GB one, ~256 GB if you want every expert resident. Write those numbers down in your own deployment notes instead of saying “plenty of RAM”.
on Linux’s exFAT driver (5x slower), and ~48 random reads per token for lookup tables – are the sort of fact that only shows up in someone else’s failure.
will report numbers that match how the model is actually used.
with its 0.8-point agreement cost, and that makes the offer usable rather than a slogan.
Xolotl: https://github.com/Indras-Mirror/Xolotl Machine requirements (the honest one): https://github.com/Indras-Mirror/Xolotl/blob/main/docs/WILL_IT_RUN.md Benchmarks and how to measure your own PC: https://github.com/Indras-Mirror/Xolotl/blob/main/docs/xolotl/BENCHMARKS.md Fast mode, what it costs: https://github.com/Indras-Mirror/Xolotl/blob/main/docs/XOLOTL_FAST.md Changelog (with PR numbers): https://github.com/Indras-Mirror/Xolotl/blob/main/docs/xolotl/CHANGELOG.md Write-ups by the author: https://indrasmirror.au llama.cpp: https://github.com/ggml-org/llama.cpp · Strata: https://github.com/Niko1221/Strata
Leave a Reply