PolyStrata – the Strata fork that stopped being one engine, and what that means against llama.cpp, Strata and Avalon

PolyStrata (github.com/simplexai-labs/PolyStrata, MIT; created by VecSzn) is a fork of Strata that grew a second engine. Where Strata serves one model family well, and where Avalon doubles down on a single model inside a llama.cpp fork, PolyStrata went the other way: it kept Strata’s engine untouched for Qwen3.8-Flash-Next, and wrote its own engine for eleven more models – GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Flint, Qwen3.6, Clover, Ornith 1.5, Gemma 4, Tiel Coder, MiniCPM-V and Kolibri 1.

The project’s own framing is the most useful part: "Run large AI models on your own PC – models that usually need a server." Its README describes what that costs: the largest model it hosts, GLM 5.3 Flash, is a file of 79 to 114 GB whose bulk stays in RAM, and needs a 32 GB card to run at all.

How it differs from the other three

Against llama.cpp: PolyStrata’s comparison claim is specific and self-reported – on an RTX 4070 Laptop with 8 GB and 32 GB of RAM it read prompts 2.7 to 3.5 times faster than llama.cpp b11540, and wrote 1.5 to 1.8 times faster, tested on Qwen3.6 and Tiel Coder. The mechanism is the same one Strata and Avalon push: stop letting the general runtime decide where weights live. Its engine also ships a benchmark you can run on your own machine – python polystrata.py benchmark --model <name> --prompt-tokens 24000 – and a community results file, which is an unusually good habit for a project of this age.

Against Strata: PolyStrata keeps Strata’s engine unchanged for Qwen3.8-Flash-Next ("its engine is the Strata project’s, unchanged") and adds its own backends for everything else, marked experimental – they have been run on one PC with a 32 GB card, and eight of them on an 8 GB laptop card. That split is why it is a separate repository rather than a pull request: different models, different architecture handling, a different maintenance path. Its installer also does things Strata does not: START-HERE.bat or ./setup.sh, model selection by family flag, Mac (Apple Silicon) builds as a beta, multi-GPU on two or three cards, an MCP server so an AI assistant can install and start the thing itself, and a docs/AI_SETUP.md page aimed at pasting into Claude Code or Cursor.

Against Avalon: Avalon is the opposite bet – one model, hand-tuned kernels, custom quantisation types mainline cannot load, RDNA3 first. PolyStrata bets on breadth with a shared app and API in front of many models. On the same laptop-class hardware that means PolyStrata’s numbers are good but not surgical: 94-100 tokens/s for Qwen3.6-35B-A3B on an RTX 4070 Laptop, 251-485 on an RTX 5090. Where Avalon fights for every token on old AMD cards, PolyStrata is trying to make a dozen MoE models runnable at all – including ones with unusual memory profiles: LFM2.5 keeps almost no attention layers, so a 128k context costs 2 GB more than 32k; Flint looks back only 513 tokens for most layers; DeepSeek V4 Flash costs 0.7 GB extra VRAM between 32k and 128k.

The memory control is the engineering worth reading

PolyStrata’s recent work is the same lesson Avalon taught, from the other direction: how much RAM an engine keeps for itself. Since 0.1.45-poly.7 its engine no longer holds parts of the model file that are already on the graphics card – Qwen3.6-35B-A3B after a 32,000-token prompt went from holding 17.9 GiB to 13.1 GiB, at the same speed. There’s an explicit knob for it, --ram-experts-mib, which trades held RAM against write speed with numbers attached rather than promises:

• holding 6,144 MiB instead of everything: RAM held after load drops from 11.7 to 6.5 GiB, peak during a 32k prompt from 14.0 to 7.3 GiB – at the cost of 3% on the first short chat and 13-14% on later turns.
• when the experts outside VRAM do not fit in free RAM, it now budgets a hot-expert set sized to free RAM minus 4 GiB and reads the rest from SSD, instead of mapping everything and taking page faults one at a time: measured on a Linux box with an RTX PRO 6000 limited to ~8 GB and the engine capped to what a 16 GB machine has, first short chat went from 51 to 57-60 tokens/s, after a 24k prompt from 75-78 to 77-85, and in the chat after that from 94-98 to 131-139.
• --lock-experts pins those experts so the OS cannot take them back; under Linux it respects ulimit -l and reports how much it locked.

That is a direct echo of the problem Avalon wrote an entire design around (page-cache eviction and driver registration limits), and it is encouraging that both projects landed in the same place: an engine must own its memory, or the machine will take it away mid-token.

The entry point is a terminal script – and a measurement tool

The file every description of this project should mention is polystrata.py, 232 lines at the root of the repository. It is the terminal front end: python polystrata.py --list-models shows what is installed and which model is in memory; --model <name> chat talks to it; benchmark --model <name> --prompt-tokens 24000 starts (or reuses) a local server on that model’s own port and reports honest numbers: time-to-first-token, prefill and decode tokens per second, cache-hit rate, how many MTP drafts were offered versus accepted, peak VRAM and RAM of the whole machine, and average GPU and CPU load. The script is explicit about its own guesses – "Four characters a token is a guess, and a text with tables comes to more" – and trims a prompt that does not fit rather than pretending it did.

That is worth copying into any local stack: if the engine cannot be measured from a script in one command, people will not measure it.

The ecosystem signal

PolyStrata has already been forked for minimalism: robertkarl/UnbiasedStrata keeps "its two inference engines unchanged and replaces everything around them: no installer, no downloads at build or first run" – which is the counter-argument to PolyStrata’s one-click approach, and worth reading before you decide which trade you want. KVMem now lists PolyStrata among its backends, alongside llama.cpp and NInfer, which is a sign that the serving layer – not any single engine – is where these tools get plugged in.

What I’d take from all four

1. General runtimes lose on memory policy, not on models. llama.cpp’s page-cache approach is exactly what both forks fix, and they quote crash-free or slower-before-versus-faster-now numbers to prove it.
2. Forks are cheap now; the expensive part is correctness. PolyStrata’s README documents per-model licences, which sizes were measured on which machines, and what is untested. So does Avalon’s, complete with its own corrections. That documentation discipline is the thing I’d copy into our own stack.
3. Pick by scope. For "whichever model today", llama.cpp plus a router. For a desktop appliance that runs one flagship model well: Strata. For squeezing one MoE out of old AMD hardware: Avalon. For a broad set of large MoEs on an ordinary PC, with an app and API in front: PolyStrata.

PolyStrata: https://github.com/simplexai-labs/PolyStrata The terminal entry point and benchmark harness: https://github.com/simplexai-labs/PolyStrata/blob/main/polystrata.py Its benchmark and community results: https://github.com/simplexai-labs/PolyStrata/blob/main/docs/COMMUNITY_BENCHMARKS.md The minimal fork of it: https://github.com/robertkarl/UnbiasedStrata Strata: https://github.com/Niko1221/Strata · Avalon: https://github.com/freriadoc/Avalon · llama.cpp: https://github.com/ggml-org/llama.cpp