Private Local AI blog

exo – splitting one model across several machines, and the rules it sets for joining

Written by

in

I went looking at exo (github.com/exo-explore/exo) because it answers the question I keep getting about a constrained rig: what if the memory ceiling of one machine were not the ceiling? The repository’s own description reads, plainly enough: “Run frontier AI locally” – and the README states what it does in one sentence: “exo connects all your devices into an AI cluster.” Not just larger models, but, with day-0 support for RDMA over Thunderbolt 5, faster as you add machines.

The GitHub metadata as I fetched it on 11 November 2026: 48,311 stars, Apache-2.0, Python, created in June 2024, last pushed on 6 November 2026 – maintained by exo labs, and active.

What it is

exo is a peer-to-peer cluster runtime: you run the same software on each machine – a desktop, a laptop, a second box over Wi-Fi or Ethernet – and it forms a cluster without manual configuration. Devices discover each other automatically, provided they are on the same network and run the same version of exo; devices on different versions simply do not join each other. Each node serves its own API and dashboard at http://localhost:52415.

The inference backend is MLX, Apple’s framework, with MLX distributed for cluster communication. Models come from HuggingFace (search and “add a custom model” endpoints are built in; models needing trust_remote_code require an explicit opt-in). There is an offline mode (EXO_OFFLINE=true), read-only model directories for shared storage such as NFS mounts, and four API surfaces: OpenAI Chat Completions, OpenAI Responses, Claude Messages and Ollama. The docs: existing clients – OpenWebUI included – just point their base URL at exo.

One honest note: exo-labs.github.io/exo returns a GitHub Pages 404 today, so I verified nothing from it; this piece relies on the repository.

How it works

Discovery, and the rules for joining

The version rule is stricter than it sounds. Devices join only when they share a cluster name (a namespace flag), a “wire version” – a number exo bumps whenever what devices send each other changes – and a build. The macOS app numbers itself with its own release number while source checkouts keep their package version, so an app and a source build on one network do not join each other by default. A Mac and a Linux machine do not join at all, even on the same commit, because they run different builds of MLX. Guest Wi-Fi and some office networks block discovery outright; when a device is seen but cannot join, the dashboard says “DEVICE ON ANOTHER EXO VERSION”.

There is also a role knob: --no-worker runs a node that coordinates networking and orchestration but executes no inference, which the README recommends for machines without GPU resources but with good connectivity.

Placement: topology-aware, before the first token

The README calls this “Topology-Aware Auto Parallel”: exo picks how to split a model from a realtime view of the device topology, taking device resources and per-link latency and bandwidth into account. You can watch the machinery from the API: GET /instance/previews returns every valid placement for a model, with a memory delta per node, before you commit to one. Launching is asynchronous; you wait for readiness on an SSE endpoint. The README claims tensor-parallel sharding reaches “up to 1.8x speedup on 2 devices and 3.2x speedup on 4 devices” – the project’s own figures, not independent benchmarks. The bundled exo-bench tool measures prefill and generation speeds across placement configurations, treating “ring” and “jaccl” instance types and pipeline versus tensor sharding as things you choose between.

RDMA over Thunderbolt 5

RDMA is a macOS 26.2 capability; the README claims a “99% reduction in latency between devices”. It requires Macs with Thunderbolt 5, TB5 cables, every machine cabled to every other one, matching OS versions down to beta numbers, and a recovery-mode rdma_ctl enable. The benchmark set linked from the README was run by Jeff Geerling (18 December 2025): four M3 Ultra Mac Studios with 1.5 TB of combined unified memory at ~$40,000. He notes that RDMA cuts memory-access latency from about 300 microseconds to under 50 (citing an MLX pull request); found llama.cpp’s RPC method losing speed as nodes were added while exo gained; and got about 32 tokens/s on Qwen3-235B (8-bit) and “around 30” on Kimi K2 Thinking on the full cluster. He also reported stability problems with the then-prerelease software, and rebooted the cluster a lot.

What happens when a device leaves

The API reference (docs/api.md) treats a dropped machine as a first-class error: an in-flight generation ends with a device_left code – or out_of_memory, gpu_timeout, model_stalled – rather than a hung request, and a launch is refused if the devices turn out not to have the memory for it. That is behaviour worth knowing about before you put an agent on it.

What it asks of your hardware

  • The macOS app needs Apple silicon and macOS Sequoia 15.1 or later; RDMA needs macOS Tahoe 26.2 and a Thunderbolt 5 Mac (the README lists the M4 Pro Mac mini, M4 Max Mac Studio, M4 Max MacBook Pro and M3 Ultra Mac Studio).
  • From source, macOS wants Xcode, uv, Node, Rust; Linux wants the same toolchain but the README is blunt: “Currently, exo runs on CPU on Linux. GPU support for Linux platforms is under development.” It asks users to open or upvote feature requests for hardware they want supported.
  • Every device must run the same version of exo, be reachable on the same network, and download or hold the model. Mixed clusters in the consumer sense – a Mac and an NVIDIA Linux box in one group – do not join today.

For my own rig – two 16 GB consumer GPUs and about 100 GB of system RAM – what this machine would need from exo, on the README’s current terms: CPU inference on Linux (the MLX backend does not use those cards on Linux yet), a model split across shared system memory, and no RDMA path. The project does not state a supported layout for that machine, and I ran nothing, so I am quoting no numbers for it.

Where it fits / where it does not

Where it fits: a homogeneous all-Apple-silicon fleet with Thunderbolt 5 between them, where you genuinely want frontier-class models – the README’s own showcase is DeepSeek v3.1 671B (8-bit) and Kimi K2 Thinking on four Mac Studios – and where a drop-in OpenAI-, Claude- or Ollama-compatible endpoint is the integration point you care about. The placement preview API and exo-bench are the tools for deciding how many machines a given model needs before you buy them.

Where it does not fit: my family stack, which I have covered here in detail – Strata, PolyStrata, Avalon and Xolotl, which tiers MoE experts across VRAM, system RAM and NVMe. Those projects attack the same ceiling from inside one machine: keep the fast silicon you have, move weights through a memory hierarchy you control. llama.cpp’s RPC path, per Geerling’s write-up, spreads layers across nodes but loses performance as nodes are added – the exact failure mode exo’s tensor-parallel-over-RDMA design is built to avoid.

exo does something different in kind: it grows the memory pool across machines and plans from a topology view. What it does not do is any expert-tiering equivalent – there is no such mechanism in its README or API docs, and the project does not claim one. On a consumer-NVIDIA rig it is simply not yet a target: Linux is CPU-only, and Mac nodes refuse to join Linux nodes. If your fleet is Macs, that is fine; if it is this lab, exo today is an interesting read, not a deployment.

How I verified this

Sources fetched on 11 November 2026:

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

More posts