Private Local AI blog

ExLlamaV3 and TabbyAPI – the pair that ships today, and what it asks of your hardware

Written by

in

ExLlamaV3 and TabbyAPI are a pair documented by both projects: exllamav3’s repository blurb reads “An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs”, and tabbyAPI’s reads “The official API server for Exllama. OAI compatible, lightweight, and fast.” Everything below describes code that ships today; the hardware story changed with it, because ExLlamaV3 (turboderp-org/exllamav3, MIT, v1.6.6 of 31 October 2026) documents CPU offloading for mixture-of-experts weights, and TabbyAPI (theroyallab/tabbyAPI, AGPL-3.0) exposes it as a configuration knob – including on a machine like mine: two 16 GB consumer cards and ~100 GB of system RAM.

What changed from ExLlamaV2

All verified in the repos themselves:

  • The exllamav2 README carries a note at the top: “This project is archived for now. Development continues on ExLlamaV3.” The V2 line is where EXL2 files lived; it is frozen.
  • TabbyAPI removed its ExLlamaV2 backend on 14 July 2026 in a commit titled “Backends: Remove ExLlamaV2 backend and container abstraction layer”. The current tree lists only backends/exllamav3 plus an embeddings backend; the sample config’s backend comment reads “Options: exllamav3”.
  • Supported model types are now listed as “Exl3 (Highly recommended)” and FP16/BF16. EXL3 itself – quantisation “based on QTIP” – exists only in ExLlamaV3; the exllamav2 README and bundled docs never mention it.

Neither project documents that ExLlamaV3 loads older EXL2 files – third-party posts claim it does – so treat migration as a model-file decision you cannot assume away.

What it is

The engine

The engine README lists the feature set plainly: EXL3 quantisation based on QTIP plus “2-8 bit cache quantization”, “Flexible tensor-parallel and expert-parallel inference for consumer hardware setups”, and – the new one – “CPU offloading – Allows large MoE models to run with limited GPU resources. AVX2 and AVX512 support.” The library compiles a Torch C++ extension; prebuilt wheels must match platform, Python, CUDA and PyTorch versions, while the PyPI build compiles from source and needs the CUDA toolkit installed.

EXL3 is “a streamlined variant of QTIP from Cornell RelaxML”: conversion takes an HF-format model plus a target bitrate and runs in one step – “a couple of minutes for smaller models, up to a few hours for larger ones (70B+)” on one high-end consumer card. Fractional bitrates are documented: the conversion guide’s examples write directories like llama3.1-70b-instruct-exl3-3.75bpw, and an -hq flag promotes attention-class layers, the docs candid about a “(0.05 – 0.10 bpw)” size increase for a disproportionately large fidelity gain. The README maintains an architecture-support table (including DeepSeek V3/V4, some multimodal) rather than making per-model claims.

The API layer

TabbyAPI is “A FastAPI based application” that today serves text through the ExLlamaV3 backend; it also calls itself “the official API backend server for ExllamaV3”. Features: tool/function calling, JSON schema, regex and EBNF grammars, Jinja2 chat templates, speculative decoding via draft models, continuous batching with paged attention (“for Nvidia Ampere GPUs and higher”), HF model downloading and embeddings. The sample config defaults the server to port 5000. GitHub’s tags and releases API returns nothing for this repository; pyproject.toml carries version 0.0.1; installation is from the repo via pip install -U .[cu12] and friends, with CUDA Docker images (including a ROCm tag for Radeon RX 7900-series cards). The README’s disclaimer says plainly: “TabbyAPI is a hobby project made for a small amount of users. It is not meant to run on production servers.”

How the pair works

The chain starts with convert.py: an HF checkpoint plus a target bitrate becomes EXL3 safetensors – that is where quantised files on Hugging Face come from. The engine then exposes documented primitives: tensor/expert parallelism across GPUs, continuous dynamic batching, speculative decoding, 2-8 bit cache quantisation. TabbyAPI turns these into OpenAI-style endpoints governed by config.yml – which is also where the hardware switches live (next section).

The pairing is stated in both directions: ExLlamaV3’s README says “Looking for a server? TabbyAPI is the official and recommended backend server”, and TabbyAPI’s README/FAQ answers “What does TabbyAPI run?” with ExLlamaV3. Today that is the only documented combination.

What it asks of your hardware

The default placement is still weights-on-card. TabbyAPI’s config documents gpu_split (“Array of VRAM sizes to split between GPUs, in GB”), an autosplit reserve of ~96 MB per GPU for loading, and tensor parallelism with a backend choice – “Native is recommended for PCIe GPUs, NCCL is recommended for NVLink”. Its Docker notes add a hard requirement: “ExLlamaV3 uses POSIX shared memory for tensor parallelism and CPU MoE offload”, with the default 64 MiB /dev/shm called out as too small.

What changed, and what matters for a 2×16 GB + ~100 GB RAM machine, is the MoE offload path. It is documented twice – in ExLlamaV3’s doc/env_vars.md and in TabbyAPI’s server options:

  • The engine flag is “Experimental: -mcl/--moe_cpu_offload … run the routed experts of the first N block-sparse MoE layers on the CPU, expert weights resident in system RAM, freeing the VRAM those layers’ experts would have used.” It names constraints: layer-split mode only (not tensor parallelism), mul1-codebook experts, K ≤ 8, uniform per-expert biases; “ineligible layers fall back to the GPU as usual”. A worker process per model component owns its expert weights and a job ring in pinned shared memory, and during prefill “hot experts additionally stream their weights to the GPU and run there … while the CPU works the remaining tail”.
  • TabbyAPI exposes two variants: cpu_moe_offload_layers (whole layers on CPU) and cpu_moe_split_experts, which “splits every MoE layer instead of offloading whole layers, overlapping CPU and GPU compute, with dynamic placement keeping hot experts in VRAM”. A pinned-arena option keeps expert weights page-locked; its own comment states the cost: “about 0.2 s per GiB of offloaded experts at load”.

The docs carry numbers, with their own disclaimer (“These are workload-specific measurements”): fully-offloaded mistral-small-4 119B with 54 GiB of experts went from 700 to 1850 tokens/s prefill on a PCIe gen5 x16 link using the pinned arena, “decode unchanged within noise”; and about 240 tok/s decode for LFM2.5-8B on a Threadripper 7960X with tuned thread counts (versus 222-232 in default configuration). Both are one-machine measurements quoted by the project, not guarantees.

For my rig specifically: the sources do establish a documented mechanism for running large MoE models with experts partly or wholly out of system RAM, on consumer cards, placed via layer split rather than tensor parallelism – two 16 GB cards splitting weights by gpu_split plus ~100 GB of RAM for expert weights is within what that mechanism is built for. They establish no sizing numbers: neither project states VRAM/RAM minimums or worked examples on 16 GB-class cards (“not stated”). The measured 54 GiB of experts above would fit inside my ~100 GB budget, but whether a specific model fits is arithmetic I would invent if I quoted it as fact. Constraints to note: this is MoE-only (“Only affects MoE models”), dense models still have to fit in combined VRAM, and the path is marked experimental.

The >=80-100 tok/s versus GGUF-CPU-offloading figure circulating for this stack remains unverified: no project source states it or compares itself to llama.cpp at all. Third-party comparisons exist (atomic.chat’s EXL3-vs-GGUF benchmark; a dev.to run of Qwen3.8-Flash-Next with the experts on CPU under 40 GB of RAM) – links below, labelled as community figures I have not reproduced.

Where it fits / where it does not

Fits: a single-user self-hosted OpenAI-compatible API on consumer NVIDIA hardware (Ampere or newer for parallel batching; ROCm images target Radeon RX 7900-series), with quantisation under your control, and a documented route to large MoE models on modest cards – the CPU-expert knobs, experimental as they are marked, are exactly what a 2×16 GB + 100 GB machine wants.

Does not fit: production traffic (TabbyAPI’s own disclaimer); dense models larger than combined VRAM (the offload path affects “MoE models” only); the V2-era EXL2 estate – that line is archived, and I could not confirm current engines read its files. The pair publishes no sizing chart: whether a specific model fits your machine is yours to measure.

How I verified this

Primary sources fetched on 11 November 2026:

Community comparisons and claims found via search, not reproduced by me and not project sources: https://atomic.chat/blog/guides/exl3-vs-gguf , https://bestllmfor.com/guides/tabbyapi-self-hosted-openai-api/ (the latter asserts ExLlamaV3 can load EXL2 files) , https://dev.to/seppegadeyne/run-qwen38-flash-next-locally-with-exl3-and-tabbyapi-on-40-gb-of-ram-3m8d .

Not established by any project source: the >=80-100 tok/s GGUF comparison; sizing numbers for 16 GB cards or ~100 GB machines; whether ExLlamaV3 reads EXL2 checkpoints; model counts on Hugging Face.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

More posts