Private Local AI blog

Aphrodite Engine – a serving engine in the vLLM lineage, now renamed Sonar, and what my two 16 GB cards would need from it

Written by

in

First, the finding that changes how this post should be read, because I checked it first: the repository at PygmalionAI/aphrodite-engine no longer exists under that name. The GitHub API returns a permanent redirect for it (repository id 657414607 – the same id behind dphnAI/sonar, created 23 June 2023); the README at main now describes the project as Sonar, and the one at tag v0.22.0 reads “Sonar (formerly Aphrodite Engine)”. This is not a successor project that claims to continue it: it is the same repository, renamed in place. The commit that did it – “feat!: rename Aphrodite Engine to Sonar”, dated 7 November 2026 – landed in v0.25.0, released 8 November 2026, the first release whose wheels are named sonar-.whl. Until v0.24.1 (September 2026) they were aphrodite_engine-.whl.

What it is

The current README describes Sonar as “an inference engine for Hugging Face-compatible language and multimodal models” providing “continuous batching, paged KV-cache management, optimized kernels, quantization, speculative decoding, and distributed serving”. On lineage it states: “Sonar is based on vLLM. It includes additional model and quantization formats, sampling methods, kernels, platforms, and deployment features.” It says the engine “serves production workloads for the Dolphin Inference Network and PygmalionAI” (now under the dphnAI org, with PygmalionAI still listed as a served deployment).

The Aphrodite-era README, readable at old tags such as v0.10.0, used the same self-description: “Aphrodite is an inference engine that optimizes the serving of HuggingFace-compatible models at scale. Built on vLLM’s Paged Attention technology, it delivers high-performance model inference for multiple concurrent users.” It also states Aphrodite was “Developed through a collaboration between PygmalionAI and Ruliad”, serving both organisations’ chat platforms – so the project started, in its own words, as roleplay infrastructure. One v0.6.7-era README has a caution box worth quoting: “This project is not, and will never be, associated with any cryptocurrencies. Please do not fall for scams.”

Is it maintained? The GitHub API reports pushed_at 9 November 2026; the latest open release is v0.25.0 (8 November 2026). The licence is AGPL-3.0, per the LICENSE file at the repository root.

The migration notice, as the project states it

The Sonar docs have a “Migrate from Aphrodite” section: install sonar in a fresh virtual environment, because “the old aphrodite-engine PyPI package does not install the renamed release”. Imports and CLI change from aphrodite to sonar, environment variables from APHRODITE_* to SONAR_* with no fallback, identifiers like AphroditeConfig become SonarConfig. The temporary aphrodite command “forwards to Sonar and warns at startup”; native wheels and serialized objects from the two eras “are not interchangeable”.

How it works

The core is vLLM’s: continuous batching and paged KV-cache management, as both eras of the README say. What keeps the lineage claim precise is the commit history: merge commits titled “chore(sync): sync to upstream <sha>” and “port vLLM through <sha>”, plus “[sync]” commits annotated “Upstream-vLLM: <sha>”. This is a downstream fork of vLLM that periodically pulls from upstream – stated by its own release notes, not inferred.

What it adds, in the README’s key-features list: prefix caching enabled by default; tensor, pipeline, data and expert parallelism; “Multi-node multiprocessing without a Ray cluster”; prefill/decode disaggregation through NIXL and other KV connectors; quantized weights and FP8 KV cache; speculative decoding with MTP, EAGLE, DSpark, DFlash and n-gram methods; structured output, reasoning parsers and automatic tool calling; image, audio and video models; LoRA serving; Prometheus metrics; and a set of front-end APIs – OpenAI, Anthropic, pooling, scoring, reranking, transcription and Kobold.

The quantization reference is what makes the support claims checkable: a matrix generated from the current source tree lists awq, auto_awq, gptq, gptq_marlin, awq_marlin, FP8 variants, exl3, bitsandbytes, compressed-tensors, the ModelOpt family, mxfp4, torchao and more, each with a minimum NVIDIA streaming-multiprocessor capability; the page notes “NVIDIA CUDA accepts all registered methods”. The v0.10.0 README listed AQLM, AutoRound, AWQ, BitNet, Bitsandbytes, EETQ, GGUF, GPTQ, QuIP#, SqueezeLLM, Marlin, FP2-FP12 and others in its feature bullet. One caution about the older ExLlama format: the current matrix lists exl3, not EXL2, and while every old README I saw credits Exllamav2 in acknowledgements, no feature list names EXL2 as a method – so the project does not state EXL2 support anywhere I looked. The samplers it has claimed since 2024 (DRY, XTC, Mirostat) and the Kobold API stay roleplay-facing: both eras of the README put the server on port 2242, and the old README pointed at SillyTavern.

What it asks of your hardware

The README’s requirements at v0.22.0: Linux or Windows via WSL2; Python 3.10 to 3.13; CUDA 12 or newer. The current installer covers Linux x86-64 with NVIDIA CUDA, AMD ROCm or CPU, Arm64, and Apple-silicon macOS with Metal; the full installation guide adds Intel XPU, Google TPU and Docker paths. On VRAM the README is explicit: “By design, Sonar takes up 92% of your GPU’s VRAM”, pointing at --gpu-memory-utilization if that is unwanted.

For multiple cards, the parallelism page gives the design in order: “Start with one GPU. Add parallelism only when one GPU cannot meet the memory or throughput requirement.” Tensor parallelism splits each layer across GPUs and should stay “within one NVLink or high-bandwidth GPU domain”; pipeline parallelism communicates less often and can cross nodes; data parallelism makes replicas; expert parallelism moves only MoE layers. The failure-pattern table lists “More GPUs reduce throughput” – cause: “Communication dominates”.

For my own rig – two 16 GB consumer cards, about 100 GB of system RAM – everything here is framed as what that machine would need from the engine, because I ran nothing and quote no numbers. From Sonar I would need a model that fits, once quantised, in both cards’ combined VRAM – and the matrix’s per-method GPU-generation notes are where I’d check whether my cards are even permitted for the format I choose. A two-card tensor-parallel group over plain PCIe sits exactly where the docs advise care: consumer boards have no NVLink domain. The ~100 GB of system RAM buys me nothing on the pages I checked: the model-storage page covers Hugging Face caches, offline snapshots and load formats – no weight tiering into system RAM anywhere.

Where it fits / where it does not

This blog’s other engine notes – llama.cpp, Strata, Avalon, PolyStrata and Xolotl – all bet that weights belong in system RAM or on NVMe and move through the bus as needed: Xolotl’s README describes itself as “Built from Strata, and grown a long way past it” carrying MoE experts up from RAM and drives; Avalon’s runs its model with “the experts in system RAM” on one consumer card. Sonar/Aphrodite is the opposite bet, and its own README states the lineage: weights sit on GPUs, split across ranks when they must, and the engine’s job is throughput for many concurrent users rather than making a too-big model fit anyway. On my two cards that means Sonar is only the right shape if a quantised model fits in both together; for the flagship-class MoEs that family chases, nothing in the pages I fetched competes with Xolotl’s expert tiering.

Where else it does not fit: the licence is AGPL-3.0, which rules it out for stacks that cannot ship AGPL code. And name-hygiene matters now – the old Pygmalion docs host, aphrodite.pygmalion.chat, serves a certificate that does not match its hostname and returned nothing usable when I checked; anything current is at sonar.dphn.ai and github.com/dphnAI/sonar; the migration page above covers upgrades under the new name.

How I verified this

All fetched on 11 November 2026:

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

More posts