Llama.cpp – What is it?

Llama.cpp is an inference engine for large language models, written in plain C and C++ with no framework dependencies. Its stated goal is to run LLM and vision-model inference "with minimal setup and state-of-the-art performance on a wide range of hardware – locally and in the cloud". In practice that means you take a model file, point a command-line tool at it, and tokens come out of your own machine instead of someone else’s datacentre.

The repository lives at https://github.com/ggml-org/llama.cpp and is built on top of ggml, a general tensor-library with its own operator kernels. llama.cpp adds the model loading, quantisation, sampling, prompt templating and server layers above it.

How you actually use it

There are several ways in, and they all do the same thing:

# installer script
curl -LsSf https://llama.app/install.sh | sh

# download and run a model straight from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# start an OpenAI-compatible API server plus a built-in web UI
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

The last command is the one that makes it usable as a building block: llama serve exposes a local OpenAI-compatible endpoint, so anything written against that API – agent loops, chat UIs, the little assistant that answers questions on this site – can be aimed at a model running on a machine you control. Pre-built binaries and a Docker image are also published, and building from source is a matter of cloning the repository and following the build guide.

Models are usually obtained as GGUF files, which llama.cpp’s convert_hf_to_gguf.py script produces from Hugging Face checkpoints when a publisher doesn’t ship a GGUF build.

Why people reach for it

• Hardware breadth. The backend table in the README covers Metal, CUDA, HIP, Vulkan, SYCL,

OpenCL, BLAS, BLIS, CANN for Huawei Ascend, Hexagon for Snapdragon, IBM zDNN, MUSA, RPC, WebGPU and ZenDNN, plus plain CPU paths with ARM NEON, Accelerate, AVX/AVX2/AVX512/AMX and RISC-V extensions. Apple silicon is treated as a first-class target rather than an afterthought.

• Quantisation. 1.5-, 2-, 3-, 4-, 5-, 6- and 8-bit integer quantisation, so a model that

would not otherwise fit in memory or run acceptably fast can still run at all. The k-quants and i-quants families let you choose where the quality/size tradeoff sits.

• Hybrid offload. When a model is bigger than your VRAM, llama.cpp keeps part of it on the

GPU and the rest in system RAM (--n-gpu-layers for layer offload). For mixture-of-experts models there are dedicated flags: --cpu-moe keeps all expert weights in CPU memory, and --n-cpu-moe N keeps only the experts of the first N layers there while attention stays on the GPU. That distinction is what makes a large MoE usable on a single consumer card.

• No dependency chain. No CUDA toolkit installation for the CPU path, no Python runtime

required to serve a model. That matters when you maintain a small number of machines rather than a cluster.

What it does not do: it isn’t a training stack, and it isn’t trying to be the fastest possible answer on a rack of datacentre GPUs. It is optimised for the hardware people actually have.

What has changed recently

The most visible addition is speculative decoding, now documented in docs/speculative.md. The --spec-type flag accepts a list of implementations: draft models (draft-simple), EAGLE-3 drafts that read the target model’s hidden states (draft-eagle3, with supported speculator checkpoints for Llama 3.x, Qwen3 and gpt-oss), block-diffusion drafts (draft-dflash, draft-dspark), Multi Token Prediction heads from the main model itself (draft-mtp), and a set of n-gram self-speculation implementations (ngram-simple, ngram-map-k, ngram-map-k4v, ngram-mod) that need no extra model at all because they draft from patterns already in the token history. --spec-default turns on the hash-pool n-gram variant, and each implementation prints its own acceptance statistics so you can see whether it is actually helping. The same docs point at a SPEED-Bench client in tools/server/bench/speed-bench for measuring end-to-end gain.

Other current ground: OpenAI-compatible server plus a built-in web UI for chat and vision models, the llama command wrapper and llama.app installer, native MXFP4 support for the gpt-oss models across CUDA, Vulkan, Metal and CPU (see the discussion in ggml-org/llama.cpp#15095), and MoE offload options beyond the layer-split – community benchmarks such as developersdigest.tech’s October 2026 post compare --moe-cache-mib against --n-cpu-moe and treat them as different trades for short chat turns versus long contexts.

Where that leaves us

The honest way to use llama.cpp is as an infrastructure piece: it converts a model file into a local API endpoint, and everything else – retrieval, prompts, tooling – sits on top of that endpoint. That is exactly how this site’s assistant works: a model on our own hardware behind a local API, rather than a per-token bill against someone else’s server. If you want to try it, the shortest path is the installer above and a small GGUF; the next step is llama serve and then whatever you already have that speaks the OpenAI API.