Private Local AI blog

Nix and NixOS: the four horsemen of the Nixopalypse

Written by

pendpost

in

Nix and NixOS: the four horsemen of the Nixopalypse

A word about the title first: “Nixopalypse” is our word, not the project’s. I could not find it used anywhere in the Nix project or its community, and if you use it in a sentence someone will look at you oddly. But the four-part framing is real, and it matches what actually hurts when something goes wrong on a Nix system. If you are going to run a service — including an inference stack — on NixOS, it is worth knowing which of the four horsemen you are about to meet.

Nixpkgs today carries more than 140,000 packages and ships a Linux distribution built from the same language. The current release is NixOS 25.11 “Xantusia”, made with 2,742 contributors, and it ships Nix 2.31 in the 25.11 tree while Nix 2.34 is the latest toolchain. The older 25.05 “Warbler” reached end of life on 31 December

  1. So: everything below is written against the 25.11 line, and it is current as of November 2026.

Horseman one: expressions

The language really is a pure, lazy functional language, and the purity claim matters more than the academic one: it means you can ask what was built instead of guessing. Three commands that pay for themselves immediately:

$ nix-instantiate --eval -E 'with import <nixpkgs> {}; lib.version'
"25.11pre…"
$ nix derivation show /run/current-system   # the .drv text of everything in the system
$ nix why-depends nixpkgs#hello /nix/store/…-glibc-…

That is the honest advantage: a build failure in Nix is usually reproducible and inspectable. nix derivation show tells you the exact arguments a derivation was called with, including the environment, and nix why-depends shows the dependency graph as store paths rather than as an abstract concept.

What bites you is not the purity — it’s the surface syntax. Tutorials copy with import <nixpkgs> {}; or buildFHSEnv snippets that are really pkgs.… calls in disguise. Two things to keep in mind:

  • Flakes are still marked experimental. The Nix 2.34 reference manual says flake output attributes “depend on

    • both the flakes and nix-command experimental features, and subject to change without notice”. Every guide you read will use flake.nix; the manual is telling you it is not a stable interface. Decide deliberately whether you want that in a production host — I would not put it on a server I have to debug at 3 a.m.

  • Unfree and broken bits are opt-in by design. Nixpkgs splits free from unfreedom with license flags, so a

CUDA or ROCm runtime may need config.allowUnfree (or the equivalent in your host config) before it will evaluate. Do not treat that as friction: it is the project telling you where the non-licensed parts are.

Horseman two: the operating system

The OS is where NixOS’s reputation for breaking gets earned, and the release notes show why: they are organised by breaking-change category, and upgrading means reading them. In 25.11 the concrete things you must check before upgrading a fleet: Linux 5.4 kernels removed entirely (mainline EOL inside 25.11’s support window), display-manager modules now strictly using tty1 where several previously used tty7, and Nixpkgs dropping macOS versions that Apple itself does not support — 25.11 requires macOS Sonoma 14 or newer for Darwin users.

The safety rails are real, and they are in the manual, so use them:

$ nixos-rebuild switch --test          # build first, do not activate yet
$ nixos-rebuild list-generations       # what you can fall back to
$ nix-store --delete --ignorets …; nix-collect-garbage -d   # the storage you forgot about

--test is the honest habit: build, look, then switch. Generations give rollback — the boot entry pointing at a previous system configuration — which is the difference between “we know what changed” and nothing at all on every other distro I have administered. And stateVersion plus the module migrations are how NixOS forces you to notice: if your config still claims an old release, the module system tells you what has moved. Budget an afternoon per major release for a server that matters. Anyone who tells you they upgrade NixOS in one command is running a machine they do not rely on.

Horseman three: modules, and the systemd units they produce

On NixOS the init system is systemd, and the module system’s main product is unit files. Nearly everything you describe — your inference server, its worker, a timer for model updates, a database — ends up as a generated systemd unit. Say that to yourself before debugging, because it means two things at once: the option you write is a declaration about a unit, and when something goes wrong you inspect it with systemd’s own tools, not only with Nix.

$ nixos-option systemd.services.inference.serviceConfig   # what the modules finally agreed on
$ systemctl cat inference.service                         # the unit as it exists on the machine
$ systemctl status inference; journalctl -u inference -b  # what actually happened at runtime

The contract lives in the unit metadata: systemd.services.<name>.unitConfig (After=, WantedBy=, Requires=, PartOf=), .serviceConfig (ExecStart=, Restart=, resource limits), and .path, .environment, .preStart. Put ordering in the metadata rather than an ExecStartPre that sleeps — that is the difference between a unit that starts correctly and one that works on a fast week. If a package ships its own unit files, systemd.packages installs them instead of you retyping them. And if your release no longer ships nixos-option, evaluate the option from your own configuration with nix eval.

Two module-system truths bite hardest around units:

  • Reading an option in your module reads the final value, not yours. Inside a module, config.services.x.y may

    • be whatever another module set it to; your line is one input among many. If a unit behaves surprisingly, the value came from somewhere else — and the evaluator has flags for this job (--trace-verbose, plus its *-from-only filter) so you can find which file set it.

  • Types are the contract that saves you. types.enum, types.submodule, composed types: they produce good

errors for whoever uses your module. If your options accept anything, they will eventually accept nonsense, and it will surface as a unit that refuses to start rather than as an error.

The advice is unchanged and boring: declare your options before you define them, keep one “glue” module per service instead of four clever ones, and write asserts/warnings' where an invalid combination would otherwise silently produce a unit that starts too early. The module system’s laziness will not warn you; systemd will, at boot.

Horseman four: tests, built around testScript

This is the horseman most people skip, and it is the one that pays for itself. NixOS integration tests let you test a whole multi-machine system inside isolated QEMU/KVM virtual machines declared in Nix code — unlike a unit test, the thing boots a real systemd-based Linux, wires networks between machines, runs commands inside them, and checks services, APIs or GUI state.

A test has three parts: metadata (name), nodes (the VMs), and testScript. Today the function to write against is pkgs.testers.runNixOSTest; older articles you read will use makeTest/nixosTest, which wrap the same driver. Every name you give a node becomes an object inside the script, so the script reads like the topology:

{ pkgs ? import <nixpkgs> {} }:

pkgs.testers.runNixOSTest {
  name = "model-server-comes-up";          # metadata

  nodes = {                                # the machines
    server = { pkgs, ... }: {
      systemd.services.model-server = {
        wantedBy = [ "multi-user.target" ];
        # stand-in for your real inference service - a plain HTTP server is enough to prove the shape
        serviceConfig.ExecStart = "${pkgs.python3}/bin/python3 -m http.server 8080";
      };
      networking.firewall.allowedTCPPorts = [ 8080 ];
    };
    client = { pkgs, ... }: { environment.systemPackages = with pkgs; [ curl ]; };
  };

  testScript = ''                          # Python driving the machines
    start_all()
    server.wait_for_unit("model-server.service")
    server.wait_for_open_port(8080)
    client.succeed("curl -sf http://server:8080/ > /dev/null")
    server.succeed("systemctl status model-server.service | grep -q 'active (running)'")
  '';
}

Run it with nix-build test.nix, or run someone else’s before writing your own — nix build -L 'github:NixOS/nixpkgs#nixosTests.nginx' is a good read of the driver in anger.

testScript is Python: the framework generates a control script that talks to the VMs over stdio or the QEMU guest agent, with superuser rights inside them. start_all() boots every node; wait_for_unit/wait_for_open_port/ wait_until are the patience; succeed/fail are the assertions, and each node exposes the same commands you would type on a running box (systemctl, journalctl). For our inference service the test is small and worth its cost — the unit enabled on one machine, its health endpoint hit from a second machine, one line of log asserted — because it catches the failure mode that used to wake me up: a unit coming up before its network or its model files were ready, which is exactly what the ordering metadata in horseman three is for. Yes, these tests boot real VMs and are slow: scope them to what you changed, keep them out of “run everything” paths, and remember they want hardware acceleration in CI unless you tell it otherwise.

The project’s own gate is worth knowing: ofborg evaluates pull requests, Hydra builds the release and unstable jobs, and since 2025 the merge path for package changes runs through @ofborg + @nixpkgs-merge-bot, where a listed maintainer can land changes to their own packages — with backport merges and a “trusted maintainers” design still being iterated on. The Nixpkgs core team’s own updates (2025-11-04) say so plainly: the merge process is in flux, and there was even a period where the connection to GitHub had to be re-established. That is not a criticism of the machinery; it is what an honest project looks like in public, and it is why you should pin your inputs and keep your own tests.

If you want to run local AI on NixOS

The parts that matter for an LLM host: the cudaQueryLinkerPorts hook exists because CUDA’s pre-built libraries dlopen their plugins at runtime — a pure-functional package manager cannot know about that, so the build-time query makes the closure complete. GPU stacks are in nixpkgs, but under license flags you have to opt into, and the driver/runtime coupling on your host will be whatever the current kernel modules allow. Keep the model files, cache and weights on a partition you back up, not inside /nix/store — the store is reproducible inputs, not state.

What Nix is genuinely good for here: pinning a toolchain and its CUDA/ROCm closure to one revision of nixpkgs so that a working inference box can be rebuilt byte-for-byte somewhere else. What it will not do: make an unfre driver into a source-built one, or spare you the release-notes reading on upgrade day.

The checklist I would actually use

  1. Pin your nixpkgs input to a revision or channel you can name, and treat “unstable” as something you debug, not run.
  2. Read the breaking-changes section of the release notes for the version you are moving to, and again before every

    3. activation: nixos-rebuild switch --test, then switch, with a generation ready to fall back to.

  3. Write typed options for anything you expose to other modules; assume your value is not the final one. Put service

    4. ordering in unit metadata (After=, WantedBy=), never in a start-up script that sleeps.

  4. Put one NixOS integration test around any service you care about — nodes plus a testScript that waits for the

    5. unit and hits its endpoint from a second machine — and run nixpkgs-review / nix flake check before a PR, not after.

  5. Keep state out of the store — weights, caches, databases — so a rollback restores configuration, not your data.

Four horsemen, and one sentence to remember: Nix does not promise you it will never break; it hands you the tools to see what was built, what changed, which value won, and whether your test still passes. That is the whole appeal, and also the whole cost.

Sources

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

More posts