Deploy NixOS nodes with Colmena, building closures on an ephemeral x86 Hetzner builder

Decision

Run every cluster node on NixOS, declared in one flake from shared base modules with per-node overrides, and deploy the operating system layer with Colmena. OpenTofu provisions the Hetzner machines, private network, firewalls and DNS, and writes the node inventory the Colmena hive reads. nixos-anywhere with disko installs a fresh machine. The application ships separately as a Nomad job, so a Colmena deploy changes node services and configuration and leaves the running app alone.

Build the closures on a native x86 Hetzner machine created for each deploy and destroyed when the deploy exits. Colmena evaluates on the Mac with build-on-target turned off. Nix sends every derivation to the ephemeral machine as its only builder, with local jobs set to zero and the machine's SSH host key pinned in the builder spec. Colmena then copies the closures to the nodes over Tailscale and activates them. A deploy host that is already x86 Linux builds locally.

The builder is a nix builder and nothing else. It never talks to the cluster and holds no cluster credentials, so losing or leaking one exposes nothing. The app's deploy image targets the portable x86-64-v2 CPU baseline, because the builder's CPU can differ from the nodes'.

Why

Deploys started as a custom script, which grew its own node ordering, error handling and partial-failure logic. NixOS makes a node's whole configuration a reviewed build artifact: a node rebuilt from the same commit is the same system. Colmena deploys many hosts from one evaluation and reports failures per node, which is the part the script kept reinventing.

The cluster runs x86 NixOS, and we deploy from an ARM Mac that cannot build an x86 Linux closure. When the local host cannot build for the target, Colmena falls back to building on the nodes, and the nodes' nix refuses to set up a build sandbox, so every build there fails. We worked around that without finding the root cause.

Rejected alternatives

• Extending the custom deploy script. It would have reimplemented the ordering and per-node failure reporting Colmena already has, without NixOS's guarantee that a node matches its commit.
• Building on the target nodes. Their nix won't create a build sandbox. If that is ever fixed, building on the nodes becomes an option and the ephemeral builder can go.
• Emulating x86 on the Mac. Nix's emulated x86 builder crashed partway through the closure while building perl, and it would be slow even if it worked.
• A permanent x86 builder. A standing machine to run and patch for a job that happens only during a deploy.

Consequences

Server configuration is reviewed in pull requests, and no node carries hand-made changes. NixOS takes time to learn, and interactions between modules can be hard to debug.

Every deploy needs Hetzner capacity for the builder. Builder creation retries across Hetzner locations and server types, and if none has room the deploy fails; since the cluster itself lives on Hetzner, a capacity outage there already blocks most recovery work. Deploys are slower and have more moving parts than a local build.

The runbook for provisioning and recovering a node, including the traps a brand-new node falls into, lives with the infrastructure code.