30
Backend
06
Security
12
iOS
21
Infra

Run CI on a self-hosted runner that hands builds to ephemeral Hetzner builders backed by an R2 Nix cache

ADR-0105 ACCEPTED · 2026-11-11
Run CI on a self-hosted runner that hands builds to ephemeral Hetzner builders backed by an R2 Nix cache

Amends ADR-0068: the production deploy runs on an ephemeral builder. Amends ADR-0078: CI jobs reach the tailnet from a standing runner under a tag of their own; only the deploy builder still joins per run as an ephemeral node.

Decision

Run every GitHub Actions job on one self-hosted Hetzner machine outside the cluster, declared in NixOS and OpenTofu like the nodes. Its runners are ephemeral: each job gets a fresh runner, a wiped work directory and an unprivileged user with no Nix trust. The machine's tailnet tag reaches only the cluster APIs CI calls, and each of those authenticates.

Jobs on the runner build nothing themselves. Each build creates an x86 Hetzner machine for the run, builds and tests there, pushes the image and destroys the machine. Builders substitute from a Nix binary cache in Cloudflare R2. Only trusted runs, on master or on a pull request a maintainer has marked trusted, sign and upload to it; every other run reads it with read-only credentials. When Hetzner has no capacity, a scheduled workflow retries the run.

The production NixOS deploy runs on an ephemeral builder too. It joins the tailnet with a single-use key, receives the deploy credential only for that session, and is destroyed on exit; a sweeper removes any that outlive the deploy timeout.

Why

GitHub bills hosted-runner minutes on a private repository, and most of a job's time is spent waiting on the builder. Before the builder, CI ran on GitHub's x86 runners at a higher price, or inside the cluster as Nomad batch jobs that ran out of memory next to production. The cache means a build compiles only what changed.

Colmena needs a Nix daemon that honours remote builders, and Nix grants that only to trusted users. A trusted user on the shared runner is root on a machine that runs unreviewed branch jobs. On a single-use builder nothing else runs.

Rejected alternatives

  • GitHub-hosted runners. Billed by the minute for jobs that mostly wait.
  • Persistent self-hosted runners. A job could copy its runner's credentials or leave state for the next job.
  • Letting any run write the cache. A compromised dependency in a branch could push a validly signed path that a master build then substitutes into the production image.

Consequences

CI depends on Hetzner capacity. The runner is shared by every job, so it reboots into a patched kernel only when no job is running. A deploy builder holds cluster access while it lives, so the deploy and the sweeper bound its lifetime.