Encrypt node-to-node traffic with a static WireGuard mesh
Amends ADR-0078.
Decision
Carry all traffic between cluster nodes over a WireGuard mesh, and on the private network interface admit only WireGuard itself and the load balancer's connections to the HTTP entrypoint.
Every node has a static mesh address and its own keypair. The private key reaches the node through sops-nix, encrypted to the node's SSH host key, and is never written to the Nix store. The public keys and the peer list are in version control, and every peer is static. Consul, Nomad, Postgres and Patroni, Redis and Sentinel, and the telemetry collector advertise mesh addresses to each other, and where a service binds a specific address it binds the mesh one. Mesh traffic fails closed: while the tunnel is down, an unreachable route refuses it instead of letting it out through the default route.
ADR-0078 says traffic between nodes stays off the tailnet and runs over Hetzner's private network. It still stays off the tailnet, and it now crosses the private network only inside WireGuard.
The load balancer cannot be a WireGuard peer, so its leg to the entrypoint stays plain HTTP on the private network. We accept that leg as Hetzner-internal. The requests on it were TLS-terminated at the load balancer. On the private network only the load balancer can reach the entrypoint, and only the load balancer may send it PROXY protocol headers.
Why
The private network belongs to Hetzner, not to us. Postgres replication, Redis replication and Sentinel, Consul and Nomad RPC, and Nomad's API carry data and credentials across it, and we want that traffic encrypted and its senders authenticated on a network we don't control. A single mechanism below every service does that for all of them.
Rejected alternatives
- TLS on each service. Postgres, Redis, Consul and Nomad each want their own certificates, and some want their own CA, so we would run several CAs to rotate. Serf gossip would still need a separate encryption key.
- Moving cluster traffic onto Tailscale. Cluster liveness and node join would then depend on Tailscale's coordination servers and our tailnet policy. An outage or a policy mistake there could partition the database cluster.
Consequences
A new node needs a keypair before it joins, and every existing node needs a deploy that adds it as a peer.
A node whose secrets carry no WireGuard key has no tunnel and stays out of the cluster.
The mesh authenticates nodes, not processes. A compromised node can still reach every service on its peers, and each service's own authentication remains the next line of defence, as ADR-0078 requires.