Run the app on Nomad and Consul and keep stateful services on NixOS
Decision
Schedule the API container with Nomad and register services in Consul. Both run as NixOS services on every cluster node, each node a Nomad and Consul server as well as a Nomad client, and Nomad starts containers through Podman. Production runs its copies on separate nodes, and each preview environment is a further job on the same cluster (ADR-0055).
Nomad schedules only workloads that hold no data. Postgres, Redis and Sentinel, Traefik and the telemetry agent are NixOS services that ship with the node configuration (ADR-0068), and Consul and Nomad themselves are installed the same way.
Why
The cluster is three small machines, and the only thing that needs scheduling is one stateless application image in several environments. Nomad and Consul are single binaries with NixOS modules, so the orchestrator is configured, upgraded and rolled back like any other node service.
Keeping stateful services on NixOS puts each one beside its disk with a fixed placement, and a deploy of the application image cannot move or restart a database.
Rejected alternatives
- Kubernetes, including a lightweight distribution such as k3s. We judged that its control plane, network plugin and ingress controller would take a large share of three small nodes to schedule a single application, and that its value for stateful services depends on operators we would then run in place of NixOS modules.
- Running the app as a NixOS service with no scheduler. Every image change would become a node deploy, and per-branch previews would need machinery that Nomad jobs already provide.
- Running Consul and Nomad in containers. HashiCorp advises running them on the host, and the host is already declared in NixOS.
Consequences
The Consul and Nomad quorums span the three nodes, so the cluster tolerates losing one node and stops scheduling when two are gone. Running Nomad's servers beside the workloads keeps the machine count at three, at the cost that a node failure removes a server and a client together. Consul later became the consensus store for Patroni as well (ADR-0101), so one quorum backs routing and the database, and a Consul outage reaches both. Stateful services gain nothing from the scheduler: replacing a database node is a node deploy, and failover is Patroni's and Sentinel's job.