Apply every merge to production with a leader-last rolling deploy
Amends ADR-0068: CI, not an operator's Mac, runs the node deploy on merge, and a rollout script now owns node ordering and health around Colmena, which ADR-0068 had rejected.
Decision
A merge to master is the production deploy. When a merge changes what the nodes run, CI rolls it out one node at a time: replicas first and the Patroni leader last, with leadership switched away before its node is touched, and Redis masters failed back to their declared nodes afterwards. Every node must be healthy before the rollout starts and after each step, and the rollout stops at the first unhealthy node. When the kernel changes, the node is drained from Nomad and rebooted. Every production-changing workflow joins one queue, and none is cancelled once started.
Why
When deploying is a step separate from merging, it happens only when someone remembers it, and a node a deploy leaves broken stays broken until the next one. Daily patching (ADR-0092) is worth only as much as the patch reaching the nodes. Merge is already where a human approves the change.
Colmena reports failure per host, but it knows nothing about which node leads Patroni or whether the cluster is healthy, so ordering and health have to live outside it. That is the custom-script logic ADR-0068 rejected, reintroduced only for what Colmena cannot see.
Rejected alternatives
- Manual deploys after merge. Nodes can run stale or broken while master is correct.
- Colmena one host at a time with a health check, and no ordering. The leader can go first, so every rollout risks an unplanned failover under load, and a reboot of the leader becomes the switchover.
- Pull-based deploys, each node applying master itself. No ordering across nodes, so leader-last would have to be coordinated through the cluster itself.
- Never rebooting automatically. A kernel update takes effect only after a reboot, so kernel patches would wait for a person.
Consequences
A bad change to the nodes reaches production within minutes, and the health gate stops it after one node; a revert is the rollback. App deploys go through the separate Nomad deploy and are not covered by this gate. A rollout that fails partway can leave a node drained or leadership moved, and the failed job names the step. The cluster's health definition now gates deploys, so a check that is wrong either blocks every deploy or lets a broken one through.