Deliver node secrets as sops files encrypted to each host's SSH key
Amends ADR-0053: the collector's credentials arrive as sops secrets, no longer fetched from 1Password and sealed with systemd-creds at boot. Amends ADR-0078: nodes no longer fetch their tailnet auth key from 1Password at boot.
Decision
Commit every secret a cluster node reads as sops-encrypted files, and let sops-nix decrypt them at activation with an age key derived from the node's ed25519 SSH host key. 1Password stays the source of truth: an operator command reads, validates and re-encrypts every secret. Nodes hold no 1Password credential and fetch nothing at boot.
sops has one data key per file, so scoping is by file: one file for every node, and one per node for what only it may read. Evaluation fails unless each file's recipients are exactly the nodes that should read it, and the declared secrets are exactly those the sync encrypts. Host keys live in 1Password and are written onto a node at install, together with a single-use tailnet auth key, so a rebuilt node keeps the identity its secrets are encrypted to. Deploys pin the host keys and refuse a node that presents another.
Why
Fetching at boot made boot depend on the network, DNS and the 1Password API, and each consumer grew a fetch unit with retries, fallbacks to a copy on disk, and ordering that broke when activation restarted the resolver. Every node also held a long-lived token that could read every cluster vault.
Rejected alternatives
- Keep fetching from 1Password at boot. The failure modes above, and a credential on each node worth more than any secret it fetches.
- agenix. It has no templates, so every service whose config file embeds a secret would need its own unit splicing the secret in, which is the code this removes. sops-nix renders templates and restarts or reloads a secret's units only when its value changes.
- Colmena deployment keys. Secrets arrive only with a deploy, so a reprovisioned node is not correct from config alone, and CI would need a 1Password credential.
Consequences
A secret change is a commit and a deploy. Syncs rewrite files whole, so diffs are unreadable, and per-host scoping means no single reviewable file. No operator or CI key is a recipient. Because the host keys live in 1Password, anyone who can read that vault can decrypt every node's secrets, which is no more than the vault already exposes. Rotating a host key means re-encrypting first. Rotating a password that replicas share with their primary still needs a coordinated change.