Register Nomad job services under the Nomad agent's Consul token instead of workload identities
Decision
Run Consul with ACLs on and a default policy of deny. Give the Nomad agent one Consul token whose policy grants service write only on the names our jobs register, and let it register every job's services. Configure no Consul workload identity and no JWT auth method. Every other Consul consumer gets its own least-privilege token. A reconciler on each node creates the tokens from the management token, with each token's secret held in 1Password and delivered as a node secret (ADR-0106).
Why
ADR-0078 makes every service authenticate its own callers, and Consul's API was root to anything that reached it. Our jobs use Consul only to be found: they register services, and their templates read Nomad Variables, never the Consul catalog or KV. Workload identity exists to give each task a Consul token for what it reads, or a service-mesh identity, and we use neither. Nomad still registers a service with no identity block under the agent's token, so that path covers us. Workload identity would add an auth method, binding rules, and Consul trusting Nomad's JWKS endpoint, which is new surface where an auth bypass could appear, all to scope access we don't grant.
Rejected alternatives
- Workload identities with a Consul JWT auth method, which HashiCorp recommends since Nomad 1.7. Revisit when a job's template reads from Consul, or a job joins a service mesh.
- The built-in Nomad templated policies. They grant service write on every name, and the server one adds ACL write. A compromised Nomad agent could then rewrite any service's registration, including the ones that route database traffic.
- A default token for tokenless local requests. Every process on the node would inherit its rights, which is the hole this closes.
Consequences
A job whose service name falls outside the Nomad policy registers under allow and is refused under deny, so a new service name is a policy change. Consul syncs each service's health under the token it was registered with, so rotating the Nomad token means restarting the agent and removing its services for Nomad to re-register.