Grant tailnet membership reachability only, and make every service authenticate

Decision

Treat tailnet membership as network reachability and grant no authority for it. Every service reachable over the tailnet authenticates its callers itself, and the statement of who may reach it lives in version control.

A service whose only user is a person at a terminal listens on loopback and is reached through an SSH forward. That holds even if the network policy is wrong, so we prefer it wherever it fits.

Everything else is reachable only from named sources, on named ports, through the Tailscale policy file, which OpenTofu applies with the rest of the infrastructure. Nodes and CI are identified by tag, never by IP address. The policy is accept-only, so any port it does not list is closed. Its test block asserts the ports each source must reach and the ports it must not, and Tailscale evaluates the tests at plan time.

The policy file is the only source of truth. When the admin console and the repository disagree, we import the live policy and diff it, and never overwrite the live policy to resolve the conflict.

We reach the cluster from outside over one Tailscale network. The nodes join at boot with an auth key fetched from 1Password and a cluster tag. GitHub Actions joins through a Tailscale OAuth client as an ephemeral node with a CI tag. Deploys and database work from a laptop or CI go over the tailnet. Traffic between nodes stays off it: Consul, Nomad and Postgres replication run over Hetzner's private network.

Why

A mesh VPN gives every authorised machine, including a short-lived CI runner, a stable address for each node with no per-service port forwarding.

That moved the question to who is on the tailnet. Before launch, Nomad's API ran without ACLs, so reaching it meant running code on the nodes. That was safe only if being on the tailnet meant being the operator, and it never did. A tailnet collects devices, such as another project's server or a machine shared in from another account, and Tailscale's default policy lets every device reach every port on every other. That default lived only in the admin console, where no review or deploy saw it. We deferred Nomad's authorization pre-launch and later closed it by enabling Nomad ACLs.

Rejected alternatives

• SSH tunnels to each node. We ran them first. They dropped, needed reconnecting by hand, and fit ephemeral CI runners badly.
• Turning on each service's own authorization and leaving the network open. We rejected it only as a substitute for closing the network. A token rollout to every consumer can take the cluster down halfway through, and narrowing the network cannot. So the network closed first, and Nomad ACLs followed without carrying all the weight while they landed.

Consequences

All outside access to the cluster depends on Tailscale. SSH on the nodes' public addresses stays open to an allow-list as the fallback when Tailscale is down.

A device with legitimate access can lose it when the policy narrows, and the failure looks like a network fault instead of a permission error. A new consumer needs a policy edit and apply; joining the tailnet is not enough.

Tags carry the grants to cluster nodes, so a node that joins untagged is unreachable from CI and from the other nodes. The provisioning flow tags each node as it joins, and a credential that cannot assign the tag fails the join outright.

The policy covers the whole tailnet. If another project shares it, this repository's tooling owns that project's rules too, and that project's devices can't be told apart by tag until it tags them.