Follow the Sentinel election instead of declaring Redis masters
Amends ADR-0093: the rollout no longer fails Redis masters back to declared nodes.
Decision
Name no Redis master in the node configuration. Each Redis server and Sentinel asks the cluster's Sentinels for the elected master every time it starts and follows their answer: a server becomes a replica of that master unless it is the master, and a Sentinel monitors it. When the Sentinels that answer disagree, or refuse the node's credentials, the node does not start and retries until they agree. Only when no Sentinel answers, as on a cold start of the whole cluster, does a node fall back to a seed fixed in configuration. The rollout's health gate asks only that each master is elected and that every other node is linked to it as a replica, wherever it lives.
Why
Sentinel moves the master on failover, but configuration that names a master stays put, so the next restart drags the cluster back to the declared node, and a deploy gate that compares the two refuses every deploy until a person intervenes.
The configuration is still rewritten from scratch on every start, which keeps Sentinel's own config rewrites from piling duplicate monitor lines on top of the declared ones.
Rejected alternatives
- Declare masters and fail back after every deploy. Breaks every deploy after an unplanned failover, and each fail-back is a deliberate extra failover.
- Follow the answer most Sentinels give. With one Sentinel down mid-failover, the two left can split one to one, and a tie broken by anything other than the election starts a second master.
- Persist the state Sentinel rewrites into its config across restarts. A rebuilt node has no such state and would start as a lone master, and the rewritten file collides with the declared monitors.
Consequences
Masters stay where Sentinel last put them, so the cache and store Redis masters can share a node until a later failover separates them. Write protection (a master refuses writes without a connected replica) is configured on every node, since any node can be a master. A cold start of every node at once trusts the seed, whose data may be older than the last master's.