Detect faults with alerts routed to an agent, not with scheduled diagnostics
Decision
Detect every production fault with a Grafana alert rule, and route every firing alert to a Claude Code agent that triages it from Grafana's logs and metrics and opens an issue or a draft pull request. Alerts do not page a phone; the agent notifies the operator when a human has to act. When a fault is found any other way, the fix includes the alert rule that would have caught it. A full diagnostic sweep of the cluster runs by hand, to find gaps in alert coverage, never on a schedule.
Why
An alert fires within minutes of a fault. A daily sweep finds the same fault up to a day later and produces a report someone has to read. Routing alerts to an agent rather than a pager means the first response is a diagnosis and a proposed fix. One detection mechanism leaves one place to look for coverage gaps.
Rejected alternatives
- A daily agent that runs the full cluster diagnostic. Slower than an alert for every fault an alert can express, and a recurring report becomes noise. A cloud-hosted agent also cannot reach the nodes, which are reachable only over the tailnet.
- Paging a phone for every alert. The agent's triage is the useful first response, and a page adds nothing until a human is needed.
Consequences
Coverage is only as good as the rule set: a fault no rule expresses goes unseen until someone looks, so each manual diagnostic must turn its findings into rules. Nodes that stop reporting and a patch pipeline that stops patching are alerted on. Every alert reaches the operator through one webhook and one agent, so that delivery path is a single dependency for all alerting.