30
Backend
06
Security
12
iOS
21
Infra

Detect faults with alerts routed to an agent, not with scheduled diagnostics

ADR-0094 ACCEPTED · 2026-10-25
Detect faults with alerts routed to an agent, not with scheduled diagnostics

Decision

Detect every production fault with a Grafana alert rule, and route every firing alert to a Claude Code agent that triages it from Grafana's logs and metrics and opens an issue or a draft pull request. Alerts do not page a phone; the agent notifies the operator when a human has to act. When a fault is found any other way, the fix includes the alert rule that would have caught it. A full diagnostic sweep of the cluster runs by hand, to find gaps in alert coverage, never on a schedule.

Why

An alert fires within minutes of a fault. A daily sweep finds the same fault up to a day later and produces a report someone has to read. Routing alerts to an agent rather than a pager means the first response is a diagnosis and a proposed fix. One detection mechanism leaves one place to look for coverage gaps.

Rejected alternatives

  • A daily agent that runs the full cluster diagnostic. Slower than an alert for every fault an alert can express, and a recurring report becomes noise. A cloud-hosted agent also cannot reach the nodes, which are reachable only over the tailnet.
  • Paging a phone for every alert. The agent's triage is the useful first response, and a page adds nothing until a human is needed.

Consequences

Coverage is only as good as the rule set: a fault no rule expresses goes unseen until someone looks, so each manual diagnostic must turn its findings into rules. Nodes that stop reporting and a patch pipeline that stops patching are alerted on. Every alert reaches the operator through one webhook and one agent, so that delivery path is a single dependency for all alerting.