Run local Grafana and Tempo for development observability
Context
Production traces go to Grafana Cloud via Alloy (ADR-0053). But iterating on tracing instrumentation against production has a slow feedback loop and costs money per trace. During TDD, you want to see traces for the request you just made, immediately, without worrying about ingestion delays or sampling.
Decision
Run a local Grafana instance (port 8000) with Tempo (port 3200) for development. The local Alloy configuration routes traces to local Tempo instead of Grafana Cloud. Same instrumentation code, different destination.
Consequences
Free, instant trace visibility during development. Iterate on instrumentation without production costs. Same trace format as production so what you see locally is what you get in prod.
Agents can query the local Tempo API programmatically — the same MCP tools (mcp__tempo__*) work against both local and production Tempo. This gives agents access to real trace data when debugging, which is significantly better than parsing logs. Parity between local and production observability means agent workflows work identically in both environments.
The cost is running additional services locally (Grafana + Tempo), but these are lightweight.