Use Alloy as credential boundary for telemetry
Context
The backend and iOS app export OpenTelemetry traces and metrics to Grafana Cloud and Langfuse. The recommended OpenTelemetry approach is to export to a local collector rather than directly to the destination — the collector handles authentication, retries, buffering, and routing.
Decision
Grafana Alloy runs on each cluster node as the telemetry collector. Backend and iOS export to localhost Alloy with no authentication. Alloy forwards to Grafana Cloud and Langfuse with the appropriate credentials.
Credentials are fetched from 1Password at boot via a systemd service, encrypted into systemd's credential store (systemd-creds encrypt), and loaded by Alloy via LoadCredentialEncrypted. Application code never sees production telemetry tokens.
Consequences
Follows the recommended collector pattern. One place to manage and rotate credentials. Adding a new telemetry destination is an Alloy config change, not an application change.
The cost is an additional service on each node and the 1Password dependency for credential bootstrapping.