Trace from the app through to the backend via a per-node Grafana Alloy collector that holds every credential

Decision

Trace the iOS app with the OpenTelemetry Swift SDK and continue its traces on the server. Every GraphQL operation gets a client span, and an Apollo interceptor sends that span's context as a W3C traceparent header, so a tap and the server work behind it form one trace.

Run Grafana Alloy on every cluster node as the only holder of telemetry credentials. The backend and the app export OTLP to Alloy without authentication: the backend finds it through Consul, and the app reaches it through a public OTLP endpoint behind Traefik. Alloy forwards traces and metrics to Grafana Cloud, and the LLM spans to Langfuse, with the real credentials. A oneshot systemd unit fetches those credentials from 1Password at boot and encrypts them with systemd-creds, and Alloy loads them through LoadCredentialEncrypted. Application processes and the app binary never hold a production telemetry token.

Why

The backend was traced already, but every request from the app started an orphan trace. Network time, cache time, the UI work the user waited through, and mobile conditions such as a weak connection or a backgrounded app were all invisible. A standard propagation header joins the two halves without a custom correlation id.

OpenTelemetry's recommended topology exports to a nearby collector that owns authentication, retries, buffering and routing. A token compiled into an iOS binary can be extracted by anyone who downloads the app, so the app cannot hold one, and the collector is the natural place for the backend's tokens too.

Rejected alternatives

• Each application exporting straight to each destination. The app would embed a Grafana Cloud token in a public binary, the backend would carry production tokens in its Nomad environment, and neither would get the collector's retries and buffering. Adding or changing a destination would mean redeploying application code.

Consequences

We can follow a request from the tap to the database query. Rotating a credential or adding a destination touches Alloy's configuration and nothing else. Because the app holds no credential, the public endpoint cannot authenticate who sends to it; it only receives, and exposes neither data nor credentials. The app pays for an SDK dependency, per-operation overhead and battery spent exporting, and a trace-id ratio sampler is the lever on volume. Span attributes must never carry personal data. Each node runs an extra service and cannot ship telemetry until 1Password has supplied its credentials at boot.