25
Backend
04
Security
08
iOS
07
Infra

Test external API calls against recorded reqwest-vcr cassettes

ADR-0034 ACCEPTED · 2025-09-13
Test external API calls against recorded reqwest-vcr cassettes

Decision

Run tests that call OpenAI and other external HTTP APIs against cassettes recorded with reqwest-vcr. By default a test replays its cassette and touches no network. Recording is an explicit mode set by an environment variable: the test calls the real API and writes each response to the cassette. Recording strips credentials, and the date the prompt carries is blanked in the request body so a cassette still matches on a later day. When a change alters a prompt, the shape of a request or the expected behaviour, that same change re-records the affected cassettes.

Why

Once a model writes effects (ADR-0032), a test against the live API is non-deterministic and slow. It also costs money on every run and fails whenever the API is down. A cassette gives the test the model's real output, frozen. We once left re-recording for a follow-up, and a changed request shape reached production untested, which is why the re-record belongs to the change that causes it.

Rejected alternatives

  • httpmock. Better maintained than reqwest-vcr and more capable, and we need none of the extra capability.
  • Hand-written mocks. They test the code against responses we imagined. The model's real responses are what matter.
  • Calling the live API in CI. Too slow, and every run costs money.
  • Evals in place of tests. Evals answer whether the output is any good (ADR-0064). They don't replace deterministic tests against recorded responses.

Consequences

Tests run against real model output, deterministically and offline, and only recording costs money. A cassette pins one response, so it shows the code handles that response and says nothing about whether the model would still produce it. A cassette keeps answering with last month's model until someone re-records it.