Test external API calls against recorded reqwest-vcr cassettes
Decision
Run tests that call OpenAI and other external HTTP APIs against cassettes recorded with reqwest-vcr. By default a test replays its cassette and touches no network. Recording is an explicit mode set by an environment variable: the test calls the real API and writes each response to the cassette. Recording strips credentials, and the date the prompt carries is blanked in the request body so a cassette still matches on a later day. When a change alters a prompt, the shape of a request or the expected behaviour, that same change re-records the affected cassettes.
Why
Once a model writes effects (ADR-0032), a test against the live API is non-deterministic and slow. It also costs money on every run and fails whenever the API is down. A cassette gives the test the model's real output, frozen. We once left re-recording for a follow-up, and a changed request shape reached production untested, which is why the re-record belongs to the change that causes it.
Rejected alternatives
- httpmock. Better maintained than reqwest-vcr and more capable, and we need none of the extra capability.
- Hand-written mocks. They test the code against responses we imagined. The model's real responses are what matter.
- Calling the live API in CI. Too slow, and every run costs money.
- Evals in place of tests. Evals answer whether the output is any good (ADR-0064). They don't replace deterministic tests against recorded responses.
Consequences
Tests run against real model output, deterministically and offline, and only recording costs money. A cassette pins one response, so it shows the code handles that response and says nothing about whether the model would still produce it. A cassette keeps answering with last month's model until someone re-records it.