Run NixOS VM tests under emulation on pull requests that ask for them
Amends ADR-0082: NixOS VM tests, which it rejected, now run in CI, under software emulation and only on request.
Decision
Test a property of the deployed node with a NixOS VM test when it depends on the node as a whole: firewall rules between containers and host services, traffic between nodes over the WireGuard mesh, secrets decrypted through the production path, Patroni bootstrapping against a real Consul, and a cold boot of the whole cluster up to a healthy app. Each test boots the real NixOS configuration in QEMU under TCG on the CI builder, which has no KVM.
The VM tests run only on a pull request carrying a label that asks for them, and the build removes the label once it has run them. Every other pull request build and every master build skips them. A property one daemon on loopback can show stays a Rust integration test in the default suite, as ADR-0082 decided.
Why
ADR-0082 rejected VM tests because no builder has KVM and emulation looked too slow to schedule. Then a firewall mitigation had to stop containers on one node from reaching a host service on any node, across the container bridge and on both IP stacks. A daemon on loopback cannot show that, and the rule took a second revision before it held on IPv6. The WireGuard mesh cutover had the same shape. Emulation turned out slow but workable.
Running them on every build cost too much. The cluster boot test boots the app, so every app change rebuilt and reran it under emulation, which needed a larger and dearer builder for every build and held production deploys behind it. A label puts the cost on the changes that warrant it.
Rejected alternatives
- Keep VM tests out, as ADR-0082 decided. Properties of the firewall, the mesh and the boot order would go untested until production.
- Run them on every build. Every app-only change would pay for the slow run and wait on it to deploy.
- Require the label on any pull request that touches infrastructure. It forced the slow run onto every infrastructure change and every dependency sweep, and still held master deploys behind it.
Consequences
An infrastructure change can merge without a VM run when its author does not ask for one. Production then relies on the rolling deploy's health gate to stop a bad node change (ADR-0093). A VM test that nobody has run recently may have broken without anyone seeing it. Moving the builder to a host with KVM would make the tests fast enough to run by default again.