Run infrastructure probes as Rust integration tests in the CI suite

Decision

Write every infrastructure check as a test in the Rust workspace suite that CI already runs. A check someone has to remember to run is no control, because nothing reports when it stops being run.

Each test checks one property, is named after it, and starts its own daemon on a free loopback port. It waits until that daemon answers, and it fails if the daemon exits first. A check against a service that never started reports the same green as one that passed, so the readiness wait is mandatory.

Take what a test knows about the deployed configuration from Nix evaluation of the real NixOS configuration, never from a copy. A test that retypes the settings it exercises certifies a configuration nobody deploys, and a grep of the module source finds a value whether or not anything uses it. A value shared by a deployed module and a test is one expression that both read.

Keep inspection out of the suite. A tool that reads the live cluster and prints what it finds has no pass or fail, and turning it into a test would mean inventing a threshold nobody chose.

Why

The security changes to the platform, among them the Nomad ACLs, Redis auth and the Traefik CrowdSec bouncer, each shipped with a probe script that started the real daemon locally, drove it and checked the change worked. A task-runner recipe was the only thing that ran them, so each ran once, on the day it was written. Later a merged policy turned out to have done nothing in production, and the probe that would have said so hadn't run since the merge.

Rejected alternatives

• Probe scripts behind task-runner recipes. They are accurate, and they stop being run without anyone noticing.
• Folding cluster inspection into the suite. It has no pass condition anyone chose.
• NixOS VM tests (runNixOSTest). They exercise the whole deployed unit, including systemd ordering, secrets and the firewall, which a single daemon on loopback does not. They need KVM, and neither builder has it: Hetzner Cloud offers no nested virtualization, and the Mac's linux-builder already runs under software emulation. Each test would either never be scheduled or run emulated twice over. They become the better tool if the builder moves to a host with KVM.

Consequences

The suite needs the daemons it tests wherever it runs. It builds them from the nixpkgs revision the cluster's flake pins, so the tests exercise the versions the cluster deploys, and a cold machine needs network access to fetch them. Each test adds seconds to every CI build, and tests that evaluate the NixOS configuration add more.

A converted probe is done only when each test has been seen to fail with its property broken. A test that stays green through the break asserts something other than its name.