Testing a Resource Provider
NixOps4 resource providers can be tested in multiple ways.
Choose a Testing Environment
The choice of testing method depends on the desired trade-offs familiarity and convenience vs. robustness.
The following table provides an overview of the trade-offs, which are explained in more detail below.
Criteria:
📦: Is the test hermetic and reproducible?
❄️: Is it easy to set up NixOS services?
☁️: Can the test access the network?
🏗️: Can the test build derivations?
🚚: Can the test use a Nix cache?
🍏📦: Can it test a macOS build of the application under test?
🍏🧑💻: Can a macOS user test with it?
| Environment | Runner | 📦 | ❄️ | ☁️ | 🏗️ | 🚚 | 🍏📦 | 🍏🧑💻 | notes |
|---|---|---|---|---|---|---|---|---|---|
| Nix sandbox | nixops4-resource-runner | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | |
| Nix sandbox with different storeDir | nixops4 | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | Impractical |
| Nix sandbox with relocated store | nixops4 | ✅ | ❌ | ❌ | ✅ | ✅¹ | ❌ | ❌ | 🚧 Untested |
| Nix sandbox with recursive nix | nixops4 | ⚠️³ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ⚠️³ |
| NixOS VM test | nixops4 | ✅ | ✅ | ❌ | ✅ | ✅¹ | ❌ | ✅² | 🚧 In development, adds ~10s overhead |
| Unsandboxed | either | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | Not perfect, but can be good |
¹: Make sure to add expected build inputs to the check derivation or system.extraDependencies
²: Requires a "remote" builder, which can be provided by nix-darwin's nix.linux-builder.enable
³: The recursive-nix experimental feature is not planned to be supported in the long term and has problems.
Environment
The main differentiator is the environment. The benefits of picking a more restrictive environment include
- ability to run offline
- hermeticity and the ability to
git bisect
These tend to be lost when running outside the Nix sandbox.
If you are testing a provider that interacts with the network, you may have no choice.
Test runner
You may run your tests with nixops4 or nixops4-resource-runner. The latter is simpler and easy to call from a script, and is good for a "unit test" style of testing, whereas nixops4 proper makes it easy to test whole deployments.
Can build
If your test relies on building a derivation, this may be a deciding factor. The Nix sandbox does not normally allow building, but workarounds exist.
Many providers do not require building to test them.
Can use cache
This is only relevant if you are building derivations in the test. Depending on the workaround, you may be able to use pre-built dependencies.
MacOS support
We can distinguish between the ability to test a provider that is built for macOS, versus the ability to test using macOS at all.
A NixOS VM test can be run on a macOS host, but it will not test the provider on macOS.