Testing a Resource Provider

NixOps4 resource providers can be tested in multiple ways.

Choose a Testing Environment

The choice of testing method depends on the desired trade-offs familiarity and convenience vs. robustness.

The following table provides an overview of the trade-offs, which are explained in more detail below.

Criteria:

📦: Is the test hermetic and reproducible?
❄️: Is it easy to set up NixOS services?
☁️: Can the test access the network?
🏗️: Can the test build derivations?
🚚: Can the test use a Nix cache?
🍏📦: Can it test a macOS build of the application under test?
🍏🧑‍💻: Can a macOS user test with it?

¹: Make sure to add expected build inputs to the check derivation or system.extraDependencies

²: Requires a "remote" builder, which can be provided by nix-darwin's nix.linux-builder.enable

³: The recursive-nix experimental feature is not planned to be supported in the long term and has problems.

Environment

The main differentiator is the environment. The benefits of picking a more restrictive environment include

• ability to run offline
• hermeticity and the ability to git bisect

These tend to be lost when running outside the Nix sandbox.

If you are testing a provider that interacts with the network, you may have no choice.

Test runner

You may run your tests with nixops4 or nixops4-resource-runner. The latter is simpler and easy to call from a script, and is good for a "unit test" style of testing, whereas nixops4 proper makes it easy to test whole deployments.

Can build

If your test relies on building a derivation, this may be a deciding factor. The Nix sandbox does not normally allow building, but workarounds exist.

Many providers do not require building to test them.

Can use cache

This is only relevant if you are building derivations in the test. Depending on the workaround, you may be able to use pre-built dependencies.

MacOS support

We can distinguish between the ability to test a provider that is built for macOS, versus the ability to test using macOS at all.

A NixOS VM test can be run on a macOS host, but it will not test the provider on macOS.

Testing with nixops4-resource-runner

The nixops4-resource-runner tool provides a simple way to test resource providers by invoking all provider operations directly. See the resource provider interface for details about the protocol.

Example: Testing a stateless resource

nixops4-resource-runner create \
  --provider-exe nixops4-resources-local \
  --type file \
  --input-str name test.txt \
  --input-str contents "Hello, world!"

See nixops4-resource-runner create.

Example: Testing a stateful resource

# Create with state persistence
nixops4-resource-runner create \
  --provider-exe nixops4-resources-local \
  --type memo \
  --stateful \
  --input-json initialize_with '"initial value"'

# Update the stateful resource
nixops4-resource-runner update \
  --provider-exe nixops4-resources-local \
  --type memo \
  --inputs-json '{"initialize_with": "new value"}' \
  --previous-inputs-json '{"initialize_with": "initial value"}' \
  --previous-outputs-json '{"value": "initial value"}'

The --stateful flag indicates that state persistence will be provided to the resource. Resources that require state must fail if this flag is not set.

See nixops4-resource-runner create and nixops4-resource-runner update.

Example: Testing state resources

State resources provide persistent storage for resource state using JSON files and incremental updates via JSON Patch operations.

Creating a state file resource

# Create a new state file
nixops4-resource-runner create \
  --provider-exe nixops4-resources-local \
  --type state_file \
  --input-str name "deployment-state.json"

This creates a JSON file with the initial state structure containing empty resources and deployments.

Reading state from a state resource

# Read the current state
nixops4-resource-runner state-read \
  --provider-exe nixops4-resources-local \
  --type state_file \
  --inputs-json '{"name": "deployment-state.json"}' \
  --outputs-json '{}'

This returns the complete current state as reconstructed from all recorded events. Initially, this will show an empty state with no resources.

Recording state changes

# Add a resource to the state
nixops4-resource-runner state-event \
  --provider-exe nixops4-resources-local \
  --type state_file \
  --inputs-json '{"name": "deployment-state.json"}' \
  --outputs-json '{}' \
  --event "create" \
  --nixops-version "4.0.0" \
  --patch-json '[
    {
      "op": "add",
      "path": "/resources/myfile",
      "value": {
        "type": "file",
        "inputProperties": {"name": "test.txt", "contents": "hello"},
        "outputProperties": {}
      }
    }
  ]'

# Update a resource in the state
nixops4-resource-runner state-event \
  --provider-exe nixops4-resources-local \
  --type state_file \
  --inputs-json '{"name": "deployment-state.json"}' \
  --outputs-json '{}' \
  --event "update" \
  --nixops-version "4.0.0" \
  --patch-json '[
    {
      "op": "replace",
      "path": "/resources/myfile/inputProperties/contents",
      "value": "updated content"
    }
  ]'

# Remove a resource from the state
nixops4-resource-runner state-event \
  --provider-exe nixops4-resources-local \
  --type state_file \
  --inputs-json '{"name": "deployment-state.json"}' \
  --outputs-json '{}' \
  --event "destroy" \
  --nixops-version "4.0.0" \
  --patch-json '[
    {
      "op": "remove",
      "path": "/resources/myfile"
    }
  ]'

State events use JSON Patch (RFC 6902) operations to record incremental changes to the state. This enables efficient state updates and provides a complete audit trail of all state modifications.

For details about the state file format and structure, see State.

See nixops4-resource-runner state-read and nixops4-resource-runner state-event.