butterpot and vmrunner: system and test-infrastructure co-design
Since early 2024, I have been working on and off on butterpot, a Btrfs-filesystem-based backup system for - primarily - my own use.
Several alternatives already exist, but none provide or provided the exact feature set I have in mind, so I started sketching an architecture.
What butterpot does on the client can be summed up briefly as:
btrfs subvolume snapshot create $subvol $snapshots/$name
btrfs send $snapshots/$name | socat
assuming an existing subvolume $subvol on a Btrfs filesystem which also contains $snapshots.
On the server side:
socat | btrfs receive $snapshotarchive/$client/$name
assuming a Btrfs volume on $snapshotarchive.
Now, the above could all be done as non-root, but deleting the created snapshots requires root or a special mount option. Deleting snapshots is required to operate the system for a longer period of time - and hence also requires test coverage.
A test harness could be constructed around a wrapper setuid binary, with chattr +C for a file-backed Btrfs filesystem, fallocate to allocate space on another disk, losetup to make the file-backed device mountable, and mkfs.btrfs to actually set up the filesystem.
That would work, technically speaking.
Do I want to serve this to unsuspecting open-source enthusiasts, risking their data when they run an innocent cargo test?
The answer to this is unequivocally No!
On the other hand, I want repeatability and consistency, and I want to be able to try the "odd config" too.
Summing up the requirements:
- isolate operations from the user's host system
- idempotent testing
- rootless operation
- integration processes
- matrix testing against kernel and Btrfs versions (preferred)
- minimal setup required (optional)
Containers with their default runtimes are insufficient, since the filesystem is still shared. Using a container-based setup in conjunction with a virtual-machine-based isolation mechanism would work, but also require more host setup than I'd prefer - it shifts much of the burden to the developer and assumes availability of tooling.
VMs seem to be a viable path. Using root filesystem images with predefined setup scripts checks all the boxes.
Add an abstraction that makes them easy to use as a supplement to or drop-in replacement for #[test] annotations.
From system requirements to test infrastructure
Creating a VM is straightforward. Making VM-backed tests repeatable, fast~ish (as in latency between test invocation and actual test execution), frictionless, and rootless across macOS and Linux hosts - with Linux guests - is the harder problem.
Windows hosts or non-Linux guests are out of scope for the time being, mostly because I don't own a Windows device and the Btrfs driver implementation for Windows is somewhat recent. Since Btrfs is only supported on Linux, Linux guests are sufficient for end-to-end tests. We might revisit this at a later point in time.
Now we can nail down the requirements in more detail:
Repeatable
The simplest way to guarantee a known state is to never reuse an environment. Each test starts with an ephemeral guest, using a given rootfs image. The rootfs changes are discarded on VM termination; the next test must start from a fresh vanilla ice cream state.
Fast~ish
Disposable environments are practical only when creating and starting them is cheap, both in resources and time. Work that does not depend on an individual test - such as preparing the guest image and building a compatible sysroot - should happen once and be cached where possible.
The per-test path should be limited to creating fresh writable state, launching the guest, and running the test binary.
Frictionless
Rust makes ordinary tests pleasantly direct with #[test]. Conceptually, a VM-backed test should feel similar to
The injected TestCase provides the default environment and can launch multiple machines within a single test case.
Rootless
On Linux, starting a VM does not inherently require root, but parts of the setup need elevated capabilities. vmrunner uses unshare to create a user namespace and become scoped "pretend-root" instead of running the test as host root.
On macOS, the binary launching the VM must be signed.1 For a unit test, this implies signing the test binary and re-executing it before launching VMs through the macOS sandbox API.
Stretch goal: Multi-participant tests
System isolation also lets us test interactions between isolated entities by launching multiple VMs within the scope of a single test declaration.
System Architecture
The current implementation separates reusable preparation from per-test execution.
Let's put the actual rootfs preparation aside for a second and start with execution.
Execution
The Rust test body orchestrates VMs on the host, and only the requested test invocation runs inside the guest.
cargo test -> #[vmrunner::test(...)]
|
v
VM Rootfs Preparation
|
v
Process preparation
|
Linux: `unshare --user --map-root-user --net --fork`
macOS: ad-hoc sign with Hypervisor entitlement, then re-exec
|
v
+--------------------------------------------------{process}---+
| Per-test host child |
| |
| `TestSetup` -> test body -> `TestCase::launch()` |
| | |
| +--------------+--------------+ |
| | fork() | fork() |
| v v |
| +----------------{process}--+ +----------------{process}--+ |
| | VM host process A | | VM host process B | |
| | +-----------------{VM}--+ | | +-----------------{VM}--+ | |
| | | init | | | | init | | |
| | | -> test invocation | | | | -> test invocation | | |
| | +-----------------------+ | | +-----------------------+ | |
| +---------------------------+ +---------------------------+ |
| | | |
| +--- virtio-net / sockets -----+ |
| | |
| v |
| Userspace L2 switch |
| (per logical Network) |
| | |
| v |
| Host uplink (optional) |
| |
| Guest console -> host pipes -> stdout/stderr capture |
| Child exit -> wait()/wait_with_output() -> test assertions |
+--------------------------------------------------------------+
┌──────────────────────────────────────────────────────────────┐
│ VM Rootfs Preparation │
│ │
│ qcow2 image download URL → Download cache check │
│ │ hit │ miss │
│ │ ↓ │
│ │ HTTP GET → temporary file │
│ │ verify* → download path │
│ ↓ │ │
│ Cached source ←─────┘ │
│ │ │
│ Known built-in image URL? │
│ │ no │ yes │
│ │ ↓ │
│ │ Prepared-image cache check │
│ │ │ hit │ miss │
│ │ │ ↓ │
│ │ │ mkosi │
│ │ ↓ │ │
│ │ Prepared image ←───┘ │
│ │ │ │
│ ↓ ↓ │
│ ┌────────────────────────────────┐ │
│ local ────→ │ Shared base qcow2 │ │
│ path │ (reused; read-only at runtime) │ │
│ └───────┬────────────────┬───────┘ │
│ │ │ │
│ ↓ ↓ │
│ ┌──────────────────────────┐ ┌──────────────────────────┐ │
│ │ RW overlay A │ │ RW overlay B │ │
│ │ (ephemeral) │ │ (ephemeral) │ │
│ └────────────┬─────────────┘ └─────────────┬────────────┘ │
│ │ │ │
│ ↓ ↓ │
│ ┌──────────────────────────┐ ┌──────────────────────────┐ │
│ │ guest A / │ │ guest B / │ │
│ └──────────────────────────┘ └──────────────────────────┘ │
└──────────────────────────────────────────────────────────────┘
The setup cache persists across test runs; the writable overlays do not. Local paths bypass downloading and automatic image preparation. Files ending in .xz are decompressed before use, whether downloaded or local. The diagram omits that step.
For downloads, the paths relative to the invoking process's working directory are:
C = target/vmrunner-qcow2-imag
Known built-in URL:
C/sha256/<expected-image-sha256>/<URL-basename>
Other HTTP(S) URL:
C/<URL-basename>
The directory really is named vmrunner-qcow2-imag. For the Ubuntu URL in the example below, <URL-basename> is ubuntu-26.04-server-cloudimg-arm64.img. Query strings and fragments are excluded, and the filename is sanitized.
verify* means SHA-256 verification for recognized built-in URLs: cached downloads are rehashed, and new downloads must match the expected digest before being installed at the download path. A matching file in the older C/<URL-basename> location can also populate the digest-keyed cache without downloading. Download writers use a per-file lock, recheck the cache after acquiring it, and publish through a temporary file. Other URLs are cached by filename and existence only: they have no checksum or freshness check, and different URLs with the same filename can collide.
mkosi prepares recognized built-in images with the development packages needed to extract a compiler sysroot for building init.krun. On a prepared-image cache miss, vmrunner copies OS directories out of the downloaded qcow2 using libguestfs, passes that directory tree to mkosi build via --base-tree, then converts mkosi’s raw disk output back to qcow2.
The configuration lives in Rust: the built-in MKOSI_QCOW2_PREP table maps each recognized image URL to a distribution, release, architecture, tools-tree choice, and package list, which vmrunner passes as command-line arguments. Fedora adds glibc-devel, glibc-static, and kernel-headers; Ubuntu adds libc6-dev and selects --tools-tree default rather than using host tools. The preparation settings are built in and cannot be changed per test; local paths and unrecognized URLs bypass this step. The prepared output is cached separately:
<download-parent>/mkosi/<distribution>/<mkosi-architecture>/
sha256/<source-image-sha256>/
prep-sha256/<preparation-sha256>/<source-stem>.mkosi.qcow2
The preparation key includes the image URL, distro/release, architecture, tools-tree selection, package list, and a fingerprint version. If that output exists, preparation is skipped; changing a keyed input selects a new cache entry. The prepared image becomes the read-only shared base for the per-node overlays.
The compiler sysroot and guest-init sidecar are separate setup caches: extracted sysroots live at target/vmrunner-sysroot/sha256/<resolved-image-sha256>/linux-<arch>-<abi>, while .<resolved-image-filename>.init.krun sits beside the resolved image. The sysroot must pass header/libc validation; sidecar reuse requires a matching fingerprint of the init binary, image, and sysroot contents.
The boundaries have different jobs:
- Host privileges: Linux maps the invoking user to UID 0 inside a new user namespace and creates a network namespace for the test. UID 0 in that namespace does not grant host-root privileges. The orchestrator still has the user's ordinary file access, without a host-filesystem sandbox. On macOS, signing grants access to Hypervisor.framework; it does not create a namespace sandbox.
- Guest execution: Each node has its own VM, kernel instance, and memory. Filesystem operations run against the guest's virtual disk rather than a host mount. The kernel comes from
libkrunfw, independently of the distro image; changing the root image changes userspace while keeping the kernel version fixed. - Disk state: Writes go into a fresh per-node qcow2 overlay, leaving the shared base unchanged. Normal completion and waiting release the temporary overlays; subsequent launches create new ones. Neither the host root nor the workspace is exported as a guest directory share.
- Networking: Nodes on the same logical network can communicate through the userspace switch. There is no host uplink by default; TAP,
gvproxy, orHostUplinkconnectivity is explicit. On Linux, reaching the host network or the internet also requires a configured network connection and route out of the test's network namespace. Creating a guest-facing TAP interface alone does not provide that connection, and the namespace does not inherit the host's network interfaces or routes.
What runs inside the VM?
The guest runs the compiled test target. vmrunner invokes a command equivalent to:
./target/<mode>/<bin> <args>
The macro and TestCase keep orchestration in the Rust test, while filesystem-changing work runs inside the disposable Linux guest. The requested executable must already be present in the guest image: the current runtime injects init.krun, but does not automatically build or copy the test payload from the host.
Configurability
In practice, being able to specify the guest rootfs is an important configuration option.
async
Current state and performance
vmrunner is functional and usable, but still slow compared with an ordinary unit-test workflow.
Most of the first-run cost comes from generating a sufficient sysroot for the test binary. In particular, the linker needs compatible glibc symbols for the guest operating system. The process is:
- Prepare the guest by installing the required libraries.
- Use
libguestfsto extract the relevant files from the root filesystem in theqcow2image. - Use that sysroot to compile the test binary for compatibility with the guest.
- Launch the VMs and run the test.
The first three steps are the most time-consuming. All three can be cached, but CI must absorb a few minutes of preparation when that cache is cold, such as after changes to the VM image.
Host requirements
The current implementation relies on:
unshareand access to/dev/kvmon Linuxlibkrunlibguestfs- guest images in
qcow2format, e.g. from Fedora Server (look forqcow2) or Ubuntu Noble cloud images (look for.imgfile extensions)
The host requirements are likely to change as the implementation evolves. The goal is to reduce them without making compilation impractically expensive.
Next
butterpot created the need for isolated, reproducible filesystem tests, and those needs shaped vmrunner's API and lifecycle. In return, vmrunner's startup and preparation costs influence how butterpot's test suite should be divided.
Logic that does not need a full guest can remain in ordinary Rust tests. Tests that exercise real filesystem behavior can use vmrunner and accept the additional isolation cost, incurred mostly by additional I/O.
The next steps are mostly driven by butterpot and user needs.