From d7d02a7d94bbacb2ef50eb055b71ed5856f41002 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 26 Sep 2026 23:24:28 -0700 Subject: [PATCH 1/2] Repin OpenVMM for caller-owned microVM virtio-fs requests Promote the `openvmm` submodule from `c9f659c36` to `65cc1c2a5`, the head of nanvix/openvmm#97. That pull request adds `--mount-owner process|caller` for the microVM `--mount` attachment: - `lxutil: add scoped per-thread filesystem credentials` adds a guard that makes the calling thread perform filesystem operations exactly as another UID and GID. It switches the filesystem UID and GID, clears the supplementary groups, drops the effective capabilities, verifies every step, and restores the previous credentials when dropped. - `virtiofs: perform microVM HostFs requests as the guest caller` runs each FUSE request under that guard when `caller` is selected. Guest root is squashed to the export owner, and a request that cannot switch identity fails with `EPERM`. `process` remains the default. The sandbox agent and CLI changes that use the new option follow in the next commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 85d0240d-2f5e-4169-ae5d-540b7edc3b7b --- openvmm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openvmm b/openvmm index c9f659c36..65cc1c2a5 160000 --- a/openvmm +++ b/openvmm @@ -1 +1 @@ -Subproject commit c9f659c36c8cb36d5aac745f7c26808296689f3f +Subproject commit 65cc1c2a5a878eb5a0e65eef313c42de301ab19f From 450c548d3a3424c39fcd2d0f8a133b18b9adc446 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 26 Sep 2026 23:24:50 -0700 Subject: [PATCH 2/2] Mount the live virtio-fs share inside sandbox workloads In sandbox mode (`nvx_sandbox=1`), OpenVMM attaches the `--mount` share and passes `virtfs_dir`, `virtfs_tag`, and `virtfs_mode` on the kernel command line, but nothing ever mounts it. The initramfs hook runs only outside sandbox mode, and `nvx-init-agent` ignores the tokens. The capability-stripped workload cannot mount the share itself, so a host directory exported with `--mount` never reaches the sandbox (#216). Guest agent: - After assembling the overlay, and before it resolves the workload identity or starts any workload, `nvx-init-agent` mounts the `microvm` tag at `virtfs_dir` inside the container root, with the requested mode plus `nosuid,nodev`. The mount lives below the overlay, so it survives the workload's private mount namespace and `chroot`. The one-shot agent unmounts it before the overlay, and the managed agent inherits it for every exec. - The agent fails closed with `NVX-SANDBOX-ERROR` and status 125 if the tag or mode is unexpected, or if the target is not canonical. It also fails closed if the target is `/etc`, `/etc/machine-id`, or lies under `/proc`, `/sys`, `/dev`, or `/.nvx-agent`, all of which the container launcher manages or rewrites. The same applies when a component of the target in the image is a symbolic link or a non-directory, or when the mount fails. Components are checked and created one at a time before any workload exists, so an image symlink cannot redirect the mount outside the container root. Missing components are created as root-owned `0755` directories in scratch. - Without `--mount`, sandbox behavior is unchanged. CLI: - `sandbox run` and `sandbox provision` accept `--mount`, `--mount-deny`, and `--mount-owner {process,caller}`. The defaults are `caller` on Linux, so workload-created files are owned by the export owner when the workload identity owns the export, and `process` on Windows. Managed state records the share with an absolute export root. Configurations provisioned before this change still load without a share. - The launch contract rejects the reserved targets before boot, and counts the bootstrap tokens that OpenVMM appends against the sandbox's 1024-byte command-line budget. - `run` forwards an optional `--mount-owner`. Tests: - A new `sandbox-filesystem` microVM scenario boots the real sandbox agent over the Ubuntu EROFS layer. It shares `/workspace` read-write, hides `secrets/` with `--mount-deny`, and verifies what the issue asks for. The workload sees a host file. Its writes, nested directory, and `chmod` appear on the host while it still runs, with no copy-back. A host edit made during the run is visible to the workload. The denied path is absent. The outcome report records a clean virtio-fs teardown. - On Linux the workload identity owns the export and the share uses `--mount-owner caller`, so the scenario also checks the host owner of every file the workload creates. When the tests run as root they export a dedicated owner, and the scenario formats its scratch image with an upper `/etc/passwd` that defines that identity. - The scenario then verifies that a read-only share rejects workload writes with `EROFS`, and that the agent rejects a reserved target and a target that traverses the image's `/bin` symlink before any workload starts. - Unit tests cover the launch contract, CLI forwarding and validation, managed-state round-tripping, and the scenario's command construction and failure handling. Documentation: - `doc/run.md` documents the share's caching, the fact that host files are shown as-is with `--mount-deny` for credentials, the unchanged egress policy, the single-share contract, `ENOTSUP` for guest symlink creation, host ownership with `--mount-owner`, and the sandbox mount behavior. - `doc/design/current-limits.md`, the sandbox design, validation, and usage references follow. - Runner provisioning installs `e2fsprogs` for the scenario's scratch image. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 85d0240d-2f5e-4169-ae5d-540b7edc3b7b --- doc/design/current-limits.md | 16 +- ...ndbox-filesystem-and-agent-architecture.md | 19 +- doc/design/validation.md | 7 + doc/run.md | 94 ++++- doc/usage.md | 24 +- guest/common/init | 1 + guest/common/nvx-init-agent | 60 +++ scripts/nvx.py | 58 ++- .../sandbox-filesystem-read-only.sh.in | 27 ++ .../sandbox-filesystem.sh.in | 33 ++ scripts/nvx_tools/microvm_tests.py | 360 +++++++++++++++++- scripts/nvx_tools/sandbox.py | 130 ++++++- scripts/nvx_tools/sandbox_lifecycle.py | 33 +- scripts/setup/README.md | 4 +- scripts/setup/setup-linux-mshv.sh | 6 +- scripts/setup/setup-linux-runner.sh | 6 +- scripts/test_microvm_tests.py | 233 ++++++++++++ scripts/test_nvx_tools.py | 328 ++++++++++++++++ 18 files changed, 1412 insertions(+), 27 deletions(-) create mode 100644 scripts/nvx_tools/microvm_test_scripts/sandbox-filesystem-read-only.sh.in create mode 100644 scripts/nvx_tools/microvm_test_scripts/sandbox-filesystem.sh.in diff --git a/doc/design/current-limits.md b/doc/design/current-limits.md index 0d94df72c..35d09ebd5 100644 --- a/doc/design/current-limits.md +++ b/doc/design/current-limits.md @@ -18,9 +18,23 @@ The current ABI family intentionally does not provide: - public activation of the reserved control console or an authenticated guest RPC broker; - replaceable sandbox configuration, a production OCI agent, or a public sandbox snapshot workflow; - serialization of live network flows or native host handles; -- snapshotting of the contents of a live virtio-fs export; or +- snapshotting of the contents of a live virtio-fs export; +- more than one virtio-fs share, or different access modes within one share; +- guest creation of symbolic links on the live virtio-fs share; +- caller-owned virtio-fs requests on Windows, or propagation of the guest + caller's supplementary groups; +- mounting a restore-time virtio-fs attachment inside a sandbox workload root; or - compatibility with standalone NVX `MVMSNAP*` or `WHPSNAP*` files. +Sandbox mode does support the live virtio-fs share on a cold boot. The agent +mounts it inside the workload root with `nosuid,nodev`, rejects targets that it +manages or that traverse an image symbolic link, and fails closed if the mount +fails. `--mount-deny` and read-only mode are enforced by the host server as in +the standard profile. With `--mount-owner caller`, Linux hosts perform each +request as the guest caller, map guest root to the export owner, and fail +requests that cannot switch identity with `EPERM`. See +[Live host share in the sandbox](../run.md#live-host-share-in-the-sandbox). + The tier contract and post-restore gate are currently low-level OpenVMM and guest-agent primitives exercised by the native lifecycle suite. The public `nvx sandbox` command does not yet orchestrate platform builds, warm-shim diff --git a/doc/design/sandbox-filesystem-and-agent-architecture.md b/doc/design/sandbox-filesystem-and-agent-architecture.md index abbf18287..5991d13f7 100644 --- a/doc/design/sandbox-filesystem-and-agent-architecture.md +++ b/doc/design/sandbox-filesystem-and-agent-architecture.md @@ -20,11 +20,11 @@ or snapshot-tier metadata alone. ## Implemented filesystem bootstrap The public `nvx sandbox` command accepts one to three role-bearing EROFS lower -images, a preformatted ext4 scratch image, an absolute entrypoint, and -individual argument tokens. It supplies non-secret kernel-command-line -configuration; environment variables, secrets, arguments containing -whitespace, and sandbox snapshot orchestration are not supported by this -command. Lower-level OpenVMM capture and restore do support sandbox blocks. +images, a preformatted ext4 scratch image, an absolute entrypoint, individual +argument tokens, and an optional live virtio-fs share. It supplies non-secret +kernel-command-line configuration; environment variables, secrets, arguments +containing whitespace, and sandbox snapshot orchestration are not supported by +this command. Lower-level OpenVMM capture and restore do support sandbox blocks. See [Run](../run.md#experimental-single-workload-sandbox) and [`scripts/nvx_tools/sandbox.py`](../../scripts/nvx_tools/sandbox.py). @@ -46,10 +46,13 @@ device filter does not mean the current agent installs one. 4. resolve and flush scratch, mount it as ext4 with `rw,nosuid,nodev`, and create its `upper` and `work` directories; 5. mount overlayfs with top-first lower layers and `metacopy=on,xino=on`; -6. start a child behind a FIFO barrier, place that child in the workload +6. when OpenVMM attached a live virtio-fs share, validate its command-line + target, create any missing components without traversing image symbolic + links, and mount the share inside the overlay with `nosuid,nodev`; +7. start a child behind a FIFO barrier, place that child in the workload cgroup before releasing it, and retain a runtime-tmpfs machine-ID file; -7. wait for the child, unmount the overlay, layers, and scratch, and return its - status through the guest exit helper. +8. wait for the child, unmount the share, overlay, layers, and scratch, and + return its status through the guest exit helper. The assembled view is: diff --git a/doc/design/validation.md b/doc/design/validation.md index c756d14dd..03626a0cf 100644 --- a/doc/design/validation.md +++ b/doc/design/validation.md @@ -36,6 +36,13 @@ pre-entry rejection of missing, corrupt, mismatched, or wrong-geometry media. Denied-filesystem coverage verifies listing suppression, allowed writes, direct and parent-relative denial, symlink/junction alias denial, a second virtio-fs mount, and pre-boot rejection of unsafe path policies. +Sandbox live-share coverage boots the real sandbox agent over the Ubuntu EROFS +layer with a read-write share. While the workload runs, it verifies host-file +visibility, denied-path hiding, nested workload writes and `chmod` that appear +on the host immediately, and a host edit that the workload observes. On Linux +it also verifies that caller-owned requests create those files as the export +owner. It then verifies a read-only share that rejects writes, and agent +rejection of reserved and symlinked mount targets before the workload starts. The native suite targets KVM, MSHV, and WHP; a passing run on one backend is not a fresh result for the others. Coverage also includes diff --git a/doc/run.md b/doc/run.md index 79bd66cc2..e137f198b 100644 --- a/doc/run.md +++ b/doc/run.md @@ -252,7 +252,9 @@ backing and are onlined before restore readiness. ## virtio-fs host mapping The microVM reserves one mapping slot with a fixed `microvm` tag. On a cold -boot with `--mount`, the initramfs mounts it automatically: +boot with `--mount`, the initramfs mounts it automatically. In the sandbox +profile, the agent instead mounts it inside the workload root; see +[Live host share in the sandbox](#live-host-share-in-the-sandbox): ```bash python3 scripts/nvx.py run --mount "/mnt/host,/absolute/host/share,rw" @@ -283,6 +285,44 @@ mkdir -p /mnt/host mount -t virtiofs microvm /mnt/host ``` +The share is live host state with no guest caching: entry and attribute cache +lifetimes are zero and file I/O is direct. Host and guest changes are visible to +the other side immediately. In exchange, every path lookup is a round trip to +the host, which metadata-heavy workloads such as package installs will notice. +Files appear exactly as they are on the host, and NVX strips nothing from a live +share. Hide credentials and other sensitive paths with `--mount-deny`. The +device adds no network path, so the network and egress policy are unchanged. + +By default, OpenVMM performs every guest request as its own host identity, so +guest-created files are owned by the OpenVMM user. On Linux, +`--mount-owner caller` performs each request as the guest caller's UID and GID +instead, and maps guest root to the owner of the export root: + +```bash +python3 scripts/nvx.py run --mount "/mnt/host,/absolute/host/share,rw" \ + --mount-owner caller +``` + +Caller mode rejects an export root owned by UID 0 or GID 0. Each request runs +without OpenVMM's supplementary groups and effective capabilities, so a +privileged OpenVMM process cannot lend the guest its privileges. OpenVMM must +run as the export owner or hold `CAP_SETUID` and `CAP_SETGID`. Without those +capabilities, only requests from OpenVMM's own identity, including guest root, +succeed, and they keep OpenVMM's supplementary groups; any other caller gets +`EPERM`. Supplementary groups of the guest caller are not propagated. Because +the guest names the caller, guest root can act as any non-root host user inside +the export. Grant the capabilities only for exports that hold no other users' +files. Windows rejects `--mount-owner caller`: files are created by the OpenVMM +user, and the guest sees attributes derived from that user's access. The owner +mode is host policy rather than snapshot state, so a restore selects it again. + +The microVM has exactly one share, with a single access mode. For example, a +read-write workspace and a read-only tool cache cannot be exported as two +shares. A second share would need a new microVM ABI version. Guest creation of +symbolic links returns `ENOTSUP`, so tools that create links, such as +`npm ci` with `node_modules/.bin`, must run in scratch rather than in the +share. Existing host symbolic links are visible and are resolved by the guest. + ## Experimental single-workload sandbox The `sandbox` command launches the microVM with one to three compressed @@ -336,6 +376,58 @@ The outer agent retains the initramfs root; the capability-stripped child enters only the assembled root with `chroot`, because Linux cannot `pivot_root` away from an initramfs `rootfs`. +### Live host share in the sandbox + +Add `--mount GUEST_TARGET,HOST_PATH[,ro|rw]` to `sandbox run` or +`sandbox provision` to live-share one host directory with the workload: + +```bash +python3 scripts/nvx.py sandbox \ + --layer distro,build/ubuntu-distro.erofs,11111111-1111-1111-1111-111111111111 \ + --scratch /var/lib/nvx/scratch.ext4 \ + --workload-user 1001:1001 \ + --mount /workspace,/home/runner/work/repo,rw \ + --mount-deny /home/runner/work/repo/.secrets +``` + +After it assembles the overlay, and before it starts any workload, the agent +mounts the share at `GUEST_TARGET` inside the workload root with the selected +mode and `nosuid,nodev`. The mount lives below the overlay, so it survives the +workload's private mount namespace and `chroot`. The workload itself still has +no capabilities, so it cannot mount or unmount the share. The share exists only +in the workload root; the agent does not mount it anywhere else. A one-shot +agent unmounts it before the overlay. The agent fails closed with +`NVX-SANDBOX-ERROR` and status 125, never starting the workload, if any of +these holds: + +- the target is not canonical; +- the target is `/etc` or `/etc/machine-id`, or lies under `/proc`, `/sys`, + `/dev`, or `/.nvx-agent`, all of which the agent manages; +- a component of the target in the image is a symbolic link or a + non-directory; +- the mount fails. + +Missing target components are created as root-owned `0755` directories in +scratch. The marker `NVX-SANDBOX-VIRTFS: mounted microvm at GUEST_TARGET (MODE)` +reports a successful mount. Without `--mount`, the sandbox is unchanged. + +The workload accesses the share as its fixed `--workload-user` identity. The +guest kernel checks access against the host owners and modes, so choose the +identity that owns the exported directory, such as the runner user; the image +must define that user, as for any `--workload-user`. On Linux, the command +passes `--mount-owner caller` by default. OpenVMM then performs workload +requests as that identity and agent requests as the export owner, so with an +identity that owns the export, every file the workload creates on the share is +owned by that host user. The export root must not be owned by UID 0 or GID 0. +OpenVMM must run as the export owner or hold `CAP_SETUID` and `CAP_SETGID`; +otherwise the workload's share requests fail with `EPERM`. Use +`--mount-owner process` to perform every request as the OpenVMM process +instead. Windows supports only `process`. A +read-only share rejects workload writes with `EROFS`. `--mount-deny` paths stay +hidden inside the container. The policy, caching, and link limits in +[virtio-fs host mapping](#virtio-fs-host-mapping) apply unchanged. A sandbox +snapshot restore does not mount a newly attached share for the workload. + For a state-aware sandbox, provision configuration without starting a VM, start it once, run multiple workloads in the same warm guest, stop it while retaining configuration, and finally deprovision it: diff --git a/doc/usage.md b/doc/usage.md index c620ea0bb..ac511d846 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -200,11 +200,15 @@ python3 scripts/nvx.py test-microvm Runs NVX-owned Linux, SMP, virtio, sandbox, and snapshot correctness scenarios against the public OpenVMM CLI. Repeat `--scenario` to select a subset; without it, every scenario supported by the selected guest runs. Alpine remains the -default. Ubuntu rejects the Alpine-control-only `sandbox-blocks` and -`scratch-snapshot` scenarios, the Alpine-prompt-specific `console-snapshot` -scenario, and the sandbox-control-dependent `snapshot-tiers` scenario. The -command requires `build/vmlinux`, the selected initramfs, and -`openvmm/target/release/openvmm[.exe]`. +default. Ubuntu rejects the Alpine-control-only `sandbox-blocks`, +`sandbox-filesystem`, and `scratch-snapshot` scenarios, the +Alpine-prompt-specific `console-snapshot` scenario, and the +sandbox-control-dependent `snapshot-tiers` scenario. The command requires +`build/vmlinux`, the selected initramfs, and +`openvmm/target/release/openvmm[.exe]`. The `sandbox-filesystem` scenario also +requires `build/ubuntu-distro.erofs` and its manifest. On Linux it formats its +scratch image with `mkfs.ext4`; on Windows it copies +`build/ubuntu-smoke-scratch.ext4`. ### `test-adversarial` @@ -296,6 +300,8 @@ python3 scripts/nvx.py run [--memory-capacity-mib MIB] [--processors {1,2,4,8}] [--mount GUEST_TARGET,HOST_PATH[,ro|rw]] + [--mount-deny HOST_PATH]... + [--mount-owner {process,caller}] [--net IPV4/PREFIX] [--network-profile {portable}] [--network-egress {allow,deny}] @@ -322,6 +328,8 @@ python3 scripts/nvx.py run | `--memory-capacity-mib MIB` | none | Reserve an immutable, 128 MiB-aligned RAM capacity for a fresh microVM snapshot. | | `--processors {1,2,4,8}` | `1` | Select the microVM processor count. | | `--mount GUEST_TARGET,HOST_PATH[,ro\|rw]` | none | Expose one host directory to the absolute guest target. Active snapshot restore requires the same canonical path, target, and mode; a dormant-slot restore may attach a new mapping that the resumed guest mounts explicitly. | +| `--mount-deny HOST_PATH` | none | Hide an existing path inside the `--mount` export. Repeat for multiple paths; relative paths are resolved against the export root. | +| `--mount-owner {process,caller}` | OpenVMM default (`process`) | Select the host identity for guest requests on `--mount`. `caller` is Linux-only; see [virtio-fs host mapping](run.md#virtio-fs-host-mapping). | | `--net IPV4/PREFIX` | none | Enable virtio-net with the static guest IPv4 address and prefix. | | `--network-profile {portable}` | none | Select the required cross-platform network behavior contract; must be specified with `--net`. | | `--network-egress {allow,deny}` | `allow` | Set the default guest egress policy. | @@ -354,6 +362,9 @@ python3 scripts/nvx.py sandbox [--hostname NAME] [--memory-max BYTES] [--pids-max COUNT] + [--mount GUEST_TARGET,HOST_PATH[,ro|rw]] + [--mount-deny HOST_PATH]... + [--mount-owner {process,caller}] [--memory-mib MIB] [--hypervisor {auto,whp,kvm,mshv}] [--net IPV4/PREFIX] @@ -371,6 +382,9 @@ python3 scripts/nvx.py sandbox | `--hostname NAME` | `nvx-sandbox` | Set the workload UTS hostname. | | `--memory-max BYTES` | none | Set the workload cgroup memory limit. | | `--pids-max COUNT` | none | Set the workload cgroup process limit. | +| `--mount GUEST_TARGET,HOST_PATH[,ro\|rw]` | none | Live-share one host directory at the absolute target inside the workload root; valid for `run` and `provision`. The target cannot be `/etc`, `/etc/machine-id`, or a path under `/proc`, `/sys`, `/dev`, or `/.nvx-agent`. | +| `--mount-deny HOST_PATH` | none | Hide an existing path inside the export. Repeat for multiple paths; relative paths are resolved against the export root. | +| `--mount-owner {process,caller}` | `caller` on Linux, `process` on Windows | Select the host identity for workload requests on the share. | | `--memory-mib MIB` | `256` | Set guest memory in MiB. | | `--hypervisor {auto,whp,kvm,mshv}` | `auto` | Select the host hypervisor. | | `--net IPV4/PREFIX` | none | Enable virtio-net with a static guest address. | diff --git a/guest/common/init b/guest/common/init index 73dbf906b..05c07e914 100644 --- a/guest/common/init +++ b/guest/common/init @@ -111,6 +111,7 @@ fi mount_virtfs() { /sbin/nvx-hostmount || fatal "virtfs: failed to mount live host directory" } +# The sandbox agent mounts the share inside the container root instead. if [ "$sandbox_mode" = false ]; then mount_virtfs fi diff --git a/guest/common/nvx-init-agent b/guest/common/nvx-init-agent index ebfb047c5..daa7063ad 100644 --- a/guest/common/nvx-init-agent +++ b/guest/common/nvx-init-agent @@ -126,6 +126,60 @@ mount_layer() { fi } +# Mounts the optional live virtio-fs share inside the container root. The +# workload cannot mount it itself because it runs without capabilities, and a +# mount below the overlay survives the private mount namespace and chroot. +mount_virtfs_share() { + virtfs_tag=$(cmdline_value virtfs_tag || true) + [ -n "$virtfs_tag" ] || return 0 + [ "$virtfs_tag" = microvm ] || fatal "unexpected virtio-fs tag: $virtfs_tag" + virtfs_dir=$(cmdline_value virtfs_dir || true) + virtfs_mode=$(cmdline_value virtfs_mode || true) + case "$virtfs_mode" in + ro | rw) ;; + *) fatal "invalid virtio-fs mode: $virtfs_mode" ;; + esac + case "$virtfs_dir" in + /*) ;; + *) fatal "virtio-fs target must be an absolute path: $virtfs_dir" ;; + esac + # The appended slash also catches the root and trailing dot components. + case "$virtfs_dir/" in + *//* | */./* | */../*) + fatal "virtio-fs target is not a canonical path: $virtfs_dir" + ;; + /proc/* | /sys/* | /dev/* | /.nvx-agent/* | /etc/ | /etc/machine-id/) + fatal "virtio-fs target is reserved by the sandbox: $virtfs_dir" + ;; + esac + + # Nothing else runs yet, so checking and creating each component in turn + # cannot race with an image symlink that redirects the mount outside the + # container root. + virtfs_path=$rootfs + virtfs_remainder=${virtfs_dir#/} + while [ -n "$virtfs_remainder" ]; do + virtfs_component=${virtfs_remainder%%/*} + case "$virtfs_remainder" in + */*) virtfs_remainder=${virtfs_remainder#*/} ;; + *) virtfs_remainder= ;; + esac + virtfs_path=$virtfs_path/$virtfs_component + [ ! -L "$virtfs_path" ] || + fatal "virtio-fs target traverses a symbolic link: $virtfs_dir" + if [ ! -e "$virtfs_path" ]; then + mkdir -m 0755 "$virtfs_path" || + fatal "failed to create virtio-fs target: $virtfs_dir" + fi + [ -d "$virtfs_path" ] && [ ! -L "$virtfs_path" ] || + fatal "virtio-fs target is not a directory: $virtfs_dir" + done + mount -t virtiofs -o "$virtfs_mode,nosuid,nodev" "$virtfs_tag" "$virtfs_path" || + fatal "failed to mount the virtio-fs share at $virtfs_dir" + virtfs_target=$virtfs_path + echo "NVX-SANDBOX-VIRTFS: mounted $virtfs_tag at $virtfs_dir ($virtfs_mode)" +} + mkdir -p "$layers" "$scratch" "$rootfs" mount -t tmpfs -o mode=755,nosuid,nodev tmpfs /run || fatal "failed to mount runtime tmpfs" @@ -208,6 +262,8 @@ mkdir -p "$scratch/upper" "$scratch/work" mount -t overlay overlay \ -o "lowerdir=$lowerdirs,upperdir=$scratch/upper,workdir=$scratch/work,metacopy=on,xino=on" \ "$rootfs" || fatal "failed to assemble the container overlay" +virtfs_target= +mount_virtfs_share entrypoint=$(cmdline_value nvx_entrypoint || true) [ -n "$entrypoint" ] || entrypoint=/bin/sh @@ -291,6 +347,10 @@ echo "NVX-SANDBOX-READY: pid=$container_pid layers=$layer_count" status=0 wait "$container_pid" || status=$? rm -f "$runtime/container.pid" "$runtime/workload-machine-id" +if [ -n "$virtfs_target" ] && ! umount "$virtfs_target"; then + echo "NVX-SANDBOX-ERROR: failed to unmount the virtio-fs share" >&2 + status=1 +fi if ! umount "$rootfs"; then echo "NVX-SANDBOX-ERROR: failed to unmount the container overlay" >&2 status=1 diff --git a/scripts/nvx.py b/scripts/nvx.py index 2c777da2c..58a5b7cca 100644 --- a/scripts/nvx.py +++ b/scripts/nvx.py @@ -75,7 +75,13 @@ package_release, verify_source_tree, ) -from nvx_tools.sandbox import SandboxLaunch, SandboxLayer, parse_workload_identity +from nvx_tools.sandbox import ( + MOUNT_OWNERS, + SandboxLaunch, + SandboxLayer, + SandboxMount, + parse_workload_identity, +) DEFAULT_RELEASE_REPOSITORY = "microsoft/nvx" HYPERVISORS = ("auto", "whp", "kvm", "mshv") @@ -334,8 +340,12 @@ def command_run(args: argparse.Namespace) -> None: if args.mount.count(",") not in (1, 2): raise ScriptError("--mount must be GUEST_TARGET,HOST_PATH[,ro|rw]") command.extend(["--mount", args.mount]) + elif args.mount_owner is not None: + raise ScriptError("--mount-owner requires --mount") for denied_path in args.mount_deny: command.extend(["--mount-deny", str(denied_path)]) + if args.mount_owner is not None: + command.extend(["--mount-owner", args.mount_owner]) if args.net is not None: command.extend(["--net", args.net, "--network-profile", args.network_profile]) if args.network_egress is not None: @@ -373,11 +383,30 @@ def command_sandbox(args: argparse.Namespace) -> None: raise ScriptError( "--outcome-report is only valid for one-shot run or managed exec" ) + mount_requested = ( + args.mount is not None or bool(args.mount_deny) or args.mount_owner is not None + ) + if mount_requested and operation not in ("run", "provision"): + raise ScriptError( + "--mount, --mount-deny, and --mount-owner are only valid for " + "sandbox run or provision" + ) + if args.mount is None and mount_requested: + raise ScriptError("--mount-deny and --mount-owner require --mount") if operation in ("run", "provision"): if (args.net is None) != (args.network_profile is None): raise ScriptError("--net and --network-profile must be specified together") if not args.layer or args.scratch is None: raise ScriptError(f"sandbox {operation} requires --layer and --scratch") + mount = ( + None + if args.mount is None + else SandboxMount.parse( + args.mount, + denied_paths=tuple(args.mount_deny), + owner=args.mount_owner, + ) + ) launch = SandboxLaunch( layers=tuple(args.layer), scratch=args.scratch, @@ -387,6 +416,7 @@ def command_sandbox(args: argparse.Namespace) -> None: workload_identity=args.workload_user, memory_max=args.memory_max, pids_max=args.pids_max, + mount=mount, ).validated() _validate_sandbox_systemd_policy(launch) else: @@ -718,6 +748,11 @@ def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: run.add_argument("--processors", type=int, choices=(1, 2, 4, 8), default=1) run.add_argument("--mount", help="GUEST_TARGET,HOST_PATH,ro|rw") run.add_argument("--mount-deny", action="append", type=Path, default=[]) + run.add_argument( + "--mount-owner", + choices=MOUNT_OWNERS, + help="host identity for guest --mount requests (default: OpenVMM's process)", + ) run.add_argument("--net", metavar="IPV4/PREFIX") run.add_argument("--network-profile", choices=NETWORK_PROFILES) run.add_argument("--network-egress", choices=("allow", "deny")) @@ -784,6 +819,27 @@ def sandbox_identity(value: str) -> tuple[int, int]: ) sandbox.add_argument("--memory-max", type=int) sandbox.add_argument("--pids-max", type=int) + sandbox.add_argument( + "--mount", + metavar="GUEST_TARGET,HOST_PATH[,ro|rw]", + help="live-share one host directory inside the workload root", + ) + sandbox.add_argument( + "--mount-deny", + action="append", + type=Path, + default=[], + metavar="HOST_PATH", + help="hide an existing path inside the --mount export; repeatable", + ) + sandbox.add_argument( + "--mount-owner", + choices=MOUNT_OWNERS, + help=( + "host identity for workload --mount requests " + "(default: caller on Linux, process on Windows)" + ), + ) sandbox.add_argument("--memory-mib", type=int, default=256) sandbox.add_argument( "--timeout", diff --git a/scripts/nvx_tools/microvm_test_scripts/sandbox-filesystem-read-only.sh.in b/scripts/nvx_tools/microvm_test_scripts/sandbox-filesystem-read-only.sh.in new file mode 100644 index 000000000..34ced9cbd --- /dev/null +++ b/scripts/nvx_tools/microvm_test_scripts/sandbox-filesystem-read-only.sh.in @@ -0,0 +1,27 @@ +set -eu +fail() { + code="$1" + echo "NVX-SANDBOX-FILESYSTEM-READ-ONLY-FAIL code=$code" + exit "$code" +} + +share=@TARGET@ +options=$(awk -v share="$share" '$2 == share && $3 == "virtiofs" { print $4 }' /proc/mounts) +case ",$options," in *,ro,*) ;; *) fail 60 ;; esac +case ",$options," in *,nosuid,*) ;; *) fail 61 ;; esac +case ",$options," in *,nodev,*) ;; *) fail 62 ;; esac +[ "$(cat "$share/host-seed")" = NVX-HOST-SEED ] || fail 63 +if ls -A "$share" | grep -qx secrets; then + fail 64 +fi +if cat "$share/secrets/token" >/dev/null 2>&1; then + fail 65 +fi +if error=$(touch "$share/mutation" 2>&1); then + fail 66 +fi +case "$error" in *"Read-only file system"*) ;; *) fail 67 ;; esac +if mkdir "$share/mutation-dir" 2>/dev/null; then + fail 68 +fi +echo NVX-SANDBOX-FILESYSTEM-READ-ONLY-OK diff --git a/scripts/nvx_tools/microvm_test_scripts/sandbox-filesystem.sh.in b/scripts/nvx_tools/microvm_test_scripts/sandbox-filesystem.sh.in new file mode 100644 index 000000000..c40452c8a --- /dev/null +++ b/scripts/nvx_tools/microvm_test_scripts/sandbox-filesystem.sh.in @@ -0,0 +1,33 @@ +set -eu +fail() { + code="$1" + echo "NVX-SANDBOX-FILESYSTEM-FAIL code=$code" + exit "$code" +} + +share=@TARGET@ +options=$(awk -v share="$share" '$2 == share && $3 == "virtiofs" { print $4 }' /proc/mounts) +case ",$options," in *,rw,*) ;; *) fail 40 ;; esac +case ",$options," in *,nosuid,*) ;; *) fail 41 ;; esac +case ",$options," in *,nodev,*) ;; *) fail 42 ;; esac +[ "$(id -u):$(id -g)" = @IDENTITY@ ] || fail 43 +[ "$(cat "$share/host-seed")" = NVX-HOST-SEED ] || fail 44 +if ls -A "$share" | grep -qx secrets; then + fail 45 +fi +if cat "$share/secrets/token" >/dev/null 2>&1; then + fail 46 +fi +mkdir "$share/guest-dir" || fail 47 +printf 'NVX-GUEST-NESTED\n' >"$share/guest-dir/nested" || fail 48 +printf 'NVX-GUEST-WRITE\n' >"$share/guest-file" || fail 49 +@OWNER_CHECK@ +echo NVX-SANDBOX-FILESYSTEM-WRITTEN +tries=0 +while [ ! -e "$share/host-edit" ]; do + [ "$tries" -lt 1200 ] || fail 51 + sleep 0.05 + tries=$((tries + 1)) +done +[ "$(cat "$share/host-edit")" = NVX-HOST-EDIT ] || fail 52 +echo NVX-SANDBOX-FILESYSTEM-OK diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index b018d1214..6c126c105 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -7,8 +7,11 @@ import os import queue import secrets +import shutil import socket +import stat import subprocess +import sys import tempfile import threading import time @@ -40,6 +43,7 @@ from .build_constants import ( BuildConstants, KernelBuildConstants, + UbuntuBuildConstants, ) from .ci import OPENVMM_TEST_BACKENDS, validate_openvmm_test_backend from .common import ( @@ -47,11 +51,20 @@ artifact_path, openvmm_binary_path, require_file, + require_tool, + run_checked, sha256_file, ) from .control_session import ControlSession from .guests import GUEST_NAMES, GuestDescriptor, guest_descriptor from .openvmm_process import OpenvmmProcess, TcpConsole +from .sandbox import ( + DEFAULT_WORKLOAD_IDENTITY, + SandboxLaunch, + SandboxLayer, + SandboxMount, + default_mount_owner, +) MICROVM_TEST_SCENARIOS = ( "console-exit", @@ -71,6 +84,7 @@ "restore-processors", "restore-tsc-sync", "sandbox-blocks", + "sandbox-filesystem", "scratch-snapshot", "smp", "smp-lapic", @@ -82,7 +96,13 @@ "workload-identity", ) UBUNTU_UNSUPPORTED_SCENARIOS = frozenset( - ("console-snapshot", "sandbox-blocks", "scratch-snapshot", "snapshot-tiers") + ( + "console-snapshot", + "sandbox-blocks", + "sandbox-filesystem", + "scratch-snapshot", + "snapshot-tiers", + ) ) MICROVM_PROCESSOR_COUNTS = (1, 2, 4, 8) MICROVM_TEST_SCRIPTS_DIR = Path(__file__).with_name("microvm_test_scripts") @@ -105,6 +125,16 @@ HOST_LOOPBACK_INGRESS_READY_MARKER = b"NVX-HOST-LOOPBACK-INGRESS-READY" HOST_LOOPBACK_PORT_BIND_ATTEMPTS = 16 SANDBOX_BLOCK_SIZE = 8 * 1024 * 1024 +SANDBOX_FILESYSTEM_TARGET = "/workspace" +SANDBOX_FILESYSTEM_WRITTEN_MARKER = b"NVX-SANDBOX-FILESYSTEM-WRITTEN" +SANDBOX_FILESYSTEM_OK_MARKER = b"NVX-SANDBOX-FILESYSTEM-OK" +SANDBOX_FILESYSTEM_READ_ONLY_MARKER = b"NVX-SANDBOX-FILESYSTEM-READ-ONLY-OK" +SANDBOX_FILESYSTEM_EXIT_MARKER = b"NVX-SANDBOX-EXIT: status=0" +SANDBOX_FILESYSTEM_USER = "nvx-share" +# A root test run cannot use UID 0 as the workload, so it exports this owner. +SANDBOX_FILESYSTEM_ROOT_RUN_IDENTITY = (12345, 12345) +SANDBOX_FILESYSTEM_SCRATCH_TEMPLATE_NAME = "ubuntu-smoke-scratch.ext4" +SANDBOX_FILESYSTEM_SCRATCH_SIZE = 64 * 1024 * 1024 SNAPSHOT_CORE_CONTINUED_MARKER = b"NVX-SNAPSHOT-CORE-CONTINUED" SNAPSHOT_CORE_COMPLETION_MARKER = b"NVX-SNAPSHOT-CORE-OK" CONSOLE_BINARY_MARKER = b"\0\r\n\x7f\xffNVX-CONSOLE-BINARY" @@ -1952,6 +1982,323 @@ def run_sandbox_blocks( ) +def _sandbox_filesystem_identity() -> tuple[tuple[int, int], str]: + """Returns the workload identity and mount owner for the live-share test. + + Linux runs use caller ownership with a workload identity that owns the + export, which maps guest-created files to that owner. Windows has no host + UID mapping and keeps the default identity and process ownership. + """ + if sys.platform == "win32": + return DEFAULT_WORKLOAD_IDENTITY, default_mount_owner() + uid, gid = os.getuid(), os.getgid() + if uid == 0: + return SANDBOX_FILESYSTEM_ROOT_RUN_IDENTITY, "caller" + if gid == 0 or 65534 in (uid, gid): + raise ScriptError( + "sandbox-filesystem requires a host UID and GID other than 0 and 65534" + ) + return (uid, gid), "caller" + + +def _assign_sandbox_export_owner(export: Path, identity: tuple[int, int]) -> None: + if sys.platform == "win32": + return + # This also replaces a group inherited from a set-group-ID parent. + for path in (export, *export.rglob("*")): + os.chown(path, *identity, follow_symlinks=False) + os.chmod(export, 0o700) + + +def _sandbox_filesystem_scratch(root: Path, identity: tuple[int, int]) -> Path: + """Creates the scratch template whose upper layer defines the identity.""" + template = root / "scratch-template.ext4" + if sys.platform == "win32": + shutil.copyfile( + require_file( + artifact_path(SANDBOX_FILESYSTEM_SCRATCH_TEMPLATE_NAME), + "Ubuntu sandbox scratch template", + ), + template, + ) + return template + uid, gid = identity + staging = root / "scratch-staging" + upper = staging / "upper" + etc = upper / "etc" + home = upper / "home" / SANDBOX_FILESYSTEM_USER + etc.mkdir(parents=True) + home.mkdir(parents=True) + (etc / "passwd").write_bytes( + ( + "root:x:0:0:root:/root:/bin/sh\n" + "nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin\n" + f"{SANDBOX_FILESYSTEM_USER}:x:{uid}:{gid}::" + f"/home/{SANDBOX_FILESYSTEM_USER}:/bin/sh\n" + ).encode() + ) + (etc / "group").write_bytes( + f"root:x:0:\nnogroup:x:65534:\n{SANDBOX_FILESYSTEM_USER}:x:{gid}:\n".encode() + ) + for directory in (staging, upper, etc, home.parent, home): + os.chmod(directory, 0o755) + os.chown(home, uid, gid) + with template.open("wb") as image: + image.truncate(SANDBOX_FILESYSTEM_SCRATCH_SIZE) + run_checked( + [ + require_tool("mkfs.ext4", "sandbox-filesystem requires mkfs.ext4 on Linux"), + "-q", + "-F", + "-m", + "0", + "-E", + "lazy_itable_init=0,lazy_journal_init=0", + "-d", + staging, + template, + ] + ) + return template + + +def _sandbox_filesystem_command( + executable: Path, + kernel: Path, + initrd: Path, + backend: str, + memory_mib: int, + launch: SandboxLaunch, + *, + extra_arguments: Sequence[str] = (), + report: Path | None = None, +) -> list[str]: + command = [ + str(executable), + *launch.validated().openvmm_arguments(), + *extra_arguments, + "--microvm-lifecycle", + "one-shot", + "--single-process", + "--hypervisor", + backend, + "--memory", + f"{memory_mib}M", + "--kernel", + str(kernel), + "--initrd", + str(initrd), + "--cmdline", + launch.kernel_command_line("quiet loglevel=0"), + ] + if report is not None: + command.extend(("--microvm-report", str(report))) + return command + + +def _check_sandbox_filesystem_writes(export: Path, identity: tuple[int, int]) -> None: + written = export / "guest-file" + nested = export / "guest-dir" / "nested" + if ( + written.read_bytes() != b"NVX-GUEST-WRITE\n" + or nested.read_bytes() != b"NVX-GUEST-NESTED\n" + ): + raise RuntimeError("sandbox workload writes were not visible on the host") + if sys.platform == "win32": + return + for path in (written, nested.parent, nested): + status = path.stat() + if (status.st_uid, status.st_gid) != identity: + raise RuntimeError( + f"sandbox workload created {path.name} as " + f"{status.st_uid}:{status.st_gid}, not the export owner " + f"{identity[0]}:{identity[1]}" + ) + if stat.S_IMODE(written.stat().st_mode) != 0o600: + raise RuntimeError("sandbox workload chmod did not reach the host file") + + +def run_sandbox_filesystem( + executable: Path, + kernel: Path, + initrd: Path, + backend: str, + *, + memory_mib: int, + timeout: float, + output_dir: Path, +) -> None: + layer_path = require_file( + artifact_path(UbuntuBuildConstants.DISTRO_NAME), + "Ubuntu EROFS distro layer", + ) + manifest_path = require_file( + artifact_path(UbuntuBuildConstants.DISTRO_MANIFEST_NAME), + "Ubuntu EROFS distro manifest", + ) + try: + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + layer = SandboxLayer( + role="distro", + path=layer_path, + uuid=str(uuid.UUID(str(manifest["uuid"]))), + ) + except (OSError, KeyError, TypeError, ValueError) as error: + raise ScriptError( + f"invalid Ubuntu EROFS distro manifest: {manifest_path}" + ) from error + memory_mib = max(memory_mib, 256) + target = SANDBOX_FILESYSTEM_TARGET + identity, owner = _sandbox_filesystem_identity() + with tempfile.TemporaryDirectory(prefix="nvx-sandbox-filesystem-") as temporary: + root = Path(temporary) + export = root / "export" + secrets_path = export / "secrets" + secrets_path.mkdir(parents=True) + secret = secrets_path / "token" + secret.write_bytes(b"NVX-SECRET\n") + (export / "host-seed").write_bytes(b"NVX-HOST-SEED\n") + (export / "probe.sh").write_bytes( + _render_script( + "sandbox-filesystem.sh.in", + TARGET=target, + IDENTITY=f"{identity[0]}:{identity[1]}", + # Only a caller-owned share reports the workload as the owner + # that the guest kernel requires for chmod. + OWNER_CHECK=( + 'chmod 0600 "$share/guest-file" || fail 50' + if owner == "caller" + else ":" + ), + ).encode() + ) + (export / "probe-read-only.sh").write_bytes( + _render_script("sandbox-filesystem-read-only.sh.in", TARGET=target).encode() + ) + _assign_sandbox_export_owner(export, identity) + template = _sandbox_filesystem_scratch(root, identity) + + def sandbox(name: str, access: str | None, probe: str) -> SandboxLaunch: + scratch = root / f"{name}-scratch.ext4" + shutil.copyfile(template, scratch) + return SandboxLaunch( + layers=(layer,), + scratch=scratch, + entrypoint="/bin/sh", + args=(f"{target}/{probe}",), + workload_identity=identity, + mount=( + None + if access is None + else SandboxMount( + guest_target=target, + host_path=export, + access=access, + denied_paths=(secrets_path,), + owner=owner, + ) + ), + ) + + report_path = root / "sandbox-filesystem.json" + with OpenvmmProcess( + _sandbox_filesystem_command( + executable, + kernel, + initrd, + backend, + memory_mib, + sandbox("read-write", "rw", "probe.sh"), + report=report_path, + ), + output_dir / "sandbox-filesystem.log", + ) as process: + process.wait_for_line( + f"NVX-SANDBOX-VIRTFS: mounted microvm at {target} (rw)".encode(), + timeout, + ) + process.wait_for_line(SANDBOX_FILESYSTEM_WRITTEN_MARKER, timeout) + # The workload is still running, so these checks observe the live + # share rather than a copy-back after exit. + _check_sandbox_filesystem_writes(export, identity) + staged_edit = export / ".host-edit" + staged_edit.write_bytes(b"NVX-HOST-EDIT\n") + os.replace(staged_edit, export / "host-edit") + process.wait_for_line(SANDBOX_FILESYSTEM_OK_MARKER, timeout) + result = process.wait(timeout) + if ( + result.returncode != 0 + or SANDBOX_FILESYSTEM_EXIT_MARKER not in _output_lines(result.output) + ): + raise RuntimeError("sandbox live-share workload did not exit cleanly") + if secret.read_bytes() != b"NVX-SECRET\n": + raise RuntimeError("sandbox workload modified a denied path") + report = _read_outcome_report(report_path) + _preserve_outcome_report(output_dir / "sandbox-filesystem.json", report) + if report["outcome"]["category"] != "success" or not all( + report["teardown"].values() + ): + raise RuntimeError("sandbox live-share outcome reported an unclean exit") + + with OpenvmmProcess( + _sandbox_filesystem_command( + executable, + kernel, + initrd, + backend, + memory_mib, + sandbox("read-only", "ro", "probe-read-only.sh"), + ), + output_dir / "sandbox-filesystem-read-only.log", + ) as process: + result = process.wait(timeout) + lines = _output_lines(result.output) + if ( + result.returncode != 0 + or f"NVX-SANDBOX-VIRTFS: mounted microvm at {target} (ro)".encode() + not in lines + or SANDBOX_FILESYSTEM_READ_ONLY_MARKER not in lines + ): + raise RuntimeError("read-only sandbox share did not reject workload writes") + if (export / "mutation").exists() or (export / "mutation-dir").exists(): + raise RuntimeError("read-only sandbox share was modified") + + for name, rejected_target, expected in ( + ( + "reserved", + "/dev/nvx-share", + b"virtio-fs target is reserved by the sandbox", + ), + ( + "symlink", + "/bin/nvx-share", + b"virtio-fs target traverses a symbolic link", + ), + ): + with OpenvmmProcess( + _sandbox_filesystem_command( + executable, + kernel, + initrd, + backend, + memory_mib, + sandbox(name, None, "probe-read-only.sh"), + extra_arguments=("--mount", f"{rejected_target},{export},ro"), + ), + output_dir / f"sandbox-filesystem-{name}.log", + ) as process: + result = process.wait(timeout) + if ( + result.returncode == 0 + or expected not in result.output + or b"NVX-SANDBOX-READY" in result.output + ): + raise RuntimeError( + f"sandbox agent did not reject the {name} share target " + f"{rejected_target}" + ) + + def run_smp_snapshot( executable: Path, kernel: Path, @@ -3826,6 +4173,17 @@ def run(args: argparse.Namespace) -> int: timeout=args.timeout, log_path=output_dir / "sandbox-blocks.log", ) + if "sandbox-filesystem" in scenarios: + print(f"Running microVM sandbox live filesystem on OpenVMM/{args.backend}") + run_sandbox_filesystem( + executable, + kernel, + initrd, + args.backend, + memory_mib=args.memory_mib, + timeout=args.timeout, + output_dir=output_dir, + ) if "scratch-snapshot" in scenarios: print(f"Running microVM scratch snapshot correctness on OpenVMM/{args.backend}") run_scratch_snapshot( diff --git a/scripts/nvx_tools/sandbox.py b/scripts/nvx_tools/sandbox.py index cd64fadfc..e8fce03b8 100644 --- a/scripts/nvx_tools/sandbox.py +++ b/scripts/nvx_tools/sandbox.py @@ -4,7 +4,7 @@ import os import re -from dataclasses import dataclass +from dataclasses import dataclass, replace from pathlib import Path from uuid import UUID @@ -24,6 +24,18 @@ ) _HOSTNAME = re.compile(r"(?=^.{1,63}$)(?!-)[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$") DEFAULT_WORKLOAD_IDENTITY = (65534, 65534) +MOUNT_ACCESS_MODES = ("ro", "rw") +MOUNT_OWNERS = ("process", "caller") +MOUNT_DENIED_PATHS_MAX = 128 +# Targets that the sandbox agent itself mounts or rewrites inside the container. +_RESERVED_MOUNT_TREES = ("/proc", "/sys", "/dev", "/.nvx-agent") +_RESERVED_MOUNT_TARGETS = ("/etc", "/etc/machine-id") +_INVALID_MOUNT_TARGET_CHARACTERS = frozenset("\0\\=,") + + +def default_mount_owner() -> str: + """Returns the host identity used for sandbox mounts on this host.""" + return "process" if os.name == "nt" else "caller" def parse_workload_identity(value: str) -> tuple[int, int]: @@ -66,6 +78,108 @@ def parse(cls, value: str) -> SandboxLayer: return cls(role=role, path=Path(raw_path), uuid=uuid) +@dataclass(frozen=True) +class SandboxMount: + """Live host directory mounted inside the sandbox workload root.""" + + guest_target: str + host_path: Path + access: str = "ro" + denied_paths: tuple[Path, ...] = () + owner: str = "process" + + @classmethod + def parse( + cls, + value: str, + *, + denied_paths: tuple[Path, ...] = (), + owner: str | None = None, + ) -> SandboxMount: + fields = value.split(",") + if len(fields) not in (2, 3): + raise ScriptError("--mount must be GUEST_TARGET,HOST_PATH[,ro|rw]") + guest_target, raw_path = fields[:2] + if not raw_path: + raise ScriptError("--mount host path is empty") + return cls( + guest_target=guest_target, + host_path=Path(raw_path), + access=fields[2] if len(fields) == 3 else "ro", + denied_paths=denied_paths, + owner=default_mount_owner() if owner is None else owner, + ) + + def __post_init__(self) -> None: + target = self.guest_target + components = target.split("/")[1:] + if ( + not target.startswith("/") + or target == "/" + or any(component in ("", ".", "..") for component in components) + or any(character.isspace() for character in target) + or not _INVALID_MOUNT_TARGET_CHARACTERS.isdisjoint(target) + ): + raise ScriptError( + "--mount guest target must be a canonical absolute path without " + f"whitespace, backslashes, commas, or '=': {target!r}" + ) + if target in _RESERVED_MOUNT_TARGETS or any( + target == tree or target.startswith(f"{tree}/") + for tree in _RESERVED_MOUNT_TREES + ): + raise ScriptError( + f"--mount guest target is reserved by the sandbox: {target}" + ) + if self.access not in MOUNT_ACCESS_MODES: + raise ScriptError(f"--mount mode must be ro or rw, not {self.access!r}") + if self.owner not in MOUNT_OWNERS: + raise ScriptError(f"unsupported --mount-owner {self.owner!r}") + if self.owner == "caller" and os.name == "nt": + raise ScriptError("--mount-owner caller requires a Linux host") + if len(self.denied_paths) > MOUNT_DENIED_PATHS_MAX: + raise ScriptError( + f"--mount-deny accepts at most {MOUNT_DENIED_PATHS_MAX} paths" + ) + if "," in os.fspath(self.host_path): + raise ScriptError( + f"--mount host paths containing commas are unsupported: {self.host_path}" + ) + + def validated(self) -> SandboxMount: + if not self.host_path.is_dir(): + raise ScriptError( + f"--mount host directory does not exist: {self.host_path}" + ) + for denied in self.denied_paths: + # OpenVMM resolves relative denied paths against the export root. + candidate = denied if denied.is_absolute() else self.host_path / denied + if not os.path.lexists(candidate): + raise ScriptError(f"--mount-deny path does not exist: {candidate}") + return self + + def resolved(self) -> SandboxMount: + """Returns the mount with an absolute export root for later launches.""" + return replace(self, host_path=self.host_path.resolve()) + + def openvmm_arguments(self) -> list[str]: + arguments = [ + "--mount", + f"{self.guest_target},{os.fspath(self.host_path)},{self.access}", + ] + for denied in self.denied_paths: + arguments.extend(("--mount-deny", os.fspath(denied))) + arguments.extend(("--mount-owner", self.owner)) + return arguments + + def command_line_fragment(self) -> str: + """Returns the bootstrap tokens that OpenVMM appends for this mount.""" + return ( + f"virtfs_dir={self.guest_target} virtfs_tag=microvm " + f"virtfs_mode={self.access}" + ) + + @dataclass(frozen=True) class SandboxLaunch: layers: tuple[SandboxLayer, ...] @@ -76,6 +190,7 @@ class SandboxLaunch: workload_identity: tuple[int, int] = DEFAULT_WORKLOAD_IDENTITY memory_max: int | None = None pids_max: int | None = None + mount: SandboxMount | None = None def __post_init__(self) -> None: if not 1 <= len(self.layers) <= len(LAYER_ROLES): @@ -117,6 +232,8 @@ def validated(self) -> SandboxLaunch: _reject_disk_path(layer.path) require_file(self.scratch, "ext4 scratch image") _reject_disk_path(self.scratch) + if self.mount is not None: + self.mount.validated() return self def ordered_layers(self) -> tuple[SandboxLayer, ...]: @@ -140,6 +257,8 @@ def openvmm_arguments(self) -> list[str]: f"{self.workload_identity[0]}:{self.workload_identity[1]}", ) ) + if self.mount is not None: + arguments.extend(self.mount.openvmm_arguments()) return arguments def kernel_command_line(self, user_command_line: str = "") -> str: @@ -169,7 +288,14 @@ def kernel_command_line(self, user_command_line: str = "") -> str: if self.pids_max is not None: tokens.append(f"nvx_pids_max={self.pids_max + 1}") command_line = " ".join(token for token in tokens if token) - if len(command_line.encode("utf-8")) + 1 > SANDBOX_COMMAND_LINE_MAX_SIZE: + # OpenVMM appends the share's bootstrap tokens to the same x86 budget. + mount_tokens = ( + "" if self.mount is None else f" {self.mount.command_line_fragment()}" + ) + if ( + len(command_line.encode("utf-8")) + len(mount_tokens.encode("utf-8")) + 1 + > SANDBOX_COMMAND_LINE_MAX_SIZE + ): raise ScriptError( "sandbox kernel command line exceeds its 1024-byte x86 budget" ) diff --git a/scripts/nvx_tools/sandbox_lifecycle.py b/scripts/nvx_tools/sandbox_lifecycle.py index 44898581a..7d3d5b1f0 100644 --- a/scripts/nvx_tools/sandbox_lifecycle.py +++ b/scripts/nvx_tools/sandbox_lifecycle.py @@ -26,7 +26,7 @@ ControlSession, ManagedExecResult, ) -from .sandbox import SandboxLaunch, SandboxLayer +from .sandbox import SandboxLaunch, SandboxLayer, SandboxMount CONFIG_NAME = "config.json" RUNTIME_NAME = "runtime.json" @@ -201,9 +201,38 @@ def _serialize_launch( "network_proxy": network_proxy, "host_loopback_forward": list(host_loopback_forward), "cmdline": cmdline, + "mount": _serialize_mount(launch.mount), } +def _serialize_mount(mount: SandboxMount | None) -> dict[str, Any] | None: + if mount is None: + return None + resolved = mount.resolved() + return { + "guest_target": resolved.guest_target, + "host_path": os.fspath(resolved.host_path), + "access": resolved.access, + "denied_paths": [os.fspath(path) for path in resolved.denied_paths], + "owner": resolved.owner, + } + + +def _deserialize_mount(value: object) -> SandboxMount | None: + if value is None: + return None + if not isinstance(value, dict): + raise ScriptError("sandbox configuration is malformed") + mount = cast(dict[str, Any], value) + return SandboxMount( + guest_target=str(mount["guest_target"]), + host_path=Path(str(mount["host_path"])), + access=str(mount["access"]), + denied_paths=tuple(Path(str(path)) for path in mount["denied_paths"]), + owner=str(mount["owner"]), + ) + + def _deserialize_launch(config: dict[str, Any]) -> SandboxLaunch: try: layers = tuple( @@ -224,6 +253,8 @@ def _deserialize_launch(config: dict[str, Any]) -> SandboxLaunch: None if config["memory_max"] is None else int(config["memory_max"]) ), pids_max=None if config["pids_max"] is None else int(config["pids_max"]), + # State directories provisioned before live shares have no mount. + mount=_deserialize_mount(config.get("mount")), ) except (KeyError, TypeError, ValueError) as error: raise ScriptError("sandbox configuration is malformed") from error diff --git a/scripts/setup/README.md b/scripts/setup/README.md index 82e26b404..e84be7c1d 100644 --- a/scripts/setup/README.md +++ b/scripts/setup/README.md @@ -43,7 +43,9 @@ On Linux, omit `--runner-name` to prepare host dependencies without reconciling an existing runner's service or cache directory. Pass its name in a subsequent invocation to update the service; `--check-only` still validates runner state. Rustup bootstrap binaries are versioned and SHA-256 verified before execution; -Linux provisioning also installs `zstd` for native Actions cache archives. +Linux provisioning also installs `zstd` for native Actions cache archives and +`e2fsprogs` for the scratch image that the sandbox live-share microVM test +formats. Both runner setup scripts install a pinned, SHA-256-verified `sccache` binary. Runner services use a persistent `_work/_sccache` directory with a 10-GiB limit, disable Cargo incremental compilation, and expose `sccache` through diff --git a/scripts/setup/setup-linux-mshv.sh b/scripts/setup/setup-linux-mshv.sh index afcebff5f..5c662ec2e 100644 --- a/scripts/setup/setup-linux-mshv.sh +++ b/scripts/setup/setup-linux-mshv.sh @@ -59,14 +59,14 @@ install_packages() { run_as_root apt-get update run_as_root env DEBIAN_FRONTEND=noninteractive apt-get install -y \ bc binutils bison build-essential ca-certificates cmake cpio curl \ - docker-buildx docker.io flex git gzip iproute2 iptables \ + docker-buildx docker.io e2fsprogs flex git gzip iproute2 iptables \ libarchive-tools libelf-dev libssl-dev make ninja-build patch perl \ pkg-config protobuf-compiler python3 rsync tar util-linux xz-utils \ zstd elif command -v tdnf >/dev/null 2>&1; then run_as_root tdnf install -y \ bc binutils bison ca-certificates cmake cpio curl diffutils \ - docker-buildx docker-cli elfutils-libelf-devel findutils flex gcc \ + docker-buildx docker-cli e2fsprogs elfutils-libelf-devel findutils flex gcc \ gcc-c++ git glibc-devel glibc-iconv gzip iproute iptables \ kernel-headers libarchive libarchive-devel make moby-engine \ ninja-build openssl openssl-devel patch perl perl-FindBin \ @@ -76,7 +76,7 @@ install_packages() { run_as_root tdnf install -y docker-cli elif command -v dnf >/dev/null 2>&1; then run_as_root dnf install -y \ - bc binutils bison ca-certificates cmake cpio curl elfutils-libelf-devel \ + bc binutils bison ca-certificates cmake cpio curl e2fsprogs elfutils-libelf-devel \ findutils flex gcc gcc-c++ git glibc-devel gzip iproute iptables \ kernel-headers libarchive libarchive-devel make moby-engine \ ninja-build openssl openssl-devel patch perl pkgconf \ diff --git a/scripts/setup/setup-linux-runner.sh b/scripts/setup/setup-linux-runner.sh index e2c2c98b6..12773e3ef 100644 --- a/scripts/setup/setup-linux-runner.sh +++ b/scripts/setup/setup-linux-runner.sh @@ -402,14 +402,14 @@ install_packages() { run_as_root apt-get update run_as_root env DEBIAN_FRONTEND=noninteractive apt-get install -y \ bc binutils bison build-essential ca-certificates cmake cpio curl diffutils \ - flex git gzip iproute2 iptables \ + e2fsprogs flex git gzip iproute2 iptables \ libarchive-tools libelf-dev libssl-dev make ninja-build patch perl \ pkg-config protobuf-compiler python3 rsync tar util-linux xz-utils \ zstd elif command -v tdnf >/dev/null 2>&1; then run_as_root tdnf install -y \ bc binutils bison ca-certificates cmake cpio curl diffutils \ - elfutils-libelf-devel findutils flex gcc \ + e2fsprogs elfutils-libelf-devel findutils flex gcc \ gcc-c++ git glibc-devel glibc-iconv gzip icu iproute iptables \ kernel-headers libarchive libarchive-devel lttng-ust make \ ninja-build openssl openssl-devel patch perl perl-FindBin \ @@ -419,7 +419,7 @@ install_packages() { elif command -v dnf >/dev/null 2>&1; then run_as_root dnf install -y \ bc binutils bison ca-certificates cmake cpio curl diffutils \ - elfutils-libelf-devel findutils flex gcc gcc-c++ git glibc-devel \ + e2fsprogs elfutils-libelf-devel findutils flex gcc gcc-c++ git glibc-devel \ gzip iproute iptables kernel-headers libarchive libarchive-devel \ make ninja-build openssl openssl-devel patch perl \ pkgconf pkgconf-pkg-config protobuf-compiler python3 rsync \ diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index d311a3e57..5b11444a5 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -3,6 +3,7 @@ import argparse import json +import os import queue import shutil import socket @@ -1606,6 +1607,237 @@ def test_sandbox_blocks_use_fixed_roles_and_access(self): microvm_tests.SANDBOX_BLOCKS_COMPLETION_MARKER, ) + def test_sandbox_filesystem_scripts_check_the_share_before_success(self): + writable = microvm_tests._render_script( + "sandbox-filesystem.sh.in", + TARGET=microvm_tests.SANDBOX_FILESYSTEM_TARGET, + IDENTITY="1001:1001", + OWNER_CHECK='chmod 0600 "$share/guest-file" || fail 50', + ) + read_only = microvm_tests._render_script( + "sandbox-filesystem-read-only.sh.in", + TARGET=microvm_tests.SANDBOX_FILESYSTEM_TARGET, + ) + for script in (writable, read_only): + self.assertNotIn("@", script) + self.assertIn("share=/workspace\n", script) + self.assertIn('[ "$(id -u):$(id -g)" = 1001:1001 ]', writable) + self.assertLess( + writable.index("fail 50"), + writable.index(microvm_tests.SANDBOX_FILESYSTEM_WRITTEN_MARKER.decode()), + ) + self.assertLess( + writable.index("fail 52"), + writable.index(microvm_tests.SANDBOX_FILESYSTEM_OK_MARKER.decode()), + ) + self.assertLess( + read_only.index("fail 68"), + read_only.index(microvm_tests.SANDBOX_FILESYSTEM_READ_ONLY_MARKER.decode()), + ) + + def _run_sandbox_filesystem( + self, + root: Path, + outputs: dict[str, tuple[int, bytes]], + ) -> list[list[str]]: + layer = root / "ubuntu-distro.erofs" + layer.write_bytes(b"layer") + (root / "ubuntu-distro.erofs.manifest.json").write_text( + json.dumps({"uuid": "11111111-1111-1111-1111-111111111111"}), + encoding="utf-8", + ) + output_dir = root / "logs" + output_dir.mkdir() + identity: tuple[tuple[int, int], str] + if sys.platform == "win32": + identity = ((65534, 65534), "process") + else: + identity = ((os.getuid(), os.getgid()), "caller") + commands: list[list[str]] = [] + + def scratch(scratch_root: Path, _identity: tuple[int, int]) -> Path: + template = scratch_root / "scratch-template.ext4" + template.write_bytes(b"scratch") + return template + + def artifact(name: str) -> Path: + return root / name + + class FakeProcess: + def __init__(self, command: list[str], _log_path: Path) -> None: + commands.append(command) + mount = command[command.index("--mount") + 1].split(",") + self.target = mount[0] + self.export = Path(mount[1]) + self.access = mount[2] + probe = (self.export / "probe.sh").read_text(encoding="utf-8") + chmod = 'chmod 0600 "$share/guest-file" || fail 50\n' + caller_owned = cast(str, identity[1]) == "caller" + if (chmod in probe) != caller_owned: + raise AssertionError("probe chmod does not match the owner mode") + + def __enter__(self) -> "FakeProcess": + return self + + def __exit__(self, *_arguments: object) -> None: + return None + + def wait_for_line(self, marker: bytes, _timeout: float) -> None: + if marker == microvm_tests.SANDBOX_FILESYSTEM_WRITTEN_MARKER: + (self.export / "guest-dir").mkdir() + (self.export / "guest-dir" / "nested").write_bytes( + b"NVX-GUEST-NESTED\n" + ) + (self.export / "guest-file").write_bytes(b"NVX-GUEST-WRITE\n") + os.chmod(self.export / "guest-file", 0o600) + elif marker == microvm_tests.SANDBOX_FILESYSTEM_OK_MARKER: + if (self.export / "host-edit").read_bytes() != b"NVX-HOST-EDIT\n": + raise AssertionError("host edit was not staged") + + def wait(self, _timeout: float) -> openvmm_process.OpenvmmProcessResult: + command = commands[-1] + if "--microvm-report" in command: + Path(command[command.index("--microvm-report") + 1]).write_text( + json.dumps( + { + "schema_version": 1, + "instance_id": "0" * 32, + "backend": "whp", + "outcome": { + "operation": "run", + "category": "success", + "status_code": 0, + }, + "network_policy": { + "status": "not-configured", + "status_code": 0, + "mode": "none", + "allow_rule_count": 0, + "deny_rule_count": 0, + "host_loopback": "allow", + }, + "teardown": dict.fromkeys( + microvm_tests.OUTCOME_TEARDOWN_FIELDS, True + ), + } + ), + encoding="utf-8", + ) + returncode, output = outputs[f"{self.target},{self.access}"] + return openvmm_process.OpenvmmProcessResult(returncode, output) + + with ( + patch.object(microvm_tests, "artifact_path", side_effect=artifact), + patch.object( + microvm_tests, + "_sandbox_filesystem_identity", + return_value=identity, + ), + patch.object(microvm_tests, "_assign_sandbox_export_owner"), + patch.object( + microvm_tests, + "_sandbox_filesystem_scratch", + side_effect=scratch, + ), + patch.object(microvm_tests, "OpenvmmProcess", FakeProcess), + ): + microvm_tests.run_sandbox_filesystem( + Path("openvmm"), + Path("vmlinux"), + Path("initrd"), + "whp", + memory_mib=128, + timeout=60, + output_dir=output_dir, + ) + self.assertTrue((output_dir / "sandbox-filesystem.json").is_file()) + return commands + + def _sandbox_filesystem_outputs(self) -> dict[str, tuple[int, bytes]]: + return { + "/workspace,rw": ( + 0, + b"NVX-SANDBOX-VIRTFS: mounted microvm at /workspace (rw)\n" + b"NVX-SANDBOX-FILESYSTEM-WRITTEN\n" + b"NVX-SANDBOX-FILESYSTEM-OK\n" + b"NVX-SANDBOX-EXIT: status=0\n", + ), + "/workspace,ro": ( + 0, + b"NVX-SANDBOX-VIRTFS: mounted microvm at /workspace (ro)\n" + b"NVX-SANDBOX-FILESYSTEM-READ-ONLY-OK\n" + b"NVX-SANDBOX-EXIT: status=0\n", + ), + "/dev/nvx-share,ro": ( + 125, + b"NVX-SANDBOX-ERROR: virtio-fs target is reserved by the sandbox: " + b"/dev/nvx-share\n", + ), + "/bin/nvx-share,ro": ( + 125, + b"NVX-SANDBOX-ERROR: virtio-fs target traverses a symbolic link: " + b"/bin/nvx-share\n", + ), + } + + def test_sandbox_filesystem_checks_live_share_policy_and_rejections(self): + with tempfile.TemporaryDirectory() as temporary: + commands = self._run_sandbox_filesystem( + Path(temporary), self._sandbox_filesystem_outputs() + ) + + self.assertEqual( + [ + command[command.index("--mount") + 1].split(",")[0] + for command in commands + ], + ["/workspace", "/workspace", "/dev/nvx-share", "/bin/nvx-share"], + ) + expected_owner = "process" if sys.platform == "win32" else "caller" + for command, access in zip(commands[:2], ("rw", "ro"), strict=True): + self.assertTrue(command[command.index("--mount") + 1].endswith(access)) + self.assertTrue( + command[command.index("--mount-deny") + 1].endswith("secrets") + ) + self.assertEqual( + command[command.index("--mount-owner") + 1], expected_owner + ) + self.assertEqual(command[command.index("--memory") + 1], "256M") + self.assertEqual( + command[command.index("--microvm-lifecycle") + 1], "one-shot" + ) + cmdline = command[command.index("--cmdline") + 1] + self.assertIn("nvx_sandbox=1", cmdline) + self.assertIn("nvx_entrypoint=/bin/sh", cmdline) + self.assertIn( + "nvx_arg=/workspace/probe.sh", + commands[0][commands[0].index("--cmdline") + 1], + ) + self.assertIn("--microvm-report", commands[0]) + for command in commands[2:]: + self.assertNotIn("--mount-owner", command) + self.assertNotIn("--mount-deny", command) + + def test_sandbox_filesystem_rejects_an_accepted_unsafe_target(self): + outputs = self._sandbox_filesystem_outputs() + outputs["/bin/nvx-share,ro"] = (0, b"NVX-SANDBOX-READY: pid=2 layers=1\n") + with tempfile.TemporaryDirectory() as temporary: + with self.assertRaisesRegex(RuntimeError, "did not reject the symlink"): + self._run_sandbox_filesystem(Path(temporary), outputs) + + def test_sandbox_filesystem_requires_live_visibility_while_running(self): + outputs = self._sandbox_filesystem_outputs() + with ( + tempfile.TemporaryDirectory() as temporary, + patch.object( + microvm_tests, + "_check_sandbox_filesystem_writes", + side_effect=RuntimeError("not visible"), + ), + self.assertRaisesRegex(RuntimeError, "not visible"), + ): + self._run_sandbox_filesystem(Path(temporary), outputs) + def test_smp_snapshot_reruns_probe_and_checks_two_restores(self): with tempfile.TemporaryDirectory() as temporary: output_dir = Path(temporary) / "logs" @@ -2029,6 +2261,7 @@ def require(path: Path, _description: str) -> Path: for scenario in ( "console-snapshot", "sandbox-blocks", + "sandbox-filesystem", "scratch-snapshot", "snapshot-tiers", ): diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index aab6088bf..e285eafd2 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -642,8 +642,129 @@ def test_sandbox_command_parses_typed_launch_contract(self): self.assertEqual(args.memory_max, 268435456) self.assertEqual(args.pids_max, 64) self.assertEqual(args.workload_user, (1000, 1001)) + self.assertIsNone(args.mount) + self.assertEqual(args.mount_deny, []) + self.assertIsNone(args.mount_owner) self.assertIs(args.handler, nvx.command_sandbox) + def test_sandbox_forwards_a_live_mount_to_openvmm(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + layer = root / "distro.erofs" + scratch = root / "scratch.ext4" + share = root / "share" + (share / "secrets").mkdir(parents=True) + layer.write_bytes(b"distro") + scratch.write_bytes(b"scratch") + base = [ + "sandbox", + "--layer", + f"distro,{layer},11111111-1111-1111-1111-111111111111", + "--scratch", + str(scratch), + "--workload-user", + "1001:1001", + "--mount", + f"/workspace,{share},rw", + "--mount-deny", + str(share / "secrets"), + "--dry-run", + ] + + def require(path: Path, _description: str) -> Path: + return path + + for extra, owner in ( + ([], sandbox.default_mount_owner()), + (["--mount-owner", "process"], "process"), + ): + with self.subTest(owner=owner): + with ( + patch.object(nvx, "require_file", side_effect=require), + patch.object( + nvx, "_format_command", return_value="formatted" + ) as format_command, + ): + nvx.command_sandbox(nvx.parse_args([*base, *extra])) + command = format_command.call_args.args[0] + self.assertEqual( + command[command.index("--mount") + 1], + f"/workspace,{share},rw", + ) + self.assertEqual( + command[command.index("--mount-deny") + 1], + str(share / "secrets"), + ) + self.assertEqual(command[command.index("--mount-owner") + 1], owner) + self.assertEqual( + command[command.index("--microvm-workload-identity") + 1], + "1001:1001", + ) + + def test_sandbox_mount_options_are_validated_before_launch(self): + layer_arguments = [ + "--layer", + "distro,distro.erofs,11111111-1111-1111-1111-111111111111", + "--scratch", + "scratch.ext4", + ] + for arguments, message in ( + ( + ["sandbox", *layer_arguments, "--mount-deny", "secrets"], + "require --mount", + ), + ( + ["sandbox", *layer_arguments, "--mount-owner", "process"], + "require --mount", + ), + ( + [ + "sandbox", + "exec", + "--state-dir", + "state", + "--mount", + "/workspace,share,rw", + ], + "only valid for sandbox run or provision", + ), + ( + ["sandbox", *layer_arguments, "--mount", "/proc/share,share,rw"], + "reserved by the sandbox", + ), + ): + with self.subTest(arguments=arguments): + with self.assertRaisesRegex(common.ScriptError, message): + nvx.command_sandbox(nvx.parse_args(arguments)) + + def test_run_forwards_mount_owner_only_with_a_mount(self): + args = nvx.parse_args( + [ + "run", + "--mount", + "/mnt/share,share,rw", + "--mount-owner", + "caller", + "--dry-run", + ] + ) + with ( + patch.object(nvx, "require_file", return_value=Path("artifact")), + patch.object( + nvx, "_format_command", return_value="formatted" + ) as format_command, + ): + nvx.command_run(args) + command = format_command.call_args.args[0] + self.assertEqual(command[command.index("--mount-owner") + 1], "caller") + + without_mount = nvx.parse_args(["run", "--mount-owner", "caller", "--dry-run"]) + with ( + patch.object(nvx, "require_file", return_value=Path("artifact")), + self.assertRaisesRegex(common.ScriptError, "requires --mount"), + ): + nvx.command_run(without_mount) + def test_sandbox_command_parses_state_aware_operations(self): provision = nvx.parse_args( [ @@ -4677,6 +4798,114 @@ def test_layer_parser_rejects_invalid_role_and_uuid(self): with self.assertRaisesRegex(common.ScriptError, "UUID is invalid"): sandbox.SandboxLayer.parse("distro,layer.erofs,not-a-uuid") + def test_mount_contract_forwards_live_share_and_counts_bootstrap_tokens(self): + distro = sandbox.SandboxLayer.parse( + "distro,distro.erofs,11111111-1111-1111-1111-111111111111" + ) + mount = sandbox.SandboxMount.parse( + "/workspace,share,rw", + denied_paths=(Path("share/secrets"),), + owner="process", + ) + self.assertEqual( + mount.openvmm_arguments(), + [ + "--mount", + f"/workspace,{Path('share')},rw", + "--mount-deny", + str(Path("share/secrets")), + "--mount-owner", + "process", + ], + ) + self.assertEqual(sandbox.SandboxMount.parse("/data,share").access, "ro") + self.assertEqual( + sandbox.SandboxMount.parse("/data,share").owner, + sandbox.default_mount_owner(), + ) + launch = sandbox.SandboxLaunch( + layers=(distro,), + scratch=Path("scratch.ext4"), + mount=mount, + ) + self.assertEqual( + launch.openvmm_arguments()[-6:], + mount.openvmm_arguments(), + ) + # The live-share bootstrap tokens are appended by OpenVMM, but they + # still consume the x86 command-line budget. + without_mount = sandbox.SandboxLaunch( + layers=(distro,), scratch=Path("scratch.ext4") + ) + padding = "x" * ( + sandbox.SANDBOX_COMMAND_LINE_MAX_SIZE + - len(without_mount.kernel_command_line()) + - 2 + ) + without_mount.kernel_command_line(padding) + with self.assertRaisesRegex(common.ScriptError, "1024-byte"): + launch.kernel_command_line(padding) + + def test_mount_contract_rejects_unsafe_targets_and_owners(self): + for value, message in ( + ("workspace,share,rw", "canonical absolute path"), + ("/,share,rw", "canonical absolute path"), + ("/work/../etc,share,rw", "canonical absolute path"), + ("/work//space,share,rw", "canonical absolute path"), + ("/work space,share,rw", "canonical absolute path"), + ("/work=space,share,rw", "canonical absolute path"), + ("/proc,share,rw", "reserved"), + ("/sys/share,share,rw", "reserved"), + ("/dev/shm,share,rw", "reserved"), + ("/.nvx-agent/share,share,rw", "reserved"), + ("/etc,share,rw", "reserved"), + ("/etc/machine-id,share,rw", "reserved"), + ("/workspace,share,write", "ro or rw"), + ("/workspace,,rw", "host path is empty"), + ("/workspace", "GUEST_TARGET,HOST_PATH"), + ): + with self.subTest(value=value): + with self.assertRaisesRegex(common.ScriptError, message): + sandbox.SandboxMount.parse(value, owner="process") + sandbox.SandboxMount.parse("/etc/app,share,rw", owner="process") + with self.assertRaisesRegex(common.ScriptError, "--mount-owner"): + sandbox.SandboxMount.parse("/workspace,share,rw", owner="root") + with self.assertRaisesRegex(common.ScriptError, "at most 128"): + sandbox.SandboxMount.parse( + "/workspace,share,rw", + denied_paths=tuple(Path(f"share/{index}") for index in range(129)), + owner="process", + ) + share = Path("share") + with ( + patch.object(sandbox.os, "name", "nt"), + self.assertRaisesRegex(common.ScriptError, "requires a Linux host"), + ): + sandbox.SandboxMount( + guest_target="/workspace", host_path=share, owner="caller" + ) + + def test_mount_contract_validates_host_paths_against_the_export(self): + with tempfile.TemporaryDirectory() as temporary: + share = Path(temporary) / "share" + (share / "secrets").mkdir(parents=True) + sandbox.SandboxMount( + guest_target="/workspace", + host_path=share, + denied_paths=(Path("secrets"), share / "secrets"), + ).validated() + with self.assertRaisesRegex(common.ScriptError, "does not exist"): + sandbox.SandboxMount( + guest_target="/workspace", + host_path=share, + denied_paths=(Path("missing"),), + ).validated() + with self.assertRaisesRegex(common.ScriptError, "directory does not"): + sandbox.SandboxMount( + guest_target="/workspace", + host_path=share / "missing", + ).validated() + def test_managed_lifecycle_provisions_and_deprovisions_state(self): with tempfile.TemporaryDirectory() as temporary: root = Path(temporary) @@ -4809,6 +5038,105 @@ def require(path: Path, _description: str) -> Path: ) session.ping.assert_called_once_with(10) + def test_managed_lifecycle_retains_the_live_mount(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary).resolve() + layer_path = root / "distro.erofs" + scratch_path = root / "scratch.ext4" + share = root / "share" + (share / "secrets").mkdir(parents=True) + layer_path.write_bytes(b"layer") + scratch_path.write_bytes(b"scratch") + state = root / "state" + mount = sandbox.SandboxMount( + guest_target="/workspace", + host_path=share, + access="rw", + denied_paths=(Path("secrets"),), + owner="process", + ) + sandbox_lifecycle.provision( + state, + sandbox.SandboxLaunch( + layers=( + sandbox.SandboxLayer( + role="distro", + path=layer_path, + uuid="11111111-1111-1111-1111-111111111111", + ), + ), + scratch=scratch_path, + mount=mount, + ), + hypervisor="whp", + memory_mib=256, + net=None, + network_profile=None, + network_egress=None, + network_ingress=None, + network_egress_allow=(), + network_egress_deny=(), + host_loopback=None, + network_proxy=None, + host_loopback_forward=(), + cmdline="quiet", + ) + config = json.loads( + (state / sandbox_lifecycle.CONFIG_NAME).read_text(encoding="utf-8") + ) + self.assertEqual( + config["mount"], + { + "guest_target": "/workspace", + "host_path": str(share), + "access": "rw", + "denied_paths": ["secrets"], + "owner": "process", + }, + ) + self.assertEqual(sandbox_lifecycle._deserialize_launch(config).mount, mount) + legacy = dict(config) + del legacy["mount"] + self.assertIsNone(sandbox_lifecycle._deserialize_launch(legacy).mount) + with self.assertRaisesRegex(common.ScriptError, "malformed"): + sandbox_lifecycle._deserialize_launch({**config, "mount": []}) + + process = MagicMock() + process.pid = 123 + process.stdin = io.BytesIO() + context = MagicMock() + context.__enter__.return_value = MagicMock() + + def require(path: Path, _description: str) -> Path: + return path + + with ( + patch.object( + sandbox_lifecycle, + "require_file", + side_effect=require, + ), + patch.object( + sandbox_lifecycle.subprocess, + "Popen", + return_value=process, + ) as popen, + patch.object( + sandbox_lifecycle.ControlSession, + "connect", + return_value=context, + ), + ): + sandbox_lifecycle.start(state, 10) + + command = popen.call_args.args[0] + self.assertEqual( + command[command.index("--mount") + 1], + f"/workspace,{share},rw", + ) + self.assertEqual(command[command.index("--mount-deny") + 1], "secrets") + self.assertEqual(command[command.index("--mount-owner") + 1], "process") + def test_versioned_json_reader_supports_custom_version_field(self): with tempfile.TemporaryDirectory() as temporary: path = Path(temporary) / "outcome.json"