Skip to content

Layer 1 microVM: a securecore.json edit reaches the running guest; per-site php.ini debug settings (#90) - #95

Merged
samifouad merged 12 commits into
mainfrom
claude/microvm-config-90
Nov 12, 2026
Merged

samifouad merged 12 commits into
mainfrom
claude/microvm-config-90

Conversation

@samifouad

Copy link
Copy Markdown
Member

Fixes #90: an edited securecore.json reaches the running guest, with the host's behaviour, and the per-site debug settings from php.ini work in the guest. Stacked on #92 (#87) and #91 (#86); this PR targets claude/microvm-boot-reason-87.

Design

The config: live delivery, no restart. I chose this over the issue's alternative (rebuild the site disk and restart the VM): it is what a host does, and it is faster.

  • The launcher follows the file with the extension's own securecore_config::Reloader, the same stamp, loader and messages, once a second. That is WATCH_INTERVAL, the same as the extension's RELOAD_INTERVAL.
  • Invalid → refused. The launcher logs <file>: invalid: …; keeping the last valid config in the guest (sha256 …), the extension's wording on a host.
  • Valid but not expressible in the VM → refused too. live_change runs the same plan() check a start runs, so a config the microVM can't run (for example unixSockets) is refused and logged, and the guest keeps its config.
  • Valid → delivered. It goes to the guest over the control channel as a new config <hex> line, accepted only after the start message.
    • The guest writes it to a new file on tmpfs (/run/securecore/securecore-N.json) and bind-mounts it read-only over <root>/securecore.json, on top of the one before.
    • Readers see the old file or the new one, never half of one, and the extension's stat sees a new inode, so the extension in the guest reloads it exactly as on a host. That includes its own restart_needed, and its securecore: reloaded … (sha256 …) line.
    • The stack is capped at 200 configs (each up to 128 KiB) per running guest. Past that, the launcher logs that a restart is needed.
  • Matches the host's restart_needed. Layers 2 and 3 change in place. A change to layer1 or the vault is delivered, and the launcher logs that it applies at the next start. The guest's extension reports restart_needed=1, as it does on a host, because it compares against the config it started with.
  • Egress. The host's egress filter takes layer 2's new host list at once. It is now read per connection from an RwLock.

php.ini: --guest-ini <file>. php.ini in the guest is the root image's, the same for every site, so the per-site settings come from the launcher:

  • The file holds securecore.debug_* = … lines. Any other key is refused with the reason. The rest of php.ini is the image's, and an extension= or securecore.config= line from a site has no business in the guest.
  • It is delivered in every start message, and again whenever the file changes (the ini control line). The guest writes /run/securecore/conf.d/zz-site.ini and adds it to PHP_INI_SCAN_DIR.
  • php-fpm is started again gracefully with it, which is what a php.ini change needs on a host. A value the extension refuses stops php-fpm, as it would on a host, and the launcher's log says why (Layer 1 microVM: a php-fpm that refuses to start at boot leaves no reason in the launcher's log #87).

Measured

On [xps], from the write of securecore.json to the first request served on the new config: 480–486 ms (3 runs). The launcher delivered it 427–434 ms after the write: the 1 s stat poll, about half a second on average. The control round trip and the guest's mount take 1–2 ms ("delivered … to the running guest in 1 ms").

  • A php.ini change (delivery plus a graceful php-fpm restart in the guest) takes 70–84 ms after the launcher sees it.
  • The rest is the two once-a-second polls: the launcher's stat, then the guest extension's.

Tests

  • test/ext-config.sh on the microVM. These checks were n/a and are now real:

    • an edit reaches every worker without a restart (10/10 requests, the same launcher, no boot, no php-fpm restart);
    • the php-fpm log records the reload and the new sha256;
    • a layer1 edit is delivered with restart_needed=1, and the launcher says it applies at the next start;
    • an edit the VM can't run is refused, and the guest keeps its config;
    • an invalid edit is refused, the last valid config stays live and serving, and the launcher's log names the file and the problem.

    Still n/a, each with its reason: "status reports off" (the target keeps layer 1 on); reload_error (the guest never sees the invalid file; the launcher's log check replaces it); the securecore.config php.ini cases.

  • test/ext-attribution.sh on the microVM. All 34 former n/a checks are real now: the debug settings go through --guest-ini, and an invalid value stops php-fpm with the extension's reason.

  • test/lib/target.sh. put is one rename. put_delivered puts a config into the running guest and waits for the launcher's delivered or refused line. launcher_start passes --guest-ini $tdir/guest.ini. put_live still restarts on microVM, because suites change site files with it.

  • Unit tests:

    • a_live_edit_is_refused_when_the_vm_cant_run_it_and_flags_what_needs_a_start;
    • a_sites_ini_carries_only_the_debug_settings;
    • the control lines round trip, and config is accepted only after the start;
    • the start message carries ini after env.

Evidence ([xps]; flock + sudo, logged, exit codes checked; fresh projects)

rust (clippy -D warnings + cargo test, whole workspace)   exit=0
test/ext-microvm.sh                                        exit=0 204 ok 0 FAIL
microVM target                                docker target
mvm-config      exit=0  47 ok 0 FAIL  5 n/a  (was 39 ok, 12 n/a)   docker-config      exit=0  46 ok
mvm-attribution exit=0  40 ok 0 FAIL  0 n/a  (was  6 ok, 34 n/a)   docker-attribution exit=0  40 ok
mvm-layer3      exit=0  55 ok 0 FAIL  4 n/a                         docker-layer3      exit=0  59 ok
mvm-layer1      exit=0  55 ok 0 FAIL 18 n/a                         docker-layer1      exit=0  73 ok
mvm-hosts       exit=0  37 ok 0 FAIL 59 n/a                         docker-hosts       exit=0  96 ok
mvm-enforce     exit=0  92 ok 0 FAIL 18 n/a                         docker-enforce     exit=0 110 ok
mvm-db          exit=0 152 ok 0 FAIL  0 n/a                         docker-db          exit=0 152 ok
mvm-vault       exit=0  70 ok 0 FAIL  9 n/a                         docker-vault       exit=0  78 ok
scripts/ext-build.sh 8.3 nts   exit=0     scripts/guest-build.sh   exit=0
test/release-check.sh 8.3 nts  exit=0 103 ok 0 FAIL     test/promote-check.sh  exit=0 14 ok
  • Config and attribution are from the final build. A first run found two bugs, both fixed here: the guest unlinked each config file after mounting it, and Linux refuses a mount on an unlinked file, so the second delivery failed; and the attribution fixtures needed a site_sync.
  • The other suites and the release gates ran on that first build. Its only difference is the guest's config-swap code, which they don't exercise.
  • docker-vault failed once on No such image for my stack's php container, whose image had been removed from the host while the container ran. I don't prune; someone else on [xps] did. After recreating my container it passes.
  • Clippy -D warnings (linux target): clean.

Revert proof

The same suites against a build with only the launcher's watcher removed (watch_live never started):

mvm-config      exit=1 38 ok 10 FAIL
FAIL the launcher delivered the edit
FAIL edited to mode=log: 10/10 requests … report the new sha256 and mode (got '0', want '10')
FAIL the php-fpm log records the reload and the new sha256 (got '0', want '1')
FAIL layer1 changed (a connect port added): delivered, restart_needed (got '…/0', want '…/1')
FAIL an edit the microVM can't run: the guest keeps the last config it could …
FAIL invalid edit: the last valid config stays live …
mvm-attribution exit=1 13 ok 27 FAIL
FAIL status: attribution_log both (got 'off', want 'both')
FAIL frame: the plugin opening a file itself is the plugin (no line …)
…

closes #90

-claude

The launcher follows the config with the extension's own Reloader, once a second. An invalid
file, or one the microVM can't run (the check a start makes), is refused with the reason and the
guest keeps its config. A valid one goes to the guest over the control channel (config line),
which mounts it over <root>/securecore.json from tmpfs; the extension there reloads it as on a
host. layer1 and vault changes apply at the next start and the guest's extension reports
restart_needed, as on a host; the host's egress takes layer 2's new hosts at once.

--guest-ini <file> carries the site's securecore.debug_* php.ini settings into the guest (any
other key is refused): in each start message, and again on every change, with php-fpm started
again gracefully. ext-config and ext-attribution check these on the microVM target instead of
n/a.
…s, the guest decides a delivery (#90 review)

- securecore.json and the --guest-ini file are read with the pool user's filesystem ids
  (setfsuid/setfsgid, this thread only), O_NOFOLLOW, regular files only, within a cap (128 KiB,
  16 KiB): a link to another site's config or to /dev/zero is refused and logged. The watcher
  parses with the config crate's own Config::from_slice and LoadError wording.
- --guest-ini values may hold only [a-z0-9_.,:-] (at most 64): no ${VAR}, which PHP would
  expand from php-fpm's environment, where the vault's secrets are. A refused line is never
  quoted back.
- The guest mounts a delivered config only on <root>/securecore.json opened O_PATH|O_NOFOLLOW
  and found to be a regular file, through its /proc/self/fd path: never through a link on the
  site disk.
- A delivery whose reply didn't arrive is settled by asking the guest which config it reads now
  (configsha): if it is the one sent, it stands.
- test/lib/na-expected.txt: config 5 and attribution 0 on this branch; ext-config checks that a
  link in place of securecore.json is refused while the guest keeps its config.
…file as the pool user; --guest-ini as the launcher (#90 review)

CI on 4f6… showed the launcher refusing to start: it opened --guest-ini (the operator's file,
under a 0700 folder) as the pool user. read_regular now opens the folder as the launcher and
only the file in it (openat, O_NOFOLLOW) as the pool user, the way the site disk's copy reaches
the site; --guest-ini is read as the launcher, never through a link, regular and capped. A test
as root checks that a file only root may read isn't read for the pool user.
@samifouad

Copy link
Copy Markdown
Member Author

Review fixes (#90 review), in b46f3fe (after merging #92 in 5108e40)

Finding Fix Test (fails with the fix reverted)
4. securecore.json was read as root, through links, with no cap (a link to another site's config got delivered; /dev/zero grew root's memory) read_regular reads the file as the pool user: this thread's filesystem uid and gid are switched with setfsuid/setfsgid around the open. It never follows a link (O_NOFOLLOW), takes regular files only (O_NONBLOCK, so a FIFO or device never blocks), and reads at most 128 KiB, the MAX_CONFIG the guest takes. The watcher parses with the config crate's Config::from_slice, and its refusals use the crate's LoadError wording a_site_file_is_read_as_itself_regular_and_capped: a link to another file, a link to /dev/zero, /dev/zero itself, and a file over the cap are all refused, and no error quotes the file's bytes. ext-config.sh (microVM): a link in place of securecore.json: the guest keeps its config, and the launcher says the link is refused, once
5. --guest-ini values could hold ${VAR} (PHP expands it from php-fpm's environment, where the vault's secrets are), and refusals quoted the value Values must match [a-z0-9_.,:-]{0,64} (site_ini_value_ok): no $, quotes or braces. No refusal quotes anything from the file, only the line number (and the key, once it is a known securecore.debug_* one) a_sites_ini_carries_only_the_debug_settings: ${WORDPRESS_DB_PASSWORD}, $SECRET, quotes, braces and uppercase are refused, and no message contains them
6. The --guest-ini file was read through links, uncapped, and its lines were echoed on error Read with read_regular as the launcher (the file is the operator's, named by a flag, like --config), with O_NOFOLLOW, regular files only, at most 16 KiB, both at the start and in the watcher. Errors carry only the path and line number Same tests as findings 4 and 5 (read_site_ini through a link; ${…} not echoed)
7. The mount target <root>/securecore.json could be a link on the site disk, and mount followed it regular_target opens it O_PATH | O_NOFOLLOW and requires a regular file. The bind mount, and its read-only remount, go on that file through /proc/self/fd/N a_mount_target_must_be_a_regular_file_not_a_link: a link and a folder are refused
8b. A timeout after the guest had mounted the config marked it refused while the guest ran it The guest's answer decides. An error reply is a refusal. With no reply, the launcher asks the guest which config it reads now (new configsha control line, answered sha <hex>); if it is the one sent, the delivery stands and the log says the reply was lost a_delivery_whose_reply_was_lost_is_decided_by_the_guests_config (delivery_outcome)

test/lib/na-expected.txt on this branch: config 5, attribution 0.

Evidence ([xps], head 33f4229; flock + sudo, each logged, exit codes checked; fresh projects sc86-*/sc86m-*)

test/ext-microvm.sh   exit=0 206 ok 0 FAIL
microVM target                                         docker target
mvm-config      exit=0  50 ok 0 FAIL  5 n/a            docker-config      exit=0  46 ok
mvm-attribution exit=0  41 ok 0 FAIL  0 n/a            docker-attribution exit=0  40 ok
mvm-layer3      exit=0  56 ok 0 FAIL  4 n/a            docker-layer3      exit=0  59 ok
mvm-layer1      exit=0  56 ok 0 FAIL 18 n/a            docker-layer1      exit=0  73 ok
mvm-hosts       exit=0  38 ok 0 FAIL 59 n/a            docker-hosts       exit=0  96 ok
mvm-enforce     exit=0  93 ok 0 FAIL 18 n/a            docker-enforce     exit=0 110 ok
mvm-db          exit=0 153 ok 0 FAIL  0 n/a            docker-db          exit=0 152 ok
mvm-vault       exit=0  71 ok 0 FAIL  9 n/a            docker-vault       exit=0  78 ok
scripts/ext-build.sh 8.3 nts exit=0   scripts/guest-build.sh exit=0
test/release-check.sh exit=0 103 ok   test/promote-check.sh exit=0 14 ok
cargo test -p securecore-fpm (native, [xps]): 78 passed

(Each ok count above includes the suite's new n/a count line.)

  • Rust gate (docker build --target check, clippy -D warnings + cargo test, whole workspace): green in GitHub CI. On the iMac's Colima, the same older timing-sensitive test fails as noted on Layer 1 microVM: every guest log line reaches the launcher, numbered and acknowledged (#86) #91.
  • GitHub CI on 33f4229 (run 43109553313): ext php8.3-nts-linux-x86_64 pass, microVM (test/ext-microvm.sh) pass. That job runs all eight suites on the microVM target.
  • Live reload, measured again: 536 ms from the edit to the first request on the new config (launcher delivery 490 ms).
  • Revert proofs (unit, one tree with findings 4, 5, 7 and 8b each undone): a_sites_ini_carries_only_the_debug_settings, a_mount_target_must_be_a_regular_file_not_a_link, a_delivery_whose_reply_was_lost_is_decided_by_the_guests_config and a_site_file_is_read_as_itself_regular_and_capped all FAIL (74 passed; 4 failed).
  • Found by CI on the way, fixed in b3dc9f6: reading --guest-ini as the pool user failed under its 0700 folder. The launcher now reaches each file's folder as itself and opens only the config file in it as the pool user. --guest-ini is the operator's file, so the launcher reads it as itself (no link, regular, capped).
  • Release check: 103 ok once guest-build.sh had its git checkout on [xps]. Without it, the first run skipped the microVM part (74 ok).

-claude

… does; late answers and ini retries (#90 verification review)

- S1/N2: the config watcher and --guest-ini use sitefile::read (no link anywhere in the path,
  as the folder's owner, regular only, capped), the launcher's start reading; read_regular is
  gone. The watcher's refusals use LoadError::without_contents.
- N5: the guest's configsha answer reads <root>/securecore.json the same way.
- N4: a php.ini delivery the guest never answered is sent again on the next look (its own
  refusal stands); a config delivery whose answer was lost is settled by asking the guest which
  config it reads on the next looks too (up to 10).
- ext-config's microVM checks expect the launcher's contents-free wording.
@samifouad

Copy link
Copy Markdown
Member Author

Verification-review fixes (round 3)

Finding Where Fix Revert-and-fail test
K2 a log key that every restore kept honouring #91 (fba086a) Each restore takes the log with a fresh key (restore_keys), and give_log_key sends it to the restored init right after the snapshot loads. The snapshot's key is only ever refused: quietly, counted apart, and said at the stop each_restore_gets_a_log_key_of_its_own, an_old_key_and_a_slow_hello_are_refused_and_said_at_the_stop
S1 the startup config was read as root, through links, uncapped (gelada too) #91 (fba086a); watcher in #95 (decf48d) One reader, sitefile::read/load, used by the launcher's start, the microVM's watcher, --guest-ini and the guest's configsha:
• every path component is resolved with openat2(RESOLVE_NO_SYMLINKS), which also fixes N2;
• the file is opened as the owner of its folder (setfsuid/setfsgid, this thread only);
• regular files only (O_NONBLOCK), at most 128 KiB.
Error text: LoadError::without_contents reports a JSON or schema error as line, column and kind, never serde's quoted token. The config crate's load_bytes parses what the launcher read; the extension's own load is unchanged
sitefile::…the_config_is_read_as_itself_by_its_folders_owner: a link to another site's config, a link anywhere in the path, root's 0600 file, /dev/zero and a FIFO are all refused within a second, and an error never contains the file's sk_live_SECRET. Also tests/launch.rs, and ext-config and ext-layer1 on the launcher's wording
K4 a worker could write a line that reads as securecore-init: #91 Own frame: the init's lines are their own frame kind (I), logged as guest-init: …; php-fpm's are always guest: ….
Reserved room: Outbox::init_line may use the last 256 KiB of the 4 MiB buffer, which php-fpm's lines never take.
Quoting: #92's quoted stop line labels the init's own
the_inits_lines_are_labelled_apart_and_never_crowded_out
K5 noexec everywhere it was claimed #91 The site disk, the writable disks and the site's tmpfs folders are all mounted MS_NOEXEC (nosuid and nodev as before) ext-microvm.sh: an uploads disk is noexec, the site disk is noexec (its script runs only through /bin/sh). php-fpm serves the whole site: all suites below pass
N1 unescaped guest lines #91 escaped(): control characters (the tab apart, since SQL in PHP's lines is full of tabs) and U+0085, U+2028 and U+2029 become \u{…} in every guest line a_guest_line_stays_one_line
K3 a slow key line, and eviction by churn #91 Hard deadline: a connection has one second in all for its key line (read_hello, a byte at a time).
Eviction: making room among waiting connections spares one whose whole key line is already waiting (FIONREAD), which is the init's case, since it writes its key the moment it connects
an_old_key_and_a_slow_hello…: the right key, dripped a byte every 60 ms, is cut at the deadline. Reverted: cut at the deadline, not after the whole line (4.30s)
N3 refusals in the last second went unsaid #91 LogServer::flush on the VM's drop says the unsaid refusals and the old key's count the same test
N4 a failed ini delivery wasn't retried; a late put_config was #95 An ini delivery the guest never answered is sent again on the next look; the guest's own refusal stands. A config delivery whose answer was lost is settled on the next looks too (up to 10) by asking the guest which config it reads a_delivery_whose_reply_was_lost_is_decided_by_the_guests_config (the decision); the retry path is the watcher's
N5 the guest's config_sha read through links, uncapped #95 config_sha uses sitefile::read sitefile tests
Document #91 docs/layer1-microvm.md, "The guest's log: what it guarantees, and against whom": it holds against non-root guest code, not against root in the guest —

Revert proof (one tree with K2, K4, N1, S1 undone, iMac Colima Rust gate): a_guest_line_stays_one_line, the_config_is_read_as_itself_by_its_folders_owner, the_inits_lines_are_labelled_apart_and_never_crowded_out, each_restore_gets_a_log_key_of_its_own and a_site_file_is_read_as_itself_regular_and_capped FAIL. K3 was proven on its own (above). The remaining failure on the iMac, the_site_key_changes_with_the_tree_but_not_with_uploads, is the older timing-sensitive test that fails only under Colima's coarse timestamps.

Also fixed, found by CI on the way (#91 cbb3e6d, merged up to ceae6bf): GitHub's run of the stress had 11 of 240 requests fail right after a SIGUSR2 reload. The guest answered the reload once the new master had started, so the launcher logged "reloaded" and the next requests met no listener. The guest now answers a reload only once 127.0.0.1:9000 takes connections (10 s at most), as it already did for a restart caused by a start message. Revert-and-fail: the CI run of 36ed957 (240 requests answered 200 (got '229')); with the fix, [xps] passes it.

Gates ([xps], flock + sudo, each logged and exit-checked; fresh projects)

Head 36ed957, the full set:

cargo test -p securecore-fpm (native, as root: sitefile's root-only cases run)  exit=0
test/ext-microvm.sh   exit=0 208 ok 0 FAIL      (the 2 new noexec checks)
microVM target                                   docker target
mvm-config      exit=0  52 ok  5 n/a             docker-config      exit=0  46 ok
mvm-attribution exit=0  41 ok  0 n/a             docker-attribution exit=0  40 ok
mvm-layer3      exit=0  56 ok  4 n/a             docker-layer3      exit=0  59 ok
mvm-layer1      exit=0  56 ok 18 n/a             docker-layer1      exit=0  73 ok
mvm-hosts       exit=0  38 ok 59 n/a             docker-hosts       exit=0  96 ok
mvm-enforce     exit=0  93 ok 18 n/a             docker-enforce     exit=0 110 ok
mvm-db          exit=0 153 ok  0 n/a             docker-db          exit=0 152 ok
mvm-vault       exit=0  71 ok  9 n/a             docker-vault       exit=0  78 ok
ext-build exit=0   guest-build exit=0   release-check exit=0 103 ok   promote-check exit=0 14 ok

After the reload fix (ceae6bf): ext-microvm.sh exit 0, 208 ok, including 240 requests answered 200; mvm-config exit 0, 52 ok.
CI: I checked once, on 36ed957: ext pass, microVM fail (the reload gap above, now fixed). I have not checked the run for ceae6bf.

-claude

@samifouad

Copy link
Copy Markdown
Member Author

Symlinked site roots work again (a deployment's current -> releases/<n>)

The rule refused a link anywhere in securecore.json's path, which broke ordinary deployments. It now follows a split.

Above the site root, the operator's. The folder holding the file (the root --config names) is resolved once, as the launcher, with canonicalize. A link there, such as /var/www/current -> releases/1 or /var/www/html -> /srv/site, works.

Below it, the site owner's. The file is opened from an fd on that resolved root with openat2(RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS), as the root's owner:

  • securecore.json itself as a link is refused;
  • a linked folder on the way is refused;
  • .. out of the root is refused.

One function, sitefile::read_beneath, is used by the launcher's start, the microVM's config watcher, --guest-ini (its folder resolved once the same way) and the guest's configsha.

On #91 (639ad37), merged up to #92 and #95 (988ccc7).

Test: sitefile::tests::the_root_may_be_a_deployments_link_but_nothing_below_it.

  • current -> releases/1 loads.
  • A linked securecore.json inside the root, a linked subfolder and ../../elsewhere.json are refused.
  • With the old reading (RESOLVE_NO_SYMLINKS from /), the first assertion fails: the deployment's link is refused.
  • the_config_is_read_as_itself_by_its_folders_owner still holds: a root-only file, /dev/zero, a FIFO and an oversized file are all refused.

[xps] (flock + sudo, logged, exit-checked):

cargo test -p securecore-fpm (native, as root)   exit=0
test/ext-microvm.sh                              exit=0 208 ok 0 FAIL
SC_TARGET=microvm test/ext-config.sh             exit=0  52 ok 0 FAIL 5 n/a  (incl. "a link in place of securecore.json: the guest keeps its config")
SC_TARGET=microvm test/ext-layer1.sh             exit=0  56 ok 0 FAIL 18 n/a
test/ext-config.sh (docker)                      exit=0  46 ok
test/ext-layer1.sh (docker)                      exit=0  73 ok

CI on #95 (988ccc7), checked once: ext php8.3-nts pass, microVM (all eight suites on the microVM target) pass, guest reproducibility pass.

-claude

@samifouad
samifouad changed the base branch from claude/microvm-boot-reason-87 to main November 12, 2026 00:41
@samifouad
samifouad merged commit e7541ab into main Nov 12, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Layer 1 microVM: a securecore.json edit never reaches a running guest

1 participant