Skip to content

Layer 1 microVM: every guest log line reaches the launcher, numbered and acknowledged (#86) - #91

Merged
samifouad merged 8 commits into
mainfrom
claude/microvm-log-86
Nov 12, 2026
Merged

samifouad merged 8 commits into
mainfrom
claude/microvm-log-86

Conversation

@samifouad

Copy link
Copy Markdown
Member

Fixes #86: no guest log line is lost without the launcher saying so. Stacked on #88 (this PR targets claude/microvm-suites-72); with it, #88's microVM-target CI step runs all eight ext-* suites.

Root cause

The launcher's log listener (vm.rs wire) called shutdown(Write) on every guest log connection, to keep it read-only. Firecracker's vsock (csm/connection.rs) reads that as EOF on the host stream. It moves the connection to LocalClosed, sends the guest SHUTDOWN(RCV|SEND), and from then on drops every data packet the guest sends on it (dropping invalid TX pkt).

The guest's write() had already returned by then, so pump_log thought those bytes were delivered. Which lines landed in that window was a race, so a different few went missing each run, and more often on slower runners. Three things made it worse:

  • pump_log reconnected after each EPIPE.
  • report() opened a connection per line.
  • These filled LOG_SLOTS (4), and a connection with no free slot was dropped without a word.

A restore also resets the guest's vsock connections, which loses anything still in flight.

What changed

  • New ext/fpm/src/guestlog.rs: one numbered, acknowledged stream.
    • The guest (Outbox and send) numbers every line: php-fpm's output and the init's own report() lines. It holds each line, up to 4 MiB, until the host acknowledges it, and sends on one connection. After a broken connection, a stall (2 s with nothing acknowledged) or a restore, it reconnects and sends every unacknowledged line again. A line that doesn't fit is counted, and the count goes out in order as an X <seq> <n> frame.
    • The host (Inbox and serve) logs each frame once (a resent frame is skipped by its number), marks it guest: as before, and acknowledges it (A <seq>). It never shuts the connection for writing.
    • Counted losses. A count frame, or a gap in the numbers, becomes the launcher's own line: securecore-fpm: N guest log lines lost (…).
    • Stop. A sync (sent before every stop) waits up to 1 s for the log to drain. Lines still unacknowledged then come back in the reply as N guest log lines lost (not acknowledged by the host before the stop), and the launcher logs that.
    • Refused control messages. Before the guest replies to a refused control message, it drains the log, so the reason is in the launcher's log before the launcher exits.
  • A full log channel is said, not silent. LOG_SLOTS still caps connections, but a refused connection is now logged, and the guest resends its lines.
  • PROTOCOL is 5. The log port's format changed, and snapshots of a protocol-4 init are never restored.
  • CI. ext-pr.yml's microVM-target step now runs config, attribution, layer3, layer1, hosts, enforce, db and vault.

Tests

  • Unit (guestlog::tests):

    • frames round trip;
    • the host logs a resent frame once and counts gaps;
    • a full buffer is counted and the count is delivered;
    • long lines are cut;
    • every_line_arrives_once_through_connections_that_die_or_stall: 4 writers × 500 lines through a host whose first 4 connections either close mid-stream or swallow data without acknowledging it. Every line arrives once, in order.
  • End to end: test/ext-microvm.sh section 3d (also test/ext-microvm.sh log alone). It runs on a restored guest:

    • 240 requests, 8 at a time, each logging 25 numbered lines through error_log (6000 lines);
    • 20 of the requests write over a host symlink the sync refuses;
    • a graceful reload (SIGUSR2) between the two halves.

    It checks that every line arrives once, all 20 refusals arrive, and nothing is counted lost.

Evidence ([xps], each run as flock … sudo -E env PATH=$PATH TMPDIR=$PWD/.tmp …, logged to a file, exit code checked; fresh projects sc86-* and sc86m-*)

rust (docker build --target check: clippy -D warnings + cargo test, whole workspace)  exit=0
microVM target                                docker target
mvm-config      exit=0 39 ok 0 FAIL 12 n/a     docker-config      exit=0  46 ok 0 FAIL
mvm-attribution exit=0  6 ok 0 FAIL 34 n/a     docker-attribution exit=0  40 ok 0 FAIL
mvm-layer3      exit=0 55 ok 0 FAIL  4 n/a     docker-layer3      exit=0  59 ok 0 FAIL
mvm-layer1      exit=0 55 ok 0 FAIL 18 n/a     docker-layer1      exit=0  73 ok 0 FAIL
mvm-hosts       exit=0 37 ok 0 FAIL 59 n/a     docker-hosts       exit=0  96 ok 0 FAIL
mvm-enforce     exit=0 92 ok 0 FAIL 18 n/a     docker-enforce     exit=0 110 ok 0 FAIL
mvm-db          exit=0 152 ok 0 FAIL 0 n/a     docker-db          exit=0 152 ok 0 FAIL
mvm-vault       exit=1 68 ok 1 FAIL 9 n/a      docker-vault       exit=0  78 ok 0 FAIL

Revert proof

  • Unit. Make a new connection start after the last sent frame instead of the last acknowledged one (no resend). The stress test then fails: the host acknowledged every line: 2000 (2000 lines never acknowledged).
  • End to end. The old code (Layer 1 microVM: the ext-* suites run on a microVM target (SC_TARGET=microvm) #88's branch, with only this PR's ext-microvm.sh and probe) loses lines silently, while 240 of 240 requests answered 200:
FAIL all 6000 numbered lines are in the launcher's log (got '5726', want '6000')
FAIL all 20 sync refusals are in the launcher's log (got '19', want '20')
ok   no line was lost or counted as lost      ← the old code has no count: that is the bug

Found on the way (not changed here)

A guest reload (SIGUSR2/HUP) stops php-fpm before it starts the new one. Requests that arrive in that gap fail: 13 of 240 got a non-200 in a first run of the stress, with the reload in the middle of a burst. The stress now reloads between bursts. A reload with no gap (start the new master, then quit the old one) would be its own issue.

closes #86

-claude

…ged (#86)

The launcher shut its side of each log connection for writing. Firecracker turns a host's
half-close into a full close of the connection and drops whatever the guest sends after it,
while the guest's write() has already returned: lines vanished at random, and pump_log's
reconnects and report()'s connection per line filled LOG_SLOTS, so more were refused.

The guest now numbers every line (php-fpm's output and the init's own) and holds it until the
host acknowledges it, on one connection, sending again what wasn't acknowledged after a broken
or stalled connection or a restore. The host logs each frame once and acknowledges it. A full
guest buffer, a gap in the numbers or lines unacknowledged at a stop become the launcher's own
counted line. PROTOCOL 5. ext-pr.yml runs all eight ext-* suites on the microVM target.
…owledged only once written (#86 review)

Any process in the guest could connect to host port 1514 and forge or suppress the log: one
Inbox was shared by every connection, so a frame numbered u64::MAX made every later init line
look already logged, and X frames or jumps wrote launcher-prefixed loss lines.

- The launcher makes a log key per VM (kept beside a snapshot, whose init holds it) and gives it
  to the init over the control port, which only the host can reach. A log connection must
  present it first, within a second; others are refused and counted in a launcher line at most
  once a second. At most 16 connections wait for their key (the oldest goes), and keyed ones
  have their own places, so nothing can squat the port.
- A jump larger than the guest's buffer can make closes the stream.
- The init's reconnects back off (to 1 s) while no connection gets an acknowledgement.
- A line is acknowledged only once it is written: a failed write stops the connection and
  leaves the line unlogged, so the guest sends it again.
- The vsock preload takes LD_PRELOAD and SC_* out of the environment once read, so programs PHP
  starts don't inherit them; /tmp in the guest is noexec.
- A stop whose sync reply is only unacknowledged log lines says so, not that the disks didn't
  sync.
- The microVM target gates each suite's n/a count on test/lib/na-expected.txt.
…refix

[xps] showed every suite failing the new gate with "ext-config has no line in
test/lib/na-expected.txt": the file names suites as config, layer3, …, and $0 is
test/ext-config.sh.
@samifouad

Copy link
Copy Markdown
Member Author

Review fixes (#86 review), in 6411aec (and 5c552be)

Finding Fix Test (fails with the fix reverted)
1. HIGH: any guest process could forge or suppress the log. One Inbox was shared by every connection: L 18446744073709551615 hid every later init line, X/jumps forged launcher loss lines, and 4 idle connections shut the init out Per-VM log key. The launcher makes a 32-byte key per VM (kept beside a snapshot, whose init holds it) and gives it to the init over the CID-checked control port (logkey, alone on its connection, never echoed). A log connection must present it first, within 1 s; others are refused and counted in a launcher line, said at most once a second.
No squatting. At most 16 connections wait for their key; a new one replaces the oldest. Keyed connections have their own 2 places.
Bounded numbering. A jump larger than the guest's buffer can make (MAX_JUMP) closes the stream.
Backoff. The init's reconnects back off to 1 s while no connection gets an acknowledgement.
Less to inherit. The vsock preload takes LD_PRELOAD/SC_* out of the environment once read, and the guest's /tmp is mounted noexec
a_hostile_connection_can_neither_hide_the_inits_lines_nor_write_the_launchers: squatters (3×16), a forged L u64::MAX, X frames and a forged launcher-looking line, with no key and with a wrong key. All 300 init lines arrive once, nothing forged reaches the log, and only counted refusal lines are the launcher's. With the key check disabled: FAIL a refused connection gets no acknowledgement.
ext-microvm.sh: neither PHP nor a program it starts sees LD_PRELOAD or SC_VSOCK_PORTS, and /tmp is noexec in the guest (the same script runs through /bin/sh, so it is noexec that stops it)
2. A line was acknowledged after a failed write serve acknowledges only what emit wrote. A failed write stops the connection, rolls the inbox back and acknowledges nothing, so the guest keeps the line and sends it again. The launcher says so on stderr a_line_the_log_could_not_take_is_never_acknowledged. Reverted: left: Ok(()) right: Err(Log("disk full"))
8a. A log-drain failure at stop read as "did not sync its disks" stop_sync_failed: a reply that is only unacknowledged log lines reads the guest synced its disks before the stop, but N guest log lines lost (…) a_stop_sync_reply_says_which_part_failed
9. n/a counts weren't gated (#88's harness) na_summary checks each suite's count on the microVM target against test/lib/na-expected.txt and counts a mismatch as a failed check Shown with a deliberate mismatch on [xps] (below)

Finding 9 belongs to #88's harness. I put it on this branch, the lowest one I created, because AGENTS.md says not to touch a branch I didn't create. It reaches #88 when this PR merges into it.

Evidence

  • Rust gate (docker build --target check: clippy -D warnings + cargo test, whole workspace, on this branch's tree, in the iMac's Colima): the new tests pass.
    • One test fails there, the_site_key_changes_with_the_tree_but_not_with_uploads. It is older than this stack: it rewrites a file with the same size inside one mtime tick, and the iMac's VM filesystem keeps coarse timestamps. It passes on [xps] and on GitHub.
  • Revert proofs (unit):
    • with the key check disabled: a_hostile_connection_… FAILED (a refused connection gets no acknowledgement);
    • with the ack-on-failed-write behaviour back: a_line_the_log_could_not_take_is_never_acknowledged FAILED (left: Ok(()) right: Err(Log("disk full"))).
  • End to end: this branch's code is all in Layer 1 microVM: a securecore.json edit reaches the running guest; per-site php.ini debug settings (#90) #95's head (33f4229), whose gates are below.
  • Revert proof for the n/a gate (finding 9) ([xps]): with config's expected count set one lower, nagate exit=1 … FAIL n/a count: 5 check(s) n/a on the microVM target, test/lib/na-expected.txt says 4.
  • A bug this run found in the gate: it first looked suites up as ext-config. Fixed in 5c552be.

-claude

… noexec site disks, careful config reads (#86 verification review)

- K2: every restore takes the log with a fresh key, given to the restored init at once; the
  snapshot's key is only refused (quietly, counted apart).
- K4: the init's lines are their own frame kind (I) and are logged as guest-init:, which php-fpm's
  lines (always guest:) can't produce; they may use the last 256 KiB of the guest's buffer, which
  php-fpm's never take.
- N1: control characters (the tab apart) and U+0085/U+2028/U+2029 in guest lines are escaped.
- K3: a log connection has one second in all to present its key (a byte at a time is still cut
  then); making room among the waiting ones spares a connection whose key line is already in.
- N3: the refusals not yet said, and the old key's, are said when the VM goes.
- K5: the guest mounts the site disk, the writable disks and the site's tmpfs folders noexec
  (nosuid, nodev already).
- S1: the launcher reads securecore.json at start with sitefile::load: every path component
  without a link (openat2 RESOLVE_NO_SYMLINKS), as the owner of the folder that holds it,
  regular files only (O_NONBLOCK), at most 128 KiB, and errors never quote the file
  (LoadError::without_contents: line, column and kind for a JSON or schema error). The config
  crate's load_bytes parses what the caller read. This applies to gelada too.
- docs/layer1-microvm.md says what the log guarantees: against non-root guest code, not against
  root in the guest.
GitHub's run of #86's stress found 11 of 240 requests failing right after a SIGUSR2 reload: the
guest answered the reload once the new master was started, the launcher logged "reloaded",
and requests met no listener yet. The guest now answers a reload only once 127.0.0.1:9000
takes connections (10 s at most), as it already did for a start message's restart.
@samifouad

Copy link
Copy Markdown
Member Author

Round-3 verification fixes for this PR (K2, S1, K4, K5, N1, K3, N3, the docs and a reload fix) are tabled here: #95 (comment)

-claude

…othing below it may be one

Refusing a link anywhere in securecore.json's path broke ordinary deployments
(/var/www/current -> releases/<n>, /var/www/html -> /srv/site). The folder holding the file
(the root the operator names with --config) is now resolved once with canonicalize, as the
launcher; the file is opened from an fd on it with openat2 RESOLVE_BENEATH |
RESOLVE_NO_SYMLINKS, so below the root no link (the file, a folder on the way) and no .. is
followed. The same for the watcher, --guest-ini and the guest's configsha (read_beneath).
@samifouad
samifouad changed the base branch from claude/microvm-suites-72 to main November 12, 2026 00:41
@samifouad
samifouad merged commit 6da076f into main Nov 12, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Layer 1 microVM: lines the guest's php-fpm logs go missing from the launcher's log

1 participant