Repository navigation
Layer 1 microVM: every guest log line reaches the launcher, numbered and acknowledged (#86) - #91
Merged
Merged
Conversation
…ged (#86) The launcher shut its side of each log connection for writing. Firecracker turns a host's half-close into a full close of the connection and drops whatever the guest sends after it, while the guest's write() has already returned: lines vanished at random, and pump_log's reconnects and report()'s connection per line filled LOG_SLOTS, so more were refused. The guest now numbers every line (php-fpm's output and the init's own) and holds it until the host acknowledges it, on one connection, sending again what wasn't acknowledged after a broken or stalled connection or a restore. The host logs each frame once and acknowledges it. A full guest buffer, a gap in the numbers or lines unacknowledged at a stop become the launcher's own counted line. PROTOCOL 5. ext-pr.yml runs all eight ext-* suites on the microVM target.
This was referenced Nov 11, 2026
…owledged only once written (#86 review) Any process in the guest could connect to host port 1514 and forge or suppress the log: one Inbox was shared by every connection, so a frame numbered u64::MAX made every later init line look already logged, and X frames or jumps wrote launcher-prefixed loss lines. - The launcher makes a log key per VM (kept beside a snapshot, whose init holds it) and gives it to the init over the control port, which only the host can reach. A log connection must present it first, within a second; others are refused and counted in a launcher line at most once a second. At most 16 connections wait for their key (the oldest goes), and keyed ones have their own places, so nothing can squat the port. - A jump larger than the guest's buffer can make closes the stream. - The init's reconnects back off (to 1 s) while no connection gets an acknowledgement. - A line is acknowledged only once it is written: a failed write stops the connection and leaves the line unlogged, so the guest sends it again. - The vsock preload takes LD_PRELOAD and SC_* out of the environment once read, so programs PHP starts don't inherit them; /tmp in the guest is noexec. - A stop whose sync reply is only unacknowledged log lines says so, not that the disks didn't sync. - The microVM target gates each suite's n/a count on test/lib/na-expected.txt.
…refix [xps] showed every suite failing the new gate with "ext-config has no line in test/lib/na-expected.txt": the file names suites as config, layer3, …, and $0 is test/ext-config.sh.
Member
Author
Review fixes (#86 review), in 6411aec (and 5c552be)
Finding 9 belongs to #88's harness. I put it on this branch, the lowest one I created, because AGENTS.md says not to touch a branch I didn't create. It reaches #88 when this PR merges into it. Evidence
-claude |
… noexec site disks, careful config reads (#86 verification review) - K2: every restore takes the log with a fresh key, given to the restored init at once; the snapshot's key is only refused (quietly, counted apart). - K4: the init's lines are their own frame kind (I) and are logged as guest-init:, which php-fpm's lines (always guest:) can't produce; they may use the last 256 KiB of the guest's buffer, which php-fpm's never take. - N1: control characters (the tab apart) and U+0085/U+2028/U+2029 in guest lines are escaped. - K3: a log connection has one second in all to present its key (a byte at a time is still cut then); making room among the waiting ones spares a connection whose key line is already in. - N3: the refusals not yet said, and the old key's, are said when the VM goes. - K5: the guest mounts the site disk, the writable disks and the site's tmpfs folders noexec (nosuid, nodev already). - S1: the launcher reads securecore.json at start with sitefile::load: every path component without a link (openat2 RESOLVE_NO_SYMLINKS), as the owner of the folder that holds it, regular files only (O_NONBLOCK), at most 128 KiB, and errors never quote the file (LoadError::without_contents: line, column and kind for a JSON or schema error). The config crate's load_bytes parses what the caller read. This applies to gelada too. - docs/layer1-microvm.md says what the log guarantees: against non-root guest code, not against root in the guest.
…ever the file's token
…eadline (K3 revert proof)
GitHub's run of #86's stress found 11 of 240 requests failing right after a SIGUSR2 reload: the guest answered the reload once the new master was started, the launcher logged "reloaded", and requests met no listener yet. The guest now answers a reload only once 127.0.0.1:9000 takes connections (10 s at most), as it already did for a start message's restart.
Member
Author
|
Round-3 verification fixes for this PR (K2, S1, K4, K5, N1, K3, N3, the docs and a reload fix) are tabled here: #95 (comment) -claude |
…othing below it may be one Refusing a link anywhere in securecore.json's path broke ordinary deployments (/var/www/current -> releases/<n>, /var/www/html -> /srv/site). The folder holding the file (the root the operator names with --config) is now resolved once with canonicalize, as the launcher; the file is opened from an fd on it with openat2 RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS, so below the root no link (the file, a folder on the way) and no .. is followed. The same for the watcher, --guest-ini and the guest's configsha (read_beneath).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #86: no guest log line is lost without the launcher saying so. Stacked on #88 (this PR targets
claude/microvm-suites-72); with it, #88's microVM-target CI step runs all eight ext-* suites.Root cause
The launcher's log listener (
vm.rswire) calledshutdown(Write)on every guest log connection, to keep it read-only. Firecracker's vsock (csm/connection.rs) reads that as EOF on the host stream. It moves the connection toLocalClosed, sends the guestSHUTDOWN(RCV|SEND), and from then on drops every data packet the guest sends on it (dropping invalid TX pkt).The guest's
write()had already returned by then, sopump_logthought those bytes were delivered. Which lines landed in that window was a race, so a different few went missing each run, and more often on slower runners. Three things made it worse:pump_logreconnected after eachEPIPE.report()opened a connection per line.LOG_SLOTS(4), and a connection with no free slot was dropped without a word.A restore also resets the guest's vsock connections, which loses anything still in flight.
What changed
ext/fpm/src/guestlog.rs: one numbered, acknowledged stream.Outboxandsend) numbers every line: php-fpm's output and the init's ownreport()lines. It holds each line, up to 4 MiB, until the host acknowledges it, and sends on one connection. After a broken connection, a stall (2 s with nothing acknowledged) or a restore, it reconnects and sends every unacknowledged line again. A line that doesn't fit is counted, and the count goes out in order as anX <seq> <n>frame.Inboxandserve) logs each frame once (a resent frame is skipped by its number), marks itguest:as before, and acknowledges it (A <seq>). It never shuts the connection for writing.securecore-fpm: N guest log lines lost (…).sync(sent before every stop) waits up to 1 s for the log to drain. Lines still unacknowledged then come back in the reply asN guest log lines lost (not acknowledged by the host before the stop), and the launcher logs that.LOG_SLOTSstill caps connections, but a refused connection is now logged, and the guest resends its lines.PROTOCOLis 5. The log port's format changed, and snapshots of a protocol-4 init are never restored.ext-pr.yml's microVM-target step now runs config, attribution, layer3, layer1, hosts, enforce, db and vault.Tests
Unit (
guestlog::tests):every_line_arrives_once_through_connections_that_die_or_stall: 4 writers × 500 lines through a host whose first 4 connections either close mid-stream or swallow data without acknowledging it. Every line arrives once, in order.End to end:
test/ext-microvm.shsection 3d (alsotest/ext-microvm.sh logalone). It runs on a restored guest:error_log(6000 lines);It checks that every line arrives once, all 20 refusals arrive, and nothing is counted lost.
Evidence ([xps], each run as
flock … sudo -E env PATH=$PATH TMPDIR=$PWD/.tmp …, logged to a file, exit code checked; fresh projectssc86-*andsc86m-*)opcache: the log explains the refusal). It is fixed in the PR stacked on this one, so Layer 1 microVM: the ext-* suites run on a microVM target (SC_TARGET=microvm) #88's CI step goes green with both.test/ext-microvm.sh(with the stress):exit=0, 204 ok, 0 FAIL. The stress alone (test/ext-microvm.sh log) passes 8/8:all 6000 numbered lines are in the launcher's log,and each once,all 20 sync refusals …,no line was lost or counted as lost.no Firecracker left running (got '3')came from my ownSC_TARGET=microvmsuite launchers still running on the host; the check counts every Firecracker on the host. After stopping them it passes.-D warnings(linux target): clean.Revert proof
the host acknowledged every line: 2000(2000 lines never acknowledged).ext-microvm.shand probe) loses lines silently, while 240 of 240 requests answered 200:killed by signal 9line and ext-hosts' sixlayer2 enforce … net …lines.Found on the way (not changed here)
A guest reload (SIGUSR2/HUP) stops php-fpm before it starts the new one. Requests that arrive in that gap fail: 13 of 240 got a non-200 in a first run of the stress, with the reload in the middle of a burst. The stress now reloads between bursts. A reload with no gap (start the new master, then quit the old one) would be its own issue.
closes #86
-claude