Skip to content

layer 1: auto is the microVM where KVM exists (#53) - #100

Merged
samifouad merged 6 commits into
mainfrom
claude/auto-microvm-53
Nov 14, 2026
Merged

samifouad merged 6 commits into
mainfrom
claude/auto-microvm-53

Conversation

@samifouad

@samifouad samifouad commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Part of #53. On 10 Nov Sami decided the microVM is layer 1's default. This PR flips the default: AUTO_SELECTS_MICROVM = true, so auto is the microVM wherever vm::available passes and gelada otherwise, with one line saying why. The installer's upgrade note, the docs and the tests are part of it too; the earlier description is in this PR's history.

It was held after review. Since then:

The held review, item by item

# review item status test that covers it
1 failures after the choice exit 125 under auto (the site-disk copy as the pool user, its space, the vault's env via the pool's env[], leftover state) #102 moved them before the choice, and this push fixes the fallback. Every check now runs in prepare_all, before auto commits, and #102 added Outcome::Fallback for a VM that fails before it serves. But gelada's restrict_self refuses a process with other threads. A prepared or started microVM has some (systemd's EXTEND_TIMEOUT thread, the VM's sync and log). So every auto fallback under systemd exited 125, no-KVM hosts included, and so did every after-choice fallback. Now the launcher execs itself again, same pid, with --gelada-fallback <line> (the launcher's own flag), and applies gelada there. Only explicit microvm exits 125. test/ext-microvm.sh fallback (new, also run in the full suite): auto falls back with the reason and gelada's php-fpm runs (NoNewPrivs: 1 Seccomp_filters: 2, exit 0) for: a root-only file, a missing env variable, an orphaned uploads disk, no room, a bare 3306 the guest can't reach (a name, an IP), a bare-port pool, no KVM under NOTIFY_SOCKET, and a VM that fails before serving (a busy --listen port); microvm exits 125 on the same failure; no Firecracker is left over
2 the guest ignores the host's PHP setup already fixed by #102: the guest is the host's own PHP (capture, checksums, the unpackaged record) #102's test/ext-capture.sh, and ext-microvm's capture checks (captured the host's PHP 8.3, checked against dpkg, the manifest)
3 FastCGI moved to 127.0.0.1:<port>, so a stock nginx unix:/run/php/…sock gets 502 #102's per-pool listen fixes the unix socket; this push fixes the bare port. A pool on a unix socket is served at that socket. A pool on a bare port (listen = 9000, which php-fpm binds on every address, while the microVM binds 127.0.0.1) with no --listen now keeps auto on gelada, so a remote nginx never gets 502s ext-microvm: auto, a pool on a unix socket: the microVM, listening at the pool's own socket, php-fpm in the guest answers there; auto, a pool on a bare port and no --listen: gelada; unit test auto_stays_on_gelada_for_a_pool_listening_everywhere
4 a bare 3306 or localhost socket loses its database, and the upgrade test sidestepped it #102 resolves DB_HOST among the approved targets; this push closes the rest. (a) Under auto, a bare 3306 that leaves the guest no database (another host or container, a private IP, an unreadable DB_HOST, with no --db-address/--db-socket approving it) stays on gelada, which reaches it, and says how to approve it. (b) An approved address now works with the site's own DB_HOST: before, the guest's connect hook redirected only 127.0.0.1:3306, so --db-address db:3306 with DB_HOST=db, or 127.0.0.1:33060, never reached the forward. Now sc.dbalias=<host>:<port> lets the hook send that name or IPv4 address and port to the forward. (c) The upgrade test no longer sidesteps: 7b uses ext/examples/wordpress.json exactly as it ships (a bare 3306) and the site's WORDPRESS_DB_HOST=db test/release-check.sh 7b: as the site is, auto stays on gelada (DB_HOST is db, which the guest can't reach by name) and WordPress keeps its database (front page 200, uploads unchanged); with --db-address db:3306, it is the microVM and the front page is 200 through the guest at DB_HOST=db. ext-microvm auto, a bare 3306 and DB_HOST …: gelada (2 cases). Unit tests auto_stays_on_gelada_rather_than_lose_the_database and the_sites_db_host_reaches_the_forward_in_the_guest
5 the space check ignores the site disk and the sum of the folders, and could fill /var/lib beside MySQL #102 added the sum and the margin; this push counts growth. #102's space_ok sums the site disk, the layers, the writable disks and the snapshot, and keeps a twentieth of the filesystem (at least 1 GiB) free. The writable disks are sparse, so this push counts each as it may fill (disk_room: a new one up to its size, an existing one up to its size less what it holds). auto falls back instead of filling ext-microvm auto without room for the whole start: gelada, counting the writable disks as they may fill, and the margin it keeps (a 1200 MiB state tmpfs); unit tests a_writable_disk_counts_what_it_may_still_take and #102's a_start_keeps_a_margin_free
6 a first start on a big site can exceed systemd's TimeoutStartSec and loop already fixed by #102: under NOTIFY_SOCKET the launcher sends EXTEND_TIMEOUT_USEC=30000000 every 10 s until it serves, then READY=1. The boot itself is bounded (60 s wait_ready, 60 s API calls, 120 s warm-up read; several pools: --pools-ready-within, 300 s). #102 had no revert proof for this; one is below ext-microvm and EXTEND_TIMEOUT_USEC=30000000 before it (READY=1)

Docs: ext/README.md's fallback table now lists every reason, including the new ones. Its stale sentence "a microVM auto picked that then fails to boot exits 125" is corrected. docs/layer1-microvm.md and README.md are updated, and so is the installer's upgrade note: a bare 3306 is described separately from the egress ports, along with the bare-port pool rule.

The re-review of cce0ab7 (HOLD), item by item, in 7140260

# item fix test that fails with the fix reverted
1 HIGH: the guest's size ignored the pool size_guest: memory is pm.max_children × memory_limit (the pool's php_admin_value/php_value, else PHP's ini, else 128M) + 256 MiB, at least 512 MiB and at most ¾ of the host's; one vCPU per 4 workers, 2 to the host's; --mem/--vcpus win. Under auto, a pool that can't fit, or has memory_limit = -1, stays on gelada. Prepared and Machine carry the size; the snapshot and space check use it ext-microvm auto, a pool too big for any guest here: gelada (400 × 512M) and the guest is sized from the pool: 5 × 128 MiB + 256 MiB, with cgroup memory.max = (896+256) MiB; unit test the_guest_is_sized_from_the_pool_and_the_host (R6)
2 HIGH: only 3306 was guarded auto_port_gap: any bare port other than 3306 (which has its own rule) and 53/80/443 keeps auto on gelada, naming the ports and the socket/address fix ext-microvm auto, a Redis on the loopback by a bare 6379: gelada, naming the port and and the fix (a listener on 127.0.0.1:6379); unit test auto_stays_on_gelada_for_a_bare_port_with_nowhere_said (R7)
3 MEDIUM: a hung start never timed out --start-within (600 s): the extensions stop at the deadline; tool_status_until kills mke2fs, e2fsck, resize2fs and the PHP layer's mke2fs at it; start_deadline is checked after the capture, before the space check, around the site disk, and before the boot ext-microvm auto, a start past --start-within: gelada / because of its deadline (--start-within 1, under NOTIFY_SOCKET); unit test a_disk_tool_never_outlives_the_starts_deadline (sleep 30 stopped at 300 ms) (R8)
4 MEDIUM: the space check raced other sites crate::space: one file (--space-ledger, default /var/lib/securecore/space.json) under flock. A start checks and records its needs while it builds, then only its writable disks' growth; other living starts on the same st_dev count as taken (each entry holds a pid and its start time, and a gone process counts for nothing); a failed or stopped start takes its entry out unit tests two_starts_at_once_never_both_pass_on_the_same_space (two threads, 600 each of 1000) and what_other_starts_keep_counts_as_taken; ext-microvm auto, another start keeping the space: gelada (a living process's 1 TiB entry), a gone start's space is free again, the record keeps nothing once those starts are gone, while it runs, the shared record keeps its disks' growth / and once it stops, nothing (R9)
5 LOW: relaunch edge cases (a) the relaunch and the pools exec /proc/self/exe; (b) a config now gelada/off runs as it says, only microvm refuses ext-microvm auto, its binary removed during the start: gelada (the copy is removed 0.3 s in); the config now gelada at the relaunch: gelada, as it says; the config now microvm at the relaunch: exit 125 (R10)

The installer's note, ext/README.md's fallback table and docs/layer1-microvm.md describe all five.

The second re-review of 7140260 (HOLD), item by item, in bd858fa

# item fix test that fails with the fix reverted
1 HIGH: guest memory isn't budgeted across sites the shared record (crate::space) holds each running guest's memory; under auto, all guests on the host stay within its memory less max(¼, 2 GiB) (memory_ok), else gelada. No balloon yet ext-microvm auto, the other guests holding the host's memory budget: gelada (a living process's entry holding the whole budget, on another filesystem); unit tests guests_together_stay_within_the_hosts_memory (two sites that each fit alone and not together) and guests_stay_within_the_hosts_memory_budget (R11)
2 HIGH: the snapshot left the reservation before it was written keep(writable + snapshot) at the end of prepare, and keep(writable) only after bring_up ext-microvm until the snapshot is written, the record keeps its 896 MiB besides the disks' growth (sampled between the choice and made the snapshot: 3087007744 bytes then, 2147483648 after) (R12)
3 MEDIUM: the deadline missed the in-process steps a watchdog thread (main.rs) runs gelada at --start-within whatever step is running (exec from that thread), and records it in <state>/deadline-fallback.json. Later starts go straight to gelada, saying why, until the config or the app root changes. The extensions go on until the start serves or ends. microvm and a pool's own launcher exit 125 ext-microvm auto, a start past --start-within: gelada, at the deadline (--start-within 1, during the capture's hashing), the next start after one past its deadline: straight to gelada, with nothing captured or built, a changed config tries the microVM again (R13)
4 LOW: the ledger free space is measured inside the lock; Entry.boot (boot_id), and an earlier boot's entry counts for nothing. Growth isn't shrunk as disks fill (documented: it errs toward gelada) ext-microvm an entry of an earlier boot keeps nothing (same pid and start time); unit test what_other_starts_keep_counts_as_taken (R14)
5 LOW: 587/465 a bare 25/465/587 is a public relay's by name (layer 2 polices it) unless a relay listens on this host's loopback (listens_locally) ext-microvm auto, a bare 587 with no relay on this host: … the microVM and with a relay on this host's loopback: gelada; unit test auto_stays_on_gelada_for_a_bare_port_with_nowhere_said (R15)

Evidence ([xps], under the build lock, each gate logged to a file, $? checked)

On bd858fa, [xps]:

  • docker build --target check: clippy -D warnings clean, cargo test all pass.
  • ext-build, guest-build: rc=0.
  • test/ext-microvm.sh fallback: 86 ok, 0 failed.
  • the full test/ext-microvm.sh: 423 ok, 0 failed.
  • Left to CI: the docker suites, the release check and the promote check.

On 7140260, [xps]:

  • docker build --target check: clippy -D warnings clean; cargo test 126 passed, 1 failed. The failure was a wrong expectation in the new what_other_starts_keep_counts_as_taken, fixed in this commit. The rerun is part of build 2's job.
  • scripts/ext-build.sh, scripts/guest-build.sh: rc=0.
  • test/ext-microvm.sh fallback: 70 ok, 0 failed.
  • the full test/ext-microvm.sh: 407 ok, 0 failed.
  • Left to CI: the docker-target ext-* suites, the release check and the promote check.

On cce0ab7 (before the re-review fixes):

gate result
shellcheck 0.11.0 -s sh -S info on release.yml's file list clean
docker build --target check (clippy -D warnings + cargo test, the whole workspace) rc=0; clippy clean; all tests pass, including the 4 new ones (auto_stays_on_gelada_rather_than_lose_the_database, auto_stays_on_gelada_for_a_pool_listening_everywhere, a_writable_disk_counts_what_it_may_still_take, the_sites_db_host_reaches_the_forward_in_the_guest)
scripts/ext-build.sh 8.3 nts, scripts/guest-build.sh, the kernel (bench/microvm/kernel.sh) rc=0
test/ext-microvm.sh fallback rc=0, 41 ok, 0 failed
test/ext-microvm.sh (full) rc=0, 377 ok, 0 failed
the 8 ext-* suites, SC_TARGET=microvm 8/8 rc=0
test/release-check.sh 8.3 nts, test/promote-check.sh left to CI: [xps] has no zip (scripts/package.sh), so both stopped at the release layout before any check ran
the 8 ext-* suites, SC_TARGET=docker left to CI: 7/8 pass; ext-hosts stopped in its WordPress setup ("WordPress is already installed": an old stack on [xps], not the launcher; it passes on the microVM target)

The first run found the item-1 bug: auto, after the choice failed with exit 125 (restrict_self … this process has 3 other thread(s)). It passes since the re-exec.

Revert proofs

Build 1 ([xps], one build with five reverts, cce0ab7's code):

revert what fails
R1: the re-exec removed (#102's fallback in the same process) auto, after the choice: the launcher execs php-fpm and … php-fpm runs under gelada (exit 125: restrict_self … other thread(s)); in fallback and in the full ext-microvm
R2: the auto DB and listen gaps removed auto, a bare 3306 and DB_HOST sc-no-such-db.invalid / 10.255.255.1:3306: gelada, and how to approve it, and gelada's php-fpm ran (both cases); auto, a pool on a bare port and no --listen: gelada (+2); release-check 7b the site as it is: WordPress keeps its database (egress refused db:3306), auto stays on gelada …, says how to approve it, nothing of the microVM was built, PHP runs on the host (gelada) (got guest_init 1)
R3: growth counted for new disks only not caught end to end (the 1200 MiB tmpfs falls back either way). It now has a unit test, a_writable_disk_counts_what_it_may_still_take (writable_room, an existing disk's growth), which runs in build 2
R4: the guest's DB alias removed (name and connect) release-check 7b upgrade, approved: WordPress in the guest reaches its database at its own DB_HOST, db (got 500)
R5: extend_start_timeout removed ext-microvm and EXTEND_TIMEOUT_USEC=30000000 before it (got no)

Totals under revert: fallback 16 failed (28 ok); the full ext-microvm 17 failed; the release check 6 failed (131 ok). R5 hides R1's preparation-time relaunch (with no extension thread, there is nothing to re-exec away from); R1's after-choice case is still caught.

Build 2 (R3, R6–R10, on 7140260):

revert what fails
R3: only new disks' growth counted unit test a_writable_disk_counts_what_it_may_still_take (writable_room)
R6: a fixed 512 MiB / 2 vCPU guest ext-microvm auto, a pool too big for any guest here: gelada and its gelada checks; the start served (stopped at the 900 s bound)
R7: auto_port_gap off ext-microvm auto, a Redis on the loopback by a bare 6379: layer 1 isolation auto: microvm, serving with 6379 as egress (the run was stopped there)
R8: the deadline off not reached in this run: superseded by the watchdog (R13, build 3)
R9: the record's others not counted unit tests what_other_starts_keep_counts_as_taken and two_starts_at_once_never_both_pass_on_the_same_space
R10: current_exe, a flipped config refused not reached in this run; in build 3

Build 3 (R10–R15, on bd858fa; fallback: 21 failed, 67 ok):

revert what fails
R10: current_exe, a flipped config refused auto, its binary removed during the start: gelada (+2); the config now gelada at the relaunch: gelada, as it says (+2)
R11: no memory budget auto, the other guests holding the host's memory budget: gelada, naming the budget … (+2); it served until the 240 s bound
R12: the snapshot's bytes dropped at the end of prepare not a clean proof here: the reverted R11/R13 cases served first and made the snapshot, so the unix-socket start restored it (made the snapshot and the sampled check both failed for that reason)
R13: no watchdog, no deadline record auto, a start past --start-within: gelada, at the deadline (it served until the bound); the next start after one past its deadline: straight to gelada (+2)
R14: the boot id ignored an entry of an earlier boot keeps nothing: the start gets past the space check
R15: mail ports always kept on gelada auto, a bare 587 with no relay on this host: … the microVM, and 587 is no reason to stay on gelada

For Sami

  • A site that auto keeps on gelada stays there until the operator acts: it approves the database (--db-address/--db-socket, or an address/socket in the config), or gives the pool an address (listen = 127.0.0.1:9000, or --listen). The launcher's line says which. The other way to make auto safe would be to move such a site and accept a site without its database; this PR doesn't do that.
  • A site owner who can edit wp-config.php can therefore keep their site on gelada (point DB_HOST somewhere the guest can't reach). gelada is still a sandbox and the line is logged; isolation: "microvm" doesn't allow it.
  • An IPv6 DB_HOST gets no guest alias (the connect hook handles IPv4 and names).
  • The ZTS images have no php-fpm, so release-check 7/7b can't capture one there. That predates this PR; this PR's CI runs 8.3 NTS.

-claude

…one line why (#53)

AUTO_SELECTS_MICROVM is on and pinned on. The launcher checks the host first
(no KVM is the first reason, and writes nothing), and a fallback reason is
always one line. install-ext.sh says, on a KVM host with the guest, that an
auto site moves into the microVM at its next start, what happens to uploads,
how to opt out, and which connect ports reach only public hosts there.
ext-microvm.sh checks auto as the microVM and each fallback reason;
release-check.sh checks an upgrade over a site with uploads (7b). Docs.
# Conflicts:
#	ext/fpm/src/vm.rs
#	test/ext-microvm.sh
…rt (#100 review)

- auto stays on gelada when a bare 3306 would leave the guest without the
  database the site's DB_HOST names, or a pool listens on a bare port with
  no --listen (php-fpm: every address; the microVM: 127.0.0.1 only)
- the site's own DB_HOST reaches an approved forward in the guest
  (sc.dbalias, the connect hook), so --db-address db:3306 works unchanged
- the space check counts each writable disk as it may fill
- after a fallback past the microVM's preparation (threads started:
  systemd's timeout extensions, the VM's), the launcher execs itself to
  apply gelada in a single-threaded process; before, every auto fallback
  under systemd exited 125
- tests: ext-microvm's auto fallbacks (fallback mode), release-check 7b
  runs the site as it is (example config, DB_HOST=db), unit tests
@samifouad

Copy link
Copy Markdown
Member Author

Review of cce0ab7: HOLD

I checked the six held items against the code. Items 1, 2, 3, 5 and 6 are fixed as the PR says. Item 4 is fixed only for port 3306. The relaunch is sound: it's the same pid, the state flock is CLOEXEC, signals are unblocked before the exec, the VM is dropped first, and the pools wait at most 10 s and then SIGKILL. The new tests check real effects: the stand-in's exit=0, NoNewPrivs/Seccomp, one fallback line, and the 7b front page 200 at DB_HOST=db. Five gaps are left for existing installs that upgrade:

  1. HIGH: the guest's size ignores the host pool. ext/fpm/src/main.rs:93-94 defaults to 512 MiB and 2 vCPUs, and the cgroup caps memory at mem+256 MiB. Nothing reads the pool's pm.max_children or memory_limit, or the host's cores. docs/layer1-microvm.md:161 measured only the default pool of 5 children. Failure: a WooCommerce pool that worked on gelada (pm.max_children = 30, 256M) moves at its next restart. Under load the guest OOM-kills workers (502s). If the master dies, the guest stops the VM (guest.rs:954), the launcher exits 125 after serving, and Debian's unit has no Restart=, so the site stays down. Fix: under auto, fall back when max_children × memory_limit (or a measured RSS) exceeds --mem, or derive --mem/--vcpus from the pool. Add a test where a large pool stays on gelada.

  2. HIGH: item 4 is only fixed for 3306. vm.rs:1360 and auto_db_gap (vm.rs:1167) only look at 3306. Any other bare connect port moves anyway, and the installer only prints a note (ext/install.sh:479-485) that nobody reads at restart. Failure: Redis object cache on 127.0.0.1:6379, memcached 11211, Postgres 5432 (Laravel), or an SMTP relay on localhost:25 work on gelada. In the guest they reach only public names, so the site loses its cache or database and email stops silently. Fix: apply the same auto gap to every bare port. If its target is loopback, a private address or unknown, stay on gelada and log a line naming the port and the socket/address fix.

  3. MEDIUM: a hung start never times out. extend_start_timeout (vm.rs:1257) has no overall cap, and run_tool (vm.rs:1572) runs mke2fs, e2fsck and resize2fs with no timeout. Failure: the first start of a site whose uploads are on a stalled NFS mount, or are just very large, renews EXTEND_TIMEOUT_USEC every 10 s for as long as the copy runs. The site is down for that whole time, and neither systemd's timeout nor auto's fallback ever fires. On gelada the same site starts at once. Fix: cap the total extension (e.g. 10 min, then stop extending and fall back), and give run_tool a deadline the way capture::run_in_root_out has one.

  4. MEDIUM: the space check is per site and races other sites. space_ok (vm.rs:1297) compares one start's needs, including the sparse disks' future growth (disk_room), with statvfs free space now. Failure: at boot, N sites' php-fpm units start in parallel. Each sees the same free space and passes, so together they reserve N times what is there. The sparse disks then fill /var/lib next to MySQL. Fix: keep a reservation ledger under /var/lib/securecore/vm (flock): reserved = the sum of each site's want - allocated, and subtract it from free space before checking.

  5. LOW: two relaunch edge cases.
    (a) main.rs:186 re-execs current_exe(). If the package was upgraded during the start, that path reads … (deleted), the exec fails, and the launcher exits 125. Fix: exec /proc/self/exe. The same applies to the pools at main.rs:323.
    (b) main.rs:630: if securecore.json is edited to "gelada" between the first run and the relaunch, the launcher exits 125. Fix: accept gelada there and refuse only microvm.

Evidence still missing: the body's revert proofs (R1–R5) say "results will be added here", and they aren't there yet (AGENTS.md §3). The microVM CI job was pending when I checked.

What I ran: I read the code with git diff/git show at cce0ab7. cargo test --locked -p securecore-fpm on macOS: rc=0, 40 passed. The vm.rs tests, including the four new ones, are Linux-only and didn't run here. No KVM suites were run.

-claude

…t, with a start deadline and shared space (#100 re-review)

- the guest's memory and vCPUs come from the pool (pm.max_children x
  memory_limit + 256 MiB, 512 MiB to 3/4 of the host; a vCPU per four
  workers) unless --mem/--vcpus; under auto a pool that can't fit stays on
  gelada
- a bare connect port other than 3306, 53, 80 and 443 keeps auto on gelada
  until a socket or address says where it goes
- --start-within (600 s) caps systemd's extensions; mke2fs, e2fsck and
  resize2fs run against it; each step checks it
- the space check counts what other starts on the filesystem keep, in a
  flock'd shared record (--space-ledger)
- the relaunch execs /proc/self/exe, and runs gelada or off when the config
  changed to them in between; only microvm refuses
@samifouad

Copy link
Copy Markdown
Member Author

Re-review of 7140260: HOLD

I checked the five held items against the code. Items 2 and 5 are fixed. Item 2's auto_port_gap runs only under auto (vm.rs:1508-1524). The example's 53/3306/443 still gets the microVM, so the rule isn't too broad. Item 5 is fixed both ways: /proc/self/exe, and gelada/off are accepted at the relaunch. The ledger can't deadlock: no Locked is ever nested, std opens files CLOEXEC, and Prepared is consumed in run(), so its entry is dropped before any relaunch. An entry left by a crash is ignored via pid + start time. Items 1, 3 and 4 hold as written, but two fixes introduce new ways to take a site down:

  1. HIGH (new, from fix 1): no budget for guest memory across sites. size_guest caps each guest at ¾ of the host's RAM (vm.rs:1303). Nothing adds up the guests on one host. There's no balloon, and memory.max is mem+256 per VM (jail.rs:527), so a guest never returns memory once it has touched it. Failure: a 16 GiB host runs three shops at 30 × 256M. Each fits alone (7936 MiB), so all three move. After a busy day, each guest's page cache and worker peaks are still resident on the host: 3 × ~8 GiB plus mysqld. The host OOM killer then takes the largest process, usually mysqld. On gelada, freed memory went back to the host. Fix: budget memory in the same flock'd ledger (Σ guest mem + 25% for the host ≤ MemTotal, else gelada), and/or a Firecracker balloon with free_page_reporting and deflate_on_oom. Add a test with two sites that each fit alone and don't fit together.
  2. HIGH: the snapshot leaves the reservation before it is written. space.keep(writable) (vm.rs:1682) drops the snapshot's mem_mib bytes. vm.mem is written later, in run() → bring_up → l.snapshot (vm.rs:2971, /snapshot/create). After fix 1 that file is up to ¾ of the host's RAM, not 512 MiB. Failure: at boot, N sites start for the first time after the upgrade. Each passes the check, calls keep(), then writes a 4–8 GiB Full snapshot that no later check counted, and the disk next to MySQL fills. This is item 4 again. Fix: keep writable + snapshot until the snapshot is on disk (or has failed), then keep(writable). The ledger test only asserts bytes > 0 while running, so it can't see this gap.
  3. MEDIUM: the deadline doesn't reach the start's in-process steps. At the deadline the extension thread stops (vm.rs:1401/1405), but the in-process steps never check it: the tree walks site_sha256 (vm.rs:2056), folder_usage (vm.rs:1972) and the capture's hashing. The stop signals are blocked through all of them (main.rs:290). Failure: a site tree on slow or network storage is still being walked at 600 s. About 30 s later systemd's timeout fires, SIGTERM is blocked, SIGKILL comes after TimeoutStopSec, and the unit fails. Debian's unit has no Restart=, so the site stays down, and every restart repeats it because nothing was cached. The old code at least eventually finished, and gelada starts at once. Fix: keep extending until prepare returns, and have a watchdog thread exec the gelada relaunch itself at the deadline (exec from any thread works). Or check the deadline inside the walks. Also record a deadline fallback in the state folder, so the next start goes straight to gelada instead of being down another 10 minutes.
  4. LOW: the ledger. (a) free is measured before taking the lock (vm.rs:1616). Measure it inside the closure. (b) A running VM's growth reservation is never shrunk as its disk fills, so the fill is counted twice over time and other sites' restarts fall back to gelada (safe, but oversized). (c) Entries outlive a reboot, and pid + start time can repeat with a deterministic boot order. Add boot_id to Entry.
  5. LOW: ports 587 and 465. WP Mail SMTP to a public relay keeps a site on gelada, and the only way out is "isolation": "microvm", which loses the fallback. Allow a hostname address, or a "public": true marker, for bare ports.

Tests: the new tests check real effects: the fallback line with its reason, gelada_ran, the ledger's jq counts, and the tool killed at 300 ms. Each would fail if its fix were dead. Nothing covers findings 1–3. Evidence: the body's R6–R10 revert results still say "will be added here" (AGENTS.md §3).

What I ran: I read the code with git diff cce0ab7..7140260 and git show. cargo test --locked in ext/fpm on macOS: rc=0, 40 passed. cargo check --locked --all-targets --target x86_64-unknown-linux-gnu: rc=0 with no warnings. That compiles the Linux-only vm.rs and space.rs tests, but they didn't run here. No KVM suites were run.

-claude

… a deadline watchdog (#100 second re-review)

- the shared record holds each running guest's memory; under auto all
  guests stay within the host's memory less max(25%, 2 GiB)
- the snapshot's bytes stay kept until the boot has written it
- a watchdog runs gelada (the relaunch) at --start-within whatever step the
  start is in, and records it in the state folder: later starts go
  straight to gelada until the config or the app root changes; systemd is
  extended until the start serves or ends
- free space is measured under the record's lock; entries carry the boot id
- a bare 25/465/587 is a public relay's unless one listens on loopback
@samifouad

Copy link
Copy Markdown
Member Author

Third review of bd858fa: SAFE-TO-MERGE (follow-ups below)

I checked all five held items myself, using git diff 7140260..bd858fa. All five are fixed.

  1. Memory across sites is fixed. The ledger now records mem_mib for each running guest. Under auto, memory_ok checks under the lock that the guests add up to no more than MemTotal minus max(¼, 2 GiB) (vm.rs:1536, inside reserve). The entry lives as long as run() holds space.
  2. The snapshot is fixed. The start keeps writable + snapshot and moves to keep(writable) only after bring_up returns. If the start fails, the entry is dropped.
  3. The deadline is fixed. At the deadline, the watchdog thread (main.rs:309) execs gelada from wherever the start is. It records the fallback so the next start goes straight to gelada. A double exec can't happen (execve kills the other threads). A VM can't come up beside gelada either: done is set before run(), and start_deadline() is checked again before bring_up. The extension cap is gone, but the boot is still bounded by wait_ready's timeout.
  4. Free space and boot ID are fixed. free_bytes/fs_size are now measured inside the lock, and an entry from another boot doesn't count as alive.
  5. 587/465 are fixed. A mail port keeps auto on gelada only if something listens on 127.0.0.1 on that port.

Upgrade safety: I found no new path to exit 125, a hang, a restart loop, a 502, a lost database, host OOM or a disk overcommit. Every new check fails toward gelada.

Tests: they assert effects. The suite checks that gelada ran and logs its reason, that two guests which each fit alone don't fit together, that the snapshot bytes are kept until it's written (sampled from the ledger), that an entry from an earlier boot is ignored, and that a changed config tries the microVM again.

Follow-ups, none blocking:

  • MEDIUM: the watchdog's exec skips cleanup (main.rs:309-330). The relaunch keeps the same pid, start time and boot. So the Reservation, which was never dropped, stays "alive" for the whole life of the gelada php-fpm. It holds the full byte total plus up to ¾ of the host's RAM, and other sites' auto starts fall back to gelada because of it.
    The mke2fs already copying the site (the usual slow step) is orphaned, because its kill thread dies with the exec. It keeps reading the slow mount and leaves site-*.ext4.new (vm.rs:2018) in the state folder. Nothing removes it, since every later start goes straight to gelada. That's a file as big as the site, left next to MySQL.
    Fix: before the exec, kill the tool child (keep its pid in a static), remove *.new, and drop this pid's ledger entries (flock with a short timeout). Also drop entries for its own pid on the --gelada-fallback path.
  • MEDIUM: the mail probe races the MTA at boot (vm.rs:1233). php-fpm and postfix/exim start in parallel with no ordering. If the probe runs first, a site whose SMTP is localhost:25 moves, and its mail is lost silently. A relay on a private address (10.x, 172.17.0.1) is missed too.
    Fix: keep 25 always on gelada (a public relay on 25 is rare, and most providers block it). Probe only 465/587, and also count an enabled MTA unit as local.
  • LOW: the deadline record almost never expires (vm.rs:1441). The app root's mtime rarely changes, so a single slow cold boot keeps the site on gelada for good. That's the safe direction. Fix: expire the record after N days or on a launcher version change.
  • LOW: a torn ledger write is possible. Locked::write (space.rs:87) truncates and then writes. If the watchdog's exec lands in between, the ledger is left empty and every reservation is forgotten once. Fix: write to a temp file and rename it under the lock.
  • LOW: the budget is tight on small hosts (vm.rs:1528). A host with 2 GiB or less has a budget of 0, so it never gets the microVM; it would be worth documenting. Ordinary hosts are fine: a default pool (5 × 128M + 256 = 896 MiB) gives 2 guests on 4 GiB and 6 on 8 GiB. The budget also leaves out the cgroup's extra 256 MiB and Firecracker's own memory. Fix: count mem_mib + 256.

Evidence still missing: the revert results R6–R10 in the body still say "will be added here" (AGENTS.md §3).

What I ran: I worked in a worktree at bd858fa. cargo test --locked (ext/fpm, macOS): rc=0, 40 passed. cargo check --locked --all-targets --target x86_64-unknown-linux-gnu: rc=0, 0 warnings. That compiles the Linux-only tests but doesn't run them. No KVM suites were run.

-claude

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant