Repository navigation
layer 1: auto is the microVM where KVM exists (#53) - #100
Conversation
…one line why (#53) AUTO_SELECTS_MICROVM is on and pinned on. The launcher checks the host first (no KVM is the first reason, and writes nothing), and a fallback reason is always one line. install-ext.sh says, on a KVM host with the guest, that an auto site moves into the microVM at its next start, what happens to uploads, how to opt out, and which connect ports reach only public hosts there. ext-microvm.sh checks auto as the microVM and each fallback reason; release-check.sh checks an upgrade over a site with uploads (7b). Docs.
# Conflicts: # ext/fpm/src/vm.rs # test/ext-microvm.sh
…rt (#100 review) - auto stays on gelada when a bare 3306 would leave the guest without the database the site's DB_HOST names, or a pool listens on a bare port with no --listen (php-fpm: every address; the microVM: 127.0.0.1 only) - the site's own DB_HOST reaches an approved forward in the guest (sc.dbalias, the connect hook), so --db-address db:3306 works unchanged - the space check counts each writable disk as it may fill - after a fallback past the microVM's preparation (threads started: systemd's timeout extensions, the VM's), the launcher execs itself to apply gelada in a single-threaded process; before, every auto fallback under systemd exited 125 - tests: ext-microvm's auto fallbacks (fallback mode), release-check 7b runs the site as it is (example config, DB_HOST=db), unit tests
Review of cce0ab7: HOLDI checked the six held items against the code. Items 1, 2, 3, 5 and 6 are fixed as the PR says. Item 4 is fixed only for port 3306. The relaunch is sound: it's the same pid, the state flock is CLOEXEC, signals are unblocked before the exec, the VM is dropped first, and the pools wait at most 10 s and then SIGKILL. The new tests check real effects: the stand-in's
Evidence still missing: the body's revert proofs (R1–R5) say "results will be added here", and they aren't there yet (AGENTS.md §3). The microVM CI job was pending when I checked. What I ran: I read the code with -claude |
…t, with a start deadline and shared space (#100 re-review) - the guest's memory and vCPUs come from the pool (pm.max_children x memory_limit + 256 MiB, 512 MiB to 3/4 of the host; a vCPU per four workers) unless --mem/--vcpus; under auto a pool that can't fit stays on gelada - a bare connect port other than 3306, 53, 80 and 443 keeps auto on gelada until a socket or address says where it goes - --start-within (600 s) caps systemd's extensions; mke2fs, e2fsck and resize2fs run against it; each step checks it - the space check counts what other starts on the filesystem keep, in a flock'd shared record (--space-ledger) - the relaunch execs /proc/self/exe, and runs gelada or off when the config changed to them in between; only microvm refuses
Re-review of 7140260: HOLDI checked the five held items against the code. Items 2 and 5 are fixed. Item 2's
Tests: the new tests check real effects: the fallback line with its reason, What I ran: I read the code with -claude |
… a deadline watchdog (#100 second re-review) - the shared record holds each running guest's memory; under auto all guests stay within the host's memory less max(25%, 2 GiB) - the snapshot's bytes stay kept until the boot has written it - a watchdog runs gelada (the relaunch) at --start-within whatever step the start is in, and records it in the state folder: later starts go straight to gelada until the config or the app root changes; systemd is extended until the start serves or ends - free space is measured under the record's lock; entries carry the boot id - a bare 25/465/587 is a public relay's unless one listens on loopback
Third review of bd858fa: SAFE-TO-MERGE (follow-ups below)I checked all five held items myself, using
Upgrade safety: I found no new path to exit 125, a hang, a restart loop, a 502, a lost database, host OOM or a disk overcommit. Every new check fails toward gelada. Tests: they assert effects. The suite checks that gelada ran and logs its reason, that two guests which each fit alone don't fit together, that the snapshot bytes are kept until it's written (sampled from the ledger), that an entry from an earlier boot is ignored, and that a changed config tries the microVM again. Follow-ups, none blocking:
Evidence still missing: the revert results R6–R10 in the body still say "will be added here" (AGENTS.md §3). What I ran: I worked in a worktree at bd858fa. -claude |
Part of #53. On 10 Nov Sami decided the microVM is layer 1's default. This PR flips the default:
AUTO_SELECTS_MICROVM = true, soautois the microVM wherevervm::availablepasses and gelada otherwise, with one line saying why. The installer's upgrade note, the docs and the tests are part of it too; the earlier description is in this PR's history.It was held after review. Since then:
no KVM: …), and this site's Firecracker is counted by its jail id.The held review, item by item
auto(the site-disk copy as the pool user, its space, the vault's env via the pool'senv[], leftover state)prepare_all, beforeautocommits, and #102 addedOutcome::Fallbackfor a VM that fails before it serves. But gelada'srestrict_selfrefuses a process with other threads. A prepared or started microVM has some (systemd'sEXTEND_TIMEOUTthread, the VM's sync and log). So everyautofallback under systemd exited 125, no-KVM hosts included, and so did every after-choice fallback. Now the launcher execs itself again, same pid, with--gelada-fallback <line>(the launcher's own flag), and applies gelada there. Only explicitmicrovmexits 125.test/ext-microvm.sh fallback(new, also run in the full suite):autofalls back with the reason and gelada's php-fpm runs (NoNewPrivs: 1 Seccomp_filters: 2, exit 0) for: a root-only file, a missing env variable, an orphaned uploads disk, no room, a bare 3306 the guest can't reach (a name, an IP), a bare-port pool, no KVM underNOTIFY_SOCKET, and a VM that fails before serving (a busy--listenport);microvmexits 125 on the same failure; no Firecracker is left overtest/ext-capture.sh, and ext-microvm's capture checks (captured the host's PHP 8.3,checked against dpkg, the manifest)127.0.0.1:<port>, so a stock nginxunix:/run/php/…sockgets 502listen = 9000, which php-fpm binds on every address, while the microVM binds 127.0.0.1) with no--listennow keepsautoon gelada, so a remote nginx never gets 502sauto, a pool on a unix socket: the microVM,listening at the pool's own socket,php-fpm in the guest answers there;auto, a pool on a bare port and no --listen: gelada; unit testauto_stays_on_gelada_for_a_pool_listening_everywhereDB_HOSTamong the approved targets; this push closes the rest. (a) Underauto, a bare 3306 that leaves the guest no database (another host or container, a private IP, an unreadableDB_HOST, with no--db-address/--db-socketapproving it) stays on gelada, which reaches it, and says how to approve it. (b) An approved address now works with the site's ownDB_HOST: before, the guest's connect hook redirected only127.0.0.1:3306, so--db-address db:3306withDB_HOST=db, or127.0.0.1:33060, never reached the forward. Nowsc.dbalias=<host>:<port>lets the hook send that name or IPv4 address and port to the forward. (c) The upgrade test no longer sidesteps: 7b usesext/examples/wordpress.jsonexactly as it ships (a bare 3306) and the site'sWORDPRESS_DB_HOST=dbtest/release-check.sh7b: as the site is,autostays on gelada (DB_HOST is db, which the guest can't reach by name) and WordPress keeps its database (front page 200, uploads unchanged); with--db-address db:3306, it is the microVM and the front page is 200 through the guest atDB_HOST=db. ext-microvmauto, a bare 3306 and DB_HOST …: gelada(2 cases). Unit testsauto_stays_on_gelada_rather_than_lose_the_databaseandthe_sites_db_host_reaches_the_forward_in_the_guest/var/libbeside MySQLspace_oksums the site disk, the layers, the writable disks and the snapshot, and keeps a twentieth of the filesystem (at least 1 GiB) free. The writable disks are sparse, so this push counts each as it may fill (disk_room: a new one up to its size, an existing one up to its size less what it holds).autofalls back instead of fillingauto without room for the whole start: gelada,counting the writable disks as they may fill,and the margin it keeps(a 1200 MiB state tmpfs); unit testsa_writable_disk_counts_what_it_may_still_takeand #102'sa_start_keeps_a_margin_freeTimeoutStartSecand loopNOTIFY_SOCKETthe launcher sendsEXTEND_TIMEOUT_USEC=30000000every 10 s until it serves, thenREADY=1. The boot itself is bounded (60 swait_ready, 60 s API calls, 120 s warm-up read; several pools:--pools-ready-within, 300 s). #102 had no revert proof for this; one is belowand EXTEND_TIMEOUT_USEC=30000000 before it(READY=1)Docs:
ext/README.md's fallback table now lists every reason, including the new ones. Its stale sentence "a microVMautopicked that then fails to boot exits 125" is corrected.docs/layer1-microvm.mdandREADME.mdare updated, and so is the installer's upgrade note: a bare 3306 is described separately from the egress ports, along with the bare-port pool rule.The re-review of cce0ab7 (HOLD), item by item, in 7140260
size_guest: memory ispm.max_children × memory_limit(the pool'sphp_admin_value/php_value, else PHP's ini, else 128M) + 256 MiB, at least 512 MiB and at most ¾ of the host's; one vCPU per 4 workers, 2 to the host's;--mem/--vcpuswin. Underauto, a pool that can't fit, or hasmemory_limit = -1, stays on gelada.PreparedandMachinecarry the size; the snapshot and space check use itauto, a pool too big for any guest here: gelada(400 × 512M) andthe guest is sized from the pool: 5 × 128 MiB + 256 MiB, with cgroupmemory.max= (896+256) MiB; unit testthe_guest_is_sized_from_the_pool_and_the_host(R6)auto_port_gap: any bare port other than 3306 (which has its own rule) and 53/80/443 keepsautoon gelada, naming the ports and thesocket/addressfixauto, a Redis on the loopback by a bare 6379: gelada, naming the portandand the fix(a listener on 127.0.0.1:6379); unit testauto_stays_on_gelada_for_a_bare_port_with_nowhere_said(R7)--start-within(600 s): the extensions stop at the deadline;tool_status_untilkills mke2fs, e2fsck, resize2fs and the PHP layer's mke2fs at it;start_deadlineis checked after the capture, before the space check, around the site disk, and before the bootauto, a start past --start-within: gelada/because of its deadline(--start-within 1, underNOTIFY_SOCKET); unit testa_disk_tool_never_outlives_the_starts_deadline(sleep 30stopped at 300 ms) (R8)crate::space: one file (--space-ledger, default/var/lib/securecore/space.json) underflock. A start checks and records its needs while it builds, then only its writable disks' growth; other living starts on the samest_devcount as taken (each entry holds a pid and its start time, and a gone process counts for nothing); a failed or stopped start takes its entry outtwo_starts_at_once_never_both_pass_on_the_same_space(two threads, 600 each of 1000) andwhat_other_starts_keep_counts_as_taken; ext-microvmauto, another start keeping the space: gelada(a living process's 1 TiB entry),a gone start's space is free again,the record keeps nothing once those starts are gone,while it runs, the shared record keeps its disks' growth/and once it stops, nothing(R9)/proc/self/exe; (b) a config nowgelada/offruns as it says, onlymicrovmrefusesauto, its binary removed during the start: gelada(the copy is removed 0.3 s in);the config now gelada at the relaunch: gelada, as it says;the config now microvm at the relaunch: exit 125(R10)The installer's note,
ext/README.md's fallback table anddocs/layer1-microvm.mddescribe all five.The second re-review of 7140260 (HOLD), item by item, in bd858fa
crate::space) holds each running guest's memory; underauto, all guests on the host stay within its memory lessmax(¼, 2 GiB)(memory_ok), else gelada. No balloon yetauto, the other guests holding the host's memory budget: gelada(a living process's entry holding the whole budget, on another filesystem); unit testsguests_together_stay_within_the_hosts_memory(two sites that each fit alone and not together) andguests_stay_within_the_hosts_memory_budget(R11)keep(writable + snapshot)at the end ofprepare, andkeep(writable)only afterbring_upuntil the snapshot is written, the record keeps its 896 MiB besides the disks' growth(sampled between the choice andmade the snapshot: 3087007744 bytes then, 2147483648 after) (R12)main.rs) runs gelada at--start-withinwhatever step is running (exec from that thread), and records it in<state>/deadline-fallback.json. Later starts go straight to gelada, saying why, until the config or the app root changes. The extensions go on until the start serves or ends.microvmand a pool's own launcher exit 125auto, a start past --start-within: gelada, at the deadline(--start-within 1, during the capture's hashing),the next start after one past its deadline: straight to gelada,with nothing captured or built,a changed config tries the microVM again(R13)Entry.boot(boot_id), and an earlier boot's entry counts for nothing. Growth isn't shrunk as disks fill (documented: it errs toward gelada)an entry of an earlier boot keeps nothing(same pid and start time); unit testwhat_other_starts_keep_counts_as_taken(R14)listens_locally)auto, a bare 587 with no relay on this host: … the microVMandwith a relay on this host's loopback: gelada; unit testauto_stays_on_gelada_for_a_bare_port_with_nowhere_said(R15)Evidence ([xps], under the build lock, each gate logged to a file,
$?checked)On bd858fa, [xps]:
docker build --target check: clippy-D warningsclean,cargo testall pass.ext-build,guest-build: rc=0.test/ext-microvm.sh fallback: 86 ok, 0 failed.test/ext-microvm.sh: 423 ok, 0 failed.On 7140260, [xps]:
docker build --target check: clippy-D warningsclean;cargo test126 passed, 1 failed. The failure was a wrong expectation in the newwhat_other_starts_keep_counts_as_taken, fixed in this commit. The rerun is part of build 2's job.scripts/ext-build.sh,scripts/guest-build.sh: rc=0.test/ext-microvm.sh fallback: 70 ok, 0 failed.test/ext-microvm.sh: 407 ok, 0 failed.On cce0ab7 (before the re-review fixes):
-s sh -S infoon release.yml's file listdocker build --target check(clippy-D warnings+cargo test, the whole workspace)auto_stays_on_gelada_rather_than_lose_the_database,auto_stays_on_gelada_for_a_pool_listening_everywhere,a_writable_disk_counts_what_it_may_still_take,the_sites_db_host_reaches_the_forward_in_the_guest)scripts/ext-build.sh 8.3 nts,scripts/guest-build.sh, the kernel (bench/microvm/kernel.sh)test/ext-microvm.sh fallbacktest/ext-microvm.sh(full)SC_TARGET=microvmtest/release-check.sh 8.3 nts,test/promote-check.shzip(scripts/package.sh), so both stopped at the release layout before any check ranSC_TARGET=dockerext-hostsstopped in its WordPress setup ("WordPress is already installed": an old stack on [xps], not the launcher; it passes on the microVM target)The first run found the item-1 bug:
auto, after the choicefailed with exit 125 (restrict_self … this process has 3 other thread(s)). It passes since the re-exec.Revert proofs
Build 1 ([xps], one build with five reverts, cce0ab7's code):
auto, after the choice: the launcher execs php-fpmand… php-fpm runs under gelada(exit 125:restrict_self … other thread(s)); infallbackand in the full ext-microvmautoDB and listen gaps removedauto, a bare 3306 and DB_HOST sc-no-such-db.invalid / 10.255.255.1:3306: gelada,and how to approve it, and gelada's php-fpm ran (both cases);auto, a pool on a bare port and no --listen: gelada(+2); release-check 7bthe site as it is:WordPress keeps its database(egress refuseddb:3306),auto stays on gelada …,says how to approve it,nothing of the microVM was built,PHP runs on the host (gelada)(got guest_init 1)a_writable_disk_counts_what_it_may_still_take(writable_room, an existing disk's growth), which runs in build 2upgrade, approved: WordPress in the guest reaches its database at its own DB_HOST, db(got 500)extend_start_timeoutremovedand EXTEND_TIMEOUT_USEC=30000000 before it(gotno)Totals under revert:
fallback16 failed (28 ok); the full ext-microvm 17 failed; the release check 6 failed (131 ok). R5 hides R1's preparation-time relaunch (with no extension thread, there is nothing to re-exec away from); R1's after-choice case is still caught.Build 2 (R3, R6–R10, on 7140260):
a_writable_disk_counts_what_it_may_still_take(writable_room)auto, a pool too big for any guest here: geladaand its gelada checks; the start served (stopped at the 900 s bound)auto_port_gapoffauto, a Redis on the loopback by a bare 6379:layer 1 isolation auto: microvm, serving with 6379 as egress (the run was stopped there)what_other_starts_keep_counts_as_takenandtwo_starts_at_once_never_both_pass_on_the_same_spacecurrent_exe, a flipped config refusedBuild 3 (R10–R15, on bd858fa;
fallback: 21 failed, 67 ok):current_exe, a flipped config refusedauto, its binary removed during the start: gelada(+2);the config now gelada at the relaunch: gelada, as it says(+2)auto, the other guests holding the host's memory budget: gelada,naming the budget …(+2); it served until the 240 s boundpreparemade the snapshotand the sampled check both failed for that reason)auto, a start past --start-within: gelada, at the deadline(it served until the bound);the next start after one past its deadline: straight to gelada(+2)an entry of an earlier boot keeps nothing: the start gets past the space checkauto, a bare 587 with no relay on this host: … the microVM,and 587 is no reason to stay on geladaFor Sami
autokeeps on gelada stays there until the operator acts: it approves the database (--db-address/--db-socket, or anaddress/socketin the config), or gives the pool an address (listen = 127.0.0.1:9000, or--listen). The launcher's line says which. The other way to makeautosafe would be to move such a site and accept a site without its database; this PR doesn't do that.wp-config.phpcan therefore keep their site on gelada (pointDB_HOSTsomewhere the guest can't reach). gelada is still a sandbox and the line is logged;isolation: "microvm"doesn't allow it.DB_HOSTgets no guest alias (the connect hook handles IPv4 and names).-claude