diff --git a/docs/layer1-microvm.md b/docs/layer1-microvm.md index 63dfa1f..38ad354 100644 --- a/docs/layer1-microvm.md +++ b/docs/layer1-microvm.md @@ -822,6 +822,86 @@ site's settings (`WORDPRESS_DB_HOST`, …). Until #72 the test image baked them the snapshot's `vm.mem` (as text or hex) and `vm.snap`, `console.log`, the launcher's log. It also checks that a restored guest still reaches the database. +## Density: N sites on one host (#72) + +`test/ext-microvm.sh density` runs N sites' microVMs at once on [xps] (4 cores, 7.5 GiB; other +agents' stacks running beside them), for N = 1, 4, 8 and 16. Each site is a copy of the test site +(WordPress, Classic Editor, Yoast SEO, Contact Form 7, `examples/wordpress.json`: all three +layers, Contact Form 7 isolated) in its own volume and state folder, one VM per site, as the +product runs them. +- **VM 1 serves a load** (oha, 2 concurrent) while the others restore one after another. +- **The others are idle once started.** +- **PSS and RSS** are Firecracker's (`/proc//smaps_rollup`), summed over the N VMs. +- **"KSM on"** is the same VMs after 30 s of ksmd (`/sys/kernel/mm/ksm/run` = 1, then unmerged and + set back to 0). +- **Restore** is each start's "load and php-fpm ping" and "ready" from the launcher's log. +- **p50** is VM 1's front page (bench's method: 20 warm-up, then 100 sequential requests, all + 200s) with all N up. + +Guest memory 512 MiB (the default): + +| N | total PSS | per VM | RSS | PSS with KSM on | restores while the others run (ping / ready, ms) | VM 1 p50 | MemAvailable after | +|---|---|---|---|---|---|---|---| +| 1 | 289 MiB | 289 MiB | 289 MiB | 289 MiB (0 shared) | – | 136.5 ms | 5433 MiB | +| 4 | 567 MiB | 141 MiB | 567 MiB | 569 MiB (0 shared) | 122–173 / 273–331 | 136.5 ms | 5142 MiB | +| 8 | 971 MiB | 121 MiB | 971 MiB | 975 MiB (0 shared) | 28–141 / 82–240 | 136.9 ms | 4957 MiB | +| 16 | 1383 MiB | 86 MiB | 1383 MiB | 1367 MiB (0 shared) | 58–155 / 116–251 | 137.8 ms | 4792 MiB | + +Guest memory 256 MiB (`DENSITY_MEM=256`), the same way: + +| N | total PSS | per VM | RSS | PSS with KSM on | restores (ping / ready, ms) | VM 1 p50 | MemAvailable after | +|---|---|---|---|---|---|---|---| +| 1 | 245 MiB | 245 MiB | 245 MiB | 245 MiB (0 shared) | – | 135.7 ms | 5414 MiB | +| 8 | 920 MiB | 115 MiB | 920 MiB | 920 MiB (0 shared) | 109–132 / 186–231 | 136.0 ms | 5110 MiB | +| 16 | 1397 MiB | 87 MiB | 1397 MiB | 1375 MiB (0 shared) | 36–171 / 105–287 | 137.1 ms | 4794 MiB | + +What this shows: +- **PSS equals RSS: restored VMs share nothing.** + - Each site restores its own snapshot (the key holds the site's tree), so each VM maps its own + `vm.mem`, and the page cache holds N copies of nearly the same pages. + - The cost is what each guest touches. The loaded VM 1 is about 290 MiB, and an idle restored + VM 60 to 75 MiB (16 VMs: 1383 MiB, with VM 1 the 290 of it). +- **KSM does nothing here.** ksmd merged 0 pages: it merges only memory marked `MADV_MERGEABLE`, + and Firecracker doesn't mark guest memory. A host-wide `prctl(PR_SET_MEMORY_MERGE)` on each + Firecracker would be the switch to try. That is a product change, so it isn't measured here. +- **"N VMs from one snapshot" is not something the launcher does.** The run also tried a "shared" + layout: every site's `vm.mem` a hard link to site 1's. + - Each launcher made a new snapshot instead: its start took 2.1–4.0 s to "ready", the time to + make a snapshot. The PSS was the same as with each site's own snapshot (16 VMs: 1916 MiB, + with the extra snapshot work in it). + - Sharing pages across sites needs a snapshot that doesn't hold the site: one per root image and + config, with the site disk attached after the restore, as the writable disks already are. + That is a design change (follow-up below), not a default to flip. +- **Restores stay fast under load**: 82–331 ms to "ready" while VM 1 serves its load and up to + 15 others run. The php-fpm ping is 28–173 ms. Four CPUs are shared by all of it. +- **Density doesn't slow a served site.** VM 1's p50 is 136.5 ms with 1 VM and 137.8 ms with 16. + (That p50 is with all three layers and Contact Form 7's isolated child on every request; #70's + +5 ms figure is the extension off.) +- **The memory a host needs is the busy sites', not the count.** At about 290 MiB a busy site + and 70 MiB an idle one, [xps]'s 7.5 GiB holds about 20 busy sites or 60 mostly idle ones, beside + its other work. + +### Recommendation + +- **Keep 512 MiB of guest memory and php-fpm's pool of 5 workers (`pm = dynamic`, the defaults).** + - A guest is charged only for the pages it touches, so its size costs an idle site nothing: + 16 VMs took 1383 MiB at 512 and 1397 MiB at 256. + - What 256 saves is on a busy site: 245 against 289 MiB for VM 1 under this load. But that is + 96 % of a 256 MiB guest with 5 workers, against 56 % of 512. A heavier site (WooCommerce, image + work) would hit the guest's limit there first, and its php-fpm workers would be killed instead + of the page slowing down. + - 256 MiB would want a smaller pool (3 workers, say), which wasn't measured. No default + changes. +- **Size a host by its busy sites:** + - about 300 MiB each for a busy site and 75 MiB each for an idle one, plus the launchers and the + page cache; + - restores stay under about 0.3 s to "ready" while the others run. +- **Next, if density matters more:** + - a site-independent snapshot, so restored VMs share their clean pages through one `vm.mem`; + - or marking guest memory mergeable for KSM. + + Either is a launcher change (follow-up 13 below). + ## The guest's log: what it guarantees, and against whom (#86) php-fpm's lines (layer 2's audit lines among them) and the init's own come to the launcher on @@ -881,10 +961,15 @@ logs are the launcher's own. takes 0.3 s rather than 80 ms. 10. **Run the `test/ext-*.sh` suites against a microVM target.** Their harness needs to reach php-fpm in the guest, not only through `docker compose exec php`. -11. **Density.** Run N restored VMs from one snapshot and measure PSS sharing. Pick the guest - memory and pool size: 512 MiB and 5 workers here, and 256 MiB is probably enough. +11. **Density.** Measured (#72, "Density" above): restored VMs share nothing, KSM merges + nothing, keep 512 MiB and 5 workers. 12. **arm64 hosts.** mirab pins Firecracker for x86_64 only. arm64 needs its own pin and kernel (mirab has `kernel-arm64-vz.config` for vz only). +13. **A snapshot that sites share.** Snapshot the guest before its site disk is attached (one per + root image and config), attach the site's disk after the restore as the writable disks are, + and key the snapshot without the site tree. Restored VMs would then share their clean pages + through one `vm.mem`. Or have the launcher mark guest memory mergeable + (`prctl(PR_SET_MEMORY_MERGE)` on Firecracker) and measure KSM again. ## Needs a decision diff --git a/test/ext-microvm.sh b/test/ext-microvm.sh index 7831385..4e6ceed 100755 --- a/test/ext-microvm.sh +++ b/test/ext-microvm.sh @@ -43,6 +43,8 @@ # test/ext-microvm.sh bench [n] front-page p50, host php-fpm vs the microVM, extension off on # both (bench/layer1.sh's method, #70's numbers), n rounds # test/ext-microvm.sh restore [n] the requests after a restore, timed +# test/ext-microvm.sh density N sites' VMs at once: PSS, KSM, restores under load, p50 +# (DENSITY_NS, DENSITY_MEM; docs/layer1-microvm.md "Density") # test/ext-microvm.sh log only the guest's log under load (3d, #86) # Needs: the guest kernel (KERNEL=, default .tmp/mvm/vmlinux-sc from bench/microvm/kernel.sh: # mirab's kernel plus Landlock and seccomp, until mirab#89 ships them); a host securecore-fpm @@ -337,6 +339,126 @@ if [ "$what" = restore ]; then exit 0 fi +if [ "$what" = density ]; then + # Density (#72, docs/layer1-microvm.md "Density"): N sites' microVMs on this host, for N in + # DENSITY_NS (default "1 4 8 16", stopping early when MemAvailable would drop under 700 MiB), + # guest memory DENSITY_MEM MiB (default 512). Each site is a copy of this one in its own Docker + # volume (sc-density-), with its own state folder: one VM per site, as the product runs them. + # Two layouts: "own" (each site restores its own snapshot, as today) and "shared" (every site's + # vm.mem a hard link to site 1's: N VMs restored from one snapshot, so its clean pages are one + # copy in the page cache). Per N: Firecracker's total and per-VM PSS and RSS (smaps_rollup), with + # KSM off and then on (30 s of ksmd, pages_sharing), each start's restore time while the earlier + # VMs run and VM 1 serves a load, and VM 1's front-page p50 (oha, bench's method) with all N up. + mem=${DENSITY_MEM:-512} + ns=${DENSITY_NS:-"1 4 8 16"} + max=$(echo "$ns" | tr ' ' '\n' | sort -n | tail -1) + dd=$dir/density + mkdir -p "$dd" + oha() { docker run --rm --network "${project}_default" "$OHA" --no-tui --output-format json -H "Host: 127.0.0.1:$port" "$@"; } + avail() { awk '/^MemAvailable:/ { print int($2 / 1024) }' /proc/meminfo; } + # The sites: copies of this one without its uploads (the writable disk is sized from them). + find "$site/wp-content/uploads" -mindepth 1 -maxdepth 1 -name 'sc-big.bin' -delete + k=1 + while [ "$k" -le "$max" ]; do + docker volume create "sc-density-$k" > /dev/null + s=$(docker volume inspect -f '{{.Mountpoint}}' "sc-density-$k") + [ -f "$s/wp-load.php" ] || cp -a "$site/." "$s/" + cp "$dir/microvm.json" "$s/securecore.json" + mkdir -p "$dd/state-$k" "$dd/run-$k" + k=$((k + 1)) + done + # dstart : site k's launcher in the background (VM 1 listens where mvmweb sends PHP). + dstart() { + s=$(docker volume inspect -f '{{.Mountpoint}}' "sc-density-$1") + l=$dd/run-$1/fpm.sock + [ "$1" = 1 ] && l=$run/fpm.sock + fpm_env --config "$s/securecore.json" --vm-dir "$vm" --state-dir "$dd/state-$1" --listen "unix:$l" \ + --listen-owner 0 --listen-group 101 --listen-mode 0660 --vm-user "$vmuser" --pool-user "$pool" \ + --guest-root "$html" --mem "$mem" > "$2" 2>&1 & + echo $! > "$dd/pid-$1" + i=0 + until grep -q 'securecore-fpm: serving FastCGI' "$2"; do + kill -0 "$(cat "$dd/pid-$1")" 2>/dev/null || { echo " site $1's launcher exited: $(tail -2 "$2" | tr '\n' ' ')"; return 1; } + i=$((i + 1)); [ "$i" -lt 1800 ] || return 1 + sleep 0.1 + done + } + dstop() { for f in "$dd"/pid-*; do [ -f "$f" ] || continue; kill -TERM "$(cat "$f")" 2>/dev/null; wait "$(cat "$f")" 2>/dev/null; rm -f "$f"; done; } + trap 'dstop; [ -z "$PID" ] || kill -TERM "$PID" 2>/dev/null; kill "$peer_pid" 2>/dev/null; echo 0 > /sys/kernel/mm/ksm/run 2>/dev/null' EXIT + fcpid() { sed -n 's/.*Firecracker runs jailed as uid [0-9]* gid [0-9]* (pid \([0-9]*\)).*/\1/p' "$1" | tail -1; } + # pss : " " of the n Firecracker processes. + pss() { + t=0 r=0 k=1 + while [ "$k" -le "$1" ]; do + f=$(fcpid "$dd/run-$k.log") + t=$((t + $(awk '/^Pss:/ { print $2 }' "/proc/$f/smaps_rollup"))) r=$((r + $(awk '/^Rss:/ { print $2 }' "/proc/$f/smaps_rollup"))) + k=$((k + 1)) + done + echo "$((t / 1024)) $((r / 1024))" + } + # Every site's own snapshot first, one at a time. + k=1 + while [ "$k" -le "$max" ]; do + dstart "$k" "$dd/prep-$k.log" || die "site $k's first start" + dstop + k=$((k + 1)) + done + echo "density: guest memory $mem MiB, $(nproc) CPUs, $(awk '/^MemTotal:/ { print int($2 / 1024) }' /proc/meminfo) MiB, KSM $(cat /sys/kernel/mm/ksm/run 2>/dev/null || echo absent)" + for layout in own shared; do + if [ "$layout" = shared ]; then + one=$(find "$dd/state-1" -maxdepth 2 -path '*/snap-*/vm.mem' | head -1) + k=2 + while [ "$k" -le "$max" ]; do + ln -f "$one" "$(find "$dd/state-$k" -maxdepth 2 -path '*/snap-*/vm.mem' | head -1)" + k=$((k + 1)) + done + fi + for n in $ns; do + dstart 1 "$dd/run-1.log" || die "VM 1" + # VM 1 under load while the others restore (a named container, stopped by its name). + docker run --rm --name "$project-density-load" --network "${project}_default" "$OHA" --no-tui \ + -H "Host: 127.0.0.1:$port" -z 600s -c 2 http://mvmweb/ > /dev/null 2>&1 & + load=$! + restores="" k=2 got=1 + while [ "$k" -le "$n" ]; do + if [ "$(avail)" -lt $((700 + mem / 2)) ]; then echo " stopping at $got VMs: $(avail) MiB available"; break; fi + dstart "$k" "$dd/run-$k.log" || break + restores="$restores $(sed -n 's/.*restored from snapshot [0-9a-f]* (load and php-fpm ping \([0-9.]*\) ms); ready \([0-9]*\) ms.*/\1\/\2/p' "$dd/run-$k.log")" + got=$k k=$((k + 1)) + done + docker stop "$project-density-load" > /dev/null 2>&1; wait "$load" 2>/dev/null + sleep 5 + # shellcheck disable=SC2046 # two numbers + set -- $(pss "$got") + off_t=$1 off_r=$2 + oha -n 20 -c 1 http://mvmweb/ > /dev/null + oha -n 100 -c 1 http://mvmweb/ > "$dd/p50-$layout-$n.json" + p50=$(jq -r '.metrics.latency_ms.p50' "$dd/p50-$layout-$n.json") + ok=$(jq -r '.statusCodeDistribution["200"] // 0' "$dd/p50-$layout-$n.json") + on_t=- share=- + if [ -w /sys/kernel/mm/ksm/run ]; then + echo 1 > /sys/kernel/mm/ksm/run; sleep 30 + # shellcheck disable=SC2046 # two numbers + set -- $(pss "$got"); on_t=$1 + share=$(( $(cat /sys/kernel/mm/ksm/pages_sharing) * 4 / 1024 )) + echo 2 > /sys/kernel/mm/ksm/run; sleep 2; echo 0 > /sys/kernel/mm/ksm/run + fi + printf '%s N=%s: PSS %s MiB (%s/VM), RSS %s MiB | KSM on: PSS %s MiB, sharing %s MiB | restores (ping/ready ms):%s | VM 1 p50 %.2f ms (%s/100 200s) | %s MiB available\n' \ + "$layout" "$got" "$off_t" "$((off_t / got))" "$off_r" "$on_t" "$share" "${restores:- -}" "$p50" "$ok" "$(avail)" + dstop + done + done + k=1 + while [ "$k" -le "$max" ]; do + s=$dd/state-$k + awk -v s="$s/" 'index($5, s) == 1 { print $5 }' /proc/self/mountinfo | sort -r | while read -r m; do umount -l "$m"; done + find "$s" -mindepth 1 -xdev -delete + docker volume rm "sc-density-$k" > /dev/null + k=$((k + 1)) + done + exit 0 +fi + if [ "$what" = bench ]; then # bench/layer1.sh's method (#70): 20 warm-up and 100 sequential requests by oha on the stack's # network, extension off on both sides; host php-fpm is the php container with layer 1 off.