Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 87 additions & 2 deletions docs/layer1-microvm.md
Original file line number Diff line number Diff line change
Expand Up @@ -822,6 +822,86 @@ site's settings (`WORDPRESS_DB_HOST`, …). Until #72 the test image baked them
the snapshot's `vm.mem` (as text or hex) and `vm.snap`, `console.log`, the launcher's log. It
also checks that a restored guest still reaches the database.

## Density: N sites on one host (#72)

`test/ext-microvm.sh density` runs N sites' microVMs at once on [xps] (4 cores, 7.5 GiB; other
agents' stacks running beside them), for N = 1, 4, 8 and 16. Each site is a copy of the test site
(WordPress, Classic Editor, Yoast SEO, Contact Form 7, `examples/wordpress.json`: all three
layers, Contact Form 7 isolated) in its own volume and state folder, one VM per site, as the
product runs them.
- **VM 1 serves a load** (oha, 2 concurrent) while the others restore one after another.
- **The others are idle once started.**
- **PSS and RSS** are Firecracker's (`/proc/<pid>/smaps_rollup`), summed over the N VMs.
- **"KSM on"** is the same VMs after 30 s of ksmd (`/sys/kernel/mm/ksm/run` = 1, then unmerged and
set back to 0).
- **Restore** is each start's "load and php-fpm ping" and "ready" from the launcher's log.
- **p50** is VM 1's front page (bench's method: 20 warm-up, then 100 sequential requests, all
200s) with all N up.

Guest memory 512 MiB (the default):

| N | total PSS | per VM | RSS | PSS with KSM on | restores while the others run (ping / ready, ms) | VM 1 p50 | MemAvailable after |
|---|---|---|---|---|---|---|---|
| 1 | 289 MiB | 289 MiB | 289 MiB | 289 MiB (0 shared) | – | 136.5 ms | 5433 MiB |
| 4 | 567 MiB | 141 MiB | 567 MiB | 569 MiB (0 shared) | 122–173 / 273–331 | 136.5 ms | 5142 MiB |
| 8 | 971 MiB | 121 MiB | 971 MiB | 975 MiB (0 shared) | 28–141 / 82–240 | 136.9 ms | 4957 MiB |
| 16 | 1383 MiB | 86 MiB | 1383 MiB | 1367 MiB (0 shared) | 58–155 / 116–251 | 137.8 ms | 4792 MiB |

Guest memory 256 MiB (`DENSITY_MEM=256`), the same way:

| N | total PSS | per VM | RSS | PSS with KSM on | restores (ping / ready, ms) | VM 1 p50 | MemAvailable after |
|---|---|---|---|---|---|---|---|
| 1 | 245 MiB | 245 MiB | 245 MiB | 245 MiB (0 shared) | – | 135.7 ms | 5414 MiB |
| 8 | 920 MiB | 115 MiB | 920 MiB | 920 MiB (0 shared) | 109–132 / 186–231 | 136.0 ms | 5110 MiB |
| 16 | 1397 MiB | 87 MiB | 1397 MiB | 1375 MiB (0 shared) | 36–171 / 105–287 | 137.1 ms | 4794 MiB |

What this shows:
- **PSS equals RSS: restored VMs share nothing.**
- Each site restores its own snapshot (the key holds the site's tree), so each VM maps its own
`vm.mem`, and the page cache holds N copies of nearly the same pages.
- The cost is what each guest touches. The loaded VM 1 is about 290 MiB, and an idle restored
VM 60 to 75 MiB (16 VMs: 1383 MiB, with VM 1 the 290 of it).
- **KSM does nothing here.** ksmd merged 0 pages: it merges only memory marked `MADV_MERGEABLE`,
and Firecracker doesn't mark guest memory. A host-wide `prctl(PR_SET_MEMORY_MERGE)` on each
Firecracker would be the switch to try. That is a product change, so it isn't measured here.
- **"N VMs from one snapshot" is not something the launcher does.** The run also tried a "shared"
layout: every site's `vm.mem` a hard link to site 1's.
- Each launcher made a new snapshot instead: its start took 2.1–4.0 s to "ready", the time to
make a snapshot. The PSS was the same as with each site's own snapshot (16 VMs: 1916 MiB,
with the extra snapshot work in it).
- Sharing pages across sites needs a snapshot that doesn't hold the site: one per root image and
config, with the site disk attached after the restore, as the writable disks already are.
That is a design change (follow-up below), not a default to flip.
- **Restores stay fast under load**: 82–331 ms to "ready" while VM 1 serves its load and up to
15 others run. The php-fpm ping is 28–173 ms. Four CPUs are shared by all of it.
- **Density doesn't slow a served site.** VM 1's p50 is 136.5 ms with 1 VM and 137.8 ms with 16.
(That p50 is with all three layers and Contact Form 7's isolated child on every request; #70's
+5 ms figure is the extension off.)
- **The memory a host needs is the busy sites', not the count.** At about 290 MiB a busy site
and 70 MiB an idle one, [xps]'s 7.5 GiB holds about 20 busy sites or 60 mostly idle ones, beside
its other work.

### Recommendation

- **Keep 512 MiB of guest memory and php-fpm's pool of 5 workers (`pm = dynamic`, the defaults).**
- A guest is charged only for the pages it touches, so its size costs an idle site nothing:
16 VMs took 1383 MiB at 512 and 1397 MiB at 256.
- What 256 saves is on a busy site: 245 against 289 MiB for VM 1 under this load. But that is
96 % of a 256 MiB guest with 5 workers, against 56 % of 512. A heavier site (WooCommerce, image
work) would hit the guest's limit there first, and its php-fpm workers would be killed instead
of the page slowing down.
- 256 MiB would want a smaller pool (3 workers, say), which wasn't measured. No default
changes.
- **Size a host by its busy sites:**
- about 300 MiB each for a busy site and 75 MiB each for an idle one, plus the launchers and the
page cache;
- restores stay under about 0.3 s to "ready" while the others run.
- **Next, if density matters more:**
- a site-independent snapshot, so restored VMs share their clean pages through one `vm.mem`;
- or marking guest memory mergeable for KSM.

Either is a launcher change (follow-up 13 below).

## The guest's log: what it guarantees, and against whom (#86)

php-fpm's lines (layer 2's audit lines among them) and the init's own come to the launcher on
Expand Down Expand Up @@ -881,10 +961,15 @@ logs are the launcher's own.
takes 0.3 s rather than 80 ms.
10. **Run the `test/ext-*.sh` suites against a microVM target.** Their harness needs to
reach php-fpm in the guest, not only through `docker compose exec php`.
11. **Density.** Run N restored VMs from one snapshot and measure PSS sharing. Pick the guest
memory and pool size: 512 MiB and 5 workers here, and 256 MiB is probably enough.
11. **Density.** Measured (#72, "Density" above): restored VMs share nothing, KSM merges
nothing, keep 512 MiB and 5 workers.
12. **arm64 hosts.** mirab pins Firecracker for x86_64 only. arm64 needs its own pin and kernel
(mirab has `kernel-arm64-vz.config` for vz only).
13. **A snapshot that sites share.** Snapshot the guest before its site disk is attached (one per
root image and config), attach the site's disk after the restore as the writable disks are,
and key the snapshot without the site tree. Restored VMs would then share their clean pages
through one `vm.mem`. Or have the launcher mark guest memory mergeable
(`prctl(PR_SET_MEMORY_MERGE)` on Firecracker) and measure KSM again.

## Needs a decision

Expand Down
122 changes: 122 additions & 0 deletions test/ext-microvm.sh
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,8 @@
# test/ext-microvm.sh bench [n] front-page p50, host php-fpm vs the microVM, extension off on
# both (bench/layer1.sh's method, #70's numbers), n rounds
# test/ext-microvm.sh restore [n] the requests after a restore, timed
# test/ext-microvm.sh density N sites' VMs at once: PSS, KSM, restores under load, p50
# (DENSITY_NS, DENSITY_MEM; docs/layer1-microvm.md "Density")
# test/ext-microvm.sh log only the guest's log under load (3d, #86)
# Needs: the guest kernel (KERNEL=, default .tmp/mvm/vmlinux-sc from bench/microvm/kernel.sh:
# mirab's kernel plus Landlock and seccomp, until mirab#89 ships them); a host securecore-fpm
Expand Down Expand Up @@ -337,6 +339,126 @@ if [ "$what" = restore ]; then
exit 0
fi

if [ "$what" = density ]; then
# Density (#72, docs/layer1-microvm.md "Density"): N sites' microVMs on this host, for N in
# DENSITY_NS (default "1 4 8 16", stopping early when MemAvailable would drop under 700 MiB),
# guest memory DENSITY_MEM MiB (default 512). Each site is a copy of this one in its own Docker
# volume (sc-density-<k>), with its own state folder: one VM per site, as the product runs them.
# Two layouts: "own" (each site restores its own snapshot, as today) and "shared" (every site's
# vm.mem a hard link to site 1's: N VMs restored from one snapshot, so its clean pages are one
# copy in the page cache). Per N: Firecracker's total and per-VM PSS and RSS (smaps_rollup), with
# KSM off and then on (30 s of ksmd, pages_sharing), each start's restore time while the earlier
# VMs run and VM 1 serves a load, and VM 1's front-page p50 (oha, bench's method) with all N up.
mem=${DENSITY_MEM:-512}
ns=${DENSITY_NS:-"1 4 8 16"}
max=$(echo "$ns" | tr ' ' '\n' | sort -n | tail -1)
dd=$dir/density
mkdir -p "$dd"
oha() { docker run --rm --network "${project}_default" "$OHA" --no-tui --output-format json -H "Host: 127.0.0.1:$port" "$@"; }
avail() { awk '/^MemAvailable:/ { print int($2 / 1024) }' /proc/meminfo; }
# The sites: copies of this one without its uploads (the writable disk is sized from them).
find "$site/wp-content/uploads" -mindepth 1 -maxdepth 1 -name 'sc-big.bin' -delete
k=1
while [ "$k" -le "$max" ]; do
docker volume create "sc-density-$k" > /dev/null
s=$(docker volume inspect -f '{{.Mountpoint}}' "sc-density-$k")
[ -f "$s/wp-load.php" ] || cp -a "$site/." "$s/"
cp "$dir/microvm.json" "$s/securecore.json"
mkdir -p "$dd/state-$k" "$dd/run-$k"
k=$((k + 1))
done
# dstart <k> <log>: site k's launcher in the background (VM 1 listens where mvmweb sends PHP).
dstart() {
s=$(docker volume inspect -f '{{.Mountpoint}}' "sc-density-$1")
l=$dd/run-$1/fpm.sock
[ "$1" = 1 ] && l=$run/fpm.sock
fpm_env --config "$s/securecore.json" --vm-dir "$vm" --state-dir "$dd/state-$1" --listen "unix:$l" \
--listen-owner 0 --listen-group 101 --listen-mode 0660 --vm-user "$vmuser" --pool-user "$pool" \
--guest-root "$html" --mem "$mem" > "$2" 2>&1 &
echo $! > "$dd/pid-$1"
i=0
until grep -q 'securecore-fpm: serving FastCGI' "$2"; do
kill -0 "$(cat "$dd/pid-$1")" 2>/dev/null || { echo " site $1's launcher exited: $(tail -2 "$2" | tr '\n' ' ')"; return 1; }
i=$((i + 1)); [ "$i" -lt 1800 ] || return 1
sleep 0.1
done
}
dstop() { for f in "$dd"/pid-*; do [ -f "$f" ] || continue; kill -TERM "$(cat "$f")" 2>/dev/null; wait "$(cat "$f")" 2>/dev/null; rm -f "$f"; done; }
trap 'dstop; [ -z "$PID" ] || kill -TERM "$PID" 2>/dev/null; kill "$peer_pid" 2>/dev/null; echo 0 > /sys/kernel/mm/ksm/run 2>/dev/null' EXIT
fcpid() { sed -n 's/.*Firecracker runs jailed as uid [0-9]* gid [0-9]* (pid \([0-9]*\)).*/\1/p' "$1" | tail -1; }
# pss <n>: "<total PSS MiB> <total RSS MiB>" of the n Firecracker processes.
pss() {
t=0 r=0 k=1
while [ "$k" -le "$1" ]; do
f=$(fcpid "$dd/run-$k.log")
t=$((t + $(awk '/^Pss:/ { print $2 }' "/proc/$f/smaps_rollup"))) r=$((r + $(awk '/^Rss:/ { print $2 }' "/proc/$f/smaps_rollup")))
k=$((k + 1))
done
echo "$((t / 1024)) $((r / 1024))"
}
# Every site's own snapshot first, one at a time.
k=1
while [ "$k" -le "$max" ]; do
dstart "$k" "$dd/prep-$k.log" || die "site $k's first start"
dstop
k=$((k + 1))
done
echo "density: guest memory $mem MiB, $(nproc) CPUs, $(awk '/^MemTotal:/ { print int($2 / 1024) }' /proc/meminfo) MiB, KSM $(cat /sys/kernel/mm/ksm/run 2>/dev/null || echo absent)"
for layout in own shared; do
if [ "$layout" = shared ]; then
one=$(find "$dd/state-1" -maxdepth 2 -path '*/snap-*/vm.mem' | head -1)
k=2
while [ "$k" -le "$max" ]; do
ln -f "$one" "$(find "$dd/state-$k" -maxdepth 2 -path '*/snap-*/vm.mem' | head -1)"
k=$((k + 1))
done
fi
for n in $ns; do
dstart 1 "$dd/run-1.log" || die "VM 1"
# VM 1 under load while the others restore (a named container, stopped by its name).
docker run --rm --name "$project-density-load" --network "${project}_default" "$OHA" --no-tui \
-H "Host: 127.0.0.1:$port" -z 600s -c 2 http://mvmweb/ > /dev/null 2>&1 &
load=$!
restores="" k=2 got=1
while [ "$k" -le "$n" ]; do
if [ "$(avail)" -lt $((700 + mem / 2)) ]; then echo " stopping at $got VMs: $(avail) MiB available"; break; fi
dstart "$k" "$dd/run-$k.log" || break
restores="$restores $(sed -n 's/.*restored from snapshot [0-9a-f]* (load and php-fpm ping \([0-9.]*\) ms); ready \([0-9]*\) ms.*/\1\/\2/p' "$dd/run-$k.log")"
got=$k k=$((k + 1))
done
docker stop "$project-density-load" > /dev/null 2>&1; wait "$load" 2>/dev/null
sleep 5
# shellcheck disable=SC2046 # two numbers
set -- $(pss "$got")
off_t=$1 off_r=$2
oha -n 20 -c 1 http://mvmweb/ > /dev/null
oha -n 100 -c 1 http://mvmweb/ > "$dd/p50-$layout-$n.json"
p50=$(jq -r '.metrics.latency_ms.p50' "$dd/p50-$layout-$n.json")
ok=$(jq -r '.statusCodeDistribution["200"] // 0' "$dd/p50-$layout-$n.json")
on_t=- share=-
if [ -w /sys/kernel/mm/ksm/run ]; then
echo 1 > /sys/kernel/mm/ksm/run; sleep 30
# shellcheck disable=SC2046 # two numbers
set -- $(pss "$got"); on_t=$1
share=$(( $(cat /sys/kernel/mm/ksm/pages_sharing) * 4 / 1024 ))
echo 2 > /sys/kernel/mm/ksm/run; sleep 2; echo 0 > /sys/kernel/mm/ksm/run
fi
printf '%s N=%s: PSS %s MiB (%s/VM), RSS %s MiB | KSM on: PSS %s MiB, sharing %s MiB | restores (ping/ready ms):%s | VM 1 p50 %.2f ms (%s/100 200s) | %s MiB available\n' \
"$layout" "$got" "$off_t" "$((off_t / got))" "$off_r" "$on_t" "$share" "${restores:- -}" "$p50" "$ok" "$(avail)"
dstop
done
done
k=1
while [ "$k" -le "$max" ]; do
s=$dd/state-$k
awk -v s="$s/" 'index($5, s) == 1 { print $5 }' /proc/self/mountinfo | sort -r | while read -r m; do umount -l "$m"; done
find "$s" -mindepth 1 -xdev -delete
docker volume rm "sc-density-$k" > /dev/null
k=$((k + 1))
done
exit 0
fi

if [ "$what" = bench ]; then
# bench/layer1.sh's method (#70): 20 warm-up and 100 sequential requests by oha on the stack's
# network, extension off on both sides; host php-fpm is the php container with layer 1 off.
Expand Down
Loading