Skip to content

Layer 1 microVM: density measured (N VMs, PSS, KSM, restores under load) - #89

Merged
samifouad merged 2 commits into
mainfrom
claude/microvm-density-72
Nov 12, 2026
Merged

samifouad merged 2 commits into
mainfrom
claude/microvm-density-72

Conversation

@samifouad

Copy link
Copy Markdown
Member

Part 5 of #72: density, measured. This is measurement only; no product default changes. Stacked on #88 (it uses that PR's mvm_files in test/ext-microvm.sh), so review that one first. closes #72

What changed

  • test/ext-microvm.sh density runs N sites' microVMs at once (DENSITY_NS, default 1 4 8 16; DENSITY_MEM, default 512). Each site is a copy of the test site in its own volume and state folder, one VM per site, as the product runs them. For each N it records:
    • Firecracker's total and per-VM PSS and RSS;
    • the same with KSM on for 30 s, then unmerged and set back to 0;
    • each restore's ping and "ready" time, while the earlier VMs run and VM 1 serves a load;
    • VM 1's front-page p50 (bench's method) with all N up.
    • It stops early if MemAvailable would drop too low, and it removes its volumes and state folders at the end.
  • docs/layer1-microvm.md has a new section, "Density: N sites on one host", with the tables and the recommendation. Follow-up 11 now points to it, and follow-up 13 (a snapshot that sites share) is new.

Results ([xps]: 4 cores, 7.5 GiB, other agents' stacks running; all three layers on, Contact Form 7 isolated)

512 MiB guests:

N total PSS per VM PSS with KSM on restores (ping / ready, ms) VM 1 p50
1 289 MiB 289 289 (0 shared) – 136.5 ms
4 567 MiB 141 569 (0 shared) 122–173 / 273–331 136.5 ms
8 971 MiB 121 975 (0 shared) 28–141 / 82–240 136.9 ms
16 1383 MiB 86 1367 (0 shared) 58–155 / 116–251 137.8 ms

256 MiB guests: 245 MiB at N=1, 920 MiB at N=8 and 1397 MiB at N=16, with the same p50 (135.7–137.1 ms, 100/100 200s).

  • PSS equals RSS: restored VMs share nothing. Each site restores its own snapshot (the key holds the site's tree), so the page cache holds N copies of vm.mem.
  • KSM merges 0 pages. Firecracker doesn't mark guest memory mergeable.
  • "N VMs from one snapshot" isn't something the launcher can do. The run tried hard-linking every site's vm.mem to site 1's. Each launcher rebuilt its own snapshot instead, taking 1.6–4.0 s to "ready".
  • Restores stay at 0.08–0.33 s to "ready" under load, and VM 1's p50 doesn't move with N.

Recommendation: keep 512 MiB and php-fpm's 5 workers.

  • An idle VM costs only what it touches, so a smaller guest saves an idle site nothing: 16 VMs took 1383 MiB at 512 and 1397 MiB at 256.
  • 256 MiB saves about 45 MiB on a busy site, but at 245 MiB it runs at 96 % of the guest with 5 workers. A heavier site would have its workers killed first.
  • Size a host by its busy sites: about 300 MiB each for a busy site, about 75 MiB each for an idle one.

More density needs a snapshot that sites share (taken before the site disk is attached), or mergeable guest memory for KSM. Both are launcher changes (follow-up 13).

Evidence

density512=0   (.tmp/runs/density-512.log)
density256=0   (.tmp/runs/density-256.log)

There is no revert proof: this PR adds a measurement mode and no check.

-claude

…ity)

PSS and RSS of 1, 4, 8 and 16 VMs at 512 MiB and 256 MiB guests, KSM off
and on, restores while the others run and one serves a load, and that
VM's front-page p50. Restored VMs share nothing (each site restores its
own snapshot), KSM merges nothing (guest memory isn't mergeable), and a
foreign vm.mem is rebuilt, not shared. Written up in
docs/layer1-microvm.md: keep 512 MiB and 5 workers; a snapshot sites
share is follow-up 13.
@samifouad
samifouad changed the base branch from claude/microvm-suites-72 to main November 12, 2026 00:41
…y-72

# Conflicts:
#	docs/layer1-microvm.md
#	test/ext-microvm.sh
@samifouad
samifouad merged commit e96dfa2 into main Nov 12, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Layer 1 microVM: egress, secrets, uploads, and the ext suites against the guest

1 participant