ci(browser): split journeys across nine shards per engine

[jameskraus]

Summary

Playwright splits browser-journey shards by test count, not cost, so WebKit's heaviest shard runs close to the 15-minute job timeout. main runs 8 shards per engine (#801). Since then, its slowest WebKit job takes 9.9–13.5 min per run, with webkit 5/8 at 12–13.5 min.

This PR uses 9 shards. Its slowest WebKit job takes 9.5–11.9 min per run. Tests, assertions, retries (0), workers (2) and the timeout are unchanged. The existing coverage guard still requires the 18 Linux selections plus macOS recording to cover every functional test exactly once.

Stack: #999 → #997 → this PR. With nine shards, inbox-edit runs first on a cold worker, which exposes the preview-text selector that #997 replaces.

Hosted timings at 8 and 9 shards

Slowest WebKit job per CI run, including ~66 s of setup:

Layout	Runs	Slowest WebKit job per run (min)
main, 8 shards	6 (38478053678 … 38490649059)	13.5, 12.9, 12.4, 12.3, 12.0, 9.9
this PR, 9 shards	4 (38451842483 … 38490789166)	11.9, 11.0, 10.2, 9.5

On the slowest runners measured (1.32–1.44×), the two layouts diverge:

• At 8 shards, webkit 4/8 hits the 15-minute timeout with 81 of 87 tests done (test(voice-notes): keep each attachment's own type in the upload fixture #999, run 38492346858, 1.38× runner).
• At 9 shards, the slowest WebKit job finishes in 13.0 min (run 38492512107, 1.44× runner).

Shard cost on main at 6 shards

Environment: GitHub-hosted ubuntu-24.04 (public repo, 4 vCPU), mcr.microsoft.com/playwright:v1.63.0-noble container, 2 workers:

./bin/pnpm test:browser:ci --project <engine> --no-deps --shard=N/6 --reporter=list,json

Sources: the browser-journeys-* artifacts (ci-report.json, ci-timing.json) of 9 main runs (38350111813 … 38445985225), plus job metadata and logs of 40 main runs.

WebKit per shard across the 9 runs. Wall time is the elapsed ci-test-report time; job setup adds ~66 s:

Shard	Tests	Elapsed wall (min)	Summed test execution (min)	Slowest files (median summed)
1/6	118–121	8.2–11.0	16.1–21.4	layout 110 s, history-loading 75 s, channel-lifecycle 73 s
2/6	116–117	8.1–12.0	15.0–22.2	navigation-sidebar 215 s, navigation-scroll-intent 141 s, sidebar-unread 98 s
3/6	110–118	7.9–12.0	14.8–23.1	nested-replies 286 s, message-navigation 271 s, buzz-links 115 s
4/6	107–115	9.1–13.5	16.6–24.2	navigation-groups 276 s, agent-activity 153 s, unread 127 s
5/6	116–118	4.1–6.3	7.8–11.9	channel-members-focus 95 s, typeahead 84 s, profiles 58 s
6/6	112	6.5–10.1	12.3–19.1	workflows.journey.mjs 141 s, messages 113 s, layout 68 s

The slowest WebKit tests by median duration:

• nested-replies "crowded capped branches expand once with readable actions" at 1492/1280/1024/700 px: 61/50/49/44 s, 2.0–2.4× Chromium.
• unread "a viewport receipt survives other-channel pressure and app reload": 52 s.
• image-scroll partial gestures: 46 s.
• navigation-groups row-menu/Star/Create cases: 33–39 s, the same in Chromium.

Timed-out webkit 4/6 jobs. These have no JSON report, so the numbers come from the job logs' list reporter. All three run at 1.15–1.16× the per-test medians:

Run	Tests	Completed	Summed execution of completed	Last result after start
38317924175 (main)	109	108	23.5 min	12.5 min
38355319857 (main)	114	113	24.6 min	13.1 min
38425952803 (#570), 1.16× runner	114	113	24.6 min	13.1 min
38425952803 (#570), same commit on a 0.87× runner	114	114, passed	19.1 min	10.6 min; 11.9 min job

Runner variance. For identical tests, a WebKit job's summed time relative to the cross-run median clusters at 0.69–0.80× (20 jobs) and 0.94–1.16× (33 jobs), with 5 jobs at 0.84–0.89× (n = 58). Chromium has the same pattern at a smaller scale: chromium 4/6 has a median of 11.9 min and runs ≥12 min in 14 of 40.

Choosing the shard count

The numbers below come from a model of Playwright 1.63's shard selection:

• Groups are ordered by worker hash, then by file.
• The partition is count-based.
• Each shard dispatches its groups in order to its 2 workers.

Validation:

• The model reproduces Playwright's own --list --shard=k/N counts exactly.
• Across 63 WebKit shard runs, actual Playwright duration is a median 6 s above the modeled wall time, within −9% to +3%.

Inputs:

• Per-file median cost per test, taken from 11 runs' artifacts.
• The current corpus: 696 WebKit tests in 165 groups, 114 min summed at median runner speed.

"Growth p95" adds 1–60 random tests to existing files over 300 trials:

WebKit shards	Slowest shard wall at main	Growth p95	Worst job at p95 (×1.16 runner + 1.2 min setup)
6	12.1 min	14.5	18.0
7	10.6	11.6	14.7
8	10.0	11.4	14.4
9	8.7	9.8	12.5
10	8.5	10.2	13.0
12	9.3	10.3	13.2

Count-based shards stay lumpy: 9 is the smallest count that leaves margin, and 10–12 are no better.

Not chosen:

• Duration-balanced assignment via Playwright's reporter.preprocess and testRun.skipSharding(). It needs a committed per-file cost table that drifts over time. The fixed in-shard dispatch order still leaves a 10.5–10.9 min slowest WebKit shard at 6 shards.
• fullyParallel. Six journey files share beforeAll/afterAll state and five keep module-level state; these would be duplicated per worker and lose in-file ordering. The partition also stays count-based.
• Shortening the slowest cases. The nested-replies crowded-branch matrix and navigation-groups are deliberate real-pointer and persistence journeys, so trimming them drops coverage.
• Raising the timeout. It hides the imbalance instead of fixing it.

Expected timings

Modeled on the 696-test WebKit corpus above. All 18 selections are non-empty.

Engine	Slowest shard: test wall	Job, median runner	Job, fast runner (×0.75)	Job, slow runner (×1.16)
WebKit, 6 → 9 shards	12.1 → 8.7 min	13.3 → 9.9 min	10.3 → 7.7 min	15.2 → 11.3 min
Chromium, 6 → 9 shards	11.2 → 8.0 min	12.4 → 9.2 min	9.6 → 7.2 min	14.2 → 10.5 min

Cost: 2 more runners per CI run than 8 shards, about +2 runner-minutes of setup.

Hosted timings with 9 shards

Run 38451842483, with the same environment and --shard=N/9. The runner factor is summed time relative to the cross-run median for the same tests.

Shard	Tests	Elapsed wall (min)	Summed execution (min)	Job (min)	Runner factor	Slowest files
webkit 1/9	80	6.3	12.1	7.6	0.93×	history-loading 68 s, channel-lifecycle 67 s
webkit 2/9	83	8.3	14.4	9.2	0.96×	layout 118 s, navigation-scroll-intent 92 s
webkit 3/9	76	7.7	13.6	8.8	1.01×	navigation-sidebar 209 s, sidebar-unread 93 s
webkit 4/9	76	6.9	12.5	8.1	0.91×	message-navigation 221 s, buzz-links 103 s
webkit 5/9	81	9.1	16.9	10.2	1.21×	nested-replies 355 s, startup 128 s
webkit 6/9	69	7.9	13.8	9.2	0.89×	navigation-groups 230 s, agent-activity 136 s
webkit 7/9	93	4.2	7.3	5.5	0.97×	typeahead 65 s, profiles 56 s
webkit 8/9	60	6.3	11.2	7.2	1.08×	channel-members-focus 112 s, messages 104 s
webkit 9/9	77	6.2	10.1	7.2	1.09×	workflows.journey.mjs 153 s, layout 91 s

• WebKit: jobs take 5.5–10.2 min, 73 runner-min in total. The slowest, 5/9, runs on a 1.21× runner and still finishes 4.8 min inside the limit.
• Chromium: jobs take 4.6–9.8 min, 58 runner-min in total.
• Test failures in webkit 2/9 are not timeouts. inbox-edit is fixed by test(inbox): open the edit journey's conversation by name #997. me-workspace "Me sidebar shares working activity…" is a clock-pause flake, fixed on main by test: fix six cross-branch CI flakes #1002.

Local measurement

Environment: macOS on Apple M4 Max (16 cores), native Playwright WebKit 26.6 (v2359), fixture prebuilt with cargo build --locked -p buzzodz-plugins --example fixture-bridge, shared machine (load average 25–35). Command, 2 workers:

PLAYWRIGHT_JSON_OUTPUT_FILE=<dir>/ci-report.json ./bin/node scripts/ci-test-report.mjs kind=playwright report=<dir>/ci-report.json evidence=<dir>/ci-timing.json title="Local WebKit <N/M>" -- ./bin/pnpm test:browser:ci --project webkit --no-deps --shard=<N/M> --reporter=list,json

Shard	Tests	Result	Elapsed wall	Summed test execution	Slowest files
4/6 (heaviest on CI at 6 shards)	115	passed	307 s	540 s	navigation-groups 85 s, unread 79 s, startup 72 s
3/9 (modeled heaviest at 9 shards)	84	passed	180 s	323 s	presence 58 s, navigation-sidebar 54 s, navigation-scroll-intent 37 s

Local macOS WebKit runs these shards 1.8–2.7× faster than hosted Linux WebKit, with a different per-file profile. These numbers show the direction of the change, not CI headroom.

Tests

• No browser cases added, removed, moved or edited.
• tests/integration/browser-ci.test.mjs expands the workflow's literal engine × shard matrix through the real command. With this change, 18 selections (Chromium 65–87 tests each, WebKit 60–94) plus 4 macOS recording tests cover full local functional discovery exactly once.
• Fail-then-pass: the guard fails with the list [1, …, 8] and /9, and passes with [1, …, 9].

Readiness

• Agent review (no blockers)
• test(voice-notes): keep each attachment's own type in the upload fixture #999 and test(inbox): open the edit journey's conversation by name #997 merged first
• Hosted CI green on the stacked head
• Human check of the hosted job summaries

Known gaps

• The expected timings are a model. Runners at 1.32–1.44× exist, and on them the slowest WebKit job reaches ~13 min.
• The suite grows: 686 → 696 WebKit tests in ~13 h. Count-based shards stay lumpy, so revisit when a shard's job on main passes ~12 min.
• Moving files between shards changes which file runs first on a cold worker, so other order-dependent setups can surface.
• Out of scope: the Chromium 3/6 timeout in 38291174623 is a 9.3-min Hermit ./bin/node --version setup stall.