feat(test-runner): add --shuffle to run tests in random order

[Pavel Feldman (pavelfeldman)]

Summary

• npx playwright test --shuffle [seed] randomizes the order in which test groups are dispatched: files, and individual tests in parallel mode. Serial suites and phase/dependency ordering are preserved.
• The seed is printed in the "Running N tests" line and recorded in metadata.shuffleSeed; pass it back to reproduce the order.

Fixes #9297
Fixes #7040
Fixes #12512

[github-actions]

Test results for "tests 1"

8 flaky

⚠️ [installation tests] › playwright-test-package-managers.spec.ts:63 › pnpm: @playwright/test should work `@package-installations-macos-latest`
⚠️ [chromium-library] › library/chromium/chromium.spec.ts:179 › serviceWorker(), and fromServiceWorker() work `@chromium-ubuntu-22.04-arm-node20`
⚠️ [chromium-library] › library/video.spec.ts:725 › screencast › should capture full viewport on hidpi `@chromium-ubuntu-22.04-arm-node20`
⚠️ [chromium-library] › library/browsercontext-page-event.spec.ts:173 › should work with Ctrl-clicking `@realtime-time-library-chromium-linux`
⚠️ [chromium-library] › library/video.spec.ts:725 › screencast › should capture full viewport on hidpi `@chromium-ubuntu-22.04-node24`
⚠️ [chromium-library] › library/chromium/chromium.spec.ts:436 › should produce network events, routing, and annotations for Service Worker (advanced) `@chromium-ubuntu-22.04-node20`
⚠️ [firefox-library] › library/browsercontext-cookies-third-party.spec.ts:257 › third party 'Partitioned;' cookies `@firefox-ubuntu-22.04-node20`
⚠️ [firefox-library] › library/browsercontext-cookies-third-party.spec.ts:470 › top level 'Partitioned;' cookie and same origin iframe `@firefox-ubuntu-22.04-node20`

52096 passed, 1241 skipped

---

Merge workflow run.

[github-actions]

Test results for "MCP"

3 failed
❌ [chromium] › mcp/cli-webmcp.spec.ts:162 › webmcp-call disambiguates duplicate tool names by frame @mcp-ubuntu-latest-chromium
❌ [chromium] › mcp/sse.spec.ts:196 › sse transport browser lifecycle (persistent) @mcp-ubuntu-latest-chromium
❌ [firefox] › mcp/annotate.spec.ts:446 › should switch screencast to -s session on show --annotate @mcp-windows-latest-firefox

8702 passed, 1474 skipped

---

Merge workflow run.

[github-actions]

Hi, I'm the Playwright bot and I took a look at the red CI here.

🟢 All three MCP failures are pre-existing flakes — the PR is clear

Each one has a history of failing on other SHAs and PRs in the aggregated CI results, and none of them touch the code this PR changes: --shuffle is opt-in, so shuffleTestGroups() and the shuffleSeed metadata are never reached on a CI run that doesn't pass the flag.

Details

Pre-existing flake / infra

• [firefox] › mcp/annotate.spec.ts:446 › should switch screencast to -s session on show --annotate @mcp-windows-latest-firefox — a well-established flake on this bot specifically: failed 79 of 699 runs (11%), passing the other 620, on pushes to main and on unrelated PRs (fix(html-reporter): show the remaining test count in the speedboard "Show more" button #42856, docs(triage): also check previous similar issues and related PRs #42844, fix(expect): report target close reason instead of internal protocol error #42861, fix(mcp): rebind page-registered WebMCP tools when the tab or frame changes #42821, fix(clock): make Temporal.Now follow page.clock #42838, …). Every other bot running the same test is at 0–2 failures out of ~715, so it's the Windows/Firefox screencast timing, not the test's subject.

• [chromium] › mcp/cli-webmcp.spec.ts:162 › webmcp-call disambiguates duplicate tool names by frame @mcp-ubuntu-latest-chromium — flaky since it landed: 8 failures across ~95 runs per bot, spread over chromium and chrome on ubuntu and macOS, on main pushes (4afcfb17, 724115f5, 0a2a33c4) and unrelated PRs (fix(selectors): reset regex state before matching each element #42818, fix(mcp): rebind page-registered WebMCP tools when the tab or frame changes #42821, feat(content): add includeShadow option to page.content() #42845, fix(firefox): preserve response header field boundaries #42832, fix(network): resolve response.finished() when the request fails after the response #42787, fix(skills): narrow npm and npx grants #42789).

• [chromium] › mcp/sse.spec.ts:196 › sse transport browser lifecycle (persistent) @mcp-ubuntu-latest-chromium — rarer, but the same test failed on the same bot on a main push at 22dc1227 (run 35785203077), plus ubuntu/webkit at 3847b985 and msedge on feat(firefox): roll Firefox to M155 #42445. ~4 failures across ~8700 runs, all on SHAs this PR can't be responsible for.

Why the diff can't reach them

The runner change is gated on configCLIOverrides.shuffle, which is only set by the new --shuffle flag; CI doesn't pass it, so test group ordering and the reporter's "Running N tests" line are byte-identical to main. The rest of the diff is docs, the CLI option declaration, and a new tests/playwright-test/shuffle.spec.ts — nothing the MCP suite loads.

_{Triaged by the Playwright bot - agent run}

[Yury Semikhatsky (yury-s)]

Don't forget to add it to api review items.

[Yury Semikhatsky (yury-s)]

Tests with beforeAll/afterAll can be placed in the same group and won't be shuffled with this approach. Maybe it's okay but perhaps you want to shuffle them too?