Skip to content

ci: parallelize test suites and shard Web E2E - #642

Merged
tt-a1i merged 4 commits into
mainfrom
codex/ci-speed
Oct 2, 2026
Merged

tt-a1i merged 4 commits into
mainfrom
codex/ci-speed

Conversation

@tt-a1i

@tt-a1i tt-a1i commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The last successful integration run took about eight minutes: Web E2E took 8m01s and Windows took 7m45s. Node and Vitest run sequentially on every platform, browser cases use one worker, and Windows separately repeats terminal lifecycle coverage already present in its full suite.

Value

Reduce time to receive merge feedback while retaining all existing Node versions, Windows coverage, browser cases, package smoke tests and provider handoff acceptance.

Approach

  • Add all, node, ui, node-parallel and node-serial selectors to the existing test runner. No argument still runs the full suite, and all selectors retain the non-empty discovery guard. The full command keeps Windows terminal files serialized after unrelated Node files. CI runs two disjoint ordinary-file shards (96 + 96 files) and the eight serialized terminal files on three independent Windows VMs, so unrelated processes cannot interfere. Invalid or empty shards fail before execution.
  • Run Node and UI suites in independent jobs on Linux Node 22.19.0/24/26 and Windows Node 22.19.0. Remove the redundant standalone Windows terminal invocation.
  • Split browser acceptance into two isolated runners. Enable case-level sharding while retaining one worker per server: 99 cases divide into 50 + 49, instead of the file-level split of 80 + 19. Run Plan Ready provider handoff on shard 1 and give each shard a unique failure artifact name.
  • Cache Playwright downloads, Chromium system-package downloads and Windows Bun downloads, keyed by OS/architecture/lockfile. Browser runners stay on Ubuntu 24.04; apt still resolves dependencies against signed repositories and cached .deb archives. Bound network retries/timeouts and cap browser provisioning at ten minutes (the first cold run took just over seven minutes). Continue installing Linux system dependencies and using frozen-lockfile installs; never cache node_modules. Linux dependency installation was already about four seconds, so it does not get a download cache step.
  • Preserve existing status-check names with explicit aggregation. Required Node gates include every runtime/UI job, Windows runtime and both browser shards. Failed, skipped or cancelled dependencies fail the gates.
  • Drain the provider-failure test's snapshot mock before disposing its request context, avoiding an asynchronous teardown error exposed during isolated execution. Pin thinking capability in the two model-selection fixtures so their menu does not depend on the real Host or previous cases.

Validation

  • actionlint .github/workflows/ci.yml: passed.
  • bun run check: passed.
  • Local bun run test: passed, 2,124 Node tests (8 skips) and 1,120 Vitest tests across 79 files.
  • Runner regression fixtures verify default/split selection, forwarding, failure propagation, invalid arguments, empty discovery, and isolated Windows group execution. A real-discovery coverage invariant proves the 96 + 96 + 8 file groups are disjoint and equal all 200 Node test files.
  • Playwright list comparison: original 99 cases equal the two disjoint shards, 50 + 49, with no omissions or duplicates.
  • Gate shell validation: success passes; each failed, skipped or cancelled dependency fails.
  • Provider-failure browser regression: 3/3 consecutive isolated executions passed after draining route handlers.
  • Complete local browser shard acceptance passed: shard 1 50/50, shard 2 49/49. Both model-selection cases also pass independently with explicit thinking-capability fixtures.
  • The first CI run passed all jobs, but slow Ubuntu package downloads stretched shard 1 to 10m01s. Later browser jobs passed in around three minutes with Chromium restored. Corrected cache paths after detecting that GitHub rejects parent-relative glob paths; final-head cache save/hit checks and total duration will be recorded below.

Impact

  • User-visible behavior: none; application source and committed Web assets are unchanged.
  • Model-visible context/tools: none.
  • Runtime/lifecycle: production unchanged; test selection and CI scheduling only.
  • Persisted config/data: none.
  • Compatibility/risk: all original platform/version coverage retained. More independent runner jobs increase setup overhead and peak concurrency. Windows process tests remain isolated; each browser runner has its own server/settings/filesystem. Cache hits do not replace dependency installation or test execution. Branch-protection settings are unchanged.

Previous revision CI results

Previous tested head: 833a7300d0184bcc3b423bb455d6d962c5d595fc (before the Windows sharding follow-up).

Run Wall time from first job start through all gates Result
Original integration CI (36884455275) 8m01s Passed
Previous head, cache population (36893319501 attempt 1) 6m24s All 15 jobs passed
Same previous head, cache reuse (attempt 2) 6m36s All 15 jobs passed

These two previous-head observations are about 18–20% shorter than the original run; they are observations, not a guaranteed duration. Windows native Node tests remain the critical path. Browser jobs now take around three minutes instead of eight.

Cache evidence: Playwright, apt archive and Windows Bun restore keys were actually hit. Apt reported Need to get 0 B/32.1 MB of archives; Windows dependency installation completed from the restored download cache and Effect verification succeeded. The corrected paths produce no parent-relative glob rejection. Dependencies still install against the lockfile; apt still installs/checks system packages.

The first optimization run's cold apt download took 6m40s for 32.1 MB and its overall CI took 10m10s, showing why download caching and runner variability matter. All its tests passed; final cache save/reuse was subsequently verified on the frozen head above. The linked Actions runs retain logs and per-step timing evidence.

Windows sharding follow-up

Local check and full suite passed. The follow-up retains all standard hosted runners and existing aggregation gates, adding two Windows VMs (12 test jobs + 5 gates). Windows Node matrix aggregation requires all three groups to succeed. Frozen head 4a31a17b2e4611fa04b8b1547e9a3e3f637e7609 passed all 17 CI jobs in run 36961846334. Wall time from first job start through all aggregation gates was 4m08s, versus the previous revision's 6m36s (~37% shorter) and the original 8m01s (~48% shorter). This is one observed run, not a guaranteed duration. Windows ordinary shards finished in 3m11s / 3m43s and the isolated process group in 2m06s, including setup and cleanup. Required legacy check names remain successful. No additional rerun was needed. Standard hosted runners on this public repository have no compute-minute charge: GitHub runner documentation. The plan concurrency limit still applies.

@tt-a1i
tt-a1i requested a review from a team as a code owner October 1, 2026 16:09
@github-actions github-actions Bot added the area:github GitHub workflows, templates, ownership, or tests label Oct 1, 2026
@tt-a1i
tt-a1i merged commit c7862b8 into main Oct 2, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:github GitHub workflows, templates, ownership, or tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant