Skip to content

CI: cap one bats test at 180s, fit a run's macOS jobs into the 5 that run at once - #1983

Open
fujibee wants to merge 4 commits into
mainfrom
ci/cap-macos-jobs
Open

fujibee wants to merge 4 commits into
mainfrom
ci/cap-macos-jobs

Conversation

@fujibee

@fujibee fujibee commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

What

Two changes to tests.yml so a run stops waiting on things that are not tests:

  1. One bats test is capped at 180 s (BATS_TEST_TIMEOUT=180 on the shard job; bats 1.13, the version npm install -g bats installs, supports it). A single hung test used to hold its shard until the 30-minute job cap: test_remote Add event log schema to SQLite driver with backward-compat reads #54 sat for 30 minutes, and the rerun on top took the run to 65-87 minutes. Now the one test fails after 180 s, is named in the log, and the shard goes on. No job cap is raised.
  2. A run asks for at most 5 long macOS jobs. (app typecheck is also a macOS job but lasts ~10 s when nothing in app/ changed; it is not one of the 5.) GitHub runs 5 macOS jobs at a time for this account. Run 43816536201 asked for 7 (6 bats shards plus the macOS storage contract; app typecheck also takes a macOS slot for ~10 s): five started, bats 4/6 and the storage leg waited 17-19 minutes for a slot, and the run took two waves.
    • macOS bats: 6 → 5 shards. Ubuntu: 6 → 10 shards (Linux has no such limit).
    • The macOS leg of the jsonl storage contract is now the last step of macOS shard 5 instead of a job of its own (it took ~2 minutes). The Ubuntu leg stays a job.
    • Per-OS totals live in the matrix (total), and the aggregate bats job re-checks coverage for each (SHARD_TOTALS: '10 5').

The storage step keeps the gate its job had: contracts_needed alone, not a reused tree (a tree a PR run recorded green without running the contract is pushed with contracts_needed=true, and the reuse must not skip it), and bats is installed for it even when the suite itself is skipped.

Required checks

Branch protection requires bats (the aggregate) and bats (windows-latest, install helpers). Neither is renamed. Shard check names change from bats (<os> n/6) to bats (<os> n/<total>), and storage contract (jsonl, macos-latest) goes away; none of those is a required context (read from the branch protection settings).

Planned shard seconds (re-measured table, macOS seconds)

  • macOS 5 shards: 1414 / 1414 / 1457 / 1413 / 1413 s
  • Ubuntu 10 shards: 685 / 685 / 729 / 729 / 729 / 729 / 729 / 684 / 728 / 684 s
  • Before, 6 shards: 1186 / 1185 / 1185 / 1185 / 1185 / 1185 s

Shard 5 also carries the storage contract (~2 min plus setup). Five macOS shards are each ~19% heavier than six were, but all five start at once; six shards plus the storage leg took two waves. The 30-minute shard cap is unchanged, so the macOS margin is thinner (see below).

Before / after (job durations, minutes)

Before: run 43816536201 (macOS shards 19-20 min, 4/6 started 18 min late and finished at 26 min; whole run 26 min). After: this PR's run (filled in below).

Longest single test (measured)

From the per-test timestamps of run 43816536201 (macOS 2932 tests, Ubuntu 3403): longest macOS test 59.9 s (test_remote_sync #54, 10,000-path roster sweep), then 54.3 s (test_agmsgd_daemon_sh #15), 53.5 s (test_install #106), 46.9 s (test_watch #1), 42.1 s (test_codex_bridge_launcher #8). Longest Ubuntu test 39.7 s. 59.9 s x 1.5 = 90 s, under the 180 s cap, so no passing test is expected to hit it; no test needs splitting for this.

Risk

macOS runner speed varied by up to 1.35x between runs in earlier measurements, so a 1414 s plan can reach ~32 minutes on a slow runner, past the 30-minute cap. The 180 s per-test cap removes the hang case, not slowness. If a macOS shard hits the cap in practice, the fix is to run fewer macOS tests per PR (next change), not a longer cap.

Process spawns

No per-test or per-tick spawns are added. One extra step on macOS shard 5 (bats on one file, already run by the old job). One bats timeout watcher per test is bats' own (a background sleep).

Tests

tests/test_ci_sharding.bats now checks the per-OS totals against the matrix, the excluded macOS shards, the 5-job macOS limit and SHARD_TOTALS; tests/test_ci_workflow.bats follows the new shard name and the new step. The four check-*.sh gates and the private-name check pass.

…_TEST_TIMEOUT does not stall them

bats 1.13 runs its timeout watcher as a child of the test shell, so a bare
wait in a test body waits for the watcher and holds the test until the limit.
The storage fan-outs and the sync autostart race now wait on the PIDs they
started, and the nested bats run in test_spawn no longer inherits the timeout.
…s timeout watcher

A test that defines kill() as a no-op leaves it in the shell bats runs the test
in, and bats uses that same kill to stop its BATS_TEST_TIMEOUT watcher at the
end of the test. The watcher then ran its whole 180 s, so files that stub kill
(teardown_reap, instance_id, resolve_project, delivery, ...) were held for
minutes and pushed macOS shards past the 30 minute cap. teardown_test_env now
drops those function stubs first, and the teardown_reap stubs live in a
subshell.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant