Skip to content

ci: balance the mutation shards by the time their mutants took on main - #468

Merged
ExtraToast merged 1 commit into
mainfrom
ci/mutation-shard-by-cost
Oct 16, 2026
Merged

ExtraToast merged 1 commit into
mainfrom
ci/mutation-shard-by-cost

Conversation

@ExtraToast

Copy link
Copy Markdown
Contributor

Summary

The mutation shards are now balanced by how long their mutants take, not by how many there are. The first full run on the statement split (#467) took 25 minutes against 20 before it: shard 1 ran 80,753 tests where the others ran about 52,000, because it held most of the mutants made when a module loads, and each of those runs every suite.

  • Time weights. A top-level statement weighs the tests its mutants ran in the last green report on main, one more per mutant (testsCompleted + 1). Divided by each runner's dry-run speed, that tracks the six shard durations of run 38921533628 closely. A file changed since the report weighs by syntax nodes, scaled to what a node cost in the files the report does hold; a run with no report weighs by nodes alone.
  • Whole-tree balance. The planner now hands the heaviest statement of the whole tree to the least-loaded shard, rather than splitting each file evenly first. Per-file splitting is what put INVARIANTS and a sixth of every other file in one shard.
  • One report per run. Every shard of a run must plan from the same report, or their shares overlap. mutation-scope (and a new baseline job in the weekly run) chooses the last green run once and hands its id to every shard.
  • The merge reads what ran. Each shard writes its ranges to reports/mutation/ranges.txt. The merge reads those instead of planning again, and refuses ranges that overlap or leave a statement out.

To know:

  • Projected on main's report, the six shards run 56,684 to 57,182 tests each, a spread of under 1 percent; by node count it was 1.30 times the mean, and the run just measured 1.45. That should bring the full run to about 15 to 17 minutes. The costliest statement (INVARIANTS, 21,000 tests) is well under a sixth, so six shards can stay level.
  • ADR 0063 and docs/architecture.md say the shares are balanced by time.

Follows #464.

Verification

  • npm run verify passes, and actionlint is clean.
  • New tests cover time weights from a report, the scaled fallback for a changed file, the whole-tree balance, writing and reading ranges, and the merge refusing a gap, an overlap and ranges that do not parse.
  • Every mutant of main's report falls in exactly one shard's ranges under the new split.

Each top-level statement now weighs the tests its mutants ran in the last
green report on main, one more per mutant, instead of its syntax node count:
a load-time mutant runs every suite, so equal counts left the shard holding
most of them half again as long as the rest. A file changed since that
report weighs by nodes, scaled to the same unit.

Every shard of a run plans from one report a single job chose, writes the
ranges it ran beside its report, and the merge reads those and refuses ranges
that overlap or leave a statement out.
@ExtraToast ExtraToast added type: feature New user-facing or operator-facing capability. area: tooling Reusable workflows, Gradle, templates, Renovate, and API tooling. component: ci Continuous integration workflow or check behavior. priority: P1 High; important and should be handled in the current iteration. labels Oct 16, 2026
@ExtraToast ExtraToast self-assigned this Oct 16, 2026
@github-actions

Copy link
Copy Markdown

Shape

bucket files additions deletions
tests 3 +207 -14
decision records 1 +10 -2
documentation 1 +2 -1
tooling 3 +163 -81
CI 2 +73 -10

Coverage

Baseline: main @ 21339e4.

metric this branch main diff
statements 99.66% 99.66% 0.00
branches 98.19% 98.14% +0.05
functions 100.00% 100.00% 0.00
lines 99.62% 99.62% 0.00

Lines this pull request changed: 65/65 covered (100.00%).

@ExtraToast
ExtraToast enabled auto-merge (squash) October 16, 2026 16:03
@ExtraToast
ExtraToast merged commit ddb367f into main Oct 16, 2026
30 checks passed
@ExtraToast
ExtraToast deleted the ci/mutation-shard-by-cost branch October 16, 2026 16:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: tooling Reusable workflows, Gradle, templates, Renovate, and API tooling. component: ci Continuous integration workflow or check behavior. priority: P1 High; important and should be handled in the current iteration. type: feature New user-facing or operator-facing capability.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant