Repository navigation
ci: balance the mutation shards by the time their mutants took on main - #468
Merged
Merged
Conversation
Each top-level statement now weighs the tests its mutants ran in the last green report on main, one more per mutant, instead of its syntax node count: a load-time mutant runs every suite, so equal counts left the shard holding most of them half again as long as the rest. A file changed since that report weighs by nodes, scaled to the same unit. Every shard of a run plans from one report a single job chose, writes the ranges it ran beside its report, and the merge reads those and refuses ranges that overlap or leave a statement out.
Shape
CoverageBaseline:
Lines this pull request changed: 65/65 covered (100.00%). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The mutation shards are now balanced by how long their mutants take, not by how many there are. The first full run on the statement split (#467) took 25 minutes against 20 before it: shard 1 ran 80,753 tests where the others ran about 52,000, because it held most of the mutants made when a module loads, and each of those runs every suite.
main, one more per mutant (testsCompleted + 1). Divided by each runner's dry-run speed, that tracks the six shard durations of run 38921533628 closely. A file changed since the report weighs by syntax nodes, scaled to what a node cost in the files the report does hold; a run with no report weighs by nodes alone.INVARIANTSand a sixth of every other file in one shard.mutation-scope(and a newbaselinejob in the weekly run) chooses the last green run once and hands its id to every shard.reports/mutation/ranges.txt. The merge reads those instead of planning again, and refuses ranges that overlap or leave a statement out.To know:
main's report, the six shards run 56,684 to 57,182 tests each, a spread of under 1 percent; by node count it was 1.30 times the mean, and the run just measured 1.45. That should bring the full run to about 15 to 17 minutes. The costliest statement (INVARIANTS, 21,000 tests) is well under a sixth, so six shards can stay level.docs/architecture.mdsay the shares are balanced by time.Follows #464.
Verification
npm run verifypasses, and actionlint is clean.main's report falls in exactly one shard's ranges under the new split.