Repository navigation
test: allow 20 seconds per test for slow CI machines - #340
Merged
Merged
Conversation
Most tests run git and the CLI several times. On a slow CI runner they take about twice as long as usual, and one went over vitest's 5 second default. Raise the default to 20 seconds instead of patching that test: others were close behind it. The signal tests' own 20 second timeout is now the default and goes away.
The timeout comment claimed most tests run the CLI and git, and gave a measurement that will not stay true: many tests run git several times, which is the reason. The 30 second tests that did not say why now do. The doc comment of git() moves back above it, from above gitSetting().
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Raises vitest's default test timeout from 5 to 20 seconds, so tests stop failing on slow CI runners for reasons that have nothing to do with the code.
Context. configfile is a CLI that deploys dotfiles from a git repository. Many of its tests are end-to-end: each creates a sandbox with a git repository standing in for the remote and clones of it, then runs the CLI a few times. The CLI runs inside the test process, but it runs git, and so do the tests to set things up: one test easily starts a dozen git processes.
What failed. On #339, a docs-only change, one macOS job failed:
tests/repository-hooks.test.ts› "says after adopt when the sync brought changes the post_update script did not see" took 5323 ms, over vitest's 5000 ms default. Rerunning the job passed.Why the config, not that test. In that run, the whole machine was slow: tests across the suite took about twice their usual time. In the last four green runs, the slowest test relying on the default timeout took about 2.8 s. In the failing run, several
tests/adopt.test.tstests took 3 to 4 s (up to 4.15 s), close behind the one that failed. Raising the timeout of that one test would only move the failure to the next.The change.
vitest.config.ts:testTimeout: 20_000, with a comment saying why.tests/e2e.test.ts: the "signals during scripts run" block drops its own{ timeout: 20_000 }, which is now the default.tests/e2e.test.ts,tests/lock.test.ts: the tests that start 8 or 16 processes at once keep their 30 seconds. Four of them did not say why; they now do, like the others ("Eight processes: allow for slow, busy CI machines.").tests/helpers.ts: the doc comment ofgit()sat abovegitSetting(), which has its own, so it documented nothing. It moves back abovegit(). Found during review; no behavior change.Trade-off. A test that hangs now fails after 20 seconds instead of 5. It is still reported as a timeout; the suite takes about 30 seconds locally and 2 minutes on CI.
No change to configfile itself, so no CHANGELOG entry.
Other Informations
Review order:
vitest.config.ts, thentests/e2e.test.tsandtests/lock.test.ts(comments and timeouts only), thentests/helpers.ts.Checking the new default. From a clone, after
pnpm install, a test that takes 6 seconds passes on this branch and times out ondevelop:Tested:
pnpm lint,pnpm typecheckandpnpm test(811 tests) pass locally on macOS.