Repository navigation
Port the checker tests from bats to Rust - #9
Merged
Merged
Conversation
fail.rs, pass.rs, and fixtures.rs hold the tests of fail.bats, pass.bats, and fixtures.bats. Each test runs the checker as a separate process and reads the existing fixtures and .expected files. The CHECK environment variable selects the checker, as in the bats suite, so the same tests also run the bash checker. A fail test compares stdout with the .expected file byte for byte and requires an empty stderr. The bats suite still runs, and the conformance test still runs it.
cli.rs, config.rs, and suppress.rs hold the tests of cli.bats, config.bats, and suppress.bats. A test that needs a git repository uses Repo, a temporary directory from the tempfile crate with a .git directory. Each test runs against the Rust binary, and against the bash checker when CHECK names it.
The Rust integration tests now cover the checker tests of the bats suite, so cargo test and cargo mutants no longer run bats. cargo mutants reports no missed mutants without it. A mutant now takes about 0.7 s to test instead of about 11 s. The bats suite still runs in CI against the release binary. The copy of the tree for cargo mutants no longer needs the .git entry, because no test runs git.
A pull request runs cargo mutants with --in-diff, on the diff between its merge commit and the base branch. A push to main runs every mutant. The checkout fetches two commits, so that the first parent of the merge commit is available. A pull request that changes only tests or the workflow tests no mutants. The full run on main is then the first run that detects a test change that removes coverage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This change ports the checker tests of the bats suite to Rust integration tests in
crates/check/tests, and removes the bats suite fromcargo test. The base branch isrust-port(#8).fail.rsfail.batspass.rspass.batsfixtures.rsfixtures.batscli.rscli.batsconfig.rsconfig.batssuppress.rssuppress.batsCHECKselects the checker, as in thebats suite, so the same tests also run the bash checker.
tests/common/mod.rsholds the helpers.Repois a temporary git repository from thetempfilecrate.tempfileis the only new dependency, and only the tests use it..expectedfile byte for byte and requires anempty stderr. The bats test compares stdout and stderr together.
Not changed
tests/run, andtests/mutateare unchanged.tests/mutatestill usesthe bats suite for the bash checker.
repo.batsandsnippets.batstest repository content, not the checker. They stay inbats.
tests/runagainst the release binary.Verification
tests/runpasses 73 of 73 tests with both checkers.cargo mutants --jobs 32found 194 mutants: 185 caught, 9 unviable, no missed mutants,and no timeouts.
Mutation run time
These local runs used 32 jobs. Another process used about 5 processors during the runs,
so the times are approximate.
On 4 processors in CI, the full mutation run took 317 s on #8 and 228 s on the previous
commit of this change.
Mutation check scope
A pull request now runs
cargo mutants --in-diffon the diff between its merge commit andthe base branch. A push to
mainruns every mutant.full run on
mainthen reports the missed mutants.Debug build
cargo testruns the debug build of the binary. Without optimization, the regex cratecompiles the rule expressions slowly. One run on
idiom.mdtook 139 ms with the debugbuild and 10 ms with the release build.
An optimized dev profile for the dependencies did not help cargo mutants. Each of its
build directories then compiles
regexwith optimization, and a local run took 229 sinstead of 136 s. This change keeps the default profile.