Skip to content

Port the checker tests from bats to Rust - #9

Merged
MirzaMerdovic merged 4 commits into
rust-portfrom
rust-tests
Oct 9, 2026
Merged

MirzaMerdovic merged 4 commits into
rust-portfrom
rust-tests

Conversation

@MirzaMerdovic

@MirzaMerdovic MirzaMerdovic commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This change ports the checker tests of the bats suite to Rust integration tests in
crates/check/tests, and removes the bats suite from cargo test. The base branch is
rust-port (#8).

Rust file bats file Tests
fail.rs fail.bats 23
pass.rs pass.bats 17
fixtures.rs fixtures.bats 2
cli.rs cli.bats 7
config.rs config.bats 15
suppress.rs suppress.bats 4
  • Each test runs the checker as a separate process. CHECK selects the checker, as in the
    bats suite, so the same tests also run the bash checker.
  • tests/common/mod.rs holds the helpers. Repo is a temporary git repository from the
    tempfile crate. tempfile is the only new dependency, and only the tests use it.
  • A fail test compares stdout with the .expected file byte for byte and requires an
    empty stderr. The bats test compares stdout and stderr together.

Not changed

  • The bats suite, tests/run, and tests/mutate are unchanged. tests/mutate still uses
    the bats suite for the bash checker.
  • repo.bats and snippets.bats test repository content, not the checker. They stay in
    bats.
  • The Rust workflow still runs tests/run against the release binary.

Verification

  • The 68 Rust integration tests pass against the Rust binary and against the bash checker.
  • tests/run passes 73 of 73 tests with both checkers.
  • cargo mutants --jobs 32 found 194 mutants: 185 caught, 9 unviable, no missed mutants,
    and no timeouts.

Mutation run time

These local runs used 32 jobs. Another process used about 5 processors during the runs,
so the times are approximate.

Tests for each mutant Wall-clock time Mean test time per mutant
Unit tests and the bats suite 135.8 s 11.0 s
Unit tests and the Rust integration tests 85.3 s 0.7 s

On 4 processors in CI, the full mutation run took 317 s on #8 and 228 s on the previous
commit of this change.

Mutation check scope

A pull request now runs cargo mutants --in-diff on the diff between its merge commit and
the base branch. A push to main runs every mutant.

  • A pull request that changes no source line tests no mutants. This pull request is one.
  • A local run on a diff that changes one line tested 5 mutants with 4 jobs in 6 s.
  • A pull request that changes only tests can remove coverage without a failed check. The
    full run on main then reports the missed mutants.

Debug build

cargo test runs the debug build of the binary. Without optimization, the regex crate
compiles the rule expressions slowly. One run on idiom.md took 139 ms with the debug
build and 10 ms with the release build.

An optimized dev profile for the dependencies did not help cargo mutants. Each of its
build directories then compiles regex with optimization, and a local run took 229 s
instead of 136 s. This change keeps the default profile.

fail.rs, pass.rs, and fixtures.rs hold the tests of fail.bats, pass.bats, and
fixtures.bats. Each test runs the checker as a separate process and reads the existing
fixtures and .expected files. The CHECK environment variable selects the checker, as in
the bats suite, so the same tests also run the bash checker.

A fail test compares stdout with the .expected file byte for byte and requires an empty
stderr. The bats suite still runs, and the conformance test still runs it.
cli.rs, config.rs, and suppress.rs hold the tests of cli.bats, config.bats, and
suppress.bats. A test that needs a git repository uses Repo, a temporary directory from
the tempfile crate with a .git directory. Each test runs against the Rust binary, and
against the bash checker when CHECK names it.
The Rust integration tests now cover the checker tests of the bats suite, so cargo test
and cargo mutants no longer run bats. cargo mutants reports no missed mutants without it.
A mutant now takes about 0.7 s to test instead of about 11 s.

The bats suite still runs in CI against the release binary. The copy of the tree for
cargo mutants no longer needs the .git entry, because no test runs git.
A pull request runs cargo mutants with --in-diff, on the diff between its merge commit and
the base branch. A push to main runs every mutant. The checkout fetches two commits, so
that the first parent of the merge commit is available.

A pull request that changes only tests or the workflow tests no mutants. The full run on
main is then the first run that detects a test change that removes coverage.
@MirzaMerdovic
MirzaMerdovic merged commit 5542541 into rust-port Oct 9, 2026
2 checks passed
@MirzaMerdovic
MirzaMerdovic deleted the rust-tests branch October 9, 2026 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant