Repository navigation
Port the doc-style checker to Rust - #8
Closed
MirzaMerdovic wants to merge 25 commits into
Closed
MirzaMerdovic wants to merge 25 commits into
MirzaMerdovic wants to merge 25 commits into
Conversation
The baseline holds the mean time of the checker on one fixture, the mean time of the test suite, and the processor count. The Rust port measurements go next to it.
The crate ports the shared parts of the bash checker to Rust: the command line, the exit status, Markdown and comment stripping, inline directives, the .doc-style file, and the finding format. The rule table is empty. Each rule follows in its own commit. A file that exists and cannot be read is an error with exit status 2. The bash checker prints the read error and exits 0 for that file.
The expression is the idiom entry of PATTERNS in the bash checker, unchanged. The fixtures under tests/ need three more rules, because the repository .doc-style file names them. A copy of tests/fail/idiom.md outside the repository matches idiom.expected.
The expression is the qualifier entry of PATTERNS in the bash checker, unchanged. A copy of tests/fail/qualifier.md outside the repository matches qualifier.expected.
The expression is the filler entry of PATTERNS in the bash checker, unchanged. A copy of tests/fail/filler.md outside the repository matches filler.expected.
The expression is the anthropomorphism entry of PATTERNS in the bash checker, unchanged. A copy of tests/fail/anthropomorphism.md outside the repository matches anthropomorphism.expected.
The expression is the hype entry of PATTERNS in the bash checker, unchanged. With this rule, the repository .doc-style file names only known rules, and the fixture tests under tests/ run. The pass fixtures and the hype fixture pass.
The expression is the tone entry of PATTERNS in the bash checker, unchanged. The source code exemptions for a task marker line and a trailing ? are part of the crate since its first commit.
The expression is the chain entry of PATTERNS in the bash checker, unchanged. Every pattern rule is now ported.
The lines of a paragraph are joined and cut at each sentence terminator, as in the long_sentences awk program of the bash checker. A finding reports the line where the sentence starts. Every bats test passes against the release binary.
GNU grep, gawk, sed, and bash read [[:space:]] with the glibc class of the UTF-8 locale. That class holds the Unicode spaces U+1680, U+2000 to U+2006, U+2008 to U+200A, U+2028, U+2029, U+205F, and U+3000. It does not hold the no-break spaces. The Rust [[:space:]] class is ASCII only, and the Rust \s class holds the no-break spaces. Every expression now compiles with the glibc class, and is_space uses it. A snippet then drops leading Unicode spaces, and a Unicode space no longer counts as a word.
The tests call markdown() and sentences() directly. They cover front matter, fences, indented code, inline code, HTML comments, directives, sentence terminators, wrapped lines, list markers, and the lines that end a paragraph. The bats fixtures stay the conformance tests.
The conformance test runs tests/run with CHECK set to the binary of the crate. cargo test then runs the bats suite, and cargo mutants tests each mutant against the fixtures. The test needs bats-core on PATH. .cargo/mutants.toml excludes main.rs. It copies the .git entry into the build directory, because the bats suite runs git grep.
Paragraph::add appended start lines in a while loop. The cargo-mutants mutant that replaces <= with > in the loop condition appends without end. A unit test then grew to 20 GB before the mutant timeout, and the kernel stopped the session. resize has no mutant that grows the list without bound.
The workflow runs on a pull request that changes the crate or one of its build files: crates/**, the workspace manifest, the lock file, the toolchain file, .cargo/, and the workflow. It runs cargo fmt, cargo clippy, and a release build, then tests/run against the release binary, then cargo mutants with one job per processor. nproc has its own step, and the mutation step prints its duration. dtolnay/rust-toolchain installs the toolchain, because mise.toml does not hold Rust. taiki-e/install-action installs a pinned cargo-mutants binary. mise-action installs bats-core.
The Rust port section holds the three measurements of the baseline, taken with the release binary, and the ratio of each. The mutation testing section holds the cargo-mutants totals and the wall-clock time.
c_block_start and trailing iterate over str::match_indices, and find iterates over Path::ancestors. Before this change, a cargo-mutants mutant of the index arithmetic in these loops made a loop run without end, and cargo-mutants reported a timeout. The block comment search and the trailing comment search now give one list of comment starts, and min_by_key selects the first. The output does not change. The bats suite passes, and the previous binary and this binary give the same output on 8000 local files.
glob_match and bracket match slice patterns and recurse on the rest of the pattern, and in_set reads the members of a bracket expression. No loop depends on index arithmetic, so no cargo-mutants mutant makes the match run without end. The new matcher and the previous matcher give the same result on 139,745 generated pairs of pattern and path. Unit tests cover *, ?, bracket expressions, negation, an unterminated bracket, and the backslash. Each expected result comes from bash.
The tests cover these cases: - The -h option. - The tone exemption in source code. - The compound join. - An empty list item. - A whitespace line in an indented block. - The order of comment starts on a line. - The single-quote count before a trailing comment. Each expected value comes from the bash checker or from its strip_code and sed steps. cargo mutants now reports 194 mutants: 185 caught, 9 unviable, no missed mutants, and no timeouts.
The table now holds the run at commit b6dcc29, and each section names the commit that it measured.
fail.rs, pass.rs, and fixtures.rs hold the tests of fail.bats, pass.bats, and fixtures.bats. Each test runs the checker as a separate process and reads the existing fixtures and .expected files. The CHECK environment variable selects the checker, as in the bats suite, so the same tests also run the bash checker. A fail test compares stdout with the .expected file byte for byte and requires an empty stderr. The bats suite still runs, and the conformance test still runs it.
cli.rs, config.rs, and suppress.rs hold the tests of cli.bats, config.bats, and suppress.bats. A test that needs a git repository uses Repo, a temporary directory from the tempfile crate with a .git directory. Each test runs against the Rust binary, and against the bash checker when CHECK names it.
The Rust integration tests now cover the checker tests of the bats suite, so cargo test and cargo mutants no longer run bats. cargo mutants reports no missed mutants without it. A mutant now takes about 0.7 s to test instead of about 11 s. The bats suite still runs in CI against the release binary. The copy of the tree for cargo mutants no longer needs the .git entry, because no test runs git.
A pull request runs cargo mutants with --in-diff, on the diff between its merge commit and the base branch. A push to main runs every mutant. The checkout fetches two commits, so that the first parent of the merge commit is available. A pull request that changes only tests or the workflow tests no mutants. The full run on main is then the first run that detects a test change that removes coverage.
Port the checker tests from bats to Rust
Contributor
Author
|
Superseded by #10, which removes the checker instead of porting it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This change adds a Rust port of
skills/doc-style/scripts/check. The crate iscrates/check, and the binary isdoc-style-check. The plugin still runs the bashchecker. The bash checker, the bats suite, and
tests/mutateare unchanged.The crate is a library.
main.rsonly passes the arguments torun. The crate depends onthe
regexcrate only, and its tests also use thetempfilecrate. Each file holds onepart of the bash checker:
lib.rsstrip.rspatterns.rsPATTERNSwithout change.length.rsdirective.rsconfig.rs.doc-stylesearch, rules,allowlines, and glob matching.Each rule has its own commit. Some comments compare a Rust idiom with Go.
Behavior
The binary writes the same stdout and stderr and exits with the same status as the bash
checker. Verification:
tests/runpasses 73 of 73 tests with the bash checker and withCHECK="$PWD/target/release/doc-style-check".cases. The only difference is the gawk warning in the list below. The differential test
scripts are not part of this change.
crates/check/testsport the checker tests of the batssuite. They run against the binary, and against the bash checker when
CHECKnames it..doc-styleglob matching, and the behavior that no bats test covers. Each expected value comes from
bash.
In a UTF-8 locale, GNU grep, gawk, sed, and bash read
[[:space:]]with the glibc class.That class includes Unicode spaces such as U+3000. The Rust
[[:space:]]class is ASCIIonly, and the Rust
\sclass also matches no-break spaces such as U+00A0. The cratetherefore replaces both with the glibc class before it compiles an expression.
Known differences from the bash checker:
cannot read FILEand exits 2. The bash checkerprints the shell and awk errors and exits 0.
warning: Invalid multibyte data detectedon stderr. Thebinary prints nothing. The findings are the same.
.doc-stylepatterns: the binary supports*,?,[...], and\. Bash also acceptsextended globs and character classes such as
[[:digit:]]in a[[ == ]]pattern..doc-stylepatterns with a backslash: bash and the binary give different results in 47of 139,745 generated pairs of pattern and path. In 41 pairs, the pattern holds a
*andends with an unpaired
\. In 6 pairs, a\is inside a bracket expression, where bashreads it as an escape.
LC_ALL=C, the length snippet iscut at 56 bytes instead of 56 characters. The binary output does not depend on the
locale. It matches the bash output in a UTF-8 locale.
Measurements
BENCH.mdrecords the baseline and the Rust measurements:tests/runThe ratio increases with the number of files in one call.
BENCH.mddoes not hold thesenumbers, because the file sets are local Markdown files:
The bash checker takes about 34 ms per file. The binary takes about 4 ms to compile the
expressions, and then about 0.26 ms per file.
CI
.github/workflows/rust.ymlruns on a pull request and on a push tomain. It runs onlywhen the crate, the Cargo files, the toolchain file, the cargo-mutants configuration, or
the workflow changes. The steps are
cargo fmt --check,cargo clippy -- -D warnings, arelease build,
tests/runagainst the binary,nproc, andcargo mutants.--in-diff. Apush to
maintests every mutant.cargo testruns the unit tests and the Rust integration tests, and not the bats suite.The bats suite runs in its own step against the release binary.
36b9666tested all 194 mutants in 228 s: 185caught, 9 unviable, no missed mutants, and no timeouts.
tests/mutatehas 76 behavior mutants, and 75 of them have a Rust equivalent. The otherone changes the cut marker of the sentence splitter, and the Rust splitter uses no cut
marker. The bats suite detects each of the 75 equivalents.
Not in this change
default macOS install. The crate passes
cargo check --target aarch64-apple-darwin. Itwas not built or run on macOS.