diff --git a/src/pages/forge/testing.mdx b/src/pages/forge/testing.mdx index 1c0cfe9b5..2367a818c 100644 --- a/src/pages/forge/testing.mdx +++ b/src/pages/forge/testing.mdx @@ -298,6 +298,174 @@ function tableSwapTest(Wallet memory wallet, Swap memory swap) public pure { The same naming requirement mentioned above is relevant here. +### Mutation testing + +Mutation testing checks the strength of your test suite by making small changes, or mutants, to your source code and re-running your tests. A mutant is killed when at least one test fails. A mutant survives when the changed code still passes the selected tests. + +:::info +Mutation testing is currently an MVP. It is ready for early use and feedback, but the workflow, reporting, and supported project configurations are still expected to evolve. +::: + +Run mutation testing with `forge test --mutate`: + +```bash +$ forge test --mutate +``` + +Forge first runs the selected tests as a baseline. Mutation testing only starts if the baseline has at least one passing test and no failing tests. + +#### Selecting files + +Pass paths to mutate only those files: + +```bash +$ forge test --mutate src/Vault.sol src/Token.sol +``` + +Use `--mutate-path` to select files with a glob pattern: + +```bash +$ forge test --mutate --mutate-path 'src/**/*.sol' +``` + +Use `--mutate-contract` to select contracts by name: + +```bash +$ forge test --mutate --mutate-contract 'Vault|Token' +``` + +`--mutate-path` and `--mutate-contract` cannot be combined. `--mutate-path` also cannot be combined with explicit paths passed to `--mutate`. + +#### Selecting tests + +Regular test filters still select the baseline tests and the tests run against each mutant: + +```bash +$ forge test --mutate src/Vault.sol --match-contract VaultTest +``` + +This lets you scope a mutation run to the tests that should detect changes in a specific contract. + +#### Parallel workers + +Forge runs mutants in parallel. By default, it uses the number of logical CPU cores. + +Set the worker count with `--mutation-jobs`: + +```bash +$ forge test --mutate src/Vault.sol --mutation-jobs 4 +``` + +Passing `0` also uses the number of logical CPU cores: + +```bash +$ forge test --mutate src/Vault.sol --mutation-jobs 0 +``` + +Parallel mutation testing uses isolated temporary workspaces per mutant. Dependency directories such as `lib`, `node_modules`, and `dependencies` are symlinked into those workspaces for performance. + +#### Timeouts + +Use `--mutation-timeout` to set a best-effort wall-clock timeout, in seconds, for each mutant: + +```bash +$ forge test --mutate src/Vault.sol --mutation-timeout 30 +``` + +Timed-out mutants are reported separately from killed, survived, skipped, and invalid mutants. + +You can also configure the timeout in `foundry.toml`: + +```toml [foundry.toml] +[mutation] +timeout = 30 +``` + +#### Operators + +Mutation testing supports these operator groups: + +- `assembly` +- `assignment` +- `binary-op` +- `delete-expression` +- `elim-delegate` +- `require` +- `unary-op` + +All operator groups are enabled by default. Exclude specific operators in `foundry.toml`: + +```toml [foundry.toml] +[mutation] +exclude_operators = ["assembly", "elim-delegate"] +``` + +Use `include_operators` to re-enable operators that are excluded by default: + +```toml [foundry.toml] +[mutation] +include_operators = ["assembly"] +``` + +#### Reports + +The report includes counts for: + +- **Survived**: mutants that passed the selected tests +- **Killed**: mutants that caused a test failure +- **Invalid**: mutants that could not be compiled or run +- **Skipped**: redundant mutants on a span or expression after another mutant in that span survived +- **Timed out**: mutants that exceeded `mutation.timeout` or `--mutation-timeout` + +Skipped and invalid counts can vary with `--mutation-jobs`, because higher parallelism can start more mutants before a survivor is known. + +The mutation score is: + +```text +killed / (killed + survived) +``` + +Focus on survived mutants first. Each survived mutant points to the source location and mutation that your tests did not catch. + +Survived mutants do not currently make `forge test --mutate` fail, and there is no threshold flag yet. To gate mutation testing in CI, run with `--json` and enforce your own threshold from the JSON output. + +```bash +$ forge test --mutate --json +``` + +The JSON output has this shape: + +```json +{ + "summary": { + "total": 12, + "killed": 8, + "survived": 2, + "invalid": 1, + "skipped": 1, + "timed_out": 0, + "mutation_score": 80.0, + "duration_secs": 12.34 + }, + "survived_mutants": { + "src/Vault.sol": [ + { + "line": 42, + "column": 17, + "original": ">", + "mutant": ">=" + } + ] + } +} +``` + +#### Limitations + +Mutation testing cannot be combined with `--list`, `--debug`, `--flamegraph`, `--flamechart`, `--junit`, `--dump`, `--showmap`, or `--showmap-out`. + +Mutation testing also rejects projects with `ffi = true`, write-capable file-system permissions that can reach symlinked dependency directories, or inline per-test network overrides. + ### Testing reverts Use `vm.expectRevert(){:solidity}` to test that a call reverts: diff --git a/src/pages/reference/forge/test.mdx b/src/pages/reference/forge/test.mdx index 1235b36f6..182fe5117 100644 --- a/src/pages/reference/forge/test.mdx +++ b/src/pages/reference/forge/test.mdx @@ -145,6 +145,34 @@ Test options: [env: FORGE_ALLOW_FAILURE=] + --mutate [...] + Enable mutation testing. + + If passed with file paths, only those files will be tested + + --mutate-path + Specify which files to mutate with glob pattern matching. + + Mutually exclusive with passing explicit paths to `--mutate`; either + supply paths to `--mutate` or use this glob filter, not both + + --mutate-contract + Only mutate contracts whose name matches the specified regex pattern. + + Mutually exclusive with `--mutate-path` + + --mutation-jobs + Number of parallel workers for mutation testing. + + Defaults to the number of CPU cores + + --mutation-timeout + Best-effort per-mutant wall-clock timeout in seconds. Mutants that + exceed it are recorded as "timed out" and cleanup continues in the + background with bounded pending workers. + + Analogous to `--invariant-timeout` for invariant campaigns + --trace-depth Defines the depth of a trace @@ -605,4 +633,4 @@ Watch options: overloading disk I/O. ``` -::: \ No newline at end of file +:::