Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
168 changes: 168 additions & 0 deletions src/pages/forge/testing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -298,6 +298,174 @@ function tableSwapTest(Wallet memory wallet, Swap memory swap) public pure {

The same naming requirement mentioned above is relevant here.

### Mutation testing

Mutation testing checks the strength of your test suite by making small changes, or mutants, to your source code and re-running your tests. A mutant is killed when at least one test fails. A mutant survives when the changed code still passes the selected tests.

:::info
Mutation testing is currently an MVP. It is ready for early use and feedback, but the workflow, reporting, and supported project configurations are still expected to evolve.
:::

Run mutation testing with `forge test --mutate`:

```bash
$ forge test --mutate
```

Forge first runs the selected tests as a baseline. Mutation testing only starts if the baseline has at least one passing test and no failing tests.

#### Selecting files

Pass paths to mutate only those files:

```bash
$ forge test --mutate src/Vault.sol src/Token.sol
```

Use `--mutate-path` to select files with a glob pattern:

```bash
$ forge test --mutate --mutate-path 'src/**/*.sol'
```

Use `--mutate-contract` to select contracts by name:

```bash
$ forge test --mutate --mutate-contract 'Vault|Token'
```

`--mutate-path` and `--mutate-contract` cannot be combined. `--mutate-path` also cannot be combined with explicit paths passed to `--mutate`.

#### Selecting tests

Regular test filters still select the baseline tests and the tests run against each mutant:

```bash
$ forge test --mutate src/Vault.sol --match-contract VaultTest
```

This lets you scope a mutation run to the tests that should detect changes in a specific contract.

#### Parallel workers

Forge runs mutants in parallel. By default, it uses the number of logical CPU cores.

Set the worker count with `--mutation-jobs`:

```bash
$ forge test --mutate src/Vault.sol --mutation-jobs 4
```

Passing `0` also uses the number of logical CPU cores:

```bash
$ forge test --mutate src/Vault.sol --mutation-jobs 0
```

Parallel mutation testing uses isolated temporary workspaces per mutant. Dependency directories such as `lib`, `node_modules`, and `dependencies` are symlinked into those workspaces for performance.

#### Timeouts

Use `--mutation-timeout` to set a best-effort wall-clock timeout, in seconds, for each mutant:

```bash
$ forge test --mutate src/Vault.sol --mutation-timeout 30
```

Timed-out mutants are reported separately from killed, survived, skipped, and invalid mutants.

You can also configure the timeout in `foundry.toml`:

```toml [foundry.toml]
[mutation]
timeout = 30
```

#### Operators

Mutation testing supports these operator groups:

- `assembly`
- `assignment`
- `binary-op`
- `delete-expression`
- `elim-delegate`
- `require`
- `unary-op`

All operator groups are enabled by default. Exclude specific operators in `foundry.toml`:

```toml [foundry.toml]
[mutation]
exclude_operators = ["assembly", "elim-delegate"]
Comment thread
decofe marked this conversation as resolved.
```

Use `include_operators` to re-enable operators that are excluded by default:

```toml [foundry.toml]
[mutation]
include_operators = ["assembly"]
```

#### Reports

The report includes counts for:
Comment thread
decofe marked this conversation as resolved.

- **Survived**: mutants that passed the selected tests
- **Killed**: mutants that caused a test failure
- **Invalid**: mutants that could not be compiled or run
- **Skipped**: redundant mutants on a span or expression after another mutant in that span survived
- **Timed out**: mutants that exceeded `mutation.timeout` or `--mutation-timeout`

Skipped and invalid counts can vary with `--mutation-jobs`, because higher parallelism can start more mutants before a survivor is known.

The mutation score is:

```text
killed / (killed + survived)
```

Focus on survived mutants first. Each survived mutant points to the source location and mutation that your tests did not catch.
Comment thread
decofe marked this conversation as resolved.

Survived mutants do not currently make `forge test --mutate` fail, and there is no threshold flag yet. To gate mutation testing in CI, run with `--json` and enforce your own threshold from the JSON output.

```bash
$ forge test --mutate --json
```

The JSON output has this shape:

```json
{
"summary": {
"total": 12,
"killed": 8,
"survived": 2,
"invalid": 1,
"skipped": 1,
"timed_out": 0,
"mutation_score": 80.0,
"duration_secs": 12.34
},
"survived_mutants": {
"src/Vault.sol": [
{
"line": 42,
"column": 17,
"original": ">",
"mutant": ">="
}
]
}
}
```

#### Limitations

Mutation testing cannot be combined with `--list`, `--debug`, `--flamegraph`, `--flamechart`, `--junit`, `--dump`, `--showmap`, or `--showmap-out`.

Mutation testing also rejects projects with `ffi = true`, write-capable file-system permissions that can reach symlinked dependency directories, or inline per-test network overrides.

### Testing reverts

Use `vm.expectRevert(){:solidity}` to test that a call reverts:
Expand Down
30 changes: 29 additions & 1 deletion src/pages/reference/forge/test.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,34 @@ Test options:

[env: FORGE_ALLOW_FAILURE=]

--mutate [<PATH>...]
Enable mutation testing.

If passed with file paths, only those files will be tested

--mutate-path <PATTERN>
Specify which files to mutate with glob pattern matching.

Mutually exclusive with passing explicit paths to `--mutate`; either
supply paths to `--mutate` or use this glob filter, not both

--mutate-contract <REGEX>
Only mutate contracts whose name matches the specified regex pattern.

Mutually exclusive with `--mutate-path`

--mutation-jobs <JOBS>
Number of parallel workers for mutation testing.

Defaults to the number of CPU cores

--mutation-timeout <TIMEOUT>
Best-effort per-mutant wall-clock timeout in seconds. Mutants that
exceed it are recorded as "timed out" and cleanup continues in the
background with bounded pending workers.

Analogous to `--invariant-timeout` for invariant campaigns

--trace-depth <TRACE_DEPTH>
Defines the depth of a trace

Expand Down Expand Up @@ -605,4 +633,4 @@ Watch options:
overloading disk I/O.
```

:::
:::