Skip to content

Add a local-only cross-shell differential fuzzer - #16

Merged
mitchellnemitz merged 11 commits into
mainfrom
impl-2026-07-25-cross-shell-differential-fuzzer
Jul 26, 2026
Merged

mitchellnemitz merged 11 commits into
mainfrom
impl-2026-07-25-cross-shell-differential-fuzzer

Conversation

@mitchellnemitz

Copy link
Copy Markdown
Owner

Adds a local-only cross-shell differential fuzzer for the wisp compiler. It defends the byte-identity invariant that hand-authored golden fixtures cannot exhaustively cover: hunting programs whose emitted shell behaves differently across dash, busybox ash, bash, and zsh.

New package internal/fuzz plus a cmd/wisp-fuzz CLI. Go stdlib only, and deliberately never wired into CI (it is a local/nightly soak job, run by hand).

How it works:

  • Type-directed AST generation builds well-typed programs by construction, then pretty-prints them, so nearly everything compiles. Entropy is spent payload-centric: on the adversarial values flowing through a program (embedded $(...), newlines, %s, backslashes, leading -, glob metacharacters, arithmetic edge values), where cross-shell bugs actually live.
  • v1 scope is the pure core: arithmetic, bool, string, int, float, arrays, dicts, control flow, and non-effectful builtins (93 across 10 families). All effectful/nondeterministic surface is excluded from the generator.
  • Two oracles: the four shells must agree on raw stdout bytes and exit status, and the same source must recompile to byte-identical output (a recompile-stability self-oracle). INSTABILITY and DIVERGENCE are reported distinctly.
  • On a divergence, an AST-level type-preserving shrinker reduces to a 1-minimal reproducer, persisted with deterministic provenance for replay.
  • Runs are deterministic: a fixed seed reproduces a run exactly, so found bugs replay from a committed regression corpus.

Design B, one documented carve-out: the generator deliberately emits INT_MIN, and the oracle carves zsh for the single INT_MIN-in-$(( )) case (a documented, source-cited zsh limitation, not a wisp bug). The carve is a narrow source-level data-flow check (INT_MIN reaching an arithmetic operand via variable flow); it is not widened to arith-lowering builtins, so it cannot mask a real divergence. Reverting the INT_MIN compiler fix makes the fuzzer produce a failing reproducer.

Built task-by-task with a review gate after each and a final whole-branch review. go test ./... is green. Four-shell tests are behind a fuzzshell build tag, so the default go test spawns no shells; exercising the fuzzer end-to-end needs all four shells installed (busybox included).

mitchellnemitz and others added 11 commits July 25, 2026 21:46
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…enerable

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add gen.go: GenConfig/DefaultGenConfig/Generate plus the witness prelude
and payload-centric random walk. Every generated program opens with an
unconditional, deterministic prelude witnessing all 10 admitted families
(convert, parse, collection, debug, option, math, array, dict, string,
regex) and all 14 pure-core/payload categories, so SC-009 coverage holds
by construction from any single program and every tag reaches stdout.

Reconciled against the real Task 4 catalog: dropped the plan's illustrative
json/result cases (no such families), and witnessed option via
is_some(parse_int(...)) rather than the non-admitted Some(...) constructor
so the generator's Call nodes only ever name admitted builtins.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01STfkQ5S99NLbifDZPMpdgd
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…k, docs

Task 11 (final): pure-Go CI tests plus fuzzshell-gated real-shell tests,
the curated INT_MIN-via-variable regression corpus entry, and the
local-only fuzzer README with the done-signal / regression-direction
procedure. AGENTS.md pointer (plan Step 5) intentionally skipped as
untraced scope creep.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mitchellnemitz
mitchellnemitz merged commit c7c8f2f into main Jul 26, 2026
8 checks passed
@mitchellnemitz
mitchellnemitz deleted the impl-2026-07-25-cross-shell-differential-fuzzer branch July 26, 2026 03:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant