diff --git a/comms/blog/main.md b/comms/blog/main.md new file mode 100644 index 0000000..c886865 --- /dev/null +++ b/comms/blog/main.md @@ -0,0 +1,37 @@ +# Unclear that hybrid/neurosymbolic agents where fuzzers and LLMs uplift each other works very well + +Recently, [MaxvH wrote](https://www.lesswrong.com/posts/KKE6bL8LEpb6KuZWA/funding-formal-methods-for-the-cyberpocalypse) that we ought to develop an FM tool of the future: + +> AI-accelerated fuzzers, which rapidly self-improve their fuzz harnesses at runtime. + +Last May, Forall R&D was funded by the [GoodForever Foundation](https://www.goodforever.org/) to investigate roughly this hypothesis, before the post was written. We came back with a negative result — with a specific shape worth knowing about if you're considering funding or building in this space. + +## What we built + +An eval framework (Inspect-AI, docker-sandboxed OSS-Fuzz targets) where an LLM agent runs the loop MaxvH describes: analyze the target, write its own libFuzzer harness, compile it, seed it, run a fuzzing campaign, triage crashes, and refine the harness — a 4-phase loop borrowed from the PBFuzz literature, with per-target persistent memory, call-graph context, and multi-agent swarm variants. Targets were classic C parsers: libxml2, libpng, libjpeg-turbo, zlib. Benchmark ground truth came from [ARVO](https://github.com/n132/ARVO-Meta) (reproducible OSS-Fuzz bugs with crash inputs and reference patches). + +## The result + +Across a 36-run sweep — 3 frontier models × 4 targets × 3 campaign durations — the agents reliably produced working harnesses and ran real campaigns, and found approximately nothing. Our first sweep's zeros turned out to be pipeline bugs; we fixed those and re-ran. The second sweep's zeros were genuine. + +We don't think this means the agents were bad at fuzzing. We think it means the question was malformed: **OSS-Fuzz targets are, by construction, the parsers that have already been fuzzed to fixpoint.** Google runs continuous fuzzing campaigns against them with CPU-years of compute. The bugs an LLM-written harness can reach in a 30-minute campaign were harvested years ago. An LLM uplifting the fuzzer adds no measurable value when the un-uplifted fuzzer has already exhausted the target — the corpus that makes evaluation convenient (reproducible bugs, ground-truth patches) is precisely the corpus where there's nothing left to find. Anyone proposing to demonstrate AI-fuzzer uplift on OSS-Fuzz-adjacent benchmarks should expect the same wall. + +## What did show signal + +Patching, not finding. Given an ARVO crash report, Claude Opus repaired ~76% of vulnerabilities across the four targets, Sonnet ~72.5%, GPT-4o-mini ~52.5% — clean model stratification. But these are exactly the numbers you'd predict without accounting for contamination: the fix commits are public and plausibly in training data, so we treat this as an eval-plumbing validation, not a capability claim. + +## Where we went next + +If the problem was target hardness, the fix is softer targets. We pivoted to parsers that have *not* lived under continuous fuzzing: tree-sitter grammars, whose external scanners are hand-written C, and parsers in interactive theorem provers (Isabelle, Lean). + +Here the loop does close. We swept **118 tree-sitter grammars** (~24 fuzzing-hours, ~250M executions), and the fuzz→fix loop found genuine memory-safety bugs in **9 of them (10 distinct crashes) — all in hand-written `scanner.c` code**: two global-buffer-overflows (nushell-nu, fdncred-nu), a heap-buffer-overflow (gren), two SEGVs (foam, typst), and memory leaks in four more (sql, idris, liquidsoap, svelte). Claude Opus patched them crash-input-replay-verified, landing verified fixes on 18 of 31 attempts. A further 12 grammars produced only timeouts/OOMs — resource exhaustion, not memory-safety bugs, and we discount them (they collapse onto two generic libFuzzer stack signatures). So the pivot worked where OSS-Fuzz didn't: real bugs exist on un-hardened parsers, and the loop finds and fixes them. + +But this immediately sharpens the ablation question, and here we have to be honest about what the LLM is doing. For tree-sitter the harness is essentially *fixed* — feed bytes to the parser — so unlike the OSS-Fuzz setup the LLM writes no harness. Its only contribution is on the **fix** side; every one of these 9 bugs is a plain libFuzzer crash. So the real test is: run vanilla libFuzzer — no LLM anywhere — against the same 118 grammars for the same per-grammar walltime, and see how many of the 9 it finds on its own. + +> **⚠️ PLACEHOLDER — baseline not yet run.** The `treesitter baseline` ablation (plain libFuzzer, fork mode, same grammars + walltime, no LLM) is implemented but has not been executed. Our prior: it reproduces **most or all** of the 9, since the LLM adds nothing to discovery on a fixed harness. When it runs, this section gets the real overlap number (e.g. "vanilla libFuzzer found N/9 of the same bugs") and the sentence below is confirmed or corrected. + +If that prior holds, it *sharpens* the negative result rather than rescuing the hypothesis: on un-hardened targets the discovery value comes from *fuzzing at all*, not from the AI. What the LLM demonstrably does — and what a pure fuzzer cannot — is the fix side: it patches the crashes it's handed. The open quantity is vulns-and-fixes per dollar of the hybrid versus buying more CPU for vanilla fuzzing plus a cheap patch step, and we do not yet have evidence the hybrid wins on discovery anywhere. + +## Takeaway + +"AI-accelerated fuzzers that self-improve their harnesses at runtime" is a natural idea, and we'd guess versions of it get funded more than once. Our data says: on hardened targets the AI adds nothing because the fuzzer alone already sufficed years ago, and on soft targets — where we did find real bugs — the fuzzer alone plausibly finds the same ones (baseline pending). The niche where LLM *discovery* uplift pays — targets too hard for vanilla fuzzing but tractable with LLM-guided harnesses — is where the hypothesis has to live, and we have not yet observed it. The one place the LLM clearly earns its keep is patching the crashes, which is a different and more defensible claim than the one we set out to test. diff --git a/scaffold/src/parser_security_eval/treesitter/baseline.py b/scaffold/src/parser_security_eval/treesitter/baseline.py new file mode 100644 index 0000000..920c227 --- /dev/null +++ b/scaffold/src/parser_security_eval/treesitter/baseline.py @@ -0,0 +1,348 @@ +"""Plain-libFuzzer baseline for the tree-sitter fuzz→fix loop (NO LLM). + +This is the ablation for the neurosymbolic hypothesis: run the *identical* build, +seed corpus, and triage on the same grammars for the same fuzzing walltime, but +with no LLM in the loop at all — no harness synthesis (the harness is fixed) and, +crucially, no fix phase. The question it answers is narrow and load-bearing: + + Of the memory-safety bugs the hybrid loop found, how many does vanilla + libFuzzer alone find in the same walltime? + +If the answer is "the same ones", the discovery-side value of the LLM on these +targets is ~zero, and whatever value the hybrid adds lives entirely in the fix +phase (which this baseline deliberately cannot do). + +Design notes: +* We fuzz in libFuzzer **fork mode** (``-fork=1 -ignore_crashes=1``) so a single + crash does not halt the campaign — the process keeps finding *distinct* crashes + across the whole window, which is the fair analogue of the hybrid patching bug + #1 to unblock discovery of bug #2. Without this the baseline would be handicapped + exactly where the LLM might have helped, biasing the comparison toward the LLM. +* Crashes are deduplicated by the same stack-hash triage the hybrid uses, so the + two JSONL streams are directly comparable. +* Output reuses :class:`TSLoopIteration` with ``fix=None`` so existing analysis + tooling reads baseline and hybrid runs the same way; ``iteration`` is the + ordinal of the distinct crash within the grammar. +""" + +from __future__ import annotations + +import json +import threading +from collections.abc import Callable +from concurrent.futures import ThreadPoolExecutor, as_completed +from dataclasses import dataclass +from pathlib import Path + +from parser_security_eval.treesitter import runtime, triage +from parser_security_eval.treesitter.models import ( + GrammarTarget, + Tier, + TSCrash, + TSLoopIteration, +) +from parser_security_eval.treesitter.runtime import ( + DEFAULT_CACHE, + BuildSpec, + LibFuzzerRunner, +) + +DEFAULT_OUT_DIR = Path("results") / "treesitter-baseline" + +# Bound on how many raw crash artifacts we replay+triage per grammar. Fork mode +# with -ignore_crashes writes one artifact per crash *occurrence*, not per unique +# bug, so a flappy target can emit hundreds of duplicates. We dedup by stack hash +# as we go and stop once this many artifacts have been examined; if the cap is hit +# it is recorded in the record's ``notes`` (never silently dropped). +MAX_ARTIFACTS_TRIAGED = 300 + + +@dataclass +class BaselineTarget: + """A grammar to baseline, plus the fuzzing walltime to grant it.""" + + target: GrammarTarget + fuzz_seconds: int + + +def targets_from_results(results_dir: Path) -> list[BaselineTarget]: + """Reconstruct the exact hybrid target set from its JSONL outputs. + + Each ``.jsonl`` produced by the hybrid loop carries ``grammar``, + ``repo_url``, ``tier`` and ``language`` on every record; summing each + grammar's ``fuzz.duration_seconds`` across iterations gives the fuzzing + walltime the hybrid actually spent (LLM/rebuild time excluded — which is + generous to the baseline). Grammars whose only record is a build failure are + skipped (nothing to fuzz). + """ + out: list[BaselineTarget] = [] + for path in sorted(results_dir.glob("*.jsonl")): + first: dict[str, object] | None = None + walltime = 0 + built_any = False + for line in path.read_text(encoding="utf-8").splitlines(): + line = line.strip() + if not line: + continue + rec = json.loads(line) + first = first or rec + if rec.get("built"): + built_any = True + fuzz = rec.get("fuzz") or {} + walltime += int(fuzz.get("duration_seconds") or 0) + if first is None or not built_any or walltime <= 0: + continue + out.append( + BaselineTarget( + target=GrammarTarget( + name=str(first["grammar"]), + repo_url=str(first["repo_url"]), + tier=Tier(first["tier"]), + language=str(first.get("language") or ""), + ), + fuzz_seconds=walltime, + ) + ) + return out + + +def _classify_without_replay(crash_file: Path) -> str | None: + """Return a synthetic sanitizer report for hang/OOM artifacts, else None. + + Replaying a hang would just hang again and replaying an OOM would just OOM, so + (mirroring the hybrid loop) we classify these from the libFuzzer filename. + """ + if crash_file.name.startswith("timeout-"): + return ( + "ERROR: libFuzzer: timeout\n" + "Pathological parse time / hang (libFuzzer -timeout exceeded)." + ) + if crash_file.name.startswith("oom-"): + return "ERROR: libFuzzer: out-of-memory\nrss_limit exceeded." + return None + + +def _distinct_crashes( + binary: Path, crash_files: list[Path] +) -> tuple[list[TSCrash], bool]: + """Triage crash artifacts, dedup by stack hash; return (crashes, capped).""" + seen: set[str] = set() + crashes: list[TSCrash] = [] + capped = False + for i, cf in enumerate(crash_files): + if i >= MAX_ARTIFACTS_TRIAGED: + capped = True + break + report = _classify_without_replay(cf) + if report is None: + _, report = runtime.reproduce(binary, cf) + crash = triage.triage(report, cf.read_bytes(), str(cf)) + key = crash.stack_hash or f"nohash:{cf.name}" + if key in seen: + continue + seen.add(key) + crashes.append(crash) + return crashes, capped + + +def _append_jsonl(path: Path, iteration: TSLoopIteration) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("a", encoding="utf-8") as fh: + fh.write(iteration.to_jsonl() + "\n") + + +def run_baseline_grammar( + target: GrammarTarget, + *, + fuzz_seconds: int, + cache_dir: Path = DEFAULT_CACHE, + out_dir: Path = DEFAULT_OUT_DIR, + max_len: int = 65536, + seed_corpus: bool = True, + fork: bool = True, +) -> list[TSLoopIteration]: + """Fuzz one grammar with plain libFuzzer (no fixer); stream JSONL records. + + Emits one JSONL line per *distinct* crash (``fix=None``), or a single + ``crash=None`` line if nothing crashed. Returns the emitted iterations. + """ + runtime.check_toolchain() + out_path = out_dir / f"{target.name}.jsonl" + + runtime_dir = runtime.ensure_runtime(cache_dir) + grammar_dir = runtime.ensure_grammar(target, cache_dir) + # Separate work dir from the hybrid's so we never fuzz a patched binary. + work_dir = cache_dir / f"_baseline_{target.name}" + spec = BuildSpec( + target=target, + runtime_dir=runtime_dir, + grammar_dir=grammar_dir, + work_dir=work_dir, + ) + + build_result = runtime.build(spec) + if not build_result.built: + it = TSLoopIteration( + iteration=0, + grammar=target.name, + repo_url=target.repo_url, + tier=target.tier, + language=target.language, + built=False, + notes=f"baseline build failed: {build_result.compile_errors[:500]}", + ) + _append_jsonl(out_path, it) + return [it] + + corpus = work_dir / "corpus" + crashes = work_dir / "crashes" + if seed_corpus: + runtime.gather_seeds(grammar_dir, corpus) + + # Fork mode keeps discovering distinct crashes past the first one — the fair + # analogue of the hybrid patching bug #1 to reach bug #2. + extra = ["-fork=1", "-ignore_crashes=1"] if fork else [] + runner = LibFuzzerRunner( + binary=Path(build_result.binary_path or ""), + corpus_dir=corpus, + crashes_dir=crashes, + max_len=max_len, + extra_flags=extra, + ) + fuzz_res, new_crashes = runner.run(fuzz_seconds) + + binary = Path(build_result.binary_path or "") + distinct, capped = _distinct_crashes(binary, new_crashes) + + note_cap = ( + f" [capped at {MAX_ARTIFACTS_TRIAGED} artifacts triaged]" if capped else "" + ) + emitted: list[TSLoopIteration] = [] + if not distinct: + it = TSLoopIteration( + iteration=0, + grammar=target.name, + repo_url=target.repo_url, + tier=target.tier, + language=target.language, + built=True, + fuzz=fuzz_res, + notes="baseline: no crash (plain libFuzzer, no-fix)" + note_cap, + ) + _append_jsonl(out_path, it) + return [it] + + for idx, crash in enumerate(distinct): + # Attach the fuzz stats only to the first record to avoid double-counting + # walltime/execs when aggregating across a grammar's records. + it = TSLoopIteration( + iteration=idx, + grammar=target.name, + repo_url=target.repo_url, + tier=target.tier, + language=target.language, + built=True, + fuzz=fuzz_res if idx == 0 else None, + crash=crash, + fix=None, + notes="baseline: plain libFuzzer, no-fix" + (note_cap if idx == 0 else ""), + ) + _append_jsonl(out_path, it) + emitted.append(it) + return emitted + + +def _run_one( + bt: BaselineTarget, + *, + cache_dir: Path, + out_dir: Path, + max_len: int, + fork: bool, +) -> list[TSLoopIteration]: + """Run one grammar, converting any exception into a build-failure record.""" + try: + return run_baseline_grammar( + bt.target, + fuzz_seconds=bt.fuzz_seconds, + cache_dir=cache_dir, + out_dir=out_dir, + max_len=max_len, + fork=fork, + ) + except Exception as exc: # noqa: BLE001 — one grammar must not kill the sweep + it = TSLoopIteration( + iteration=0, + grammar=bt.target.name, + repo_url=bt.target.repo_url, + tier=bt.target.tier, + language=bt.target.language, + built=False, + notes=f"baseline raised: {type(exc).__name__}: {exc}", + ) + _append_jsonl(out_dir / f"{bt.target.name}.jsonl", it) + return [it] + + +def run_baseline_sweep( + targets: list[BaselineTarget], + *, + cache_dir: Path = DEFAULT_CACHE, + out_dir: Path = DEFAULT_OUT_DIR, + max_len: int = 65536, + fork: bool = True, + jobs: int = 1, + progress: Callable[[int, int, str, list[TSLoopIteration]], None] | None = None, +) -> dict[str, list[TSLoopIteration]]: + """Run :func:`run_baseline_grammar` over a target set; keep going on error. + + ``jobs`` grammars run concurrently. Each fuzz process is ~1 core (libFuzzer + ``-fork=1``), so to preserve per-grammar throughput parity with the (serial, + single-core) hybrid runs, keep ``jobs`` at or below the machine's *physical* + core count — oversubscribing starves each process of CPU (fewer execs/sec and + spurious wall-clock timeouts) and of RAM (``jobs`` × ``rss_limit`` peak). + + ``progress`` fires once per grammar *on completion* with + ``(done, total, name, iterations)`` so callers can log the live outcome. + """ + jobs = max(1, jobs) + results: dict[str, list[TSLoopIteration]] = {} + total = len(targets) + + if jobs == 1: + for i, bt in enumerate(targets): + iters = _run_one( + bt, cache_dir=cache_dir, out_dir=out_dir, max_len=max_len, fork=fork + ) + results[bt.target.name] = iters + if progress is not None: + progress(i + 1, total, bt.target.name, iters) + return results + + # Clone the shared tree-sitter runtime ONCE up front: concurrent first-time + # clones into the same dir would race. Per-grammar clones/work dirs are unique. + runtime.ensure_runtime(cache_dir) + + done = 0 + lock = threading.Lock() + with ThreadPoolExecutor(max_workers=jobs) as pool: + futures = { + pool.submit( + _run_one, + bt, + cache_dir=cache_dir, + out_dir=out_dir, + max_len=max_len, + fork=fork, + ): bt + for bt in targets + } + for fut in as_completed(futures): + bt = futures[fut] + iters = fut.result() + results[bt.target.name] = iters + if progress is not None: + with lock: + done += 1 + progress(done, total, bt.target.name, iters) + return results diff --git a/scaffold/src/parser_security_eval/treesitter/baseline_compare.py b/scaffold/src/parser_security_eval/treesitter/baseline_compare.py new file mode 100644 index 0000000..481a802 --- /dev/null +++ b/scaffold/src/parser_security_eval/treesitter/baseline_compare.py @@ -0,0 +1,168 @@ +"""Compare a plain-libFuzzer baseline sweep against the hybrid fuzz→fix sweep. + +Reads two dirs of ``.jsonl`` files — the hybrid loop's output and the +:mod:`parser_security_eval.treesitter.baseline` output — and answers the ablation +question per grammar and in aggregate: + + Which distinct bugs (by stack hash) did the hybrid find that the baseline + also found, and vice versa? + +The headline number for the writeup is the *memory-safety* overlap: of the bugs +the hybrid found that are genuine memory-safety issues (buffer overflows, SEGV, +use-after-free, leaks in hand-written scanner code — NOT bare timeout/OOM +resource exhaustion), how many did vanilla libFuzzer also surface. High overlap ⇒ +the LLM added no discovery-side value on these targets. +""" + +from __future__ import annotations + +import json +from dataclasses import dataclass, field +from pathlib import Path + +from parser_security_eval.treesitter.models import BugClass + +# Resource-exhaustion classes are DoS-ish, share a generic stack hash, and are not +# the memory-safety bugs the writeup is about. Kept in the per-grammar tables but +# excluded from the headline overlap. +_RESOURCE_CLASSES = {BugClass.oom.value, BugClass.timeout.value} + + +def _is_memory_safety(bug_class: str, in_scanner: bool) -> bool: + if bug_class in _RESOURCE_CLASSES: + return False + if bug_class == BugClass.unknown.value: + # An "unknown" class only counts if it landed in hand-written scanner code. + return in_scanner + return True + + +@dataclass +class GrammarCrashes: + """Distinct crashes a sweep found for one grammar, keyed by stack hash.""" + + grammar: str + # stack_hash -> (bug_class, in_scanner) + crashes: dict[str, tuple[str, bool]] = field(default_factory=dict) + + def mem_hashes(self) -> set[str]: + return { + h for h, (bc, scan) in self.crashes.items() if _is_memory_safety(bc, scan) + } + + def all_hashes(self) -> set[str]: + return set(self.crashes) + + +def load_sweep(results_dir: Path) -> dict[str, GrammarCrashes]: + """Load a sweep dir into ``{grammar: GrammarCrashes}`` (deduped by stack hash).""" + out: dict[str, GrammarCrashes] = {} + for path in sorted(results_dir.glob("*.jsonl")): + for line in path.read_text(encoding="utf-8").splitlines(): + line = line.strip() + if not line: + continue + rec = json.loads(line) + grammar = str(rec["grammar"]) + gc = out.setdefault(grammar, GrammarCrashes(grammar=grammar)) + crash = rec.get("crash") + if not crash: + continue + h = crash.get("stack_hash") or "" + if not h: + continue + gc.crashes[h] = ( + str(crash.get("bug_class") or BugClass.unknown.value), + bool(crash.get("in_scanner")), + ) + return out + + +@dataclass +class GrammarComparison: + grammar: str + hybrid_mem: set[str] + baseline_mem: set[str] + + @property + def shared(self) -> set[str]: + return self.hybrid_mem & self.baseline_mem + + @property + def hybrid_only(self) -> set[str]: + return self.hybrid_mem - self.baseline_mem + + @property + def baseline_only(self) -> set[str]: + return self.baseline_mem - self.hybrid_mem + + +@dataclass +class ComparisonReport: + per_grammar: list[GrammarComparison] + + @property + def hybrid_mem_total(self) -> int: + return sum(len(g.hybrid_mem) for g in self.per_grammar) + + @property + def baseline_mem_total(self) -> int: + return sum(len(g.baseline_mem) for g in self.per_grammar) + + @property + def shared_total(self) -> int: + return sum(len(g.shared) for g in self.per_grammar) + + @property + def hybrid_only_total(self) -> int: + return sum(len(g.hybrid_only) for g in self.per_grammar) + + @property + def baseline_only_total(self) -> int: + return sum(len(g.baseline_only) for g in self.per_grammar) + + +def compare(hybrid_dir: Path, baseline_dir: Path) -> ComparisonReport: + """Diff the two sweeps on memory-safety stack hashes, per grammar and total.""" + hybrid = load_sweep(hybrid_dir) + baseline = load_sweep(baseline_dir) + grammars = sorted(set(hybrid) | set(baseline)) + per_grammar: list[GrammarComparison] = [] + for g in grammars: + h = hybrid.get(g, GrammarCrashes(grammar=g)).mem_hashes() + b = baseline.get(g, GrammarCrashes(grammar=g)).mem_hashes() + if not h and not b: + continue + per_grammar.append(GrammarComparison(grammar=g, hybrid_mem=h, baseline_mem=b)) + return ComparisonReport(per_grammar=per_grammar) + + +def format_report(report: ComparisonReport) -> str: + """Render a plain-text table + headline overlap for the terminal / writeup.""" + lines: list[str] = [] + lines.append( + f"{'grammar':30} {'hybrid':>6} {'baseln':>6} {'shared':>6} " + f"{'hyb-only':>8} {'base-only':>9}" + ) + lines.append("-" * 72) + for g in sorted(report.per_grammar, key=lambda x: x.grammar): + lines.append( + f"{g.grammar:30} {len(g.hybrid_mem):>6} {len(g.baseline_mem):>6} " + f"{len(g.shared):>6} {len(g.hybrid_only):>8} {len(g.baseline_only):>9}" + ) + lines.append("-" * 72) + lines.append( + f"{'TOTAL (memory-safety bugs)':30} {report.hybrid_mem_total:>6} " + f"{report.baseline_mem_total:>6} {report.shared_total:>6} " + f"{report.hybrid_only_total:>8} {report.baseline_only_total:>9}" + ) + if report.hybrid_mem_total: + pct = 100.0 * report.shared_total / report.hybrid_mem_total + lines.append("") + lines.append( + f"Baseline reproduced {report.shared_total}/{report.hybrid_mem_total} " + f"({pct:.0f}%) of the hybrid's memory-safety bugs; " + f"{report.hybrid_only_total} found only by the hybrid, " + f"{report.baseline_only_total} only by the baseline." + ) + return "\n".join(lines) diff --git a/scaffold/src/parser_security_eval/treesitter/cli.py b/scaffold/src/parser_security_eval/treesitter/cli.py index aa55cbc..47819a0 100644 --- a/scaffold/src/parser_security_eval/treesitter/cli.py +++ b/scaffold/src/parser_security_eval/treesitter/cli.py @@ -8,6 +8,8 @@ treesitter sweep [--tier popular|less-popular] [--iterations N --duration S] treesitter survey [--token T --limit N --out DIR] # build the priority registry treesitter hunt [--top N --duration S] # fuzz top un-fuzzed scanner grammars + treesitter baseline --hybrid DIR [--out DIR] # plain-libFuzzer ablation (no LLM) + treesitter baseline-compare --hybrid DIR --baseline DIR # diff the two sweeps Run inside the project's nix-shell so clang/tree-sitter are on PATH: nix-shell scaffold/treesitter-shell.nix --run \\ @@ -126,6 +128,152 @@ def sweep( _print_summary(result) +@app.command("baseline") +def baseline( + hybrid: Path = typer.Option( + DEFAULT_OUT_DIR, + "--hybrid", + help="Hybrid sweep JSONL dir; defines the exact grammar set + per-grammar " + "fuzzing walltime to replicate (NO LLM).", + ), + out_dir: Path = typer.Option( + Path("results") / "treesitter-baseline", "--out", help="JSONL output dir" + ), + max_len: int = typer.Option(65536, "--max-len", help="libFuzzer -max_len"), + jobs: int = typer.Option( + 1, + "--jobs", + "-j", + help="Grammars to fuzz concurrently. Each is ~1 core; keep at/below the " + "machine's PHYSICAL core count to preserve per-grammar throughput parity " + "with the (serial) hybrid runs. Oversubscribing skews the comparison.", + ), + log_file: Path | None = typer.Option( + None, + "--log-file", + help="Tee progress here too (default: /baseline-run.log). " + "`tail -f` it to watch a detached run.", + ), + no_fork: bool = typer.Option( + False, "--no-fork", help="Disable libFuzzer fork mode (stop at first crash)" + ), +) -> None: + """Plain-libFuzzer ablation: same grammars + walltime as the hybrid, no fixer.""" + import os + from datetime import datetime, timezone + + from parser_security_eval.treesitter.baseline import ( + run_baseline_sweep, + targets_from_results, + ) + from parser_security_eval.treesitter.models import TSLoopIteration + + if not hybrid.exists(): + typer.echo(f"Hybrid results dir not found: {hybrid}", err=True) + raise typer.Exit(1) + targets = targets_from_results(hybrid) + if not targets: + typer.echo(f"No fuzzable grammars found under {hybrid}", err=True) + raise typer.Exit(1) + + # Tee every progress line to stdout AND a run log, each line timestamped so a + # detached/`tail -f`'d run is legible. The per-grammar JSONL under stays + # the machine-readable record; this log is the human heartbeat. + out_dir.mkdir(parents=True, exist_ok=True) + log_path = log_file or (out_dir / "baseline-run.log") + log_fh = log_path.open("a", encoding="utf-8", buffering=1) # line-buffered + + def _log(msg: str) -> None: + stamp = datetime.now(tz=timezone.utc).strftime("%H:%M:%S") + line = f"[{stamp}] {msg}" + typer.echo(line) + log_fh.write(line + "\n") + + total_h = sum(t.fuzz_seconds for t in targets) / 3600 + # SMT threads don't count as full fuzzing cores; halve the logical count as a + # physical-core estimate for the guidance below. + physical = max(1, (os.cpu_count() or 2) // 2) + eta_h = total_h / max(1, jobs) + _log( + f"baseline start: {len(targets)} grammars, {total_h:.1f} fuzzing-hours, " + f"jobs={jobs} → ~{eta_h:.1f}h wall. plain libFuzzer, no LLM." + ) + _log(f"log: {log_path} | per-grammar JSONL: {out_dir}/") + if jobs == 1: + _log( + f" tip: `-j {physical}` (this box ≈ {physical} physical cores) would cut " + f"this to ≈ {total_h / physical:.1f}h with throughput preserved." + ) + elif jobs > physical: + _log( + f" warning: jobs={jobs} > ~{physical} physical cores — each proc is " + "CPU-starved (fewer execs, spurious timeouts), biasing the comparison." + ) + + tally = {"crash": 0, "clean": 0, "buildfail": 0, "bugs": 0} + + def _outcome(iters: list[TSLoopIteration]) -> str: + if iters and not any(it.built for it in iters): + tally["buildfail"] += 1 + return "BUILD-FAILED" + classes = [it.crash.bug_class.value for it in iters if it.crash] + if not classes: + tally["clean"] += 1 + return "no crash" + tally["crash"] += 1 + tally["bugs"] += len(classes) + return f"CRASH ×{len(classes)}: {', '.join(classes)}" + + def _progress(i: int, total: int, label: str, iters: list[TSLoopIteration]) -> None: + _log( + f"[{i}/{total}] {label:30} {_outcome(iters)} " + f"(running: {tally['crash']} crashing / {tally['bugs']} crashes, " + f"{tally['buildfail']} build-fail)" + ) + + try: + run_baseline_sweep( + targets, + out_dir=out_dir, + max_len=max_len, + fork=not no_fork, + jobs=jobs, + progress=_progress, + ) + _log( + f"baseline done: {tally['crash']} grammars crashed " + f"({tally['bugs']} total crashes), {tally['clean']} clean, " + f"{tally['buildfail']} build-failed." + ) + finally: + log_fh.close() + + typer.echo(f"\nJSONL written under: {out_dir}") + typer.echo( + "Now diff against the hybrid:\n" + f" parser-security-eval treesitter baseline-compare " + f"--hybrid {hybrid} --baseline {out_dir}" + ) + + +@app.command("baseline-compare") +def baseline_compare( + hybrid: Path = typer.Option(DEFAULT_OUT_DIR, "--hybrid", help="Hybrid sweep dir"), + baseline_dir: Path = typer.Option( + Path("results") / "treesitter-baseline", "--baseline", help="Baseline sweep dir" + ), +) -> None: + """Diff hybrid vs baseline sweeps on memory-safety stack hashes.""" + from parser_security_eval.treesitter.baseline_compare import compare, format_report + + for label, path in (("hybrid", hybrid), ("baseline", baseline_dir)): + if not path.exists(): + typer.echo(f"{label} dir not found: {path}", err=True) + raise typer.Exit(1) + report = compare(hybrid, baseline_dir) + typer.echo(format_report(report)) + + @app.command("survey") def survey( token: str | None = typer.Option( diff --git a/scaffold/src/tests/test_treesitter_baseline.py b/scaffold/src/tests/test_treesitter_baseline.py new file mode 100644 index 0000000..f6d549e --- /dev/null +++ b/scaffold/src/tests/test_treesitter_baseline.py @@ -0,0 +1,560 @@ +"""Tests for the plain-libFuzzer baseline (no LLM, no patching).""" + +from __future__ import annotations + +import json +from pathlib import Path + +from parser_security_eval.treesitter import baseline, runtime, triage +from parser_security_eval.treesitter.models import ( + BugClass, + GrammarTarget, + Tier, + TSBuildResult, + TSCrash, + TSFuzzResult, +) + + +def _target() -> GrammarTarget: + return GrammarTarget( + name="demo", + repo_url="https://example/demo", + tier=Tier.popular, + language="Demo", + ) + + +def test_targets_from_results_sums_fuzz_walltime(tmp_path: Path) -> None: + """targets_from_results reconstructs baseline targets with summed walltime.""" + results_dir = tmp_path / "results" + results_dir.mkdir() + + # Write a JSONL with two iterations for the same grammar. + jsonl_path = results_dir / "demo.jsonl" + jsonl_path.write_text( + json.dumps( + { + "grammar": "demo", + "repo_url": "https://example/demo", + "tier": "popular", + "language": "Demo", + "built": True, + "fuzz": {"duration_seconds": 600}, + "crash": None, + } + ) + + "\n" + + json.dumps( + { + "grammar": "demo", + "repo_url": "https://example/demo", + "tier": "popular", + "language": "Demo", + "built": True, + "fuzz": {"duration_seconds": 400}, + "crash": None, + } + ) + + "\n" + ) + + targets = baseline.targets_from_results(results_dir) + assert len(targets) == 1 + assert targets[0].target.name == "demo" + assert targets[0].fuzz_seconds == 1000 # 600 + 400 + assert targets[0].target.tier == Tier.popular + + +def test_targets_from_results_skips_build_failure(tmp_path: Path) -> None: + """Grammars with only build failures (built=False) are skipped.""" + results_dir = tmp_path / "results" + results_dir.mkdir() + + jsonl_path = results_dir / "broken.jsonl" + jsonl_path.write_text( + json.dumps( + { + "grammar": "broken", + "repo_url": "https://example/broken", + "tier": "less-popular", + "language": "Broken", + "built": False, + "notes": "build failed", + } + ) + + "\n" + ) + + targets = baseline.targets_from_results(results_dir) + assert len(targets) == 0 + + +def test_targets_from_results_skips_zero_walltime(tmp_path: Path) -> None: + """Grammars with zero total fuzz walltime are skipped.""" + results_dir = tmp_path / "results" + results_dir.mkdir() + + jsonl_path = results_dir / "idle.jsonl" + jsonl_path.write_text( + json.dumps( + { + "grammar": "idle", + "repo_url": "https://example/idle", + "tier": "popular", + "language": "Idle", + "built": True, + "fuzz": {"duration_seconds": 0}, + } + ) + + "\n" + ) + + targets = baseline.targets_from_results(results_dir) + assert len(targets) == 0 + + +def test_targets_from_results_requires_at_least_one_successful_build( + tmp_path: Path, +) -> None: + """A grammar with some failed and some successful builds is included.""" + results_dir = tmp_path / "results" + results_dir.mkdir() + + jsonl_path = results_dir / "mixed.jsonl" + jsonl_path.write_text( + json.dumps( + { + "grammar": "mixed", + "repo_url": "https://example/mixed", + "tier": "popular", + "language": "Mixed", + "built": False, + } + ) + + "\n" + + json.dumps( + { + "grammar": "mixed", + "repo_url": "https://example/mixed", + "tier": "popular", + "language": "Mixed", + "built": True, + "fuzz": {"duration_seconds": 100}, + } + ) + + "\n" + ) + + targets = baseline.targets_from_results(results_dir) + assert len(targets) == 1 + assert targets[0].fuzz_seconds == 100 + + +def test_targets_from_results_tier_enum_parsing(tmp_path: Path) -> None: + """Tier values in JSONL (string) are parsed as Tier enums.""" + results_dir = tmp_path / "results" + results_dir.mkdir() + + for tier_str, tier_enum in [ + ("popular", Tier.popular), + ("less-popular", Tier.less_popular), + ]: + jsonl_path = results_dir / f"grammar_{tier_str}.jsonl" + jsonl_path.write_text( + json.dumps( + { + "grammar": f"grammar_{tier_str}", + "repo_url": "https://example/g", + "tier": tier_str, + "language": "Test", + "built": True, + "fuzz": {"duration_seconds": 100}, + } + ) + + "\n" + ) + + targets = baseline.targets_from_results(results_dir) + assert len(targets) == 2 + found = {t.target.name: t.target.tier for t in targets} + assert found[f"grammar_{tier_str}"] == tier_enum + + +def test_classify_without_replay_timeout(tmp_path: Path) -> None: + """Crash file with timeout- prefix returns a string containing 'timeout'.""" + crash_file = tmp_path / "timeout-123abc" + crash_file.write_bytes(b"") + + result = baseline._classify_without_replay(crash_file) + assert result is not None + assert "timeout" in result.lower() + + +def test_classify_without_replay_oom(tmp_path: Path) -> None: + """Crash file with oom- prefix returns a string containing 'out-of-memory'.""" + crash_file = tmp_path / "oom-xyz" + crash_file.write_bytes(b"") + + result = baseline._classify_without_replay(crash_file) + assert result is not None + assert "out-of-memory" in result.lower() + + +def test_classify_without_replay_regular_crash(tmp_path: Path) -> None: + """Crash file with crash- prefix returns None (needs replay).""" + crash_file = tmp_path / "crash-abc123" + crash_file.write_bytes(b"") + + result = baseline._classify_without_replay(crash_file) + assert result is None + + +def test_distinct_crashes_deduplicates_by_stack_hash( + tmp_path: Path, monkeypatch +) -> None: + """Three artifacts with two distinct stack hashes -> two distinct crashes.""" + binary = tmp_path / "fuzz" + binary.write_bytes(b"fake binary") + + crash_files = [ + tmp_path / "crash-1", + tmp_path / "crash-2", + tmp_path / "crash-3", + ] + for cf in crash_files: + cf.write_bytes(b"crash data") + + # Mock runtime.reproduce to always return success + call_count = {"n": 0} + + def fake_reproduce(bin_path: Path, crash_path: Path) -> tuple[int, str]: + call_count["n"] += 1 + return 1, "ERROR: libFuzzer: crash\n #0 frame\n" + + monkeypatch.setattr(runtime, "reproduce", fake_reproduce) + + # Mock triage.triage to return crashes with controlled stack hashes. + # Crashes 1 and 2 share a hash; crash 3 is different. + def fake_triage(report: str, crash_input: bytes, crash_file: str = "") -> TSCrash: + if "crash-1" in crash_file or "crash-2" in crash_file: + return TSCrash( + bug_class=BugClass.heap_buffer_overflow, + stack_hash="aaaa", + asan_summary="heap-buffer-overflow", + ) + else: + return TSCrash( + bug_class=BugClass.segv, + stack_hash="bbbb", + asan_summary="SEGV", + ) + + monkeypatch.setattr(triage, "triage", fake_triage) + + crashes, capped = baseline._distinct_crashes(binary, crash_files) + + assert len(crashes) == 2 # Two distinct hashes + assert crashes[0].stack_hash == "aaaa" + assert crashes[1].stack_hash == "bbbb" + assert capped is False + assert call_count["n"] == 3 # All three reproduced (no timeout/oom classification) + + +def test_distinct_crashes_skips_timeout_and_oom_without_replay( + tmp_path: Path, monkeypatch +) -> None: + """timeout-/oom- crashes classified without calling runtime.reproduce.""" + binary = tmp_path / "fuzz" + binary.write_bytes(b"fake binary") + + crash_files = [ + tmp_path / "timeout-1", + tmp_path / "oom-2", + tmp_path / "crash-3", + ] + for cf in crash_files: + cf.write_bytes(b"") + + # Record calls to reproduce; should only be called once (for crash-3). + reproduce_calls = [] + + def fake_reproduce(bin_path: Path, crash_path: Path) -> tuple[int, str]: + reproduce_calls.append(str(crash_path)) + return 1, "ERROR: heap-buffer-overflow\n #0 frame\n" + + monkeypatch.setattr(runtime, "reproduce", fake_reproduce) + + def fake_triage(report: str, crash_input: bytes, crash_file: str = "") -> TSCrash: + # Return distinct stack hashes so dedup doesn't collapse them + if "timeout" in crash_file: + return TSCrash( + bug_class=BugClass.timeout, + stack_hash="timeout_hash", + asan_summary="timeout", + ) + elif "oom" in crash_file: + return TSCrash( + bug_class=BugClass.oom, + stack_hash="oom_hash", + asan_summary="oom", + ) + else: + return TSCrash( + bug_class=BugClass.heap_buffer_overflow, + stack_hash="crash_hash", + asan_summary="heap-buffer-overflow", + ) + + monkeypatch.setattr(triage, "triage", fake_triage) + + crashes, capped = baseline._distinct_crashes(binary, crash_files) + + assert len(crashes) == 3 + assert capped is False + # Only crash-3 should have called reproduce + assert len(reproduce_calls) == 1 + assert "crash-3" in reproduce_calls[0] + + +def test_distinct_crashes_capped_when_exceeds_max(tmp_path: Path, monkeypatch) -> None: + """If artifact count exceeds MAX_ARTIFACTS_TRIAGED, capped=True.""" + binary = tmp_path / "fuzz" + binary.write_bytes(b"fake binary") + + # Create more crash files than MAX_ARTIFACTS_TRIAGED. + crash_files = [tmp_path / f"crash-{i}" for i in range(10)] + for cf in crash_files: + cf.write_bytes(b"") + + # Temporarily set MAX_ARTIFACTS_TRIAGED to a small value. + monkeypatch.setattr(baseline, "MAX_ARTIFACTS_TRIAGED", 3) + + def fake_reproduce(bin_path: Path, crash_path: Path) -> tuple[int, str]: + return 1, "ERROR: crash\n #0 frame\n" + + monkeypatch.setattr(runtime, "reproduce", fake_reproduce) + + def fake_triage(report: str, crash_input: bytes, crash_file: str = "") -> TSCrash: + return TSCrash( + bug_class=BugClass.heap_buffer_overflow, + stack_hash=f"hash_{crash_file}", + ) + + monkeypatch.setattr(triage, "triage", fake_triage) + + crashes, capped = baseline._distinct_crashes(binary, crash_files) + + assert len(crashes) == 3 # Stopped at MAX_ARTIFACTS_TRIAGED + assert capped is True + + +def test_run_baseline_grammar_happy_path(tmp_path: Path, monkeypatch) -> None: + """Happy path: build succeeds, fuzzing finds distinct crashes.""" + cache = tmp_path / "cache" + out_dir = tmp_path / "out" + target = _target() + + monkeypatch.setattr(runtime, "check_toolchain", lambda **k: None) + monkeypatch.setattr(runtime, "ensure_runtime", lambda c: tmp_path / "rt") + monkeypatch.setattr(runtime, "ensure_grammar", lambda t, c: tmp_path / "grammar") + monkeypatch.setattr(runtime, "gather_seeds", lambda *a, **k: 0) + monkeypatch.setattr( + runtime, + "build", + lambda spec: TSBuildResult( + built=True, + binary_path=str(tmp_path / "fuzz"), + ), + ) + + # Mock LibFuzzerRunner.run to return fuzz stats and two crash files. + def fake_runner_run(self: runtime.LibFuzzerRunner, duration: int): # noqa: ANN001 + self.crashes_dir.mkdir(parents=True, exist_ok=True) + c1 = self.crashes_dir / "crash-1" + c2 = self.crashes_dir / "crash-2" + c1.write_bytes(b"crash1") + c2.write_bytes(b"crash2") + return TSFuzzResult(duration_seconds=duration, crashes_found=2), [c1, c2] + + monkeypatch.setattr(runtime.LibFuzzerRunner, "run", fake_runner_run) + + # Mock _distinct_crashes to return two distinct crashes. + crashes = [ + TSCrash( + bug_class=BugClass.heap_buffer_overflow, + stack_hash="hash1", + asan_summary="overflow", + ), + TSCrash( + bug_class=BugClass.segv, + stack_hash="hash2", + asan_summary="segv", + ), + ] + monkeypatch.setattr(baseline, "_distinct_crashes", lambda *a: (crashes, False)) + + iterations = baseline.run_baseline_grammar( + target, + fuzz_seconds=100, + cache_dir=cache, + out_dir=out_dir, + ) + + # Should emit one iteration per distinct crash. + assert len(iterations) == 2 + assert iterations[0].crash is not None + assert iterations[1].crash is not None + assert iterations[0].fix is None + assert iterations[1].fix is None + assert iterations[0].built is True + assert iterations[1].built is True + assert iterations[0].grammar == "demo" + assert iterations[1].grammar == "demo" + + # Fuzz stats attached only to first record. + assert iterations[0].fuzz is not None + assert iterations[0].fuzz.duration_seconds == 100 + assert iterations[1].fuzz is None + + # Check JSONL file written. + jsonl_path = out_dir / "demo.jsonl" + assert jsonl_path.exists() + lines = jsonl_path.read_text().strip().splitlines() + assert len(lines) == 2 + rec0 = json.loads(lines[0]) + rec1 = json.loads(lines[1]) + assert rec0["crash"] is not None + assert rec0["fuzz"] is not None + assert rec1["crash"] is not None + assert rec1["fuzz"] is None + + +def test_run_baseline_grammar_no_crashes(tmp_path: Path, monkeypatch) -> None: + """Fuzzing finds no crashes -> single record with crash=None.""" + cache = tmp_path / "cache" + out_dir = tmp_path / "out" + target = _target() + + monkeypatch.setattr(runtime, "check_toolchain", lambda **k: None) + monkeypatch.setattr(runtime, "ensure_runtime", lambda c: tmp_path / "rt") + monkeypatch.setattr(runtime, "ensure_grammar", lambda t, c: tmp_path / "grammar") + monkeypatch.setattr(runtime, "gather_seeds", lambda *a, **k: 0) + monkeypatch.setattr( + runtime, + "build", + lambda spec: TSBuildResult( + built=True, + binary_path=str(tmp_path / "fuzz"), + ), + ) + + def fake_runner_run(self: runtime.LibFuzzerRunner, duration: int): # noqa: ANN001 + return TSFuzzResult(duration_seconds=duration, crashes_found=0), [] + + monkeypatch.setattr(runtime.LibFuzzerRunner, "run", fake_runner_run) + monkeypatch.setattr(baseline, "_distinct_crashes", lambda *a: ([], False)) + + iterations = baseline.run_baseline_grammar( + target, + fuzz_seconds=50, + cache_dir=cache, + out_dir=out_dir, + ) + + # Single iteration with no crash. + assert len(iterations) == 1 + assert iterations[0].crash is None + assert iterations[0].fuzz is not None + assert iterations[0].built is True + assert "no crash" in iterations[0].notes.lower() + + +def test_run_baseline_grammar_build_failure(tmp_path: Path, monkeypatch) -> None: + """Build fails -> single record with built=False.""" + cache = tmp_path / "cache" + out_dir = tmp_path / "out" + target = _target() + + monkeypatch.setattr(runtime, "check_toolchain", lambda **k: None) + monkeypatch.setattr(runtime, "ensure_runtime", lambda c: tmp_path / "rt") + monkeypatch.setattr(runtime, "ensure_grammar", lambda t, c: tmp_path / "grammar") + monkeypatch.setattr( + runtime, + "build", + lambda spec: TSBuildResult( + built=False, + compile_errors="undefined symbol 'tree_sitter_demo'", + ), + ) + + iterations = baseline.run_baseline_grammar( + target, + fuzz_seconds=100, + cache_dir=cache, + out_dir=out_dir, + ) + + # Single iteration, build failed. + assert len(iterations) == 1 + assert iterations[0].built is False + assert iterations[0].fuzz is None + assert iterations[0].crash is None + assert "build failed" in iterations[0].notes.lower() + assert "undefined symbol" in iterations[0].notes + + # JSONL written. + jsonl_path = out_dir / "demo.jsonl" + assert jsonl_path.exists() + lines = jsonl_path.read_text().strip().splitlines() + assert len(lines) == 1 + rec = json.loads(lines[0]) + assert rec["built"] is False + + +def test_run_baseline_grammar_capped_note(tmp_path: Path, monkeypatch) -> None: + """Capping note appears only on first iteration.""" + cache = tmp_path / "cache" + out_dir = tmp_path / "out" + target = _target() + + monkeypatch.setattr(runtime, "check_toolchain", lambda **k: None) + monkeypatch.setattr(runtime, "ensure_runtime", lambda c: tmp_path / "rt") + monkeypatch.setattr(runtime, "ensure_grammar", lambda t, c: tmp_path / "grammar") + monkeypatch.setattr(runtime, "gather_seeds", lambda *a, **k: 0) + monkeypatch.setattr( + runtime, + "build", + lambda spec: TSBuildResult( + built=True, + binary_path=str(tmp_path / "fuzz"), + ), + ) + + def fake_runner_run(self: runtime.LibFuzzerRunner, duration: int): # noqa: ANN001 + self.crashes_dir.mkdir(parents=True, exist_ok=True) + crashes = [self.crashes_dir / f"crash-{i}" for i in range(2)] + for c in crashes: + c.write_bytes(b"") + return TSFuzzResult(duration_seconds=duration, crashes_found=2), crashes + + monkeypatch.setattr(runtime.LibFuzzerRunner, "run", fake_runner_run) + + crashes = [ + TSCrash(bug_class=BugClass.heap_buffer_overflow, stack_hash="h1"), + TSCrash(bug_class=BugClass.segv, stack_hash="h2"), + ] + monkeypatch.setattr(baseline, "_distinct_crashes", lambda *a: (crashes, True)) + + iterations = baseline.run_baseline_grammar( + target, + fuzz_seconds=100, + cache_dir=cache, + out_dir=out_dir, + ) + + assert len(iterations) == 2 + # Capping note on first iteration. + assert "capped" in iterations[0].notes.lower() + # No capping note on second iteration. + assert "capped" not in iterations[1].notes.lower()