agent-ab

A small, honest harness for answering one question: does this Claude Code configuration actually make outcomes better — or does it just feel like it?

You give it two configurations (a control and a treatment that differ by exactly one thing), a set of tasks with machine-checkable graders, and a number of repetitions. It runs them interleaved, records every run, and tells you whether the difference between the two configs is bigger than the run-to-run noise.

Why this exists

Coding agents are non-deterministic. The same prompt with the same config gives different token counts, different costs, and sometimes different pass/fail outcomes. So "I ran it once with my new CLAUDE.md and it felt better" proves nothing — you measured noise.

This harness is built around three ideas that make the measurement trustworthy:

1. Repetition over anecdote. Every (task, arm) cell runs N times. We compare distributions, not single runs.
2. An A/A noise floor. Before trusting any result, run the aa experiment: two identical configs against each other. The true effect is zero, so whatever gap you see is your noise floor. A real change only counts if its effect clears that floor. This is the keystone — without it, every "win" is suspect.
3. Interleaving + pinning. Arms are interleaved round-robin so a mid-run model update or API hiccup can't bias one arm. The model snapshot is pinned per experiment and recorded on every run, so drift is visible, not silent.

Graders are plain commands that exit 0 or 1 (here, bun test) — never an LLM judge. "Verifiable" means a machine decided pass/fail.

Usage

bun install

# 1. establish your noise floor first (this is not optional)
bun run index.ts run aa
bun run index.ts analyze aa

# 2. then test a real change
bun run index.ts run claude-md --budget 5
bun run index.ts analyze claude-md

--help is generated by commander — run bun run index.ts --help or bun run index.ts run --help for the full flag reference. Useful flags at a glance:

run <experiment> — --budget <usd> (hard spend cap), --runs <n> (override repetitions), --model <id> (override pinned model), --tasks-root <dir> (custom task directory), --dry-run (print the pending plan as JSON without spawning any agent), --json (print fresh results as JSON instead of the markdown report).

analyze <experiment> — writes reports/<exp>.md by default and prints to stdout. --output <dir> changes the output directory, --no-write prints only (skips the file), --json prints the raw report data as JSON instead of markdown.

list [--json] — lists all registered experiments from the registry.

Runs are append-only in results/runs.jsonl and resumable: re-running run skips cells already completed, so a crash or a budget stop never loses progress.

How an experiment is defined

// src/experiments.ts
'claude-md': {
  name: 'claude-md',
  tasks: ['hello-fn', 'fix-bug', 'refactor'],
  arms: [baseline, withClaudeMd], // differ by ONE thing: a seeded CLAUDE.md
  runs: 10,
  model: 'claude-sonnet-4-6', // pinned; identical across arms
}

An arm is a named config: extra CLI args and/or files seeded into the task's scratch directory. The only difference between control and treatment should be the one variable under test.

Reading the report

The per-arm table shows pass rate and the mean/median/spread of cost, tokens, turns, and wall time. The delta table shows treatment-minus-baseline with a 95% bootstrap confidence interval. If the interval excludes 0, the effect is larger than noise. Calibrate "large enough to care about" against the aa report, where the real effect is 0 by construction.

Design notes

• Auth: the child claude process inherits the parent environment, so OAuth credentials / ANTHROPIC_API_KEY survive. --bare is opt-in per arm (it can drop credentials → "Not logged in"). For fully reproducible cross-machine runs, set ANTHROPIC_API_KEY and flip bare: true.
• Isolation: each run executes in a fresh temp dir containing only the pristine fixture plus the arm's seed files, then the dir is destroyed. Runs cannot contaminate each other.
• Cross-platform: all filesystem and process work goes through node:* and Bun APIs (no shell-isms), so it runs the same on macOS, Linux, and Windows.

Extending it

• New task: add tasks/<name>/task.json (prompt + grade command) and a fixture/ directory. Any language works as long as the grader is a command that exits 0/1 — e.g. a Rust task graded by ["cargo", "test"].
• New arm: add an ArmConfig in src/arms.ts and reference it in an experiment.
• New backend: runClaude is the only place that shells out to a specific agent. Swap it behind an interface to benchmark other agent CLIs on the same tasks.

Layout

src/
  claude.ts       parse `claude -p --output-format json` + spawn the agent
  task.ts         one cell: scratch -> seed -> run -> grade -> record
  plan.ts         interleaved (task x arm x run) execution order
  experiment.ts   orchestrator: resume, budget cap, auth abort
  stats.ts        mean/median/stddev + seeded bootstrap CI (pure, tested)
  report.ts       markdown tables + deltas with CIs
  arms.ts         arm definitions
  experiments.ts  experiment registry (aa, claude-md, plan-hint)
tasks/            task fixtures + graders
test/             unit tests for every pure piece (the trust anchor)
results/          append-only run log (intentionally committed; not gitignored)
reports/          generated markdown reports from `analyze` (gitignored)