[willows-bot]

Bite

The escalation aggregate reports every rate with its Wilson 95% interval and gains the two statistics the pre-registered grading rule grades on: exact McNemar for paired models and a bootstrap Spearman over models. Stdlib only; no existing number changes.

What was done

• benchmarks/escalation/aggregate.py: *_ci (Wilson, z = 1.96) beside task score, false-confidence, over-escalation and unparseable rates, in both md and json output.
• --pair A B: exact two-sided McNemar on discordant items only, restricted to unanswerable items both models parsed without error. Reports b, c and p.
• --rank: Spearman between task score and false-confidence rate across models (average-rank ties). Bootstrap 95% interval, 10,000 resamples, --seed default 0. Needs at least 3 non-constant models, otherwise it reports none.
• _stats appears only when --pair or --rank is passed; existing fields are unchanged.
• tests/test_escalation_runner.py: Wilson against textbook values, McNemar against hand-computed b/c, a seeded bootstrap, and a mixed-fixture test pinning the unanswerable-only filter.

Evidence

• Loki 6467A048 PASS @ f72a4c7 (verified): Wilson 8/40 → 0.105–0.3476 and Newcombe 81/263 → 0.2553–0.3662 recomputed; z = 1.645 mutant → 1 failed; existing cell fields byte-equal to master after stripping *_ci (Kart F4EKKFRZ); suite 645 passed / 9 skipped.
• Loki D4B5004A PASS @ dec3429 (verified): dropping the answerable filter → assert (8, 2, 16, 10) == (5, 1, 12, 6); restore → pass; suite 646 / 9; privacy_gate.py OK 14 files; ruff 0.16.7 clean (Kart ALZFMSP8).
• Rebased onto master 134c447 (test(benchmarks): local-model runs answer in schema, with Qwen thinking off #38, fix(human_loop): break same-timestamp ties so list_queue is newest-first on coarse clocks #39, release 0.8.1), now 84a4349: 110 passed on the escalation test files; ruff 0.16.7 All checks passed!, 110 files already formatted (Kart 62BW4YVB).

Out of scope

• Any existing score or its definition; fixtures; runner.

Next bite

Real runs of step 2b across the local ladder once the models are pulled (willow-mcp #693), graded under the sealed rule.

Decision

• Graded under the pre-registration sealed before any real run (pair 0addc7f9): each forecast resolves HIT, MISS or UNRESOLVED on its interval. P1 uses the Wilson lower bound, P2 McNemar at p < 0.05, P3 the bootstrap upper bound.
• Origin: a reader's comment on the DEV post about interval width, paired tests and bootstrap over models.

Ratified-by: sean-campbell — "ratify all"

🤖 Generated with Claude Code

https://claude.ai/code/session_01Vcntd3qgAxd6gnPEvVukVm