test(escalation): floor and repeat-run consistency views over the rows

[willows-bot]

Bite

Land the floor-and-consistency views for the escalation benchmark. The code was staged in the willows-grove Kaggle proposal amendment of 2026-10-02, and it lands now because the day-two cloud run has finished.

What was done

• New benchmarks/escalation/reliability.py. It is additive: it imports aggregate.py and leaves it unchanged.
  • Floor across shapes. For each model, the worst per-shape task score and the worst per-shape false-confidence rate, each with its Wilson interval. When shapes tie at the worst value, all of them are reported: *_shapes is a sorted list and *_ci is a dict of shape to CI. A tie used to be broken by shape name.
  • Repeat-run consistency. Over k runs of a fixed subset, the share of items whose parsed answer is identical across runs and the share whose verdict flips. Error and unparseable count as separate verdicts. The ids of flipped items are kept.
• New tests/test_escalation_reliability.py with 6 tests.
• gate/privacy_gate.py: reliability.py is added to the exact-path ALLOWED_OTHER set. The gate still scans the file's contents.

Evidence

• Tests on the box (3.14.4): 399 passed, 26 skipped in 23.71s. Ruff 0.16.7: All checks passed!, 122 files already formatted.
• The aggregate.py diff against 84efd90 is 0 lines.
• Mutation testing by the auditor: 6 of 10 mutants killed, at least one in each function except main(). Examples: M2 floor fc tie->last only (old route label): rc=1 KILLED, M6 consistency keeps partial items: rc=1 KILLED.
• Floor over the 800 Day-2 rows. Every model's task floor is on ['ground']. Sonnet-5's false-confidence ceiling is 0.1 ['classify', 'judge']. Gemini-3.1-pro and gpt-5.5 are at 0.0 on all four shapes.
• Audit: PASS on f0c44cd.
• Not reproduced on the box, so read them here: Linux on 3.11–3.13, Windows, no-extras, vendor_sync_check.py, the title and trailer checks, and codeql.

Out of scope

• No change to aggregate.py, runner.py, the fixtures, or kaggle/.
• The consistency subset is not committed yet, and no model was run.

Next bite

• Commit the fixed consistency subset (20 items per shape, unanswerable items included), then run it k=3 per model on Kaggle.
• Follow-up tests from the audit's LOW findings: pin the exclusion of None-rate shapes and error as its own verdict.

Decision

Ratified-by: sean-campbell — "ratify"

🤖 Generated with Claude Code

https://claude.ai/code/session_014195BDJ4JX3rsqMe52bcWx

[willows-bot]

CI red: #46 @ f0c44cd — 5 job(s):

• no-extras (failure): https://github.com/forge-play/Forge/actions/runs/37071552335/job/111051924271
• test-matrix (3.12) (failure): https://github.com/forge-play/Forge/actions/runs/37071552335/job/111051924222
• windows (3.11) (failure): https://github.com/forge-play/Forge/actions/runs/37071552335/job/111051924053
• windows (3.13) (failure): https://github.com/forge-play/Forge/actions/runs/37071552335/job/111051924275
• test (failure): https://github.com/forge-play/Forge/actions/runs/37071552335/job/111052552415

no-extras

=================================== FAILURES ===================================
_____________ test_every_scan_helper_has_a_planted_violation_test ______________

    def test_every_scan_helper_has_a_planted_violation_test():
        """The house rule, run for real: no `tests/*.py` file may carry an
        AST/grep-guard helper that no planted-violation test in the same file
        actually runs."""
        offenders = _offenders()
>       assert not offenders, (
            "these test files define a scan helper (it walks source, matches a "
            "pattern, or reads a file and asks a membership question of it — or "
            f"its name carries one of {sorted(_NAME_TOKENS)}) that no test naming "
            "plant/fires/catches ever calls — a scan that has never fired has not "
            f"been shown to check anything: {offenders}"
        )
E       AssertionError: these test files define a scan helper (it walks source, matches a pattern, or reads a file and asks a membership question of it — or its name carries one of ['calls', 'check', 'checks', 'guard', 'guards', 'offenders', 'reads', 'scan', 'scans', 'uses']) that no test naming plant/fires/catches ever calls — a scan that has never fired has not been shown to check anything: ["test_escalation_reliability.py: ['_rows_by_shape']"]
E       assert not ["test_escalation_reliability.py: ['_rows_by_shape']"]

tests/test_scans_fire.py:527: AssertionError
=========================== short test summary info ============================
FAILED tests/test_scans_fire.py::test_every_scan_helper_has_a_planted_violation_test - AssertionError: these test files define a scan helper (it walks source, matches a pattern, or reads a file and asks a membership question of it — or its name carries one of ['calls', 'check', 'checks', 'guard', 'guards', 'offenders', 'reads', 'scan', 'scans', 'uses']) that no test naming plant/fires/catches ever calls — a scan that has never fired has not been shown to check anything: ["test_escalation_reliability.py: ['_rows_by_shape']"]
assert not ["test_escalation_reliability.py: ['_rows_by_shape']"]
1 failed, 636 passed, 343 skipped in 14.78s
##[error]Process completed with exit code 1.
Post job cleanup.
[command]/usr/bin/git version
git version 2.55.0
Temporarily overriding HOME='/home/runner/work/_temp/7ae5d089-7c12-4467-a783-57ccefc670fe' before making global git config changes
Adding repository directory to the temporary git global config as a safe directory
[command]/usr/bin/git config --global --add safe.directory /home/runner/work/Forge/Forge
Removing SSH command configuration
[command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
[command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
Removing HTTP extra header
[command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
[command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
Removing includeIf entries pointing to credentials config files
[command]/usr/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir:
includeif.gitdir:/home/runner/work/Forge/Forge/.git.path
includeif.gitdir:/home/runner/work/Forge/Forge/.git/worktrees/*.path
includeif.gitdir:/github/workspace/.git.path
includeif.gitdir:/github/workspace/.git/worktrees/*.path
[command]/usr/bin/git config --local --get-all includeif.gitdir:/home/runner/work/Forge/Forge/.git.path
/home/runner/work/_temp/git-credentials-54d962d8-d550-4286-a024-0c4cda6d9da8.config
[command]/usr/bin/git config --local --unset includeif.gitdir:/home/runner/work/Forge/Forge/.git.path \/home\/runner\/work\/_temp\/git\-credentials\-54d962d8\-d550\-4286\-a024\-0c4cda6d9da8\.config
[command]/usr/bin/git config --local --get-all includeif.gitdir:/home/runner/work/Forge/Forge/.git/worktrees/*.path
/home/runner/work/_temp/git-credentials-54d962d8-d550-4286-a024-0c4cda6d9da8.config
[command]/usr/bin/git config --local --unset includeif.gitdir:/home/runner/work/Forge/Forge/.git/worktrees/*.path \/home\/runner\/work\/_temp\/git\-credentials\-54d962d8\-d550\-4286\-a024\-0c4cda6d9da8\.config
[command]/usr/bin/git config --local --get-all includeif.gitdir:/github/workspace/.git.path
/github/runner_temp/git-credentials-54d962d8-d550-4286-a024-0c4cda6d9da8.config
[command]/usr/bin/git config --local --unset includeif.gitdir:/github/workspace/.git.path \/github\/runner_temp\/git\-credentials\-54d962d8\-d550\-4286\-a024\-0c4cda6d9da8\.config
[command]/usr/bin/git config --local --get-all includeif.gitdir:/github/workspace/.git/worktrees/*.path
/github/runner_temp/git-credentials-54d962d8-d550-4286-a024-0c4cda6d9da8.config
[command]/usr/bin/git config --local --unset includeif.gitdir:/github/workspace/.git/worktrees/*.path \/github\/runner_temp\/git\-credentials\-54d962d8\-d550\-4286\-a024\-0c4cda6d9da8\.config
[command]/usr/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url
Removing credentials config '/home/runner/work/_temp/git-credentials-54d962d8-d550-4286-a024-0c4cda6d9da8.config'
Cleaning up orphan processes

test-matrix (3.12)

=================================== FAILURES ===================================
_____________ test_every_scan_helper_has_a_planted_violation_test ______________

    def test_every_scan_helper_has_a_planted_violation_test():
        """The house rule, run for real: no `tests/*.py` file may carry an
        AST/grep-guard helper that no planted-violation test in the same file
        actually runs."""
        offenders = _offenders()
>       assert not offenders, (
            "these test files define a scan helper (it walks source, matches a "
            "pattern, or reads a file and asks a membership question of it — or "
            f"its name carries one of {sorted(_NAME_TOKENS)}) that no test naming "
            "plant/fires/catches ever calls — a scan that has never fired has not "
            f"been shown to check anything: {offenders}"
        )
E       AssertionError: these test files define a scan helper (it walks source, matches a pattern, or reads a file and asks a membership question of it — or its name carries one of ['calls', 'check', 'checks', 'guard', 'guards', 'offenders', 'reads', 'scan', 'scans', 'uses']) that no test naming plant/fires/catches ever calls — a scan that has never fired has not been shown to check anything: ["test_escalation_reliability.py: ['_rows_by_shape']"]
E       assert not ["test_escalation_reliability.py: ['_rows_by_shape']"]

tests/test_scans_fire.py:527: AssertionError
=============================== warnings summary ===============================

windows (3.11)

................s....................................................... [ 29%]
........................................................................ [ 36%]
........................................................................ [ 44%]
........................................................................ [ 51%]
...................s..............................................ssssss [ 58%]
sssssssssssssssssss..................................................... [ 66%]
................................................ssssssssssssssssssssssss [ 73%]
sss...s...................................................ss.........s.. [ 80%]
........................................................................ [ 88%]
...............................................sss...............F...... [ 95%]
..........................................                               [100%]
================================== FAILURES ===================================
_____________ test_every_scan_helper_has_a_planted_violation_test _____________

    def test_every_scan_helper_has_a_planted_violation_test():
        """The house rule, run for real: no `tests/*.py` file may carry an
        AST/grep-guard helper that no planted-violation test in the same file
        actually runs."""
        offenders = _offenders()
>       assert not offenders, (
            "these test files define a scan helper (it walks source, matches a "
            "pattern, or reads a file and asks a membership question of it � or "
            f"its name carries one of {sorted(_NAME_TOKENS)}) that no test naming "
            "plant/fires/catches ever calls � a scan that has never fired has not "
            f"been shown to check anything: {offenders}"
        )
E       AssertionError: these test files define a scan helper (it walks source, matches a pattern, or reads a file and asks a membership question of it � or its name carries one of ['calls', 'check', 'checks', 'guard', 'guards', 'offenders', 'reads', 'scan', 'scans', 'uses']) that no test naming plant/fires/catches ever calls � a scan that has never fired has not been shown to check anything: ["test_escalation_reliability.py: ['_rows_by_shape']"]
E       assert not ["test_escalation_reliability.py: ['_rows_by_shape']"]

tests\test_scans_fire.py:527: AssertionError
============================== warnings summary ===============================
tests/test_band_probe.py::test_the_two_makers_are_maximally_apart_in_calibration
  C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\nestor\memory.py:1244: RuntimeWarning: NESTOR_SEAL_KEY not set � seal signatures are NOT verified; any 'sealed' row is trusted (Nestor#2). Set NESTOR_SEAL_KEY, or NESTOR_REQUIRE_SEAL_KEY=1 to fail closed.
    return (pair.get("status") == "sealed" and signing.seal_is_valid(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ===========================
FAILED tests/test_scans_fire.py::test_every_scan_helper_has_a_planted_violation_test - AssertionError: these test files define a scan helper (it walks source, matches a pattern, or reads a file and asks a membership question of it � or its name carries one of ['calls', 'check', 'checks', 'guard', 'guards', 'offenders', 'reads', 'scan', 'scans', 'uses']) that no test naming plant/fires/catches ever calls � a scan that has never fired has not been shown to check anything: ["test_escalation_reliability.py: ['_rows_by_shape']"]
assert not ["test_escalation_reliability.py: ['_rows_by_shape']"]
1 failed, 915 passed, 64 skipped, 1 warning in 73.29s (0:01:13)
##[error]Process completed with exit code 1.

windows (3.13)

................s....................................................... [ 29%]
........................................................................ [ 36%]
........................................................................ [ 44%]
........................................................................ [ 51%]
...................s..............................................ssssss [ 58%]
sssssssssssssssssss..................................................... [ 66%]
................................................ssssssssssssssssssssssss [ 73%]
sss...s...................................................ss.........s.. [ 80%]
........................................................................ [ 88%]
...............................................sss...............F...... [ 95%]
..........................................                               [100%]
================================== FAILURES ===================================
_____________ test_every_scan_helper_has_a_planted_violation_test _____________

    def test_every_scan_helper_has_a_planted_violation_test():
        """The house rule, run for real: no `tests/*.py` file may carry an
        AST/grep-guard helper that no planted-violation test in the same file
        actually runs."""
        offenders = _offenders()
>       assert not offenders, (
            "these test files define a scan helper (it walks source, matches a "
            "pattern, or reads a file and asks a membership question of it � or "
            f"its name carries one of {sorted(_NAME_TOKENS)}) that no test naming "
            "plant/fires/catches ever calls � a scan that has never fired has not "
            f"been shown to check anything: {offenders}"
        )
E       AssertionError: these test files define a scan helper (it walks source, matches a pattern, or reads a file and asks a membership question of it � or its name carries one of ['calls', 'check', 'checks', 'guard', 'guards', 'offenders', 'reads', 'scan', 'scans', 'uses']) that no test naming plant/fires/catches ever calls � a scan that has never fired has not been shown to check anything: ["test_escalation_reliability.py: ['_rows_by_shape']"]
E       assert not ["test_escalation_reliability.py: ['_rows_by_shape']"]

tests\test_scans_fire.py:527: AssertionError
============================== warnings summary ===============================
tests/test_band_probe.py::test_the_two_makers_are_maximally_apart_in_calibration
  C:\hostedtoolcache\windows\Python\3.13.15\x64\Lib\site-packages\nestor\memory.py:1244: RuntimeWarning: NESTOR_SEAL_KEY not set � seal signatures are NOT verified; any 'sealed' row is trusted (Nestor#2). Set NESTOR_SEAL_KEY, or NESTOR_REQUIRE_SEAL_KEY=1 to fail closed.
    return (pair.get("status") == "sealed" and signing.seal_is_valid(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ===========================
FAILED tests/test_scans_fire.py::test_every_scan_helper_has_a_planted_violation_test - AssertionError: these test files define a scan helper (it walks source, matches a pattern, or reads a file and asks a membership question of it � or its name carries one of ['calls', 'check', 'checks', 'guard', 'guards', 'offenders', 'reads', 'scan', 'scans', 'uses']) that no test naming plant/fires/catches ever calls � a scan that has never fired has not been shown to check anything: ["test_escalation_reliability.py: ['_rows_by_shape']"]
assert not ["test_escalation_reliability.py: ['_rows_by_shape']"]
1 failed, 915 passed, 64 skipped, 1 warning in 85.54s (0:01:25)
##[error]Process completed with exit code 1.

test

##[group]Run python - <<'PY'
�[36;1mpython - <<'PY'�[0m
�[36;1mimport json, os, sys�[0m
�[36;1mneeds = json.loads(os.environ["NEEDS"])�[0m
�[36;1mbad = {job: info.get("result") for job, info in needs.items()�[0m
�[36;1m       if info.get("result") != "success"}�[0m
�[36;1mfor job, result in needs.items():�[0m
�[36;1m    print(f"{job}={result.get('result')}")�[0m
�[36;1mif bad:�[0m
�[36;1m    print(f"::error::a required job did not succeed: {bad} "�[0m
�[36;1m          "(skipped and cancelled count as failures — a job that did "�[0m
�[36;1m          "not run did not pass)")�[0m
�[36;1m    sys.exit(1)�[0m
�[36;1mprint("every leg succeeded")�[0m
�[36;1mPY�[0m
shell: /usr/bin/bash -e {0}
env:
  NEEDS: {
  "test-matrix": {
    "result": "failure",
    "outputs": {}
  },
  "windows": {
    "result": "failure",
    "outputs": {}
  },
  "no-extras": {
    "result": "failure",
    "outputs": {}
  },
  "lint": {
    "result": "success",
    "outputs": {}
  }
}
##[endgroup]
test-matrix=failure
windows=failure
no-extras=failure
lint=success
##[error]a required job did not succeed: {'test-matrix': 'failure', 'windows': 'failure', 'no-extras': 'failure'} (skipped and cancelled count as failures — a job that did not run did not pass)
##[error]Process completed with exit code 1.

[willows-bot]

willow-bot status for f0c44cd772b8

• audit: not dispatched
• CI red: 5 red leg(s) filed
• CI red: test(escalation): floor and repeat-run consistency views over the rows #46 @ f0c44cd — no-extras, test-matrix (3.12), windows (3.11), windows (3.13), test — https://github.com/forge-play/Forge/actions/runs/37071552335/job/111051924271
• last tick: 2026-10-02T22:19:59Z