test(escalation): hosted Kaggle arm of the escalation benchmark — tasks, converter, smoke-tested on five models by willows-bot[bot] · Pull Request #44 · forge-play/Forge

willows-bot · 2026-10-01T04:28:06Z

Bite

The hosted half of the escalation benchmark: four Kaggle Benchmarks tasks (escalation-bench-{route,classify,judge,ground}) that collect raw replies from hosted models under the same prompts, answer envelope and 512-token cap as the local runner, plus kaggle_rows.py, which turns a kaggle b t download tree into runner-format rows that the unchanged aggregate.py scores.

What was done

Tasks (benchmarks/escalation/kaggle/*_task.py): one item, one call, at temperature 0 / seed 0. A failed call is a result, not a retry. The exception is a 429 rate limit, which is retried up to 6 calls with 4/8/16/32/60 s waits and records attempts.
- Typed answer schemas. Route has one object per catalogue tool or ESCALATE. Classify has {status, severity, needs_human} or ESCALATE. Judge is a literal verdict set. Ground is a string. Every object is closed and inlined, with no $defs, so OpenAI strict mode and Gemini structured output both accept them. The old open dict[str, Any] made Gemini return {} and OpenAI return 400.
- One settings block, identical in all four files (a drift test enforces it): reasoning low for google/gemini- and openai/ slugs, no temperature for openai/gpt-5* (they reject 0), and max_completion_tokens for openai/.
- Reply handling. A leading deepseek <think> block is stripped after a failed parse, and the raw reply is kept. A schema-violating reply scores as unparseable (text="" plus schema_violation) instead of being dropped as an error row.
- grok-4.6 is out of this batch: the proxy returned 404 for it.
Converter (kaggle_rows.py): accepts both escalation-bench-<shape> and escalation-<shape>. Passes attempts, temperature and schema_violation through.
CI: pydantic added to the test extra, so the typed-schema tests run on the .[test] legs.
Merge: origin/master (test(escalation): thinking off for Gemma 4 as well as Qwen, through both backends #43, Gemma 4 thinking off) merged in at 1aef4cc.

Evidence

Loki audits, each PASS on its own commit: D4B02BB2 (8c7eff6), 4917FD62 (7df3c34), 37410321 (6b0a557), 4915E060 (230a695), 44B451D9 (07e29b4), 23A827B1 (6284cdc), BD6DED92 (278b871), BC6303CD (0ae6935), 9BFE3FE1 (b2e1f99).
Tests after the merge (Kart AY4Y50CB, at 1aef4cc): 391 passed, 26 skipped in 15.83s; ruff 0.16.7 All checks passed!.
Real-SDK tests under the step3 Python 3.13 venv (Kart 9SCUS2KA, builder 2200D2F3): 401 passed.
Mutation proof, each mutant red with a green control on a copy that keeps .git:
- Converter prefix: assert 0 == 60.
- Open dict restored (dict[str, Any]): 7 failed.
- No temperature reject set: 8 failed.
- Retry everything: 20 failed.
- User turn resent on retry: 5 failed.
- Open dict restored in judge: 5 failed.
- Error-row behaviour restored: 12 failed.
Live smoke on Kaggle (rows under ~/.forge/escalation-runs/kaggle-smoke{2,3}):
- gemini-3.7-flash: 96.2% overall (route 0/33 → 97.8% after the typed schema).
- gpt-oss-20b: 86.2%, 0 errors.
- gpt-5.4-nano: route and classify 85.9% after the temperature fix. Judge and ground returned 400 under the old open schema, which is fixed in 0ae6935.
- deepseek-r1: 67.3%. 429 errors fell from 169 to 1, and items took 1–6 attempts. 27% of its replies are cut mid-JSON at the 512 cap; that is a measured condition, not a defect.

Out of scope

runner.py's local schema and scoring, and the 512 cap value.
Recovering the raw text of a schema-violating reply from the chat (gap bea73d7262fd).
The paid 11-model run and the DEV.to writeup; both follow this PR.

Next bite

Merge, then run the paid 11-model set in two days under the $10/day cap (cheap models first). Then group the four tasks into a public Kaggle benchmark for the DEV.to #kagglechallenge entry, due 2026-10-11.

Decision

Sealed Nestor decisions this PR implements:

279aedee: the ratified hosted model set.
9c9ecdfc: grok-4.6 is out of this batch, which amends 279aedee to eleven models.
cf803012: a 429 alone is retried with bounded backoff and each row records attempts; every other failure is recorded once.
d227ee33 / d84fa045: the standing Kaggle grant, which lives in the willow-bot box. Until the box is built, each Kaggle call is a per-task seal.

The push and PR ride envelopes env-git.push-d3291939c70d and env-pr.open-b2cd03b02854, both ratified by the operator.

Ratified-by: sean-campbell — "ratify"

🤖 Generated with Claude Code

https://claude.ai/code/session_016jPFpbxqZYKT1gTfw8pKkD

… and dataset gate Everything the desk needs to run the benchmark on a hosted runner and score it with the unchanged aggregate.py, built and tested offline. - kaggle_export.py writes one JSON line per fixture item (id, shape, messages, schema). messages come from runner.build_chat_payload, so the hosted prompt and the local one cannot drift; the answer key is never exported. - kaggle/{route,classify,judge,ground}_task.py: one kaggle-benchmarks 0.6.1 task per shape. A per-item sub-task (store_task=False) sends system + user at seed 0 and temperature 0 and records the raw reply; the top-level task evaluates it over a DataFrame with on_failure="continue" and returns completed/errored counts. Nothing is parsed on the host. - kaggle_rows.py reads a `kaggle b t download` tree and writes runner-format rows, reusing runner.make_row so parsing is identical; errors and missing items become error rows. - kaggle_stage.py builds the dataset upload directory and refuses, deleting it, unless every gate passes: file set and line counts against manifest.json, the answer key, everything privacy_gate.py checks, store ids, Unicode lookalikes, compressed payloads. The deny-list and the staging directory must both be outside every git tree. - privacy_gate.py ALLOWED_OTHER gains the new files; README documents the flow. Tests: export parity for every item, converter round-trip into aggregate.py on a hand-built tree in the download layout, and the stage gate against a planted email, file URI, store id, deny-list term, lookalike name, count mismatch and answer key. Four mutants (dropped user message, unfolded confusables, skipped line counts, unflagged store ids) each turn the suite red. Persona: hanuman Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vcntd3qgAxd6gnPEvVukVm

…, output cap, unknown done_reason The hosted task files called respond() with its default str schema, never sent MAX_OUTPUT_TOKENS, and left done_reason null, so a hosted model was scored on easier terms than a local one (Loki D4B02BB2 parity finding). - Each task file passes an Answer type (runner.ANSWER_SCHEMA as a pydantic model) to respond(). The parsed reply is written back out as the JSON text aggregate.py parses, with the SDK's raw text kept as raw_text. A reply that breaks the schema is kept as unparseable text, like a local one. ESCALATION_SCHEMA=none sends none; each item records which. - The 512-token cap is sent as max_tokens, or max_output_tokens for Google GenAI models; ESCALATION_CAP_PARAM overrides the name or turns it off. - kaggle-benchmarks 0.6.1 reports no finish reason, so kaggle_rows writes done_reason "unknown" and aggregate.py counts it apart from "length" and shows Truncated as unknown. Local rows and their output are unchanged. Persona: hanuman Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vcntd3qgAxd6gnPEvVukVm

…ape> task names The tasks were renamed to escalation-bench-<shape> when pushed, so RunDir.shape stripped "escalation-", saw "bench-<shape>", and skipped every task: the 204-file smoke download converted 0 rows (gap 2c4c46da429d). shape now tries escalation-bench- then escalation-; any other name is still skipped. Persona: hanuman Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016jPFpbxqZYKT1gTfw8pKkD

…a; tasks take the live names The hosted route and classify tasks declared answer as str | dict[str, Any], an open object with no properties. Gemini's structured output drops additionalProperties, so the model could only emit {}: every route and classify row in the smoke run parsed to answer {} and scored 0/33 and 0/38 (gap a84645278b60). - route_task.py: answer is one explicit object per catalogue tool (tool fixed to its name, args limited to that tool's arguments, copied from catalog/tools.json and pinned to it by a test) or the string ESCALATE. An optional catalogue argument is a required nullable key, which OpenAI strict output needs; a null is dropped from the recorded reply, so it scores like a local reply that omits it. - classify_task.py: answer is {status, severity, needs_human} with the values prompts/classify.txt allows, or the string ESCALATE. - Both models are inlined (no $defs), because kaggle_benchmarks 0.6.1 treats a schema with $defs as nested and for an OpenAI-style model drops response_format and only pastes the schema into the prompt. - Each row records schema "answer-typed"; ESCALATION_SCHEMA=none still records "none". The reply text is the JSON aggregate.py parses for a local reply; a test feeds typed replies through the task, kaggle_rows and aggregate and gets the same cell as the equivalent runner rows. A reply that breaks the typed schema stays raw text and is scored like a local one. - The load_frame guard no longer compares the exported schema to the model field for field. It requires the same envelope fields, the same required set, the same confidence type, and that the typed answer admits exactly the JSON types the export's answer admits (object and string); a dropped ESCALATE arm, a dropped object arm or a new type is refused. - Hidden reasoning shares the 512-token cap. route and classify send respond(reasoning=) by model-name prefix: "low" for google/ and openai/, nothing for other vendors, which keep their default. ESCALATION_REASONING overrides it. The cap is not raised. - Task names are now escalation-bench-<shape> and escalation-bench-<shape>-item, as live on Kaggle. judge and ground differ from kaggle-escalation/step3/push2 in nothing; route and classify differ from it only by the changes above. Hosted/local difference, deliberate (Loki 4917FD62): the local runner and the exported dataset keep runner.ANSWER_SCHEMA (answer is a string or an open object); only the two hosted task files type it. The module header of both says so. Tests: the typed schema has no open object; the route schema matches the catalogue; the classify schema matches the prompt; every fixture answer is expressible; typed replies score like local ones; the guard accepts the runner schema and refuses six mismatches. Restoring the open dict[str, Any] in a scratch copy turns 5 route tests and 2 classify tests red; the unmutated copy is green. Persona: hanuman Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016jPFpbxqZYKT1gTfw8pKkD

… cap per slug, send one reasoning table on all four shapes Closes the four open items Loki raised on 230a695 (4915E060, gap 5f1d65e364a2) so every slug in the ratified set can be run. deepseek-r1 <think>. kaggle_benchmarks 0.6.1 strips a leading <think> block only inside OpenAI._call_api and only when a reasoning level was sent. A probe through the real SDK over a canned transport (deepseek-ai/deepseek-r1-0528, both parse paths) showed: with no level the reply fails the schema; with level "low" the text path parses, but the structured-output path (beta.chat.completions.parse) still raises a pydantic ValidationError before the strip runs, and an inlined schema takes that path on all four shapes. deepseek is built with support_structured_outputs=False (model_proxy.py), so its live path is the text one, but whether the proxy accepts reasoning_effort for it is unverifiable offline and a refusal would cost every row. So the task strips it instead: when the SDK's parse has already failed, a closed leading <think> block is taken off, the rest is validated as the shape's Answer, and the text is the answer JSON (or the stripped text when it does not validate). The reply as sent is kept as raw_text. A reply cut off inside its think block has nothing to strip and stays unparseable text. A ValidationError with no think block is still an error row. Cap parameter. The SDK passes the name through to the proxy unchanged; it has no per-model mapping. The openai/ prefix (gpt-5.5, gpt-5.4-nano, gpt-oss-20b) now sends max_completion_tokens (the OpenAI API deprecates max_tokens and refuses it for reasoning models); everything else keeps max_tokens, and a Google GenAI client keeps max_output_tokens. The value stays 512. ESCALATION_CAP_PARAM still overrides it. Uniform reasoning. One settings table (reasoning level and cap name by model-name prefix, plus the think helpers) now sits in all four task files, copied identically because each file is standalone on Kaggle, and judge and ground now send and record the same reasoning level as route and classify. A test fails if the four copies drift. google/ is narrowed to google/gemini-: gemma does not think and was being sent a thinking level. CI. pydantic is added to the test extra: the Linux matrix and Windows legs install .[test], so the typed kaggle tests run there instead of skipping at importorskip. A separate CI leg would add a job to the required aggregate for one dependency; the no-extras leg stays dependency-free and still skips them. A test holds the extra and the two .[test] installs. The real-SDK tests importorskip kaggle_benchmarks, so they run only where the SDK is installed (not in CI). grok-4.6: nothing changed. Tests: every ratified slug gets its cap name and reasoning level on every shape; the think block is stripped on the stub path, the pydantic path, the schema-off path, and through the real SDK (16 cases); a cut-off think block and a plain ValidationError behave as before; the four settings blocks are identical; the test extra lists pydantic. Each new test was made red in a scratch copy that keeps .git (strip_think a no-op, cap table emptied, one copy drifted, pydantic removed from the extra, google/ prefix restored, ValidationError no longer caught); the unmutated copy is green. Persona: hanuman Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016jPFpbxqZYKT1gTfw8pKkD

… a 429, drop grok-4.6 from the batch Smoke2 blockers (gap 2b2d3c896251). Temperature: openai/gpt-5 (nano confirmed by smoke2 400, gpt-5.5 inferred, SDK has no per-model rule) sends no temperature; every other slug sends 0; each row records temperature 0 or "default". 429 retry: only an exception class named RateLimitError, up to 6 calls, waits 4/8/16/32/60 s (120 s max), only llm.respond is repeated so the chat holds one user turn (checked through the real SDK), row records attempts; any other error is recorded once. grok-4.6 leaves the ratified set for this batch (operator: "For this batch, retire and drop Grok"). The four task files keep one identical settings block; runner.py untouched. Persona: hanuman Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016jPFpbxqZYKT1gTfw8pKkD

…onverted rows Sealed cf803012 says each row records how many calls a 429 took. 6284cdc put attempts and temperature in the hosted task's result, but make_rows dropped them, so rows.jsonl never showed either (Loki 23A827B1 F1). Copy both from the run-file result when it recorded them; rows without them are unchanged. Persona: willow Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016jPFpbxqZYKT1gTfw8pKkD

… strict mode accepts smoke3: every gpt-5.4-nano judge and ground call (90 of 90) was a 400, "Invalid schema for response_format 'Answer': ... 'additionalProperties' is required to be supplied and to be false", because both still declared answer: str | dict[str, Any]. A judge answer is one of four strings (SUPPORTS, CONTRADICTS, UNRELATED, ESCALATE) and a ground answer is a string (a phrase from the passage, or ESCALATE); aggregate.answers_match scores a string for both and never credits an object. So judge types answer as the four labels and ground as str, both with extra=forbid, and rows now record the answer-typed schema label like route and classify. check_exported_schema is one rule in all four task files. The export keeps the runner's open schema (string or object), so a typed answer narrower than it is allowed: the export's arms must stay within (object, string), and the typed answer must keep the string arm (ESCALATE) and add no arm the export lacks. A typed object arm the export lacks, a lost string arm, a new export type and changed envelope fields are still refused. Equality would refuse the string-only shapes; route and classify behave as before. Tests: openai_strict_problems() (no open object, every object closed and fully required, no $ref) runs on every shape's generated schema and on the request body the real kaggle_benchmarks SDK sends for each shape; a control proves it fails on the old str | dict[str, Any], on the schema and on the real SDK request. The four task files are held identical outside the typed block. The three gpt-5.4-nano ValidationErrors were route items whose reply matched no catalogue arm (a mis-shaped object); they are unrelated to this schema. Persona: hanuman Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016jPFpbxqZYKT1gTfw8pKkD

…le, not a dropped error row smoke3: nano route-017, route-038 and route-060 raised a pydantic ValidationError that is not json_invalid. reply_in_error returned None for it, the task recorded an error row, and aggregate leaves error rows out of every denominator, so hosted accuracy was inflated against local, where a schema-violating reply stays unparseable text and counts wrong on an answerable item. The reply text is not recoverable for these: the SDK's structured-output parse raises before it appends the reply to the chat, and a non-json_invalid error carries only the offending sub-values. So the task now records text empty plus a bounded schema_violation (str(exc), 2000 characters), which converts to parse_ok False and error None. A json_invalid reply with no think block is kept as text and raw_text instead of an error row. A RateLimitError, BadRequestError or any other failure stays an error row. kaggle_rows passes schema_violation through like attempts and temperature. Same change in all four task files (the settings block stays identical). Re-converting kaggle-smoke3 with the old run files still gives error rows for the three items, since those results were written by the old task code. Under the new code nano route would go from 40/45 (0.889, 3 errors) to 40/48 (0.833, 3 unparseable). Persona: hanuman Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016jPFpbxqZYKT1gTfw8pKkD

willows-bot · 2026-10-01T04:30:10Z

CI red: #44 @ 1aef4cc — 3 job(s):

windows (3.11) (failure): https://github.com/forge-play/Forge/actions/runs/36815254079/job/110218807400
windows (3.13) (failure): https://github.com/forge-play/Forge/actions/runs/36815254079/job/110218807071
test (failure): https://github.com/forge-play/Forge/actions/runs/36815254079/job/110219181718

windows (3.11)

E         Full diff:
E           [
E               'README.md',
E         -     'catalog/tools.json',
E         ?             ^
E         +     'catalog\\tools.json',
E         ?             ^^
E               'classify.jsonl',
E               'dataset-metadata.json',
E               'ground.jsonl',
E               'judge.jsonl',
E               'route.jsonl',
E           ]

tests\test_escalation_kaggle.py:1567: AssertionError
============================== warnings summary ===============================
tests/test_band_probe.py::test_the_two_makers_are_maximally_apart_in_calibration
  C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\nestor\memory.py:1244: RuntimeWarning: NESTOR_SEAL_KEY not set � seal signatures are NOT verified; any 'sealed' row is trusted (Nestor#2). Set NESTOR_SEAL_KEY, or NESTOR_REQUIRE_SEAL_KEY=1 to fail closed.
    return (pair.get("status") == "sealed" and signing.seal_is_valid(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ===========================
FAILED tests/test_escalation_kaggle.py::test_stage_builds_the_dataset_and_passes_every_gate - AssertionError: assert ['README.md',...e.jsonl', ...] == ['README.md',...e.jsonl', ...]
  
  At index 1 diff: 'catalog\\tools.json' != 'catalog/tools.json'
  
  Full diff:
    [
        'README.md',
  -     'catalog/tools.json',
  ?             ^
  +     'catalog\\tools.json',
  ?             ^^
        'classify.jsonl',
        'dataset-metadata.json',
        'ground.jsonl',
        'judge.jsonl',
        'route.jsonl',
    ]
1 failed, 907 passed, 64 skipped, 1 warning in 58.20s
##[error]Process completed with exit code 1.

windows (3.13)

E         Full diff:
E           [
E               'README.md',
E         -     'catalog/tools.json',
E         ?             ^
E         +     'catalog\\tools.json',
E         ?             ^^
E               'classify.jsonl',
E               'dataset-metadata.json',
E               'ground.jsonl',
E               'judge.jsonl',
E               'route.jsonl',
E           ]

tests\test_escalation_kaggle.py:1567: AssertionError
============================== warnings summary ===============================
tests/test_band_probe.py::test_the_two_makers_are_maximally_apart_in_calibration
  C:\hostedtoolcache\windows\Python\3.13.15\x64\Lib\site-packages\nestor\memory.py:1244: RuntimeWarning: NESTOR_SEAL_KEY not set � seal signatures are NOT verified; any 'sealed' row is trusted (Nestor#2). Set NESTOR_SEAL_KEY, or NESTOR_REQUIRE_SEAL_KEY=1 to fail closed.
    return (pair.get("status") == "sealed" and signing.seal_is_valid(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ===========================
FAILED tests/test_escalation_kaggle.py::test_stage_builds_the_dataset_and_passes_every_gate - AssertionError: assert ['README.md',...e.jsonl', ...] == ['README.md',...e.jsonl', ...]
  
  At index 1 diff: 'catalog\\tools.json' != 'catalog/tools.json'
  
  Full diff:
    [
        'README.md',
  -     'catalog/tools.json',
  ?             ^
  +     'catalog\\tools.json',
  ?             ^^
        'classify.jsonl',
        'dataset-metadata.json',
        'ground.jsonl',
        'judge.jsonl',
        'route.jsonl',
    ]
1 failed, 907 passed, 64 skipped, 1 warning in 64.42s (0:01:04)
##[error]Process completed with exit code 1.

test

##[group]Run python - <<'PY'
�[36;1mpython - <<'PY'�[0m
�[36;1mimport json, os, sys�[0m
�[36;1mneeds = json.loads(os.environ["NEEDS"])�[0m
�[36;1mbad = {job: info.get("result") for job, info in needs.items()�[0m
�[36;1m       if info.get("result") != "success"}�[0m
�[36;1mfor job, result in needs.items():�[0m
�[36;1m    print(f"{job}={result.get('result')}")�[0m
�[36;1mif bad:�[0m
�[36;1m    print(f"::error::a required job did not succeed: {bad} "�[0m
�[36;1m          "(skipped and cancelled count as failures — a job that did "�[0m
�[36;1m          "not run did not pass)")�[0m
�[36;1m    sys.exit(1)�[0m
�[36;1mprint("every leg succeeded")�[0m
�[36;1mPY�[0m
shell: /usr/bin/bash -e {0}
env:
  NEEDS: {
  "test-matrix": {
    "result": "success",
    "outputs": {}
  },
  "windows": {
    "result": "failure",
    "outputs": {}
  },
  "no-extras": {
    "result": "success",
    "outputs": {}
  },
  "lint": {
    "result": "success",
    "outputs": {}
  }
}
##[endgroup]
test-matrix=success
windows=failure
no-extras=success
lint=success
##[error]a required job did not succeed: {'windows': 'failure'} (skipped and cancelled count as failures — a job that did not run did not pass)
##[error]Process completed with exit code 1.

willows-bot · 2026-10-01T04:30:14Z

willow-bot status for 1aef4cc48684

audit: not dispatched
CI red: 3 red leg(s) filed
CI red: test(escalation): hosted Kaggle arm of the escalation benchmark — tasks, converter, smoke-tested on five models #44 @ 1aef4cc — windows (3.11), windows (3.13), test — https://github.com/forge-play/Forge/actions/runs/36815254079/job/110218807400
last tick: 2026-10-01T04:30:03Z

…ws passes The stage test built paths with str(relative_to), which is catalog\tools.json on Windows against EXPECTED_FILES' catalog/tools.json; Forge#44 windows (3.11/3.13) went red on that one assertion. Use as_posix(). Persona: willow Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016jPFpbxqZYKT1gTfw8pKkD

rudi193-cmd and others added 10 commits September 30, 2026 17:01

Merge origin/master into test/escalation-kaggle

1aef4cc

willows-bot Bot requested a review from rudi193-cmd October 1, 2026 04:28

willows-bot Bot assigned rudi193-cmd Oct 1, 2026

willows-bot Bot added the willow-bot/audit-dispatched label Oct 1, 2026

rudi193-cmd merged commit 7c67f55 into master Oct 1, 2026
17 checks passed

rudi193-cmd deleted the test/escalation-kaggle branch October 1, 2026 04:44

rudi193-cmd approved these changes Oct 1, 2026

View reviewed changes

Navigation Menu

Repository navigation

test(escalation): hosted Kaggle arm of the escalation benchmark — tasks, converter, smoke-tested on five models - #44

test(escalation): hosted Kaggle arm of the escalation benchmark — tasks, converter, smoke-tested on five models#44
rudi193-cmd merged 11 commits into
masterfrom
test/escalation-kaggle

willows-bot Bot commented Oct 1, 2026

Uh oh!

willows-bot Bot commented Oct 1, 2026

Uh oh!

willows-bot Bot commented Oct 1, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

willows-bot Bot commented Oct 1, 2026

Bite

What was done

Evidence

Out of scope

Next bite

Decision

Uh oh!

willows-bot Bot commented Oct 1, 2026

Uh oh!

willows-bot Bot commented Oct 1, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant