Repository navigation
test(escalation): a Unix-socket backend, a one-model ladder with --resume/--tail, and an explicit output cap - #42
Merged
Conversation
Where the benchmark runs with no route to the model server but a delegate on a Unix socket can reach it, socket_backend.py sends the same request as ollama_http (temperature 0, ANSWER_SCHEMA as the format, Qwen /no_think) through the delegate's chat op. The socket path comes from ESCALATION_SOCKET; no path is written in the tree. `ladder` runs models one at a time and asks the delegate to unload each before the next loads, so one model is held in memory. It refuses to overwrite a rows file that has rows. --resume continues one: rows already there are skipped (an error row counts as done; a failure is never retried), a torn last line is dropped, the run id is kept. --tail prints one progress line per row. The privacy gate's file allow-list gains socket_backend.py. Persona: willow Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vcntd3qgAxd6gnPEvVukVm
…every row Nothing set an output limit, so the server's own default decided it: a small default cuts a route answer mid-JSON, an unbounded one lets a model pad under a schema until the call times out. Either way the rows could not tell a cut-off answer from a wrong one. - runner: MAX_OUTPUT_TOKENS = 512, sent as options.num_predict by build_chat_payload, so ollama_http and the socket backend send the same cap. A done_reason row field carries the server's own stop reason. - socket_backend: sends max_tokens from the same payload and reads done_reason back. - aggregate: n_truncated per cell (done_reason == "length", error rows excluded) and a Truncated column in the table. Truncated rows are scored as usual; the count keeps them from passing silently as unparseable. Persona: willow Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vcntd3qgAxd6gnPEvVukVm
rudi193-cmd
approved these changes
Sep 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bite
The escalation benchmark can run its local ladder through a Unix-socket delegate, one model in memory at a time, resumable, with truncation visible on every row.
Decision
Build the socket route rather than run Ollama over HTTP from the sandbox: the benchmark runs where there is no route to the model server, and a delegate on a Unix socket can reach it. Ratified by the operator ("you build it"); the explicit output cap and truncation count ratified separately ("yes on the fix"). The socket request is held equal to
ollama_http's, so rows from either backend are the same measurement.What was done
socket_backend.py(stdlib): sends the same request asollama_http(temperature 0,ANSWER_SCHEMA, Qwen/no_think, the output cap) through the delegate'schatop; socket path fromESCALATION_SOCKETonly.ladder: runs models one at a time and unloads each before the next; refuses to overwrite a rows file;--resumeskips rows already present (an error row counts as done), drops a torn last line, ends an unterminated row, keeps the run id;--tailprints one progress line per row.MAX_OUTPUT_TOKENS = 512sent asoptions.num_predictby both backends; newdone_reasonrow field.n_truncatedper cell (done_reason == "length", error rows excluded) and a Truncated column.socket_backend.py; README documents the backend and flags.Evidence
python3 -m pytest -q tests/test_escalation_socket_backend.py tests/test_escalation_runner.py tests/test_escalation_fixtures.py→135 passed in 2.12spython3 benchmarks/escalation/gate/privacy_gate.py→privacy gate: OK, 15 file(s) scanned5 failed; tail off →2 failed; newline guard off →1 failed; unload removed →3 failed; cap removed →12 failed, 68 passed; row done_reason unset →2 failed; truncation count off →1 failed; socket done_reason dropped →2 failed; socket max_tokens dropped →1 failedOut of scope
0addc7f9); truncated rows stay in the unparseable count they already fed.Next bite
After this and willow-bot#82 merge: smoke run (
ladder --limit 2, all 8 models, batch lane) to measure latency, tokens_in and truncation per model, then the full ladder with--resume.Ratified-by: sean-campbell — "you build it" / "yes on the fix"
🤖 Generated with Claude Code
https://claude.ai/code/session_01Vcntd3qgAxd6gnPEvVukVm