Skip to content

Hill-climb lab: tune app settings against scored, held-out tests - #1791

Draft
claude[bot] wants to merge 20 commits into
mainfrom
claude/hill-climb-lab-4nbq57
Draft

Hill-climb lab: tune app settings against scored, held-out tests#1791
claude[bot] wants to merge 20 commits into
mainfrom
claude/hill-climb-lab-4nbq57

Conversation

@claude

@claude claude Bot commented Sep 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Requested by Justin · project thread

Before: tuning Transcripted meant guessing a threshold, running one bench by hand, and eyeballing the numbers. Almost every speed or speaker-naming setting was a hardcoded constant, nothing stopped a lucky run from looking like a win, and an agent had no way to drive the real app.

After: an agent can run python3 scripts/hillclimb/hillclimb.py climb <objective> --budget 30 --confirm on the Mac. It tries one setting change at a time, keeps only statistically clear, safe wins on the dev tests, checks the final winner once against a locked holdout set, and writes a recommendation with the source line to change. In a local lab build (build.sh --lab) it can also drive the real app (start/stop dictation and meetings, import audio).

Why

Justin asked for a lab where an agent can drive the app, turn its knobs, score the results, and hill-climb toward better speaker naming, faster dictation, and faster meeting notes, built the way a careful eval team would.

Product Impact

  • Affects: dictation / meetings (tooling; release builds don't contain the control channel, and knob overrides do nothing unless a lab env var is set at launch)
  • Lane: agent workflow
  • Why this matters: settings changes get evidence (paired permutation tests, a budgeted holdout, hard safety gates) instead of vibes.

What changed

  • scripts/hillclimb/ (Python stdlib): knob/objective registry, hash-stable dev/holdout splits with optional clusters, one-sided paired sign-flip tests (exact up to 16 units; p < 0.05 on dev, p < 0.01 on holdout) with min_effect, non-inferiority guardrails that fail closed, hard gates, interleaved A/B timing runs, A/A calibration, simulate-null false-accept rates, a holdout budget counted by item overlap and recorded before each check runs, sealed holdout rows, checkpoint + --resume, coordinate-ascent climber, append-only ledger and leaderboard. Benches run in one process group per trial, and a timeout kills everything in it, including what the adapter started. A synthetic demo registry runs the whole loop anywhere.
  • config/hillclimb/: 108 knobs with verified defaults and source lines (12 live, 26 bench-only, 70 mapped but hardcoded), five objectives, bench commands, suites. Includes Speaker lab: try Nemotron speaker detection side by side (after 1.1.62) #1789's speaker-lab bench, knobs and objective (its adapter lives on Speaker lab: try Nemotron speaker detection side by side (after 1.1.62) #1789).
  • Bench adapters: dictation stop path (real DictationStopBenchmarkRunner, 64 phrases, WER from saved Markdown), meeting turnaround (transcripted-cli import-audio, fresh speaker DB per run, word recall and speaker count vs truth), speaker naming across calls (SpeakerEvalHarness autoeval, every safety counter a hard gate), speech-model accuracy (plugs into Speech-to-text model shootout harness #1788's shootout).
  • Sources/TranscriptedCore/Utilities/LabKnobOverrides.swift: reads TRANSCRIPTED_LAB_KNOBS_FILE once; without it nothing changes. Wired 8 diarization/clustering constants with identical defaults. Unknown ids are dropped with a warning.
  • Sources/Support/LabControlChannel.swift + LabControlCommand.swift: file-drop control channel compiled only under #if TRANSCRIPTED_LAB_CONTROL (build.sh --lab). build-beta.sh refuses the lab flag and fails if the binary contains the channel. In lab builds the control folders must be private 0700 dirs owned by the user; command files open O_NOFOLLOW|O_NONBLOCK. Paste is off unless asked for, and starting dictation never brings another app forward. Client: scripts/hillclimb/lab_control.py (needs --container, refuses a relocated library, and turns telemetry off for the lab process only via launch arguments, never touching saved Settings). Doc: docs/lab-control-channel.md.
  • scripts/dev/typecheck-lab-build.sh type-checks the app with the lab flag on. docs/hill-climb-lab.md and test-matrix rules.
  • No workflow edits. Claude's GitHub app can't merge .github/workflows changes, so the CI hooks (a repo-hygiene step for the lab self-test + validate, and an app-build step running the lab type-check) are a separate patch for Justin: /mnt/project-files/hill-climb/workflows.patch.

Deep review fixes (/mnt/project-files/reviews/next-release/1791.md)

All of B1, S1-S8, M1-M9 and the re-check's N1-N5 are addressed. S1 (the #1789 FluidAudio 0.17 conflict) is a merge note with exact resolution steps and a grep hazard check at /mnt/project-files/hill-climb/merge-note-1789.md. S7 makes speaker-naming-across-calls refuse to climb for now: its 48 items are 16 audio qualities of the same people, so they collapse to 4 dev and 2 holdout independent units, under the 10/8 minimum.

How I checked it

  • scripts/dev/agent-preflight.sh
  • python3 scripts/hillclimb/hillclimb.py --self-test, validate, lab_control.py --self-test (36 tests), a demo climb that finds the three planted wins and confirms on holdout
  • check-build-source-lists.py, check-duplicate-declarations.py, test-matrix-checks.py --self-test, bash -n on the touched scripts
  • bash build.sh --no-open (plain and --lab), bash scripts/dev/typecheck-lab-build.sh, bash run-tests.sh, swift test, SKIP_NOTARIZATION=1 bash build-beta.sh: not run. No Swift toolchain in the session; the Swift here has never been compiled. Commits carry [skip ci] until the coordinator lifts the hold on this PR.
  • Manual check: first real runs on the Mac.

Risk Review

  • Privacy: bench results hold numbers, hashes and ids only; override values and control commands are never logged or sent off-device.
  • Release builds can't contain the control channel (compile flag + build-beta.sh binary check).
  • No private transcripts, audio, tokens, personal paths, or customer data are included.
  • Agent PR stays draft until human review.

Notes

  • Speaker naming needs the fingerprint caches (data/eval/qmatrix, ~11 GB, not produced in this repo) on the Mac, and per-person harness rows before it can climb.
  • transcripted-cli import-audio ignores the encoder compute-units setting and the model choice today, so those are left out of meeting-turnaround.

🤖 Generated with Claude Code

https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS

@claude
Add hill-climb lab core: knob registry, held-out splits, paired stats…
7cc2b90 
…, climber [skip ci]

Python stdlib tool under scripts/hillclimb that tunes app knobs against
scored benches. Dev/holdout splits are hash-stable, verdicts need a
bootstrap-CI win above min_effect with no guardrail or hard-gate
regression, timing benches interleave A/B runs, and holdout checks are
budgeted. A synthetic demo registry backs the self-tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS
@claude
Let the lab override 8 meeting-pipeline knobs from a file [skip ci]
52555d9 
LabKnobOverrides reads TRANSCRIPTED_LAB_KNOBS_FILE once per process. With
no env var nothing changes: every call returns today's default and there is
no file I/O. Wired: diarizer clustering threshold, VBx Fa/Fb, min segment
duration, and same-voice consolidation / small-cluster absorb per embedder.
Not compiled yet (no Swift toolchain in session).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS
@claude
Add dictation stop-latency bench adapter and 64-phrase suite [skip ci]
db23fbe 
Synthesizes fixtures with say, runs the real DictationStopBenchmarkRunner
once per repetition in an isolated HOME, and reports per-phrase latency,
word error rate (from the saved Markdown), and missing-text, silence-text
and unstable-output gates. Self-test now also runs bench adapter tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS
@claude
Add real knob registry, objectives, speaker naming bench, and lab gui…
668eff5 
…de [skip ci]

101 knobs (10 live, 21 bench-only, 70 mapped but hardcoded) with verified
defaults and source lines; three objectives (dictation stop latency,
meeting turnaround, speaker naming across calls); a SpeakerEvalHarness
autoeval adapter with per-cache items and every safety counter as a hard
gate; docs/hill-climb-lab.md; a test-matrix rule and a repo-hygiene step
that runs the lab self-test and registry validation.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS
@claude claude Bot assigned r3dbars Sep 23, 2026
@claude
claude Bot requested a review from r3dbars September 23, 2026 21:32
@claude
Add meeting turnaround bench adapter and corpus suite builder [skip ci]
c0b915c 
Runs transcripted-cli import-audio per corpus item with a fresh empty
speaker database, times Stop-to-transcript per second of audio, scores
word recall and speaker count against truth, and passes lab knob
overrides through TRANSCRIPTED_LAB_KNOBS_FILE (erroring if the CLI did not
confirm them). One untimed warmup import absorbs model load.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS
@claude
Add a lab control channel so an agent can drive the real app [skip ci]
22e924e 
Off unless the app is launched with TRANSCRIPTED_LAB_CONTROL_DIR. Commands
dropped as JSON into <dir>/inbox (ping, status, start/stop dictation,
start/stop meeting, import audio) call the same entry points the menus
use; responses go to <dir>/responses.jsonl. Nothing from the channel goes
off-device. scripts/hillclimb/lab_control.py launches the app and sends
commands. Swift not compiled yet (no toolchain in session).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS
@claude claude Bot mentioned this pull request Sep 23, 2026
Draft
9 tasks
@claude
Plug the speech model shootout into the lab [skip ci]
9b06398 
Adds the stt-shootout bench (script lives in PR #1788), a stt.engine knob
with the shootout's engine list, a Mac-local stt-clips suite, and a
speech-model-accuracy objective: fewer wrong words, at most ~35% slower,
no engine failures or empty text.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS
@claude claude Bot mentioned this pull request Sep 23, 2026
Draft
15 tasks
claude Bot pushed a commit that referenced this pull request Sep 23, 2026
@claude
Hill-climb bench adapter for the speaker lab [skip ci]
4db4217 
scripts/hillclimb/benches/speaker_lab.py speaks the hill-climb request/result
protocol (#1791) and drives run_speaker_lab.sh --single once per trial over
the requested AMI series, then splits scores.json + recognition-events.json
back into per-series items: recognition_rate, recognized, asked_again,
pipeline/raw DER, speaker-count error, objective, plus int gates
wrong_person and new_person_false_match. Missing/incomplete series and
series without returning speakers are item errors, never zeros. Driver env
twins are scrubbed so shell exports can't leak into a trial, and scores.json
must echo every knob that was set. app_revision hashes the harness binary,
the requested RTTM/audio, and the lab scripts.

Imports hc_benches when the climber is on the branch, else uses a local copy
of the protocol validator, so it works before and after the merge.

Also: config/hillclimb/suites/speaker-lab-ami.json (the 16 download_ami.sh
lab series, 12 dev / 4 holdout stratified by site) and
speaker_lab.README.md with the benches/knobs/objectives JSON for #1791.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
claude Bot pushed a commit that referenced this pull request Sep 23, 2026
@claude
Test matrix: speaker-lab bench adapter checks [skip ci]
9c0d250 
Changes to the adapter, its tests/README/suite, or the lab driver/scorer it runs now select py_compile + speaker_lab.py --self-test. Kept as its own rule above the SpeakerEvalHarness block so it doesn't collide with #1791's scripts/hillclimb/** rule.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
hillclimb: kill the whole process group when a bench adapter times ou…
afac88b 
…t [skip ci]

subprocess.run(timeout=) only kills the direct child, so the app, the CLI or
the speaker harness kept running into the next trial and skewed timings.
Adds hc_proc.run_group (new session + killpg) and routes every adapter
subprocess through it, with grandchild-survival tests per adapter.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS
@claude
hillclimb: exact sign-flip tests, minimum suite sizes, fail-closed gu…
4eee3bc 
…ardrails [skip ci]

Fixes the deep review's stats and holdout findings on #1791:
- S4: verdicts use a one-sided paired sign-flip permutation test (exact up
  to 16 units) instead of the percentile bootstrap; the holdout check uses
  p < 0.01. Climb and confirm refuse suites under 10 dev / 8 holdout
  independent units. simulate-null reports false-accept rates at real sizes.
- S5: a guardrail measured on fewer than half the primary's units, or that
  lost items, rejects. Metrics can declare the item field they need
  (truth, speakers, text) and the lab refuses suites that can't feed them.
- S6: holdout peeks count by holdout item overlap (>50% = same holdout), so
  adding an item no longer resets the budget. Holdout per-item values are
  sealed out of trials.jsonl and bench work dirs go to holdout-sealed/.
- S7: suite items can carry a cluster; clusters count once and never
  straddle the holdout line. Speaker items cluster by family and identity
  split, which blocks that objective until the harness emits per-person
  rows. Its recommendation now lists the per-bucket contract as required.
- S8: CommandBench runs through hc_proc.run_group.
- M1: the inconclusive re-measure pools with the first run at alpha/2.
- M2: every repetition's build/host/OS must match.
- M3: climb-result.json is checkpointed after each decision; climb --resume.
  Malformed bench results become item errors instead of crashing.
- M4: forced holdout checks are recorded as forced.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS
@claude
lab control: compile the channel only into local lab builds, lock its…
572ef8a 
… folders [skip ci]

Fixes the deep review's B1, S2, S3, M6, M8 and M9 on #1791:
- B1: LabControlChannel and its launch hook sit behind
  #if TRANSCRIPTED_LAB_CONTROL, set only by build.sh --lab. build-beta.sh
  refuses TRANSCRIPTED_LAB_BUILD and fails if the binary contains the
  channel's env var name. In lab builds the control dir, inbox/ and done/
  must be real 0700 dirs owned by this uid, re-checked every poll.
- M6: command files open O_NOFOLLOW|O_NONBLOCK and are fstat-checked
  (regular, ours, <= 64 KB) before reading; done/ moves use rename(2).
- S3: stop_dictation pastes only when paste is explicitly true;
  start_dictation calls the session directly and never activates an app.
- S2: lab_control.py launch needs --container (or --use-real-library),
  refuses a relocated capture library and telemetry-on unless overridden.
- M9: LabKnobOverrides drops unknown ids with one stderr line.
- M8: Support and Core CLAUDE.md list the lab files and Core's env var.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS
@claude
hillclimb: #1789 merge note hooks, source revision in recommendations…
913554c 
…, models-dir note [skip ci]

Review S1/M5/M7 on #1791: the clustering knob's notes say it's in cosine
units and point at the merge note for #1789's FluidAudio 0.17 distance
change; recommendation.json records the revision its source line numbers
came from; the guide explains the shared FluidAudio model cache.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS
@claude
hillclimb: nested timeouts kill the real work, telemetry off per run,…
13f5a1a 
… lab typecheck [skip ci]

Second deep-review round on #1791:
- N1: only the outermost run_group starts a new session; it marks the
  child env so an adapter's own run_group keeps the app/CLI/harness in the
  same group. The climber's timeout now kills the real work too. Two-level
  tests (the outer one fails on the old code).
- S2 follow-up: lab_control launch turns analytics and crash reporting off
  for the launched process only, as NSArgumentDomain launch arguments, so
  the person's saved Settings are never needed or changed.
- N2: scripts/dev/typecheck-lab-build.sh type-checks the app with
  -D TRANSCRIPTED_LAB_CONTROL; app-build CI runs it after the normal build.
- N3: a holdout check writes a started row before it runs, so a crash still
  uses up budget.
- N4/N5: guide notes on #1788's audio naming and LabKnobOverrides' env var.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS
@claude
hillclimb: add #1789's speaker-lab bench, knobs and objective [skip ci]
00516ee 
Pasted from #1789's speaker_lab.README.md: the speaker-lab command bench,
6 new knobs (diarization backend, Nemotron preset, match mode and floor,
replay dedup, write-path fixes), speaker-lab-recognition, and an identical
copy of its suite. The adapter lives on #1789, so trials are item errors
until it merges. The objective reports BLOCKED: 4 holdout series, need 8.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS
@claude
hillclimb: take #1789's 24-series speaker-lab suite (8 holdout) [skip…
70595bb 
… ci]

Byte-for-byte copy of config/hillclimb/suites/speaker-lab-ami.json from
#1789 at ffe5871 (salt speaker-lab-ami-v2, 16 dev / 8 holdout series), so
speaker-lab-recognition is no longer BLOCKED.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS
@claude
Merge origin/main into claude/hill-climb-lab-4nbq57 (brings #1785's C…
4f21d06 
…I freeze fix) [skip ci]

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS
@claude
hillclimb: speaker-lab objective is no longer blocked; update notes […
cc90274 
…skip ci]

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS
r3dbars pushed a commit that referenced this pull request Sep 24, 2026
@claude
Restore inline repo-hygiene steps; add linux-checks.sh as one extra step
6fba39f 
Four open PRs (#1791, #1803, #1810, #1812) append steps to the inline
repo-hygiene workflow, so replacing those steps would conflict and a lazy
resolution would silently drop their checks. Keep the original steps
byte-for-byte and insert a single "Linux checks (scripts/dev/linux-checks.sh)"
step right after Checkout, away from where the other PRs append. The
overlap in runtime is intentional.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GhNAWeNXuLGdq4ioddevdp
@claude
Merge origin/main into claude/hill-climb-lab-4nbq57; move workflow ed…
935e39f 
…its out [skip ci]

Resolves the Sources/Support/CLAUDE.md conflict (main's updated mic
processing line plus the lab control-channel line).

Drops this PR's two .github/workflows edits (repo-hygiene hill-climb step,
app-build lab type-check step): Claude's GitHub app can't merge workflow
changes, so they go in their own PR for Justin. Patch kept at
/mnt/project-files/hill-climb/workflows.patch. The checks stay in
.agents/test-matrix.yml for local runs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS
@claude
Merge origin/main into claude/hill-climb-lab-4nbq57 [skip ci]
ca3db9c 
Resolves Sources/TranscriptedCore/CLAUDE.md: takes main's per-file map (no
counts) and adds LabKnobOverrides.swift to the Utilities line; the
"Environment variables Core reads" section is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS
@claude
Merge origin/main into claude/hill-climb-lab-4nbq57 [skip ci]
44c76eb 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS
@claude
Merge origin/main into claude/hill-climb-lab-4nbq57
882fdbf 
First CI run for the hill-climb lab after the 1.1.63 freeze.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS
@r3dbars r3dbars mentioned this pull request Sep 25, 2026
Merged
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@r3dbars r3dbars Awaiting requested review from r3dbars

Assignees

@r3dbars r3dbars

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@r3dbars @claude