Skip to content

Speaker lab: try Nemotron speaker detection side by side (after 1.1.62) - #1789

Draft
claude[bot] wants to merge 34 commits into
mainfrom
claude/project-thread-l3ripi
Draft

Speaker lab: try Nemotron speaker detection side by side (after 1.1.62)#1789
claude[bot] wants to merge 34 commits into
mainfrom
claude/project-thread-l3ripi

Conversation

@claude

@claude claude Bot commented Sep 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Requested by Justin · project thread

Before: meetings split speakers with one model (pyannote through FluidAudio 0.15.4). There was no way to try another speaker model, or to measure how often a returning person gets recognized on their next call.

After: FluidAudio is on 0.17.0, with the diarizer behavior and model caches kept as they were. NVIDIA's Nemotron 3 speaker model can be swapped in behind a hidden switch that's off by default. A speaker lab scores every combination on the same meetings: who spoke when, speaker counts, and cross-call recognition (recognized, wrong person, asked to name again).

Draft. CI is green on macOS with FluidAudio 0.17.0 (main merged in, including the MLX removal). It can't merge until the Mac runs below pass.

Why

FluidVoice shipped local meeting recording with speaker labels on Nemotron 3. Before switching anything, we should know whether it beats our diarizer, and which knobs raise cross-call recognition.

Product Impact

  • Affects: meetings, and dictation through the FluidAudio bump
  • Lane: meeting reliability
  • Why this matters: fewer wrong-speaker turns, and fewer people to name again on every call.

What changed

  • FluidAudio 0.15.4 → 0.17.0.
    • build-deps' generated manifest moves to tools 6.2 with traits: [], so the new Rust text-normalization library isn't linked. swiftLanguageModes: [.v5].
    • FluidAudioCompatibility (Core) keeps what shipped:
      • It resolves the diarizer model repo at main. 0.17 pins it to a commit and deletes markerless caches, which would include the copy bundled in the app.
      • The tuned pyannote config is converted to 0.17's distance threshold (sqrt(0.8)) with constrained assignment off. A test pins every value.
    • build-beta.sh deletes any .fluidaudio-revision marker from the bundled diarizer models, so a pinned-build cache can't ship a marker into the app.
    • CLI diarize/batch keep the 0.15.x defaults, and config files keep cosine thresholds. The CLI's offline guard for local model folders (Parakeet Ultra) moves from DownloadUtils.enforceOffline (removed in 0.16) to ModelHub.offlineMode.
    • Parakeet keeps the two new long-form defaults off.
    • Not pinnable: 0.17 also changed >15 s chunk merging and retries blank v3 decodes up to 5 extra times (FluidAudio [operator] Treat Sparkle no-update cycles as successful #909), which also hits dictation. The bump must not merge until scripts/stt_fluidaudio_ab.sh passes on a Mac (see below).
  • Nemotron backend in Core (DiarizationBackend, NemotronDiarizationRunner, pure NemotronTurnBuilder → non-overlapping turns).
    • Voiceprints come from the injected embedder (ERes2Net), or else FluidAudio's online WeSpeaker model.
    • Those vectors use their own speakers_wespeaker-fluid-online.sqlite until parity with the saved people is measured.
  • Hidden switch: TRANSCRIPTED_DIARIZATION_BACKEND=nemotron or the diarization-backend-preference default. No UI, and it's read at launch.
  • Speaker lab (scripts/run_speaker_lab.sh, score_speaker_lab.py, harness dump --backend/--embedder, replay knobs for thresholds and fingerprint updates).
    • It scores raw DER, pipeline DER, speaker-count error and cross-call recognition on 24 AMI series (same people across meetings, no one in two series). The hill-climb suite splits them 16 dev / 8 holdout.
    • Own-calls mode writes a local timeline page.
    • --single writes scores.json for the hill-climbing lab.
    • Replay deletes its temp database with the -wal/-shm side files.
  • Harness review pass plus --embedding-parity (online vs offline WeSpeaker, with an "interchangeable or not" verdict in the report).
  • Hill-climb bench adapter for Hill-climb lab: tune app settings against scored, held-out tests #1791 (scripts/hillclimb/benches/speaker_lab.py; registry snippets in its README). Merge steps with Hill-climb lab: tune app settings against scored, held-out tests #1791 are written down; whichever PR lands second follows them.
  • FluidAudio A/B (scripts/stt_fluidaudio_ab.sh): builds the CLI on origin/main with 0.15.4 and on this branch with 0.17.0, then compares Parakeet V3 on about 27 min of captioned lecture (WER) and on generated silence/noise clips (how fast a blank result comes back, which is what decides how fast dictation stops). Gate: WER no more than +0.5 pp, and silence/noise at most 1.25x and +300 ms slower. Writes result.json + report.md.

How I checked it

  • CI green on macOS (app-build, checks, spm-tests, repo-hygiene), built against FluidAudio 0.17.0.
  • Main's newer FluidAudio calls checked against the 0.17.0 source; the DownloadUtils one above was the only break. 0.17 pins only the diarizer repo, so existing Parakeet caches and the Ultra folder aren't deleted.
  • Mac runs, written as steps for after the release: the speaker lab bake-off with embedding parity, then the FluidAudio A/B on its own.
  • Linux-runnable checks:
    • scripts/dev/linux-checks.sh (52/52), which now also runs test_score_speaker_lab.py, test_speaker_lab.py and test_stt_fluidaudio_ab.py
    • check-source-pins.py --changed-only, agent-preflight
  • Independent code-reading reviews against the FluidAudio 0.15.4 and 0.17.0 sources. Blockers and S4 are fixed. S1 is covered by the A/B above.

Risk Review

  • Privacy / local-first reviewed. Models run on device. Lab outputs are gitignored, hold meeting ids only, and contain no audio or transcript text.
  • No private transcripts, audio, tokens, personal paths, or customer data.
  • The dependency bump touches every meeting and dictation path. It stays blocked on a passing FluidAudio A/B. Consider shipping it on its own.

Notes

  • Nemotron caps at 8 speakers, builds features for the whole recording in memory (~370 MB per 2 h), and can't be cancelled mid-run. That's fine for an opt-in lab switch; revisit before it's ever the default.
  • The Nemotron model license is OpenMDW 1.1 per the FluidAudio 0.17.0 release notes. Confirm on the model card before shipping.
  • SwiftPM still downloads FluidAudio's NemoTextProcessing binary with traits: []; it isn't linked.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ

@claude
Bump FluidAudio to 0.17.0 with today's behavior pinned [skip ci]
0e1758b 
0.17.0 adds Nemotron 3 Diarization, which the speaker lab needs. The bump
itself should change nothing users see, so:

- build-deps: tools 6.2 manifest so FluidAudio can opt out of its default
  NemoTextProcessing trait (a prebuilt Rust static lib we never call and
  would not archive). swiftLanguageModes [.v5] keeps in-tree targets on
  the language mode they built under before.
- DiarizationService: clusteringThreshold is now a Euclidean cut distance,
  so the tuned 0.6 cosine becomes sqrt(0.8); constrainedAssignment pinned
  off to match 0.15.x assignment.
- Parakeet (app + CLI): keep the 0.15.x long-form chunking
  (melChunkContext on, no seam-gap repair).
- Integration fake FluidAudio grows the ASRConfig shape.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude claude Bot assigned r3dbars Sep 23, 2026
@claude
claude Bot requested a review from r3dbars September 23, 2026 21:12
claude added 25 commits September 23, 2026 21:14
@claude
Add a hidden diarization backend preference, off by default [skip ci]
963d000 
DiarizationBackendPreferences reads TRANSCRIPTED_DIARIZATION_BACKEND, then
the diarization-backend-preference default, then falls back to pyannote.
No Settings UI. Not wired into meetings yet; that lands with the Core
Nemotron backend.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
Add opt-in Nemotron 3 diarization backend to DiarizationService [skip…
ef9c3e9 
… ci]

DiarizationBackend (.pyannote default, .nemotron experimental) picks the
model. The pyannote path is unchanged. The Nemotron path loads a FluidAudio
Nemotron3 preset (fast128 by default, TRANSCRIPTED_NEMOTRON_PRESET for the
lab), runs it on a private serial queue, turns frame probabilities into
exclusive speaker turns with the pure NemotronTurnBuilder, and embeds each
turn with the injected embedder or a new FluidWeSpeakerSegmentEmbedder,
since Nemotron emits no voiceprints.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
Test Nemotron turn building and the diarization backend switch [skip ci]
a0e4535 
NemotronTurnBuilder: exclusivity, argmax on overlap, ties, gap bridging,
min-duration drop and blip rejoin, first-appearance remapping, empty and
malformed input. DiarizationService: pyannote default, backend round-trip,
not-loaded error, thresholds, shared reembed path. Nemotron preset override
and the WeSpeaker fallback embedder's pure helpers.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
Document the diarization backends in the Core CLAUDE.md [skip ci]
ee223f9 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
Rename the explicit-embedder reembed test [skip ci]
5645bbb 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
Merge branch 'nemotron-core' into claude/project-thread-l3ripi
8e483af
@claude
Wire the hidden diarization switch into meetings [skip ci]
c71ddd7 
MeetingSessionController passes the chosen backend to DiarizationService.
With Nemotron and no ERes2Net, voiceprints come from FluidAudio's online
WeSpeaker model, so they go to their own speakers_wespeaker-fluid-online
database (meetings and Settings > People agree) until the lab proves they
match the offline pipeline's vectors. Default pyannote is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
Harness: dump by diarizer backend and embedder, replay with per-model…
9956d01 
… thresholds and fingerprint-update knobs [skip ci]

dump takes --backend pyannote|nemotron and --embedder native|eres2net
(--eres2net-model for a path-based model) and records backend, embedder,
dimension, wall time, and audio length. Old dumps still decode.

replay takes --thresholds auto|weSpeaker|eRes2Net, --match adaptive,
--same-voice, --dedup, and the write-back EMA knobs. Defaults reproduce
the old behavior for WeSpeaker dumps. Per meeting it now reports raw
cluster count and whether each cluster matched an existing profile.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
Speaker eval: move scoring math into a shared, dependency-free module…
da2c346 
… [skip ci]

speaker_eval_common.py now holds RTTM parsing, overlap helpers, the
identity metrics (fragmentation, false merge, re-ID curve), and a pure
Python DER/JER that matches pyannote.metrics to float precision
(collar convention, UEM, overlap, Hungarian mapping). It is about 40x
faster, which matters once the lab scores many replays.

score_speaker_eval.py uses it and no longer needs pyannote installed.
Its markdown and JSON output are unchanged on a synthetic fixture.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
Speaker lab scorer: raw vs pipeline DER, speaker-count error, returni…
99f5144 
…ng-speaker recognition [skip ci]

score_speaker_lab.py scores a lab run directory into scores.json and
REPORT.md: per variant, raw diarizer DER/JER and count error, pipeline
DER after clustering and DB matching, and the recognition scoreboard
(recognized / wrong person / asked again / undetected, plus new people
false-matched to a known profile). It picks each variant's best knob
setting and shows which knob values moved recognition. Own-calls mode
reports behavior without ground truth, agreement against a baseline
variant, and writes a self-contained timeline.html.

Also has helper subcommands the driver uses: grid, dump-ok, and
own-calls-list. Unit tests run on Linux with synthetic fixtures.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
Speaker lab driver: one command for the diarizer bake-off, sweep or s…
d4ea70d 
…ingle trial [skip ci]

run_speaker_lab.sh dumps each meeting once per variant (backend x
embedder x Nemotron preset) into its own cache, replays every knob
setting, and scores it all into reports/speaker-lab/<stamp>/. Every
knob is a flag with an env twin, the run is non-interactive, fails loud,
and prints the scores.json path as its last stdout line. --single runs
one variant at one setting for an outer optimizer. --own-calls runs the
same variants on saved meetings (call track read in place) and writes a
timeline page.

Tests drive the script end to end against a fake harness, so the
orchestration is covered on Linux too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
download_ami.sh: add a 'lab' preset spanning three AMI recording site…
0004606 
…s [skip ci]

16 scenario series (8 Edinburgh, 4 Idiap, 4 TNO), 4 sessions each with
the same 4 people, for the speaker lab's cross-call recognition test.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
Docs: speaker lab section, scores.json schema, and test-matrix rules …
86eff01 
…[skip ci]

README covers what the lab measures, how to run it on AMI and on your
own saved calls, every knob, how an optimizer drives --single trials,
the scores.json schema, and how to add a diarizer or embedder. The
test matrix now runs the lab's syntax checks and unit tests when the
lab files change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
Keep 0.15.x diarizer caches and CLI config meaning under FluidAudio 0…
2c7d3f0 
….17 [skip ci]

Review of #1789 found the bump was not as behavior-neutral as claimed:

- FluidAudio 0.17 pins speaker-diarization-coreml to one commit and
  deletes any cache without a matching revision marker. Every 0.15.x
  cache, and the offline-diarizer-models copy bundled in the app, has
  none, so the first meeting would delete inside the signed bundle (or
  fail offline). FluidAudioCompatibility.keepUnpinnedDiarizerCaches()
  resolves that repo at main, as 0.15.x did; DiarizationService,
  FluidWeSpeakerSegmentEmbedder and the CLI call it before loading.
- The tuned pyannote config moves to
  FluidAudioCompatibility.tunedOfflineDiarizerConfig() and a test pins
  the distance threshold, constrainedAssignment=false and every tuned
  value.
- CLI diarize/batch start from the 0.15.x default (cosine 0.6, no
  constrained assignment), and a config file's clusteringThreshold stays
  a cosine similarity, converted for 0.17.
- Parakeet comments now say only the two new defaults are pinned: 0.17
  also changed >15 s chunk merging with no switch.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
Note FluidAudio 0.17's new attributions and unpinned behavior changes…
f17f455 
… [skip ci]

- THIRD_PARTY_LICENSES: FluidAudio 0.17's Japanese G2P (Misaki ports,
  UniDic) and Spanish/French lexicon notices; NeMo text processing is not
  linked.
- Parakeet comment: 0.17 retries blank v3 decodes up to five more times,
  so near-silent dictation can take longer and return text.
- build-deps: FluidAudio's resource bundle never ships, so LuxTts G2p must
  stay unused.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
Harness: fail loudly on unknown Nemotron presets, bad on/off values, …
be2f68d 
…and valueless flags [skip ci]

A dump with an unknown TRANSCRIPTED_NEMOTRON_PRESET recorded the typo while
Core's runner silently ran fast128, so the cached dump lied about its variant.
Validate with FluidAudio's Nemotron3Config.preset(named:) and record 'default'
for unset. --write-path-fixes now rejects anything but on|off (a typo used to
run the legacy path), and a flag with no value is an error instead of the default.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
Harness: embedding-parity compares pyannote's WeSpeaker vectors with …
c9c338a 
…the online WeSpeaker embedder [skip ci]

Diarizes one file with today's pipeline, re-embeds the same segments with
FluidWeSpeakerSegmentEmbedder (what Nemotron falls back to), and reports
per-segment, within-model, cross-model and per-cluster cosine stats against the
WeSpeaker match floor, plus a looksInterchangeable heuristic. Answers whether
Nemotron voiceprints could share speakers.sqlite instead of a separate DB.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
Record the Nemotron preset that actually ran in harness dumps [skip ci]
d46cd19 
Add DiarizationService.resolvedNemotronPresetName(environment:), a public
wrapper over the runner's internal resolver, so the harness records 'fast128'
instead of 'default' when TRANSCRIPTED_NEMOTRON_PRESET is unset, and refuses a
name Core would silently replace. The lab scorer treats unset, 'default' (older
dumps) and 'fast128' as one variant so existing caches stay reusable.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
Hill-climb bench adapter for the speaker lab [skip ci]
4db4217 
scripts/hillclimb/benches/speaker_lab.py speaks the hill-climb request/result
protocol (#1791) and drives run_speaker_lab.sh --single once per trial over
the requested AMI series, then splits scores.json + recognition-events.json
back into per-series items: recognition_rate, recognized, asked_again,
pipeline/raw DER, speaker-count error, objective, plus int gates
wrong_person and new_person_false_match. Missing/incomplete series and
series without returning speakers are item errors, never zeros. Driver env
twins are scrubbed so shell exports can't leak into a trial, and scores.json
must echo every knob that was set. app_revision hashes the harness binary,
the requested RTTM/audio, and the lab scripts.

Imports hc_benches when the climber is on the branch, else uses a local copy
of the protocol validator, so it works before and after the merge.

Also: config/hillclimb/suites/speaker-lab-ami.json (the 16 download_ami.sh
lab series, 12 dev / 4 holdout stratified by site) and
speaker_lab.README.md with the benches/knobs/objectives JSON for #1791.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
Test matrix: speaker-lab bench adapter checks [skip ci]
9c0d250 
Changes to the adapter, its tests/README/suite, or the lab driver/scorer it runs now select py_compile + speaker_lab.py --self-test. Kept as its own rule above the SpeakerEvalHarness block so it doesn't collide with #1791's scripts/hillclimb/** rule.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
Speaker lab: --embedding-parity runs the WeSpeaker parity check and r…
e3a559f 
…eports it [skip ci]

run_speaker_lab.sh --embedding-parity (EMBEDDING_PARITY=1) runs the harness's
embedding-parity per meeting (RTTM labels on corpora, pyannote clusters on own
calls), cached in data/eval/<corpus>/parity/. The scorer pools the reports via
their 200-bin histograms into an additive embeddingParity block in scores.json
and an 'Embedding parity' section in REPORT.md, with the same interchangeability
rule as the harness. schemaVersion stays 1; runs without the flag are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
Merge the speaker-lab hill-climb bench adapter [skip ci]
74951c3
@claude
Never bundle a diarizer revision marker into the app [skip ci]
5c8267b 
A cache written by a pinned FluidAudio 0.17 build carries a
.fluidaudio-revision marker naming a commit. The app resolves the
diarizer repo at main, so a shipped marker naming anything else would make
it delete files inside its own signed bundle. build-beta drops the marker
after copying the cache.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
speaker-lab bench: track the driver's new preset + parity behavior [s…
b255f61 
…kip ci]

Rebased onto e3a559f. The dump now records the resolved Nemotron preset and the scorer treats unset/default/fast128 as one variant, so the knob echo check normalizes presets the same way. fast128 is still not passed because the driver names variants after the preset string (passing it would fork the dump cache). EMBEDDING_PARITY joins the scrubbed env twins; --embedding-parity is a diagnostic and never passed per trial.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
Speaker lab replay: also delete the SQLite -wal/-shm side files [skip…
25d46be 
… ci]

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
claude Bot pushed a commit that referenced this pull request Sep 24, 2026
@claude
hillclimb: #1789 merge note hooks, source revision in recommendations…
913554c 
…, models-dir note [skip ci]

Review S1/M5/M7 on #1791: the clustering knob's notes say it's in cosine
units and point at the merge note for #1789's FluidAudio 0.17 distance
change; recommendation.json records the revision its source line numbers
came from; the guide explains the shared FluidAudio model cache.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS
@claude claude Bot mentioned this pull request Sep 24, 2026
Draft
9 tasks
@claude
Add a one-command FluidAudio 0.15.4 vs 0.17.0 A/B for Parakeet V3 [sk…
92baa0c 
…ip ci]

The 0.17 bump can't merge until WER is no worse and blank-audio dictation
stops aren't slower (PR #909 retries blank V3 decodes up to 5 times).

scripts/stt_fluidaudio_ab.sh builds transcripted-cli twice in git worktrees
under ~/stt-fluidaudio-ab: the baseline from origin/main (pre-bump source,
since 0.15.4 can't compile the new ASRConfig call) with FLUID_AUDIO_VERSION
0.15.4, the candidate from HEAD with 0.17.0. Builds are reused while their
inputs are unchanged. It refuses a baseline ref that already uses the 0.17
API and checks the version SwiftPM actually resolved.

scripts/stt_fluidaudio_ab.py then measures, reusing the STT shootout's
lecture download, caption parsing and WER scorer (read from git when the
shootout isn't in the checkout):
- WER: first 20 min of the lecture plus 10 caption-aligned ~45 s pieces
- stop time: silence 1/3/8 s, near-silence, low noise, room rumble, a
  0.5 s noise burst (gated), plus `say` clips (reported), ABBA rounds,
  median of CLI processingSeconds
Gate flags: --max-wer-delta-pp 0.5, --max-time-ratio 1.25,
--max-time-delta-ms 300. Writes result.json + report.md; last stdout line
is the result.json path.

scripts/test_stt_fluidaudio_ab.py covers the math, fixtures and a fake-CLI
end-to-end run, and drives the bash wrapper against a throwaway repo with
stubbed Mac tools. Runs on Linux.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
Test matrix + scripts README: FluidAudio A/B checks [skip ci]
0d0501e 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
claude Bot pushed a commit that referenced this pull request Sep 24, 2026
@claude
hillclimb: add #1789's speaker-lab bench, knobs and objective [skip ci]
00516ee 
Pasted from #1789's speaker_lab.README.md: the speaker-lab command bench,
6 new knobs (diarization backend, Nemotron preset, match mode and floor,
replay dedup, write-path fixes), speaker-lab-recognition, and an identical
copy of its suite. The adapter lives on #1789, so trials are item errors
until it merges. The objective reports BLOCKED: 4 holdout series, need 8.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS
@claude
Speaker lab: 24 AMI series so the hill-climb holdout has 8 independen…
ffe5871 
…t groups [skip ci]

The hill-climb lab needs >= 10 dev and >= 8 holdout independent units before an
objective can confirm anything; 16 series (12/4) left speaker-lab-recognition BLOCKED.
Adds ES2004 ES2011 ES2012 ES2013 IS1006 IS1007 TS3007 TS3008 (all four sessions have
pyannote RTTMs, and no participant appears in two series). Holdout re-picked per site by
lowest unit_hash under a new salt (speaker-lab-ami-v2): 4 ES, 2 IS, 2 TS.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
claude Bot pushed a commit that referenced this pull request Sep 24, 2026
@claude
hillclimb: take #1789's 24-series speaker-lab suite (8 holdout) [skip…
70595bb 
… ci]

Byte-for-byte copy of config/hillclimb/suites/speaker-lab-ami.json from
#1789 at ffe5871 (salt speaker-lab-ami-v2, 16 dev / 8 holdout series), so
speaker-lab-recognition is no longer BLOCKED.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8VU2yBBnE8eH5f18mubhS
@claude
Merge main into the Nemotron speaker lab branch [skip ci]
cc385eb 
Resolves the Core and SpeakerEvalHarness CLAUDE.md file maps: keeps main's
per-file map and adds this branch's diarization backend, FluidAudio
compatibility and speaker lab entries.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
CLI: use ModelHub.offlineMode, FluidAudio 0.17 dropped DownloadUtils …
41ad10a 
…[skip ci]

main's --models-dir guard (keeps FluidAudio from deleting a Parakeet Ultra
install that fails to load) used DownloadUtils.enforceOffline, which 0.16
renamed to ModelHub.offlineMode. Same semantics: loadModels rethrows
instead of purging and re-downloading. Also corrects the local-install
layout comment: v3 file names are unchanged through 0.17.0.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
Merge main (1.1.63 freeze) into the Nemotron speaker lab branch
de7d0a6 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
linux-checks: run the speaker lab and FluidAudio A/B test suites
cdd8bb6 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@claude
Merge main (MLX removal #1834) into the Nemotron speaker lab branch
d125dde 
build-deps.sh: keep FluidAudio 0.17.0 with traits: [] and take main's
removal of mlx-swift-lm.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvYQfvK9JYWs3DB2fNkVNQ
@r3dbars r3dbars mentioned this pull request Sep 25, 2026
Merged
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@r3dbars r3dbars Awaiting requested review from r3dbars

Assignees

@r3dbars r3dbars

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@r3dbars @claude