Skip to content

Stream dictation with LocalAgreement, fix mid-session stalls, refresh UI - #6

Merged
adavila0703 merged 1 commit into
mainfrom
angel/voice-text-blog-improvements-bff205
Nov 13, 2026
Merged

adavila0703 merged 1 commit into
mainfrom
angel/voice-text-blog-improvements-bff205

Conversation

@adavila0703

Copy link
Copy Markdown
Member

Summary

Makes talking out a blog post feel smooth: sentences come out properly punctuated, words appear as you speak, and dictation no longer goes dead mid-session.

Streaming transcription (LocalAgreement-2)

  • Replaces pause-based chunking, which handed Whisper sentence fragments and broke punctuation. Whisper now re-reads a rolling buffer about every 0.5s, a word is committed only once two passes agree on it, and the buffer is trimmed at committed sentence ends. Earlier text is passed as the prompt. Based on Whisper-Streaming (paper).
  • Whisper's guesses are flaky in a few specific ways, and each now has a fix:
    • Already-committed words are found by matching text, because whisper.cpp timestamps shift between passes.
    • The newest word is held back until the next one arrives.
    • Trailing ... is stripped.
    • A period Whisper adds late is still committed.
  • In main, PCM goes straight to whisper.cpp with no temp WAV, the model is warmed up ahead of time, and calls run one at a time.
  • Saying "new paragraph" or "new line" inserts a break, and a trailing "new"/"next" waits to see whether "paragraph" follows.

Fix: dictation going dead mid-session

  • The VAD could stay stuck in "speaking" mode in a slightly noisy room because its hold threshold never adapted. It then never finalized, and the Whisper buffer grew without limit until passes returned nothing. In a simulation of a noisy room, the old VAD detected 1 start of speech in 2 minutes; the new one detected 14 of 14.
    • The noise floor is now the quietest frame of the last 3 seconds, and the thresholds are capped.
    • Past 15s, the buffer is forced to commit and trim.
  • A suspended AudioContext (Bluetooth or AirPods switching) is resumed, and a lost mic track now shows an error.

Logging

  • Writes to the terminal and to userData/logs/lyra.log, rotated at 5 MB, with:
    • a pipeline heartbeat every 3s (frames, levels, noise floor, VAD state, buffer length, pass time, audio context and track state), flagged STALLED;
    • Whisper timing and text for every pass, including dropped hallucinations;
    • a warning when a whisper.cpp call is still running after 10s.

UI

  • New sidebar with search, a serif editor, and a floating dictation dock with a live waveform.
  • Unconfirmed words appear as gray text at the insertion point.
  • Dark mode and a hidden-inset title bar on macOS.
  • Fixes a lost save when switching pages during autosave.
  • Copy exports Markdown.
  • Shortcuts: ⇧⌘D toggles dictation and ⌘N creates a new page.

Notes for review

  • Not yet tested with a live microphone. All testing used synthesized speech fed in real time through the real whisper.cpp addon:
    • A 28s recording came out word for word.
    • A 74s run of continuous speech over noise never stalled.
  • One stress run duplicated a phrase in the middle. It didn't happen on a rerun, and it is probably caused by the test splicing clips mid-word. The fallback that could cause it is now logged.
  • streaming.ts and vad.ts hold the main tuning knobs: STEP, MAX_BUFFER and END_PAUSE.

Checks

  • pnpm typecheck
  • pnpm smoke:ui (pnpm smoke was not run)
  • Stays local-first (no cloud APIs, chat LLMs, or telemetry; logs stay on disk)

- Replace pause-based chunking with Whisper streaming (LocalAgreement-2):
  re-transcribe a rolling buffer, commit words two passes agree on, trim at
  sentence ends, and carry prior text as the prompt so punctuation holds up.
- Show unconfirmed words as ghost text at the insertion point.
- Fix dictation going dead mid-session: the VAD noise floor now uses minimum
  statistics with capped thresholds, and the stream buffer forces progress
  past 15s. Recover suspended audio contexts and report a lost microphone.
- Add a log file (userData/logs/lyra.log) with a 3s pipeline heartbeat,
  per-pass Whisper timing and VAD/capture events.
- Modernize the UI: sidebar, editor, floating dictation dock, autosave fix.

Co-Authored-By: Claude <noreply@anthropic.com>
@adavila0703
adavila0703 merged commit 462f2dd into main Nov 13, 2026
4 checks passed
@adavila0703
adavila0703 deleted the angel/voice-text-blog-improvements-bff205 branch November 13, 2026 16:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant