Skip to content

Fix dictation freezing after short phrases and stuck sentence ends - #7

Open
adavila0703 wants to merge 1 commit into
mainfrom
angel/lyra-server-debug-965e0e
Open

adavila0703 wants to merge 1 commit into
mainfrom
angel/lyra-server-debug-965e0e

Conversation

@adavila0703

Copy link
Copy Markdown
Member

Summary

Two bugs in the streaming transcriber (src/renderer/src/audio/streaming.ts), found in live logs while dictating:

1. Dictation freezes after a short phrase. On short clips Whisper (which pads input to 30 s) can time the last word far past the real audio. For example, "observability." on a 3.2 s clip was timed about 18 s too late. When you paused, that word's audio was trimmed away, but because its end time was past the cut, the word itself stayed in committed. On the next passes, text alignment failed and the timestamp fallback dropped every word starting before that bogus end: fresh: 0 for 30+ s while the user kept talking. The same stale word also blocked trimming, so the buffer grew to 31 s with about 1.8 s passes.
→ Word start and end times are now limited to the end of the audio that was sent to Whisper.

2. Sentence ends never commit until a pause. LocalAgreement held back the newest word, and a sentence end is always the newest word when it's first heard. Whisper often rewrote it first (years. → years, and), so no sentence end got committed. The buffer then hit the 15 s cap and was cut mid-sentence (trimmed at word (buffer full)), and pass time climbed from about 90 ms to 280 ms.
→ The newest word now commits when two consecutive passes heard it identically as a sentence end (., !, ?). Tradeoff: a period Whisper hears twice in a row mid-thought now sticks.

Checks

  • pnpm typecheck
  • pnpm smoke / pnpm smoke:ui (if this touches native, IPC, or pages): not run; renderer-only logic
  • Stays local-first (no cloud APIs, chat LLMs, or telemetry)

Also ran a replay against StreamingTranscriber with a stubbed Whisper. The old code drops all speech after a word with a bad end time and never commits years.; the new code handles both. Not yet confirmed by dictating live with a mic after the change.

…mmitting.

- Clamp Whisper word timestamps to the audio actually transcribed. On short
  clips Whisper (which pads to 30s) can time the last word far past the real
  audio; that bogus end outlived the trim, so the timestamp fallback discarded
  every new word until the buffer caught up (~18s of lost speech, and buffers
  growing to 30s+ with 2s passes).
- Commit the newest word when two passes heard it identically as a sentence
  end. Holding it back meant Whisper often rewrote "years." to "years, and"
  first, so no sentence end committed and the buffer hit the 15s cap and was
  cut mid-sentence.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant