Repository navigation
Fix dictation freezing after short phrases and stuck sentence ends - #7
Open
adavila0703 wants to merge 1 commit into
Open
adavila0703 wants to merge 1 commit into
adavila0703 wants to merge 1 commit into
Conversation
…mmitting. - Clamp Whisper word timestamps to the audio actually transcribed. On short clips Whisper (which pads to 30s) can time the last word far past the real audio; that bogus end outlived the trim, so the timestamp fallback discarded every new word until the buffer caught up (~18s of lost speech, and buffers growing to 30s+ with 2s passes). - Commit the newest word when two passes heard it identically as a sentence end. Holding it back meant Whisper often rewrote "years." to "years, and" first, so no sentence end committed and the buffer hit the 15s cap and was cut mid-sentence. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two bugs in the streaming transcriber (
src/renderer/src/audio/streaming.ts), found in live logs while dictating:1. Dictation freezes after a short phrase. On short clips Whisper (which pads input to 30 s) can time the last word far past the real audio. For example, "observability." on a 3.2 s clip was timed about 18 s too late. When you paused, that word's audio was trimmed away, but because its end time was past the cut, the word itself stayed in
committed. On the next passes, text alignment failed and the timestamp fallback dropped every word starting before that bogus end:fresh: 0for 30+ s while the user kept talking. The same stale word also blocked trimming, so the buffer grew to 31 s with about 1.8 s passes.→ Word start and end times are now limited to the end of the audio that was sent to Whisper.
2. Sentence ends never commit until a pause. LocalAgreement held back the newest word, and a sentence end is always the newest word when it's first heard. Whisper often rewrote it first (
years.→years, and), so no sentence end got committed. The buffer then hit the 15 s cap and was cut mid-sentence (trimmed at word (buffer full)), and pass time climbed from about 90 ms to 280 ms.→ The newest word now commits when two consecutive passes heard it identically as a sentence end (
.,!,?). Tradeoff: a period Whisper hears twice in a row mid-thought now sticks.Checks
pnpm typecheckpnpm smoke/pnpm smoke:ui(if this touches native, IPC, or pages): not run; renderer-only logicAlso ran a replay against
StreamingTranscriberwith a stubbed Whisper. The old code drops all speech after a word with a bad end time and never commitsyears.; the new code handles both. Not yet confirmed by dictating live with a mic after the change.