Skip to content

fix(app): gate on-device transcription hallucinations#9312

Draft
moni11811 wants to merge 2 commits into
BasedHardware:mainfrom
moni11811:agent/omi4meta-on-device-transcription
Draft

fix(app): gate on-device transcription hallucinations#9312
moni11811 wants to merge 2 commits into
BasedHardware:mainfrom
moni11811:agent/omi4meta-on-device-transcription

Conversation

@moni11811

@moni11811 moni11811 commented Jul 9, 2026

Copy link
Copy Markdown

Theory
On-device transcription accepted low-quality local Whisper/Speech output as final text. Short hallucinations, repeated filler, and weak local transcripts could leak into conversations instead of being rejected or falling back.

Scope

  • Adds on-device transcript quality gate.
  • Applies gate in socket transcription flow.
  • Tightens on-device provider handling.
  • Sets iOS speech request dictation hint and punctuation.

Cut from #9084

  • Meta glasses runtime/UI.
  • Multi-device saved-device work.
  • Native BLE/device integration churn.
  • Docs/plugin examples/auth changes.

Tests

  • Red first: flutter test test/unit/on_device_transcription_upgrade_test.dart failed on missing OnDeviceTranscriptQualityGate.
  • Green: flutter test test/unit/on_device_transcription_upgrade_test.dart passed 5 tests.

Notes

Review in cubic

@Git-on-my-level

Copy link
Copy Markdown
Collaborator

Thanks for extracting this into a focused draft — the scope is much easier to reason about than the larger branch, and the local Whisper quality gate is a useful direction.

A couple of things I think should be addressed before this is ready for maintainer review:

  • The new gate is only wired into OnDeviceWhisperProvider. On iOS, SttProvider.onDeviceWhisper returns OnDeviceAppleProvider, so the native Apple/Speech path still accepts short/filler results as final text. The PR description says it gates on-device Whisper/Speech output, but the iOS change currently only sets dictation/punctuation hints. Please either apply the same quality gate to OnDeviceAppleProvider too, or narrow the PR description/tests to make it clear this only gates the non-iOS local Whisper path.
  • The regression tests are mostly static string checks for the iOS path. It would be stronger to add behavior-level coverage for the provider path that should be gated, especially the iOS/native path if that remains in scope.

Not approving because this is still a draft and there is no passing check summary available. Once the iOS path/scope mismatch is resolved and checks are available, this should be a good candidate for another pass.

@Git-on-my-level Git-on-my-level added needs-tests PR introduces logic that should be covered by tests needs-maintainer-review Needs human maintainer review before merge labels Jul 9, 2026
@moni11811

Copy link
Copy Markdown
Author

Addressed draft review in 160d818.

Changes:

  • Wired OnDeviceTranscriptQualityGate into OnDeviceAppleProvider, not only local Whisper.
  • Apple/native iOS path now drops low-energy filler transcripts before returning final STT text.
  • Added behavior-level mocked iOS speech-channel test for Apple provider.
  • Cleaned analyzer warnings in the on-device provider files.

Local checks:

  • flutter test test/unit/on_device_transcription_upgrade_test.dart
  • dart analyze lib/services/sockets/on_device_apple_provider.dart lib/services/sockets/on_device_whisper_provider.dart lib/services/sockets/on_device_transcript_quality_gate.dart test/unit/on_device_transcription_upgrade_test.dart

@Git-on-my-level

Copy link
Copy Markdown
Collaborator

Good progress since the last pass — the quality gate is now wired into OnDeviceAppleProvider (not just local Whisper), and the new mocked iOS speech-channel test gives much better confidence than the earlier static-string checks.

A few observations for the next iteration or for the human maintainer:

  1. Dead parameter: OnDeviceTranscriptQualityGate.filter() accepts duration but never uses it. Either add a TODO for future duration-based heuristics or remove it.
  2. Magic thresholds: The low-energy threshold (avg |sample| < 180) and the six-token filler list are uncalibrated. They're reasonable starting points, but real-device audio validation would strengthen confidence that legitimate short utterances aren't dropped.
  3. State lifecycle: _lastAcceptedFiller is per-provider-instance. If providers are short-lived the duplicate-filler suppression won't persist across polling cycles; if long-lived it may suppress fillers across unrelated conversation contexts. Worth a comment documenting the expected lifecycle.
  4. Polling window change: The iOS buffer change (5s → 3s, minBufferSizeBytes doubled) affects the transcription latency/quality tradeoff. Real-device testing would help confirm shorter windows don't regress accuracy.

Not approving: still a draft with no CI checks available. Once CI confirms the new test suite passes, the needs-tests label can be lifted. Keeping needs-maintainer-review — the thresholds and polling-window changes warrant human judgment before merge.

Thanks for the focused extraction from the larger branch — the scope is clean and the quality gate direction is valuable for Omi's transcription reliability.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-maintainer-review Needs human maintainer review before merge needs-tests PR introduces logic that should be covered by tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants