Skip to content

feat(app): add Meta glasses capture integration#9310

Draft
moni11811 wants to merge 2 commits into
BasedHardware:mainfrom
moni11811:agent/omi4meta-meta-glasses
Draft

feat(app): add Meta glasses capture integration#9310
moni11811 wants to merge 2 commits into
BasedHardware:mainfrom
moni11811:agent/omi4meta-meta-glasses

Conversation

@moni11811

@moni11811 moni11811 commented Jul 9, 2026

Copy link
Copy Markdown

Theory
Meta glasses capture failed because the app lacked a checked-in DAT bridge plus app-level registration, queue, watchdog, and capture-cache contracts. #9084 mixed that with docs/examples/auth/device-list work, so this PR keeps only the runtime path.

Scope

  • Adds local Meta Wearables DAT Flutter bridge runtime only.
  • Adds Meta glasses page, provider, capture queue/watchdog/diagnostics, onboarding step, and photo cache handoff.
  • Adds iOS remote-command gesture bridge for glasses tap gestures.
  • Adds l10n keys needed by the UI.

Cut from #9084

  • Meta reference docs and plugin example app.
  • Auth env changes.
  • Unrelated multi-device saved-device work.
  • On-device transcription quality gate.
  • Generated iOS ephemeral files.

Tests

  • Red first: flutter test test/unit/meta_glasses_relay_queue_test.dart failed before DAT/runtime files existed.
  • Green: Meta focused batch passed, including relay queue, health, display, runtime regression, session pause, watchdog, continuous vision, background stream contract, contract behavior, sanitize, onboarding widget.

Notes

Review in cubic

@Git-on-my-level Git-on-my-level added needs-maintainer-review Needs human maintainer review before merge dependency-review Touches dependencies or lockfiles; needs dependency review security-review Touches auth, provider routing, secrets, or security-sensitive surfaces workflow-review Needs maintainer review for workflow, automation, hooks, or CI behavior needs-tests PR introduces logic that should be covered by tests labels Jul 9, 2026
@Git-on-my-level

Copy link
Copy Markdown
Collaborator

Thanks for carving the Meta glasses runtime work out of the larger branch. This is directionally interesting and the scope is more coherent than the earlier mixed PR, but I do not think this is ready for maintainer approval/merge yet.

Blocking / needs human maintainer review:

  1. Privacy/product behavior needs an explicit maintainer call. The provider defaults autoCaptureEnabled to true and, once a capture controller is attached, attempts to start capture automatically whenever registered glasses are present. For a camera wearable integration that can periodically cache images to the backend, I would want this to be explicit opt-in (or at least behind a clearly reviewed product setting/consent flow) before it can ship.

  2. The lockfile changes downgrade unrelated transitive packages and SDK constraints while adding the local plugin. That is high-risk for the whole Flutter app and not explained by the Meta integration. Please regenerate the lockfile against the repo’s intended Flutter/Dart toolchain and avoid unrelated dependency downgrades unless each one is intentional and justified.

  3. app/lib/backend/http/api/conversations.dart now changes several existing API parsers away from the generated wire models. At least TranscriptsResponse.fromJson regresses from tolerant parsing to hard casts on deepgram, soniox, whisperx, and speechmatics, so a response missing one provider key will throw instead of returning an empty list. This is unrelated to Meta glasses and could break existing transcript/compare flows.

  4. The vendored third_party/meta_wearables_dat_flutter package needs provenance/licensing review. Its podspec declares s.license = { :file => '../LICENSE' }, but I do not see that LICENSE file checked in under the vendored package path. Since this also adds native Android/iOS dependency resolution through Meta DAT SDKs and GitHub Packages, maintainers should review the license/source and build implications before accepting it.

  5. Please add/attach CI results for the focused Meta tests and, ideally, the affected app-level tests after fixing the above. The PR is draft and currently has no passing check summary available here.

I’ve labeled this for maintainer/security/dependency/workflow review plus tests. I’m intentionally not asking to split this only because it is large—the feature is cohesive—but the privacy defaults, vendored dependency surface, lockfile churn, and unrelated conversation parser changes should be addressed or explicitly accepted by maintainers before this moves out of draft.

@moni11811

Copy link
Copy Markdown
Author

Addressed review blockers in b1c34d3.

Changes:

  • Meta glasses auto-capture now defaults off. Registered glasses do not start capture without persisted opt-in.
  • Added privacy regression: registered visible glasses do not auto-start without explicit opt-in.
  • Restored transcript parser tolerance for missing provider keys and added conversation_transcripts_response_test.dart.
  • Cleaned lockfile so the worktree diff only adds the local meta_wearables_dat_flutter path package, with no SDK floor/package downgrades.
  • Added vendored package LICENSE and NOTICE so the podspec license file reference exists.

Local checks:

  • flutter test test/unit/meta_glasses_autostart_regression_test.dart --plain-name "registered visible glasses do not auto-start without explicit opt-in"
  • flutter test test/unit/conversation_transcripts_response_test.dart
  • focused Meta batch: health, display, runtime regression, session pause, watchdog, continuous vision, background stream contract, contract behavior, sanitize, onboarding widget, relay queue, transcript parser
  • dart analyze lib/providers/meta_wearables_provider.dart lib/backend/http/api/conversations.dart test/unit/meta_glasses_autostart_regression_test.dart test/unit/conversation_transcripts_response_test.dart

@Git-on-my-level

Copy link
Copy Markdown
Collaborator

Thanks for putting this together — the Meta glasses capture flow is a coherent product surface and the direction looks potentially valuable for Omi.

I’m not approving this draft, and I think it needs human maintainer review before merge because it is a large device/privacy integration with a vendored native plugin, background camera/BLE streaming, wake-lock/foreground-service behavior, permission handling, and a new photo-ingestion path.

A few things I’d want validated before this is considered merge-ready:

  • Provenance/dependency review for app/third_party/meta_wearables_dat_flutter: please make it easy to verify which upstream commit/tag this vendored copy came from, what Omi-specific changes were made, and why vendoring is preferred over a pinned package/git dependency.
  • Privacy/security review for the capture behavior: the flow can continuously capture frames from glasses and buffer them locally before upload. The user-facing consent, foreground/background notification behavior, retention/cleanup of queued frames, and failure modes should be reviewed explicitly.
  • Backend/API contract validation: the production ingestion path appears to rely on websocket image_chunk uploads, while cacheMetaWearablesPhoto() references v1/meta-wearables/photos/cache. Please confirm there is no dead or environment-only path left that could silently strand frames in production.
  • Device/runtime validation on both iOS and Android hardware: this touches Meta DAT registration callbacks, camera permission, background streaming, display status, media-remote gestures, and watchdog recovery. The added tests are helpful, but this needs maintainer/device testing evidence before merge.

I did not see an obvious malicious/supply-chain attack in the static review, but the size and native/device surfaces mean this should stay under needs-maintainer-review, security-review, dependency-review, workflow-review, and needs-tests until a maintainer has verified the above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependency-review Touches dependencies or lockfiles; needs dependency review needs-maintainer-review Needs human maintainer review before merge needs-tests PR introduces logic that should be covered by tests security-review Touches auth, provider routing, secrets, or security-sensitive surfaces workflow-review Needs maintainer review for workflow, automation, hooks, or CI behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants