Skip to content

camera_livekit_bridge: validate each copy with the slot write generation - #1549

Closed
gareth-morgan-nv wants to merge 4 commits into
gmorgan/CameraStreamingLiveKitfrom
gmorgan/CameraStreamingBridgeMarkers
Closed

gareth-morgan-nv wants to merge 4 commits into
gmorgan/CameraStreamingLiveKitfrom
gmorgan/CameraStreamingBridgeMarkers

Conversation

@gareth-morgan-nv

@gareth-morgan-nv gareth-morgan-nv commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Makes the bridge validate every copy it takes from camera_receiver with the shared per-slot write-generation marker, so a picture whose slot the receiver rewrote after announcing it is dropped instead of encoded. This replaces the notification-age check as the integrity decision. It builds on the receiver-side change in #1548 and is based on the bridge PR (#1478), so the diff here is only the bridge change.

What changed

  • The IPC client refuses a publisher that does not offer slot_write_generation, maps the markers advertised in subscribed through the same native helper the receiver uses (camera_streamer/ipc_protocol), and exposes slot_unchanged(notice).
  • The stream worker checks the marker before the GPU copy (saves work) and again after the copy into the encoder input surface has completed, before encode. A changed marker drops the picture, releases the surface, counts a slot overrun and prints a prominent SLOT OVERWRITE DETECTED error (at most every 5 s per stream).
  • The notification-age heuristic is removed: its header, --max-notice-age-ms, the stale and unchecked statistics and their tests. The newer-notice check is gone with it.
  • README: the slot-reading section describes the marker check and its limits.

Validation

  • Bridge builds standalone from a clean configure (RTX PRO 5000 host, CUDA, local LiveKit SDK).
  • Bridge tests against the real bridge binary and a real publisher with CUDA slots: a picture whose marker is unchanged is encoded and counted; a marker advanced after the notice is dropped, counted and reported with the prominent error (checked before the copy); alternating valid and overwritten pictures drop only the overwritten ones; a publisher without write generations, a missing descriptor, a wrong epoch and garbled descriptors are each refused. 7 passed.
  • Existing bridge tests that need no LiveKit client library: 4 of 4 CTest targets and the IPC probe tests pass. The pipeline and timing tests skip in this environment (no receiver venv with the LiveKit Python SDK), so they were not run.
  • The during-copy drop path is not driven deterministically from Python; it shares the check and reporting with the before-copy path.
  • Live run against the receiver with a real LiveKit subscriber and Orin sender.

Known limitations / follow-ups

  • Interim protection, as in camera_streamer: expose slot write generations for copy validation #1548: CUDA makes a read that races a write undefined behavior, so this relies on the race only changing the bridge's private copy, which it discards, and is validated by tests rather than proven. The proper fix is for the receiver to exclude overlapping access (short reader reservations it honours), planned as follow-up work that would replace this check.
  • The first commit is the receiver-side change from camera_streamer: expose slot write generations for copy validation #1548, carried here until it merges; after that merge this branch drops it. The launcher files from that PR are not carried because this base does not have them yet.
  • Time sync, integration and sine warp PRs restack on this once it lands.

Test plan

  • CI
  • Reviewer: confirm the bridge's check ordering (copy complete, then marker load, then encode) and that shipping this as interim protection is acceptable.

🤖 Generated with Claude Code

gareth-morgan-nv and others added 2 commits November 18, 2026 17:07
camera_receiver rewrites a CUDA IPC slot ring_depth pictures after announcing
it and never waits for a reader, and the replacement notice only arrives after
the new write, so a reader could not tell that its copy overlapped a rewrite.

The receiver now advances a shared per-slot write-generation marker for every
eye before any GPU work touches the slot, waits for each worker's own CUDA
stream, and announces that generation only once the write is complete. A failed
attempt publishes nothing and keeps its generation. A reader copies into memory
it owns, waits for its GPU reads to finish, and keeps the copy only if the
marker still equals the notified generation. The example consumer does this and
refuses a publisher that does not offer the feature.

The marker is a small header-only native helper with a Python binding, shared
by Python and C++ readers, in a new ipc_protocol component. Readers that ignore
the marker are unchanged and outside the guarantee.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…e generation

The bridge now maps camera_receiver's shared per-slot write-generation markers
through the same native helper the receiver uses, and refuses a publisher that
does not provide them. After the GPU copy into the encoder input surface has
completed it loads the slot's marker and encodes the picture only if the marker
still equals the notified generation; otherwise it drops the picture, counts a
slot overrun and prints a prominent error. The same check before the copy only
saves work.

The notification-age check, its flag, statistics and tests are removed, so
there is a single integrity decision.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
@coderabbitai

coderabbitai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

gareth-morgan-nv and others added 2 commits November 18, 2026 17:21
CUDA treats a read racing a write as undefined behavior, so the check is a
stopgap that makes overlaps visible rather than a documented guarantee. The
writer-side exclusion that would replace it is follow-up work.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
@farbod-nv

Copy link
Copy Markdown
Contributor

Reviewed 787b9be. I have no blocking objection to the current bridge adoption of #1548 as interim protection. The code waits for the copy to finish, checks the marker, and drops a changed slot before encoding. The bridge-specific code and shared example helper are in appropriate directories.

I accept the documented residual CUDA overlap risk for this temporary integration. A writer-honoured reservation mechanism and a deterministic test of the actual bridge's during-copy rejection can remain follow-ups; I am not requesting those changes before this PR lands. Keep the existing limitation and replacement notes, and proceed with the dependency/restack plan.

Reviewed with Codex on behalf of Farbod; this is not a formal approval.

@gareth-morgan-nv

Copy link
Copy Markdown
Contributor Author

Closing this together with the receiver-side detection PR (#1548). The bridge will adopt the slot-ownership protocol that replaces it. The branch is kept for reference.

This branch was successfully deployed

1 active deployment
dev — 787b9be5 Deployed Nov 18, 2026 by gareth-morgan-nv via publish-wheel #7094
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants