Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 21 additions & 12 deletions examples/camera_livekit_bridge/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@ sequence is the track's `frame_id`, the sensor start time (when the receiver
derived one) is the `user_timestamp` in Unix microseconds, and the sensor
counter is the 8-byte `user_data`. The newest picture is encoded and older
queued ones are skipped. The receiver overwrites a slot without notice once its ring wraps, so after the GPU copy the
bridge checks for a newer notice for the same slot, or as many newer notices as the ring has other slots, and drops the
picture instead of encoding it (counted as `slot overruns`). The session also answers clock-sync requests that
bridge checks the slot's shared write-generation marker and drops the picture instead of encoding it when the receiver
began a write meanwhile (counted as `slot overruns`). The session also answers clock-sync requests that
arrive as LiveKit data messages.

## Integration boundary
Expand All @@ -33,8 +33,8 @@ wiring (`livekit_publisher/` and the session handling in `bridge/`) is temporary
behind transport adapters: a layer that carries camera video and application messages over a selectable network
transport and owns sessions, admission limits and feedback. Camera buffer ownership and encode and decode
responsibilities belong to the camera interface. A replacement must preserve browser delivery, bounded latency and
safe GPU access. Until receiver slots have read leases, the residual risk described under "Reading the receiver's
slots" applies.
safe GPU access. Until the receiver excludes overlapping slot access instead of letting the bridge detect it, the
residual described under "Reading the receiver's slots" applies.

| Current path | Intended owner or replacement | Reuse, adapt or retire | Follow-up and landing condition |
| --- | --- | --- | --- |
Expand Down Expand Up @@ -113,15 +113,24 @@ build/probe/camera_ipc_probe --socket /tmp/camera_receiver_avs1_cuda_ipc.sock \

It prints each stream's plane layout, then per frame the slot, generation,
stream sequence, session id, sensor start time and a checksum of the first
pixels. Slots are overwritten without notice when the publisher's ring wraps.
pixels. Slots are overwritten without notice when the publisher's ring wraps; the probe does not validate the pixels it
checksums.

## Reading the receiver's slots

The receiver rewrites its GPU ring slots in rotation and never waits for a reader: a slot is overwritten without
notice about `ring_depth` frames after it was announced, and the only signal is a later frame notice. A consumer
therefore has to finish with a slot within that window or skip ahead. The bridge follows this: it uses only the
newest notice, starts the GPU copy at once on its own stream, waits for it to complete, and drops (never encodes)
the picture when later notices show the ring may have wrapped; those frames count as slot overruns in the `[stats]`
line. The notices arrive asynchronously, so this check cannot prove a copy was clean: a stall of most of the ring's
window between the notice and the end of the copy can still publish a picture that mixes two frames under the earlier
frame's metadata. Only a read guard in the receiver, which skips slots a reader holds, would remove that case.
notice about `ring_depth` frames after it was announced, and the replacement notice only arrives after the new write.
It therefore advances a shared write-generation marker for the slot before it touches the pixels and announces that
generation in the frame notice once the write is complete. The bridge maps the markers through the same native helper
the receiver uses (`camera_streamer/ipc_protocol`), uses only the newest notice, starts the GPU copy into the encoder
input surface at once on its own stream, waits for it to complete, and then loads the marker. It encodes the picture
only if the marker still equals the notified generation; otherwise it drops it, prints a prominent error (at most every
5 s per stream) and counts a slot overrun in the `[stats]` line. The same check before the copy only saves work. A
publisher that does not provide write generations is refused at connect time.

This is interim protection, not a documented guarantee. CUDA's memory model makes a read that races a write undefined
behavior, and the check relies on the race only changing the bridge's private copy, which it discards. It makes every
detected overlap visible and narrows the window, but a picture that passed it is very likely intact rather than proven
intact. The proper fix is for the receiver to exclude overlapping access (short reader reservations it honours), planned
as follow-up work that would replace this check. See `camera_streamer/ipc_protocol/README.md` for the ordering
contract.
11 changes: 3 additions & 8 deletions examples/camera_livekit_bridge/bridge/main.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ void usage()
{
std::cerr
<< "usage: camera_livekit_bridge --livekit-url URL --livekit-token-file FILE --stream NAME=TRACK[:DIV] [--stream ...]\n"
" [--socket PATH] [--fps N] [--bitrate-mbps N] [--idr-period N] [--max-notice-age-ms N] [--gpu N] [--source-full-range]\n [--timestamp-basis monotonic_us|capture_monotonic_us|capture_unix_us]\n"
" [--socket PATH] [--fps N] [--bitrate-mbps N] [--idr-period N] [--gpu N] [--source-full-range]\n [--timestamp-basis monotonic_us|capture_monotonic_us|capture_unix_us]\n"
" NAME is a stream in camera_receiver's IPC catalog; TRACK is the LiveKit track name egocap expects;\n DIV shrinks the picture on the GPU by that integer factor before encoding.\n";
}

Expand Down Expand Up @@ -91,8 +91,6 @@ int main(int argc, char** argv)
defaults.bitrate_bps = static_cast<uint32_t>(std::atof(next().c_str()) * 1e6);
else if (arg == "--source-full-range")
defaults.source_full_range = true;
else if (arg == "--max-notice-age-ms")
defaults.max_notice_age_ms = static_cast<uint32_t>(std::atoi(next().c_str()));
else if (arg == "--idr-period")
defaults.idr_period = static_cast<uint32_t>(std::atoi(next().c_str()));
else if (arg == "--timestamp-basis")
Expand Down Expand Up @@ -200,12 +198,9 @@ int main(int argc, char** argv)
std::cerr << "[stats] " << workers[i]->config().track << ": " << (frames - last_frames[i]) / seconds
<< " fps, " << frames << " frames, " << stats.keyframes.load() << " keyframes, "
<< stats.skipped.load() << " skipped, " << stats.rejected.load() << " rejected, "
<< stats.slot_overruns.load() << " slot overruns, " << stats.stale_slot_reads.load()
<< " stale slot reads, encode " << mean_ms << " ms mean, bitrate "
<< stats.bitrate_bps.load() / 1000 << " kbps (transport target "
<< stats.slot_overruns.load() << " slot overruns, encode " << mean_ms
<< " ms mean, bitrate " << stats.bitrate_bps.load() / 1000 << " kbps (transport target "
<< stats.target_bps.load() / 1000 << " kbps, " << stats.rate_changes.load() << " changes)";
if (const uint64_t unchecked = stats.unchecked_slot_reads.load(); unchecked > 0)
std::cerr << ", " << unchecked << " slot reads without a notice time to check";
const uint64_t ready_samples = stats.capture_to_ready_samples.load();
if (ready_samples > 0)
{
Expand Down
108 changes: 38 additions & 70 deletions examples/camera_livekit_bridge/bridge/stream_worker.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
#include <gpu_ops/box_downscale.hpp>
#include <ipc_client/cuda_ipc_client.hpp>
#include <nvenc/nvenc_encoder.hpp>
#include <video_types/slot_age.hpp>
#include <video_types/video_types.hpp>

#include <algorithm>
Expand All @@ -15,7 +14,6 @@
#include <cuda_runtime_api.h>
#include <iostream>
#include <mutex>
#include <optional>
#include <stdexcept>
#include <utility>

Expand All @@ -27,27 +25,20 @@ namespace

constexpr auto kOverwriteReportInterval = std::chrono::seconds(5);

/// The loud error for a picture that may mix two frames. Rate limited per stream; the stats line keeps the totals.
void report_slot_overwrite_risk(const WorkerConfig& config,
const FrameNotice& notice,
std::size_t slot_count,
bool newer_notice,
bool late_copy,
int64_t age_ns,
int64_t deadline_ns,
uint64_t dropped_total)
/// The loud error for a picture whose slot the publisher rewrote after announcing it. Rate limited per stream; the
/// stats line keeps the totals.
void report_slot_overwrite(const WorkerConfig& config,
const FrameNotice& notice,
std::size_t slot_count,
bool during_copy,
uint64_t dropped_total)
{
std::cerr << "\n!!!!!!!!!! SLOT OVERWRITE RISK on stream '" << config.stream << "': picture DROPPED !!!!!!!!!!\n"
<< " The receiver's ring (" << slot_count << " slots) may have rewritten slot " << notice.slot
<< " (generation " << notice.slot_generation << ") while this bridge was still reading it, so the\n"
<< " pixels could be a mix of two frames under the first frame's ID and timestamp.\n";
if (newer_notice)
std::cerr << " Evidence: a newer notice for the same slot, or a full ring of newer notices, arrived.\n";
if (late_copy)
std::cerr << " Evidence: the copy finished " << static_cast<double>(age_ns) / 1e6
<< " ms after the notice; the ring wraps in " << static_cast<double>(deadline_ns) / 1e6 << " ms at "
<< config.fps << " fps.\n";
std::cerr
<< "\n!!!!!!!!!! SLOT OVERWRITE DETECTED on stream '" << config.stream << "': picture DROPPED !!!!!!!!!!\n"
<< " The receiver began writing slot " << notice.slot << " (generation " << notice.slot_generation
<< ") of its " << slot_count << "-slot ring "
<< (during_copy ? "while this bridge was copying it" : "before this bridge started copying it")
<< ",\n so the pixels could be a mix of two frames under the first frame's ID and timestamp.\n"
<< " Cause: this consumer is reading too slowly (starved GPU or CPU), or the receiver is delivering frames\n"
<< " faster than " << config.fps << " fps. Fix the stall; do not trust recordings of this period.\n"
<< " Pictures dropped for this reason so far: " << dropped_total << " (reported at most every "
Expand Down Expand Up @@ -166,22 +157,13 @@ struct StreamWorker::Impl

const IpcStream& stream = client.streams().front();
bool need_idr = true;
const int64_t deadline_ns = config.max_notice_age_ms != 0 ?
static_cast<int64_t>(config.max_notice_age_ms) * 1'000'000 :
slot_read_deadline_ns(config.fps, stream.slots.size());
auto last_overwrite_report = std::chrono::steady_clock::now() - kOverwriteReportInterval;
uint32_t own_frame_id = 0;
uint64_t last_session = 0;
FrameNotice notice;
std::optional<FrameNotice> pending; // a newer notice seen while checking the last slot; used next
while (running)
{
if (pending)
{
notice = std::move(*pending);
pending.reset();
}
else if (!client.next_frame(notice, std::chrono::milliseconds(200)))
if (!client.next_frame(notice, std::chrono::milliseconds(200)))
continue;
// Encode the newest picture: anything queued behind it is older, and a late frame is worse than a skipped
// one.
Expand All @@ -192,6 +174,24 @@ struct StreamWorker::Impl
++stats.skipped;
}

// A picture whose slot the publisher rewrote is dropped, counted and reported loudly (rate limited).
const auto drop_overwritten = [&](bool during_copy)
{
++stats.slot_overruns;
const auto now = std::chrono::steady_clock::now();
if (now - last_overwrite_report >= kOverwriteReportInterval)
{
last_overwrite_report = now;
report_slot_overwrite(config, notice, stream.slots.size(), during_copy, stats.slot_overruns.load());
}
};
// Check before spending a copy on a slot that is already being rewritten. This only saves work; the check
// after the copy is the one that decides.
if (!client.slot_unchanged(notice))
{
drop_overwritten(/*during_copy=*/false);
continue;
}
const IpcSlot& slot = client.slot(notice);
const auto* base = static_cast<const uint8_t*>(slot.device_ptr);
NvencEncoder::Surface surface;
Expand Down Expand Up @@ -224,51 +224,19 @@ struct StreamWorker::Impl
cudaGetErrorString(copy_status));
}

// The publisher overwrites a slot without notice once its ring wraps. The copy must have finished before
// that, and a newer notice for the same slot, or as many newer notices as the ring has other slots, means
// it may not have. Such a picture is dropped, never encoded.
// The copy must have finished before the marker is read: a write that began during it advanced the marker
// first, so an unchanged marker means no write overlapped the copy. A changed one means the picture may
// mix two frames; it is dropped, never encoded.
if (cudaStreamSynchronize(copy_stream) != cudaSuccess)
{
encoder.release(surface);
throw std::runtime_error("GPU copy from the IPC slot did not complete");
}
if (!client.slot_unchanged(notice))
{
bool overwritten = false;
std::size_t newer_count = 0;
FrameNotice newer;
while (client.next_frame(newer, std::chrono::milliseconds(0)))
{
++newer_count;
if (newer.slot == notice.slot && newer.slot_generation > notice.slot_generation)
overwritten = true;
pending = std::move(newer);
}
if (stream.slots.size() > 1 && newer_count + 1 >= stream.slots.size())
overwritten = true;
// The copy is done. If it finished later than the publisher needs to wrap its ring, the slot may have
// been overwritten during it, whether or not the replacement notice has arrived yet.
const int64_t copy_done_ns = monotonic_ns();
const SlotAge age = classify_slot_age(notice.timestamp_ns, copy_done_ns, deadline_ns);
if (age == SlotAge::Unknown && deadline_ns > 0)
++stats.unchecked_slot_reads;
const bool stale = age == SlotAge::Stale;
if (overwritten || stale)
{
encoder.release(surface);
if (overwritten)
++stats.slot_overruns;
else
++stats.stale_slot_reads;
const auto now = std::chrono::steady_clock::now();
if (now - last_overwrite_report >= kOverwriteReportInterval)
{
last_overwrite_report = now;
report_slot_overwrite_risk(config, notice, stream.slots.size(), overwritten, stale,
copy_done_ns - static_cast<int64_t>(notice.timestamp_ns),
deadline_ns, stats.slot_overruns + stats.stale_slot_reads);
}
continue;
}
encoder.release(surface);
drop_overwritten(/*during_copy=*/true);
continue;
}

const uint64_t session_id = unsigned_field(notice.metadata, "session_id");
Expand Down
10 changes: 2 additions & 8 deletions examples/camera_livekit_bridge/bridge/stream_worker.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,6 @@ struct WorkerConfig
uint32_t fps = 30;
uint32_t bitrate_bps = 8'000'000;
uint32_t idr_period = 30;
/// A read that finishes later than this after its slot's notice is treated as possibly overwritten and dropped.
/// Zero derives it from the ring: the time the publisher needs to wrap it at `fps`.
uint32_t max_notice_age_ms = 0;
/// Integer factor the picture is shrunk by on the GPU before encoding (1 keeps the source size).
uint32_t scale_div = 1;
/// The IPC pictures are full-range (0-255) rather than limited-range (16-235); the encoded stream signals it. The
Expand All @@ -44,12 +41,9 @@ struct WorkerStats
std::atomic<uint64_t> skipped{ 0 };
std::atomic<uint64_t> keyframes{ 0 };
std::atomic<uint64_t> rejected{ 0 };
/// Pictures dropped because the publisher may have overwritten the IPC slot during the copy.
/// Pictures dropped because the publisher began writing the IPC slot after announcing the picture, either before
/// the copy or while it ran.
std::atomic<uint64_t> slot_overruns{ 0 };
/// Pictures dropped because the copy finished too long after the slot's notice for the pixels to be trusted, and
/// pictures whose notice carried no time to check (those are kept).
std::atomic<uint64_t> stale_slot_reads{ 0 };
std::atomic<uint64_t> unchecked_slot_reads{ 0 };
std::atomic<uint64_t> encode_us_total{ 0 };
/// The bitrate the encoder is configured for now, and how many times congestion feedback changed it.
std::atomic<uint32_t> bitrate_bps{ 0 };
Expand Down
14 changes: 14 additions & 0 deletions examples/camera_livekit_bridge/ipc_client/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,16 +1,30 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# The slot write-generation markers are shared with camera_receiver and its other readers; the header lives with them.
set(CAMERA_IPC_PROTOCOL_DIR "${CMAKE_CURRENT_SOURCE_DIR}/../../camera_streamer/ipc_protocol"
CACHE PATH "camera_streamer's ipc_protocol directory")
if(NOT EXISTS "${CAMERA_IPC_PROTOCOL_DIR}/inc/ipc_protocol/slot_markers.hpp")
message(FATAL_ERROR "slot_markers.hpp not found under CAMERA_IPC_PROTOCOL_DIR=${CAMERA_IPC_PROTOCOL_DIR}")
endif()

add_library(ipc_client STATIC cuda_ipc_client.cpp)

target_include_directories(ipc_client
INTERFACE
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/inc>
)

# The client's own sources include the markers header, and so do its consumers through cuda_ipc_client.hpp.
target_include_directories(ipc_client
PUBLIC
$<BUILD_INTERFACE:${CAMERA_IPC_PROTOCOL_DIR}/inc>
)

target_link_libraries(ipc_client
PUBLIC
nlohmann_json::nlohmann_json
rt
PRIVATE
CUDA::cudart
)
Expand Down
Loading
Loading