Skip to content

openipc_frame_ts: expose encoder-done / ISP frame-end event for per-segment latency decomposition #176

Description

@widgetii

Follow-up to #144 / #155 (now merged).

Motivation

#155 ships per-frame capture timestamps via `/dev/openipc-frame-ts` — receivers can compute end-to-end latency as `recv_wall - reconstructed_capture_wall` once anchored in the RTCP SR (widgetii/majestic#83).

The sibling OpenIPC/PixelPilot_rk#127 (drone-side waybeam_venc + GS-side latency probe) demonstrates the next-step value: per-segment decomposition — `capture_to_encode`, `encode_to_send`, `wire`, `gs_pipeline` — instead of a single total. Knowing where the latency lives is much more useful for diagnosing pipeline regressions than knowing the sum.

Their approach uses a custom UDP sidecar protocol with per-stage hooks on the drone. Ours can deliver the same decomposition through the existing RTCP SR plumbing if the chrdev emits one additional event per frame: frame-end (ISP FEND or equivalent — the moment the back-end pipeline has the full frame ready to hand off to the encoder).

With both `FS_wall_ns` and `FEND_wall_ns` for the same frame at the receiver, the receiver can:

  • compute `FEND - FS` = ISP / capture-to-back-end-ready time
  • compare against `encoder_pts_at_VENC_GetStream` ⇒ `encode_done - FEND` = encoder buffer
  • compare against `recv_wall` ⇒ `recv - encode_done` = wire + RTP path

That's the same segmentation PixelPilot exposes, but available to any RTCP-aware receiver with zero bespoke protocol.

Design

There's already a stacked branch by @widgetii with the ABI + plumbing implemented:
`widgetii:feat/openipc-frame-ts-fend` — 8 files, +185/-60.

Shape:

  • `struct openipc_frame_ts_event` grows two fields: `__u32 event_type` (0=FS, 1=FEND), `__u32 reserved`. Total 32 bytes (was 24).
  • New `OPENIPC_FT_IOC_SET_EVENT_MASK` so consumers can subscribe to a subset.
  • `openipc_frame_ts_push(vi_chn, event_type)` — type param added.
  • ISP FEND fires from inside the existing ISP IRQ handler when `(isp_int_status & ISP_INT_FE_FEND)` is set — zero overhead when the bit isn't asserted.
  • Forward-compatible: future event types append without changing struct size.

Definition of done

  1. The FEND branch is rebased onto current main (Add openipc_frame_ts: sensor frame-start (PTS, CLOCK_REALTIME) chrdev #155 is now merged).
  2. ABI struct change documented in `include/openipc_frame_ts.h` and `kernel/openipc_frame_ts/README.md` as ABI v2 — explicit note that `sizeof(struct openipc_frame_ts_event) == 32` is frozen at this version, future event types append within that 32 bytes (use `reserved` first).
  3. Per-event-type rate validated on each supported SoC: at sensor rate (1:1 with FS) within a few percent.
  4. End-to-end hardware verified on at least hi3516ev300 — observe per-frame pairing between FS and FEND with reasonable inter-event delta (the "capture_to_encode" segment) and stable across a 60s capture.
  5. cv500 family validated separately (see cv500 ISP IRQ routing: 'ISP' platform IRQ resolves to VI_PROC0 (~1/8 fps), not VI_CAP0 (per-frame) #177).
  6. widgetii/majestic#83 updated to use the new ABI optionally — emit RTCP SR using `(FEND_wall, encoder_pts)` if FEND is available and per-frame, fall back to FS-only when not.

Out of scope here

  • Encoder-side event (`VENC_GetStream` userspace hook) — orthogonal, can be tracked separately if anyone needs it. The above three events (FS, FEND, recv) already give the most diagnostically valuable decomposition.
  • Clock-offset RTT estimator on the consumer side — see widgetii/majestic#84 / openipc-camera tooling.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions