refactor: prototype on-GPU hook path with dual ring buffers #29

Activity

1. Samfisheryu

   assigned

   Samfisheryu

   and

   XbzOnGit

   on Feb 28, 2026
2. Samfisheryu commented on Mar 2, 2026

   

   Samfisheryu

   on Mar 2, 2026

   Last edited by Samfisheryu

   CollaboratorAuthor

   More actions

   Proposal 1: Dual-Ring Design with One Large Payload Ring Buffer

   This is our current Solution 1 for issue #29.

   Goal

   • Keep CUDA Graph replay fast and stable (production-oriented path).
   • Continuously emit hook data during replay.
   • Preserve correct hook identity and ordering.
   • Support dynamic hook-copy volume per step (without re-capturing graph each step).

   Core Architecture

   • Two rings
     • Ring Table: control plane (identity/order/state)
     • Payload Ring Buffer: data plane (raw bytes)
   • One compute graph (not two)
   • Three streams
     • S_compute: in-graph model path
     • S_d2d: out-of-graph persistent copy worker (DtoD consumer)
     • S_d2h: out-of-graph host drain stream (DtoH to CPU)
   • Two custom op/kernel roles
     • Pre-Barrier Op (in graph): stream-ordered wait gate before compute
     • Post-Trigger Kernel (in graph): publish descriptors after compute
   • One persistent kernel
     • Persistent Copy Kernel (out of graph, launched once): consume descriptors and run DtoD copies

   Pre-Barrier

   Pre-Barrier Op is implemented as stream-ordered waits on S_compute (prefer cuStreamWaitValue64).

   It enforces two conditions before each hook-site compute runs:

   • Barrier A (source/computation-buffer safety): wait for copied_seq boundary (DtoD completion)
   • Barrier B (payload-ring space safety): wait for byte_free / drained_seq boundary (DtoH completion)

   So this is not a Python-side hook barrier and not a host sync point; it is a stream-level gate in the graph path.

   Identity and Mapping

   • Identity is not address-based.
   • Stable identity key is global_seq_no + hook_id.
   • Slot reuse safety uses full sequence validation (seq, not only seq % N).

   Shared Counters (64-bit atomics)

   • seq_next: next sequence to allocate
   • produced_seq: highest sequence published by Post-Trigger
   • copied_seq: highest sequence DtoD-completed by persistent worker
   • drained_seq: highest sequence DtoH-completed by host drain
   • byte_resv: payload bytes reserved by producer
   • byte_free: payload bytes released after DtoH completion

   Synchronization Contract

   • In-graph producer path (S_compute):
     • Pre-Barrier Op waits (A/B) -> compute kernel -> Post-Trigger writes descriptor -> release fence -> update produced_seq
   • Out-of-graph DtoD path (S_d2d):
     • persistent worker waits for produced_seq -> reads descriptor -> DtoD -> release fence -> updates copied_seq
   • Out-of-graph DtoH path (S_d2h + host):
     • consume ready sequences -> DtoH -> on completion update byte_free and drained_seq
   • Memory ordering:
     • publish with release, consume with acquire

   In-Graph vs Out-of-Graph

   • In graph
     • Pre-Barrier Op (stream wait)
     • model compute kernels
     • Post-Trigger descriptor publish
   • Out of graph
     • persistent DtoD copy kernel on S_d2d
     • DtoH drain + host consumer on S_d2h
3. Samfisheryu commented on Mar 2, 2026

   

   Samfisheryu

   on Mar 2, 2026

   CollaboratorAuthor

   More actions

   Proposal v2 (per-hook ring, no global mapping table)

   We propose switching to a per-hook fixed memory region with a local ring (head/tail/commit) instead of a global ring + mapping
   table.

   • Each hook owns a fixed GPU address range, so hook_id is resolved by region directly.
   • We keep a single HookCopyOp after each hook point (no pre/post split).
   • HookCopyOp does chunked SM DtoD into that hook’s local ring.
   • Backpressure rule: block only when current write needs need_bytes but free_bytes < need_bytes.
   • S_d2h drains asynchronously per hook from [head, commit) and advances head.
   • This allows overlap across steps when ring space is available, instead of forcing “wait DtoH every time”.

   Initial sizing plan:

   • Use offline profiling and set attention-hook ring capacity by P80/P90 payload size.
   • Long-tail requests use chunked writes and may block on space reclaim.

   @zhaoyboo will first merges our newest codebase into his branch. And implement this method by reusing some of his codebase
4. Samfisheryu

   assigned

   zhaoyboo

   on Mar 2, 2026
5. Samfisheryu

   closed this as a duplicateof DMI: Public Release Roadmap #32 on Apr 6, 2026