[tzulingk]

@coderabbitai summary

Description

KVCacheManagerV2 has two generic host<->GPU copy helpers, copyHostToDevice and copyDeviceToHost, that move a batch of pages with a batched copy kernel: one 128-thread CTA per page with 8 KiB in flight, which caps every page at a few GB/s whatever the link. That is fine for the many small attention pages of a batch, but a 217 MB recurrent-state page of a hybrid model such as Kimi K3 then takes tens of milliseconds, and the execution stream waits for it. This PR sends calls with large pages to the GPU copy engines instead (one cuMemcpyBatchAsync per call on the caller's stream) and leaves everything else on the kernel.

Where this applies on main. Since #17512 and #18541, hot<->host migrations (eviction, onboard, prefetch, copySlotData) go through IKvCacheColdPageCodec and BatchedPageCopier, not through these two helpers. On main they serve CopyEngine::transfer, i.e. migrations between two cold levels when one of them is a GPU memory level, and the copy_host_to_device / copy_device_to_host bindings. The end-to-end gain below (+6.0 % output tokens/s per GPU, 0.8 s less TTFT at the median) was measured on a deployment built from a branch that predates the cold-page codec, where these helpers were the only GPU<->host copy path. The change is posted as the measured, self-contained version of the idea; the section "Measurements on main's path" at the end shows what the same comparison looks like against BatchedPageCopier and what a follow-up for main would be. If the maintainers consider the helper-level branch redundant on main, closing this PR in favour of that follow-up is a fine outcome.

The problem

• The host copy kernel moves one page with one thread block. launchBatchedCopy(lowBandwidth=true, ...) sets nbSplits = 1, so each (page, pool) task gets a single 128-thread CTA. The block copies 16 B per thread per step through a ring of four shared-memory buffers, so it never has more than 4 x 128 x 16 B = 8 KiB of loads outstanding. Bandwidth is bytes in flight divided by the latency of those loads, so one task is capped at a few to low tens of GB/s however fast the link is:

  bandwidth = bytes in flight / load latency
            = 8,192 B / (0.5 to 4 us)  =  16 to 2 GB/s
  

  The measured 1.8 to 17.5 GB/s per CTA fit that range. The stores are posted, so device->host is bounded by device-memory load latency and only host->device by the link round trip, which is why the two directions differ. Keeping a 400 GB/s link busy would need about 400 GB/s x 2 us = 800 KiB in flight; one task supplies 8 KiB, about 1 % of that. A batch of many small pages is still fast because every page gets its own block; a single large page is not.

• Hybrid models have large pages. Kimi K3 (MLA attention plus KDA linear-attention layers, NVFP4) keeps two recurrent-state pages per rank, 217,055,232 B (KDA SSM state) and 15,261,696 B (conv state), next to its 884,736 B MLA pages. Every request restores one state snapshot from the host tier and later evicts one to it, so the large pages cross the link twice per request.

• What a profile of one prefill rank shows (44.9 s window at steady load, on the branch described above):

  • 1,107 single-block host copies adding up to 31.5 s of kernel time in the 44.9 s window. Half of them finished in under 1.9 ms (small pages); the slowest 5 % took longer than 136 ms (state pages), the longest 237.7 ms.
  • A 434 MB fp32 state page took 24.8 to 237.7 ms, which is 1.8 to 17.5 GB/s for its one block.
  • At those rates the 217 MB bf16 state page takes 12 to 120 ms. The copy engines of the same NVLink-C2C system move it in about a millisecond.

• The time is on the critical path. The storage manager records an event after each copy and the consumers wait on it in-stream, so a compute-bound prefill stalls until the page has landed. Per request the traffic is modest, about 458 MB out and 442 MB in; the problem is how slowly each large page moves, not how much data moves.

• Adding blocks is the wrong fix here. The prefill already uses every SM, while the copy engines sit idle. [None][perf] Speed up burst KVCM2 resize for very long sequences #18541 reached the same conclusion for PCIe parts and uses cuMemcpyBatchAsync there, "which also costs zero SMs".

The change

copyHostToDevice / copyDeviceToHost in cpp/tensorrt_llm/batch_manager/kvCacheManagerV2Utils.cu share one dispatch, copyAcrossHostLink<HostToDevice>: the copy-engine branch when the call is non-empty, numBytes > 0 and useCopyEngineForHostCopy(numBytes) is true, launchBatchedCopy(true, ...) unchanged otherwise. copyDeviceToDevice is untouched. The predicate is

!HostMem::shouldUseChunkedRegistration() && numBytes >= MIN_BYTES

knob	default	meaning
TLLM_KV_CACHE_MANAGER_V2_HOST_COPY_DMA_MIN_BYTES	4 MiB	smallest per-task size that uses the copy engines; 0 = any size, negative = never (today's behaviour)

With the default an 884,736 B attention page stays on the kernel and the two state pages go to the copy engines. The knob is read once per process; a value that is not an integer, or is out of range, is reported at WARNING and ignored. The copy-engine branch submits all tasks of the call with one cuMemcpyBatchAsync on the caller's stream and returns the CUresult; on the legacy NULL stream (0 or CU_STREAM_LEGACY), which the batched call does not accept, it issues one cuMemcpy*Async per task. The dispatch also rejects a negative byte count, and a byte count the kernel launcher's uint32_t cannot carry, with TLLM_CHECK_WITH_INFO instead of letting them wrap.

Other files:

• kvCacheManagerV2Utils.h: declarations of hostCopyDmaMinBytes(), hostCopyDmaMinBytesFor(rawValue) (the knob's semantics on a raw value, so tests can drive them without resetting the process-wide cache), kHostCopyDmaMinBytesEnv and useCopyEngineForHostCopy(), plus the contract of the two entry points: the kernel path requires a multiple of 16 and 16 B-aligned task addresses, the copy-engine path accepts any size and alignment, and the reach stated above.
• nanobind/batch_manager/kvCacheManagerV2Utils.cpp: use_copy_engine_for_host_copy(num_bytes) next to the copy helpers, so tests can ask which path a task size takes. The copy_* signatures are unchanged.
• cpp/tests/unit_tests/batch_manager/kvCacheManagerV2TestUtils.h: the CudaAllocation / allocateCuda and ScopedEnv helpers that the sibling KVCM2 tests each carried a copy of, as shared inline definitions (the siblings keep their copies for now).

Design notes

• Why one driver call per call. A call is one copyHostToDevice / copyDeviceToHost invocation; CopyEngine::transfer issues one per pool, so all tasks (pages) of a call have the same size but sit at scattered addresses. cuMemcpyBatchAsync takes the destination, source and size of every task and submits them in one driver call, so the CPU cost does not grow with the page count and no task limit is needed. It is the same call batchedPageCopy.cu uses on PCIe parts, with the same attributes, and TensorRT-LLM builds with CUDA 13, where it is always available.
• Why the same stream. The copies are enqueued on the stream the caller passes, after its prior-event waits and before the finish event it records, so the ready event consumers wait on still covers them. The batch is stream-ordered as a whole; copies inside a batch have no mutual order, which is fine because the tasks of one call read and write disjoint slots, as the kernel launch they replace already requires.
• Why chunked registration disables it. One memcpy may not span two cuMemHostRegister ranges, and HostMem registers the host tier in 2 GiB chunks on Linux 6.11-6.13. Those hosts keep the kernel, which reaches pinned memory through ordinary loads and stores. Same rule as batchedPageCopy.cu.
• Why not BatchedPageCopier. It is driven by PoolCopyArgs (base, stride, index pairs) for bulk uniform-page copies and sizes a CTA grid per platform. This path receives the pages of one migration batch as a plain task vector (one or two large state pages per request, a whole batch on eviction or prefetch), where a copy-engine enqueue needs no tuning and no SMs.
• Exceptions. Since [None][fix] KVCM2: fix remaining Python/C++ parity issues and replace abort with a poison latch #19103 the two entry points may throw (the kernel path rejects unaligned sizes with TLLM_CHECK); they keep that signature and now also throw on a negative byte count. The policy helpers (hostCopyDmaMinBytes, hostCopyDmaMinBytesFor, useCopyEngineForHostCopy) are noexcept; the warning for a bad knob value is guarded so it cannot turn into std::terminate. copyWithCopyEngine is not noexcept, because its descriptor vectors can throw std::bad_alloc, which the entry points propagate.

Measured effect

A/B on a Kimi K3 disaggregated deployment (16 single-GPU prefill workers plus a 16-GPU decode worker, 32 GPUs): fixed concurrency, 1 h windows, identical configuration and observability.

metric	before	after
output throughput per GPU	744.2 tok/s	789.2 tok/s (+6.0 %)
TTFT p50	5.98 s	5.19 s
prefill iteration p50	2.59 s	2.39 s
per-request prefill time p50	2.38 s	2.03 s
per-user decode speed p50	65.4 tok/s	64.0 tok/s (-2.1 %)
host bytes per request (offload / onboard)	458 / 442 MB	443 / 440 MB

The copies get faster, not fewer. The attention-page batches (140 x 884,736 B per request) keep the kernel and are unaffected.

Which build was measured. The A/B ran the first version of this change: one cuMemcpyHtoDAsync / cuMemcpyDtoHAsync per task on the caller's stream, with calls of more than 8 tasks kept on the kernel (a task-limit knob, since removed), compiled into a production image whose KV cache manager predates the cold-page codec (#17512), so that copyHostToDevice / copyDeviceToHost were its only GPU<->host copy path. The code in this PR submits each call as one cuMemcpyBatchAsync with CU_MEMCPY_FLAG_PREFER_OVERLAP_WITH_COMPUTE and no task limit. Per-request state migrations carry one task per pool, so the shapes behind the numbers above are single-task calls on both variants; the submitted path has been timed per page (tables below) but not A/B-tested end to end.

Measurements on main's path

To see whether the idea still has value on main, where hot<->host migrations run through BatchedPageCopier, the three paths were timed on the same shapes on one GB300 node (NVLink-C2C, 152 SMs, CUDA 13.4): this PR's helpers at the default threshold (copy engines), the same helpers with MIN_BYTES=-1 (the pre-PR single-block kernel) and BatchedPageCopier::launch, which selected its multi-CTA kernel on that coherent link (4 blocks for offload, 6 for onboard, sized for the 224 GB/s fallback link figure because NVML reported no C2C bandwidth on that node). Event-to-event, 10 iterations after warm-up, pinned HostMem buffers.

Idle GPU:

shape	copy engines (this PR)	BatchedPageCopier kernel (main)	single-block kernel (before)
1 x 217,055,232 B, host->device / device->host	211 / 193 GB/s	201 / 185 GB/s	9.6 / 15.9 GB/s
12 x 15,261,696 B	211 / 193 GB/s	202 / 185 GB/s	103 / 189 GB/s
140 x 884,736 B	209 / 192 GB/s (kernel path at the default)	203 / 186 GB/s	209 / 192 GB/s
64 x 4,194,304 B	211 / 193 GB/s	202 / 185 GB/s	211 / 193 GB/s

Both the copy engines and main's kernel sit at the link ceiling of that node (about 210 GB/s per direction per GPU); the copy engines are 4 to 5 % ahead. The single-block kernel is 13 to 22x slower on a single large page, which is the gap the A/B above closed.

Same node, every SM thread slot occupied by a spin kernel on another stream (blocks of 0.1 ms or 5 ms, launched back to back), host->device, milliseconds per copy:

shape	load	copy engines (this PR)	BatchedPageCopier kernel (main)
1 x 217 MB	none	1.03 ms	1.08 ms
1 x 217 MB	0.1 ms blocks	1.03 ms	1.2 to 2.0 ms
1 x 217 MB	5 ms blocks	1.03 ms	3.8 to 4.5 ms
12 x 15 MB	5 ms blocks	0.87 ms	3.9 to 4.5 ms

The copy engines use no SM, so their time does not change under load; the kernel's blocks wait for a free slot, and giving the copy stream the highest priority did not help, because resident blocks are not preempted. Persistent-scheduled GEMM and attention kernels in a long prefill behave like the 5 ms case. So on coherent links main's production path ties the copy engines when the GPU is idle and loses 1.2 to 4x in copy latency when it is saturated. The follow-up for main is #20426, an option that lets BatchedPageCopier prefer its existing copy-engine path on coherent links (keeping the kernel where pinned memory is registered in chunks); this PR does not touch BatchedPageCopier.

Test Coverage

New tests

cpp/tests/unit_tests/batch_manager/kvCacheManagerV2HostCopyTest.cu (gtest target kvCacheManagerV2HostCopyTest, one GPU, linked with kvCacheManagerV2PoisonCheck.cpp like its neighbours). The knob is cached per process, so the tests pick shapes on both sides of the default instead of changing the environment, and every path-dependent expectation asks the predicate first, which lets the same binary run under any knob value:

• ThresholdFromRawValue: hostCopyDmaMinBytesFor() on raw values: unset, empty, 4abc and a 20-digit value give the default, 0 gives 0, -1 gives never, 16777216 gives 16 MiB.
• ThresholdFollowsEnvironment: the cached accessor equals the parser applied to the process environment, and the predicate over {16 B, 884,736 B, 4 MiB - 1, 4 MiB, 15,261,696 B, 217,055,232 B} follows it.
• HostToDeviceToHost (parametrised): byte-exact HostMem -> cudaMalloc -> HostMem round trips in the slot layout of a host pool for (884,736 B x 1 / 140 / 300), (16 B x 300), (15,261,696 B x 1 / 12), (217,055,232 B x 1) and (4 MiB x 8 / 64). 300 > 256 exercises the kernel's multi-launch remainder; the x 12 and x 64 shapes send many large pages through one batched call. The fill is pseudo-random per 64-bit word, so no two slots carry the same bytes and a permuted or slot-0 source read is caught.
• NonMultipleOf16: 3 x 24 B and 3 x (4 MiB + 8 B). A size the predicate sends to the copy engines must round-trip (they accept any size); a size it keeps on the kernel must throw TllmException from both entry points before any CUDA work. At the default the first throws and the second round-trips.
• NegativeByteCountIsRejected, EmptyTaskListIsANoOp, NullStreamAccepted (the per-task fallback on 0 and on CU_STREAM_LEGACY with 1 and 3 large pages, and the kernel) and StreamOrderingThroughEvents (producer stream -> recorded event -> consumer stream sees the bytes; a host-function gate holds the producer until everything is enqueued, so the check is deterministic rather than won by timing).
• StateSlotBandwidthInfo: records the GB/s of one 217 MB page per direction and the path that moved it as test properties. Not asserted.

tests/unittest/kv_cache_manager_v2_tests/test_copy_engine_host_copy.py (one GPU; the directory is already in the l0_a10, l0_h100 and l0_b200 lists): round trips through the bindings for (884,736 B x 140 / 300), (16 B x 300), (4 MiB x 44), (15,261,696 B x 1), (217,055,232 B x 1) and 3 x (4 MiB + 8 B), with torch pinned and device buffers on a non-default stream; the predicate against an expectation computed from the environment and the kernel version, not from the predicate itself; a 24 B task, which the kernel rejects and the copy engines accept; the legacy NULL stream; and a consumer stream observing a copy through a recorded event. test_copy_helper_exceptions.py skips its two host-helper cases when the knob sends a 1 B task to the copy engines, so the directory runs under MIN_BYTES=0 as well.

Existing coverage

test_copy_helper_exceptions.py is the only existing test that calls copy_host_to_device / copy_device_to_host; it never performs a copy (1 B, unaligned, fake addresses) and, with the default threshold, is rejected on the kernel branch before any CUDA work. The host-tier cases of test_kv_cache_manager_v2.py do not reach the two entry points: StorageManager::submitMigrationBatch routes GPU<->host migrations through IKvCacheColdPageCodec::encode / decode. The two new tests are therefore the only coverage of the copy-engine branch and of useCopyEngineForHostCopy.

Validation

check	result
clang-format, cmake-format, codespell, ruff, ruff-format on the changed files	clean
first version of the engine change (per-task cuMemcpy*Async, 8-task limit) compiled and run in production on an earlier base; the image's unit-test lanes show no new failure against the stock image (kvv2_core 123/123)	done
end-to-end A/B of that build (not of the cuMemcpyBatchAsync submission in this PR)	table above
gtest kvCacheManagerV2HostCopyTest at the PR head (a141d550d9) on a GB300 node (aarch64, CUDA 13.4, Linux 6.17, so the copy-engine path is active; nvcr.io/nvidia/tensorrt-llm/devel:1.4.0rc0, build_wheel.py --cuda_architectures 103-real --fast_build --install), in four environments: default, MIN_BYTES=-1 (kernel only), MIN_BYTES=0 (copy engines for every size) and MIN_BYTES=abc (warning, default kept)	17 passed in each; the 4abc and 20-digit values are reported as "not an integer" and "out of range for a 64-bit integer"; neighbouring kvCacheManagerV2HostMemTest 9 passed
pytest test_copy_engine_host_copy.py and test_copy_helper_exceptions.py through the nanobind module on the same node, same four environments	17 passed (default, MIN_BYTES=-1, MIN_BYTES=abc); 15 passed, 2 skipped (MIN_BYTES=0: the existing unaligned-count test skips its two host-helper cases, as designed)

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

🤖 Generated with Claude Code

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/TensorRT-LLM/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 6ce4b6cb-5418-46d0-8f19-3a57b8246e87

📥 Commits

Reviewing files that changed from the base of the PR and between 9fd8917 and a141d55.

📒 Files selected for processing: 8

• cpp/tensorrt_llm/batch_manager/kvCacheManagerV2Utils.cu
• cpp/tensorrt_llm/batch_manager/kvCacheManagerV2Utils.h
• cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManagerV2Utils.cpp
• cpp/tests/unit_tests/batch_manager/CMakeLists.txt
• cpp/tests/unit_tests/batch_manager/kvCacheManagerV2HostCopyTest.cu
• cpp/tests/unit_tests/batch_manager/kvCacheManagerV2TestUtils.h
• tests/unittest/kv_cache_manager_v2_tests/test_copy_engine_host_copy.py
• tests/unittest/kv_cache_manager_v2_tests/test_copy_helper_exceptions.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

Host-to-device and device-to-host KV cache copies now choose between CUDA copy engines and the existing batched kernel using a configurable, process-cached threshold. The change adds a policy-query binding and C++ and Python tests for selection, data integrity, and stream ordering.

Changes

KV cache host-copy dispatch

Layer / File(s)	Summary
Policy-based host-copy dispatch cpp/tensorrt_llm/batch_manager/kvCacheManagerV2Utils.cu, cpp/tensorrt_llm/batch_manager/kvCacheManagerV2Utils.h, cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManagerV2Utils.cpp	The helpers parse and cache a configurable DMA threshold. Eligible non-empty copies use copy engines; other calls use the batched kernel. The header documents selection rules and constraints, and the Python binding exposes the policy query.
CUDA dispatch and transfer tests cpp/tests/unit_tests/batch_manager/kvCacheManagerV2HostCopyTest.cu, cpp/tests/unit_tests/batch_manager/kvCacheManagerV2TestUtils.h, cpp/tests/unit_tests/batch_manager/CMakeLists.txt	The CUDA tests cover threshold parsing, path selection, input handling, round-trip data, stream ordering, and informational bandwidth. Test utilities add CUDA allocation and environment-variable helpers.
Python host-copy integration tests tests/unittest/kv_cache_manager_v2_tests/test_copy_engine_host_copy.py, tests/unittest/kv_cache_manager_v2_tests/test_copy_helper_exceptions.py	The tests check path selection, round-trip data, NULL-stream copies, and event ordering. The exception test skips host-copy alignment assertions when the copy-engine path is selected.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant HostCopyDispatch
  participant HostCopyPolicy
  participant CopyEngineOrKernel
  Caller->>HostCopyDispatch: submit host/device copy tasks
  HostCopyDispatch->>HostCopyPolicy: evaluate size and registration policy
  HostCopyPolicy-->>HostCopyDispatch: return transfer-path decision
  HostCopyDispatch->>CopyEngineOrKernel: submit copy-engine or batched-kernel transfers

Loading

Suggested reviewers: juney-nvidia

Merge Risk: ⚪ Minimal · up to a141d

No actionable issue remains identified for this change; it is ready for normal merge checks.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 52 functions across 7 files. (1 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Title check		The title clearly and concisely identifies the main change: using copy engines for large host-to-GPU and GPU-to-host page copies in KVCacheManagerV2. It follows the repository’s ticket/type format.
Description check		The description is complete and directly related to the change. It explains the problem, implementation, design decisions, measured results, test coverage, validation, and checklist status. The requir…

Full details: Docstring Coverage

Explanation

Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 52 functions across 7 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[lucifer1004]

Thanks for spelling out the BatchedPageCopier follow-up; I opened it as #20364 (issue #20363). One data point that bears on this PR's default threshold:

On an SM107 system with NVLink-C2C (Kimi K3, 8 GPUs, CUDA graphs) we see that SM copy kernels touching registered host memory leave the whole process slower afterwards: every kernel boundary in later CUDA graph replays gets ~0.2 us longer, about +1.2 ms (~7%) per decode step, and it never recovers (details and nsys findings in #20363). It is triggered by small pages too, not only by the large recurrent-state pages:

• With TLLM_KV_CACHE_MANAGER_V2_BACKEND=python, i.e. through copyHostToDevice / copyDeviceToHost, offloading only MLA KV blocks (attention pages well below 4 MiB; the state pool never filled in that run) produced the same +1.2 ms step as soon as KV blocks started to be offloaded.
• Replacing the two helpers' kernel with per-task cudaMemcpyAsync for every size removed it, with byte-identical outputs. The cost was ~0.19 s (~9%) more TTFT when reloading ~6K KV blocks (~13 GB per rank) from the host tier.

So with the default TLLM_KV_CACHE_MANAGER_V2_HOST_COPY_DMA_MIN_BYTES of 4 MiB, the attention pages would still go through the kernel and keep this slowdown on coherent links. Would you consider defaulting to the copy engines for every size on coherent links (or making the threshold link-dependent), in line with #20364?

[lowsfer]

I think this is mostly for a deprecated code path. With recent code page codec change, the H2D and H2D of pages mostly go through the default codec, which has both kernel and DMA path, and the kernel path is also well optimized for perf/resource (not absolute perf because the migration is meant to run concurrently with inference, and I wanted it to use minimum amount of resource and achieve >80% peak throughput).

Did you see perf problems for page H2D/D2H with the default codec path?

BTW, python backend has been removed, so TLLM_KV_CACHE_MANAGER_V2_BACKEND=python has no impact.

[mikeiovine]

Stamp on behalf of runtime devs, delegating proper review to @NVIDIA/trt-llm-kv-cache-manager-devs

[nvpohanh]

[by Claude Code] @yizhang-nv Could you review this PR? Thanks!

[nvpohanh]

[by Claude Code] @yizhang-nv Could you review this PR? Thanks!