[SimengLiu-nv]

Dev Engineer Review

This forward-port groups changes across attention kernels, model loading, distributed execution, disaggregation, performance tooling, telemetry, and documentation. It adds APIs and code paths, changes defaults, and updates CI and QA lists.

Verify head-size-104 FMHA generation and dispatch across supported architectures. Check that NCCL-window suppression is restored after autotuning, shared cubin pooling handles concurrent access, and NIXL descriptor checks preserve transfer ownership. Also verify Qwen3.5 INT4 packing, LoRA identity rules, Mistral image processing, and managed-memory fallback behavior.

No current review findings or severity counts were supplied. The author reports that tests/unittest/scripts/test_perf_submit.py passed 127 tests before rebase. The supplied comments report failed pipelines #65836 and #66443, then successful pipelines #67168 and #67715; #67715 used automatic test retries. The PR_Github run for commit 7351540 was aborted. GPU tests were not run locally. Overall completion status is not established.

QA Engineer Review

The changes add and update tests for FMHA head size 104, Pixtral and sparse attention, MLA page tables, checkpoint mapping, distributed workspace caching, cubin pooling, NIXL transfers, LoRA identity, and EPLB mounts. They also update CI and QA lists, move a Qwen3 test from the core QA list to the multinode QA list, and report removal of a waiver.

The supplied test-list changes place the head-104 C++ test in test-db/l0_b200.yml, test-db/l0_gb202.yml, and test-db/l0_rtx_pro_6000.yml. They place the Pixtral test in the GB202 and RTX Pro 6000 lists, and the cubin-pool test in test-db/l0_cpu.yml. The Qwen3 test appears in qa/llm_function_multinode.txt and test-db/l0_gb300_multi_nodes_node2_gpu8.yml. Shell inspection found that the checked-out waives.txt still contains the Qwen3 test waiver, despite the supplied change summary reporting removal. The checkout therefore does not confirm that waiver change. The supplied changes do not establish list coverage for every other changed test.

Coverage verdict: needs follow-up. Confirm test-list coverage for tests without identified entries, and confirm the reported waiver removal in the intended PR revision. GPU tests were not run locally.

Per-File QA Perspective

Source and documentation files

• cpp/kernels/fmha_v2/setup.py — Verify head-size-104 mask-kernel generation across SM90, SM100, and SM120, including dtype, warp-specialization, and tiling conditions.
• cpp/tensorrt_llm/common/ncclUtils.cpp — Verify suppressed allocations return an invalid buffer before communicator and size validation.
• cpp/tensorrt_llm/common/ncclUtils.h — Verify the setter declaration matches the implementation and thread-local contract.
• cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/transferAgent.cpp — Verify descriptor splitting uses recorded VMM regions and behaves correctly when no map exists.
• cpp/tensorrt_llm/kernels/fmhaDispatcher.cpp — Verify head-size-104 dispatch uses FMHA v2 only for the specified SM100-family types and masks.
• cpp/tensorrt_llm/runtime/moeLoadBalancer/hostAccessibleDeviceAllocator.cpp — Verify managed-memory support, allocation advice, and fallback behavior across GDRCopy and direct-host-access configurations.
• cpp/tensorrt_llm/runtime/moeLoadBalancer/hostAccessibleDeviceAllocator.h — Verify the new access-mode state and documented preference order match allocator behavior.
• cpp/tensorrt_llm/thop/allreduceOp.cpp — Verify the new operator returns the prior suppression state and behaves as intended without multi-device support.
• tensorrt_llm/_torch/attention/kernels/blackwell/mla/mla_decode_fp8.py — Verify partial page-table tiles do not read beyond the table.
• tensorrt_llm/_torch/autotuner.py — Verify suppression applies only to distributed parallel tuning and is restored after profiling errors.
• tensorrt_llm/_torch/configs/nemotron_h.py — Verify the None cache-dtype default resolves through the intended model-dtype logic.
• tensorrt_llm/_torch/disaggregation/native/transfer.py — Verify WRITE preflight failures prevent submission and retire ownership correctly.
• tensorrt_llm/_torch/disaggregation/nixl/_agent_py.py — Verify MemoryDescs conversion and errors match native-agent expectations.
• tensorrt_llm/_torch/distributed/ops.py — Verify process-wide workspace reuse is safe across threads and distinct mappings.
• tensorrt_llm/_torch/flashinfer_cubin_pool.py — Verify pool eligibility, checksum validation, atomic publication, and fallback behavior.
• tensorrt_llm/_torch/flashinfer_utils.py — Verify pool installation runs only when FlashInfer is available and preserves normal loader behavior.
• tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py — Verify packed-INT4 dimensions, scales, biases, and AWQ preprocessing for split and fused layouts.
• tensorrt_llm/_torch/models/checkpoints/hf/qwen4_exp_weight_mapper.py — Verify shared TP-rank concatenation preserves projection ordering.
• tensorrt_llm/_torch/models/modeling_minimaxm3.py — Verify reshaped GQA SDPA output preserves batch, KV-head, and group ordering.
• tensorrt_llm/_torch/models/modeling_mistral.py — Verify image geometry and scheduling metadata for both processors, including text-only checkpoint behavior.
• tensorrt_llm/_torch/models/modeling_utils.py — Verify TP-rank concatenation preserves rank-major ordering and rejects invalid shapes.
• tensorrt_llm/_torch/modules/linear.py — Verify optional AWQ pre-quant scales are loaded, sharded, and cleared correctly.
• tensorrt_llm/_torch/nccl_window_graph.py — Verify the context manager restores the previous suppression state, including after exceptions.
• tensorrt_llm/bench/tuning/heuristics.py — Verify missing and auto cache dtypes use the model dtype when sizing Mamba cache memory.
• tensorrt_llm/executor/base_worker.py — Verify LoRA ID bindings reject conflicting identities while allowing aliases and retries after unsuccessful lookups.
• tensorrt_llm/llmapi/mpi_session.py — Verify check_worker_error() reports a dead manager thread but not a live or stopped pool.
• tensorrt_llm/serve/disagg_coordinator.py — Verify cluster-storage scope validation rejects unsafe host and scope combinations.
• tensorrt_llm/usage/usage_lib.py — Verify deployment IDs are normalized as UUIDs and invalid values are omitted from telemetry.
• jenkins/scripts/perf/benchmark_utils.py — Verify EPLB detection handles inline and file-based configs and adds the shared-memory mount only when needed.
• jenkins/scripts/perf/local/submit.py — Verify local submission passes the loaded config into mount generation without changing other launch arguments.
• jenkins/scripts/perf/submit.py — Verify CI submission changes container mounts only and preserves other srun arguments.
• docs/source/blogs/tech_blog/blog13_Inference_Time_Compute_Implementation_in_TensorRT-LLM.md — Documentation-only model-name capitalization change.
• docs/source/developer-guide/sparse-attention-development-guide.md — Verify the documented summary contract matches kernel weighting for partial KV blocks.
• docs/source/features/sparse-attention.md — Verify the public sparse-attention contract states means, valid-token counts, and no pre-weighting.
• docs/source/llm-api/index.md — Verify shared-cubin documentation matches implementation and environment-variable behavior.
• docs/source/overview.md — Documentation-only capitalization correction.
• docs/source/release-notes.md — Documentation-only typo correction.
• examples/wide_ep/README.md — Verify the managed-memory fallback is limited to the stated coherent platforms.
• tensorrt_llm/_torch/attention/ATTENTION_DEVELOPER_GUIDE.md — Verify developer guidance matches sparse-kernel proxy weighting.
• tensorrt_llm/usage/schemas/README.md — Verify UUID normalization guidance matches telemetry behavior.

Test-code files

• cpp/tests/unit_tests/executor/transferAgentTest.cpp — Covers NIXL descriptor subranges, unknown agents, writes, and adjacent VMM chunks; no changed test-list entry is identified.
• cpp/tests/unit_tests/kernels/fmhaHead104Test.cpp — Covers packed-QKV head-size-104 output across masks, dtypes, and ragged lengths; its integration test is listed in GPU test lists.
• cpp/tests/unit_tests/runtime/hostAccessibleDeviceAllocatorTest.cu — Covers direct host access to managed memory without GDRCopy; no changed test-list entry is identified.
• tests/integration/defs/cpp/conftest.py — Adds the head-104 C++ build fixture; its integration test is listed in GPU test lists.
• tests/integration/defs/cpp/test_unit_tests.py — Adds the head-104 integration test for targets 100 and 120; corresponding entries appear in the supplied GPU test-list changes.
• tests/unittest/_torch/attention/multi_gpu/test_mha_helix.py — Covers attention output lifetime inside an NCCL-window tensor scope; no changed test-list entry is identified.
• tests/unittest/_torch/attention/multi_gpu/test_mla_helix.py — Covers MLA output lifetime inside an NCCL-window tensor scope; no changed test-list entry is identified.
• tests/unittest/_torch/attention/sparse/test_prims_ts_block_sparse.py — Covers partial-block proxy weighting and CPU/GPU behavior; no changed test-list entry is identified.
• tests/unittest/_torch/attention/test_cute_dsl_mla_page_table.py — Covers unpadded page-table widths and partial final pages on SM100/SM103; no changed test-list entry is identified.
• tests/unittest/_torch/attention/test_fmha_v2_head_size_coverage.py — Covers generated head-size-104 kernel inclusion and exclusions; no changed test-list entry is identified.
• tests/unittest/_torch/attention/test_pixtral_head104_attention.py — Covers Pixtral head-size-104 attention on supported architectures; listed for GB202 and RTX Pro 6000.
• tests/unittest/_torch/attention/test_prims_ts_attention_backend.py — Covers lazy rescaling around the skip threshold, including CUDA graph replay; no changed test-list entry is identified.
• tests/unittest/_torch/attention/test_prims_ts_fmha.py — Updates wrapper mocks to use autospec across FMHA paths; no changed test-list entry is identified.
• tests/unittest/_torch/distributed/test_allreduce_workspace_cache.py — Covers cross-thread reuse, concurrent initialization, and distinct mappings; no changed test-list entry is identified.
• tests/unittest/_torch/flashinfer/test_flashinfer_cubin_pool.py — Covers pool eligibility, reuse, corruption, failed downloads, concurrency, and installation; listed in test-db/l0_cpu.yml.
• tests/unittest/_torch/modeling/test_modeling_mistral.py — Covers image geometry and scheduling for both Mistral processors and text-only checkpoints; no changed test-list entry is identified.
• tests/unittest/_torch/modeling/test_qwen4_exp_support.py — Tests the shared TP-rank concatenation helper; no changed test-list entry is identified.
• tests/unittest/_torch/models/checkpoints/hf/test_qwen3_5_weight_mapper.py — Covers INT4 mapping, layouts, TP sizes, scales, and invalid inputs; no changed test-list entry is identified.
• tests/unittest/_torch/models/test_minimax_m3.py — Covers dense SDPA decode against explicit GQA repetition with scattered cache pages; no changed test-list entry is identified.
• tests/unittest/_torch/models/test_qwen3_5_int4_loading.py — Covers INT4 mapping into GDN projections across TP ranks and optional scales; listed in test-db/l0_b200.yml.
• tests/unittest/_torch/modules/test_linear_uneven_tp.py — Covers optional AWQ pre-quant scales in COLUMN and ROW modes; listed in test-db/l0_b200.yml.
• tests/unittest/_torch/multi_gpu/test_allocate_output_buffer_kinds.py — Covers NCCL-window allocation after distributed parallel tuning; no changed test-list entry is identified.
• tests/unittest/_torch/test_custom_config_registration.py — Covers Nemotron-H cache dtype resolution for absent and explicit values; no changed test-list entry is identified.
• tests/unittest/disaggregated/test_agent.py — Covers Python NIXL descriptor conversion, metadata results, and errors; no changed test-list entry is identified.
• tests/unittest/disaggregated/test_coordinator_worker.py — Covers loopback-only HTTP cluster-storage validation; no changed test-list entry is identified.
• tests/unittest/disaggregated/test_kv_transfer_fail_stop.py — Updates WRITE request fixtures for descriptor validation; no changed test-list entry is identified.
• tests/unittest/disaggregated/test_retirement_deadline.py — Updates NIXL WRITE fixtures across retirement and deadline cases; no changed test-list entry is identified.
• tests/unittest/disaggregated/test_transfer_late_settlement.py — Updates sender fixtures for descriptor checks; no changed test-list entry is identified.
• tests/unittest/disaggregated/test_transfer_ownership_regressions.py — Covers preflight rejection, submission failure, retirement state, and sender ownership; no changed test-list entry is identified.
• tests/unittest/executor/test_base_worker.py — Covers LoRA identity conflicts, aliases, retry behavior, and concurrency; no changed test-list entry is identified.
• tests/unittest/executor/test_proxy_fast_death.py — Covers dead MPI manager-thread detection; no changed test-list entry is identified.
• tests/unittest/llmapi/test_mpi_launcher_shutdown.py — Covers launcher startup without site initialization; no changed test-list entry is identified.
• tests/unittest/others/test_bench.py — Covers Nemotron cache sizing for unset, auto, and float32 dtypes; no changed test-list entry is identified.
• tests/unittest/scripts/test_perf_submit.py — Covers EPLB detection and CI/local mount generation; the PR objectives report 127 tests passed before rebase.
• tests/unittest/usage/test_reporter.py — Covers UUID normalization and invalid deployment IDs across telemetry events; no changed test-list entry is identified.
• tests/test_common/session_prefetcher.py — Changes diagnostic output handling; verify messages remain available through file descriptor 1.
• tests/unittest/_torch/peft/test_lora_ownership.py — Updates runtime setup with LoRA identity and locking fields; no changed test-list entry is identified.

Test-list and test configuration files

• tests/integration/test_lists/qa/llm_function_core.txt — Removes the Qwen3 NVFP4 test entry; the case moves to the multinode QA list.
• tests/integration/test_lists/qa/llm_function_multinode.txt — Adds the Qwen3 NVFP4 test entry; CI coverage is also listed in test-db/l0_gb300_multi_nodes_node2_gpu8.yml.
• tests/integration/test_lists/test-db/l0_b200.yml — Adds the head-104 C++ test and Qwen3.5 INT4/AWQ tests to pre-merge coverage.
• tests/integration/test_lists/test-db/l0_cpu.yml — Adds the FlashInfer cubin-pool unit test.
• tests/integration/test_lists/test-db/l0_gb202.yml — Adds head-104 and Pixtral attention tests.
• tests/integration/test_lists/test-db/l0_rtx_pro_6000.yml — Adds head-104 and Pixtral attention tests.
• tests/integration/test_lists/waives.txt — The supplied change summary reports removal of the Mistral Large 3 waiver. The checked-out file still contains the Qwen3 NVFP4 waiver, so verify the intended PR revision and exact waiver change.
• tests/scripts/perf-sanity/disaggregated/gb300_glm-5-fp4_1k1k_con1_ctx1_dep2_gen1_tep4_eplb0_mtp3_ccb-NIXL.yaml — Adds NUMA-aware context-worker affinity; verify the launcher accepts the setting.
• tests/scripts/perf-sanity/disaggregated/gb300_glm-5-fp4_1k1k_con4096_ctx1_dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL.yaml — Adds NUMA-aware context-worker affinity; verify the launcher accepts the setting.
• tests/scripts/perf-sanity/disaggregated/gb300_glm-5-fp4_1k1k_con512_ctx1_dep2_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml — Adds NUMA-aware context-worker affinity; verify the launcher accepts the setting.
• tests/scripts/perf-sanity/disaggregated/gb300_glm-5-fp4_8k1k_con1024_ctx1_dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL.yaml — Adds NUMA-aware context-worker affinity; verify the launcher accepts the setting.
• tests/scripts/perf-sanity/disaggregated/gb300_glm-5-fp4_8k1k_con1_ctx1_dep2_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml — Adds NUMA-aware context-worker affinity; verify the launcher accepts the setting.
• tests/scripts/perf-sanity/disaggregated/gb300_glm-5-fp4_8k1k_con512_ctx1_dep2_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml — Adds NUMA-aware context-worker affinity; verify the launcher accepts the setting.

Description

Second mass merge of release/1.3 fixes that are still missing from main, following #20232. It forward-ports the 23 release/1.3 PRs that, as of 2026-10-22, have no counterpart on main and no open PR to main. Each original PR is one commit, and each commit message links back to the original release PR and commit and keeps the original author and sign-off. Commits that needed changes for main carry an "Adapted to main" note in their message. The branch was rebased onto main 7924c15a61 on 2026-11-03.

Release PR	Ticket	Author	Change	Port	Commit
#19788	6665906	Pranav Shrestha	Restore Mistral Large 3 multimodal inference	Adapted: uses main's MultimodalItemMetadata contract (#20391); the HF tokenizer load honors trust_remote_code; the head-104 gtest fixture uses dynamic NVRTC and plain arch numbers like its siblings and always regenerates FMHA sources; fmhaHead104Test is registered with add_tllm_gtest (#21022)	ccf9a83c48
#19815	6562942	Pranav Shrestha	Load Qwen3.5 packed INT4 checkpoints	Clean cherry-pick	74df862d60
#20029	6877539	Chenfei Zhang (via TRT-LLM agent)	Fold GQA group into SDPA query axis in MiniMax-M3 decode attention	Adapted: also updates a stale main comment	6aaef38e7b
#20132	—	Ivy Zhang	Move cases to multinode test list	Clean cherry-pick	43f46be0fd
#20136	6463987	Chenfei Zhang	Pin GB300 GLM-5 ctx workers to their GPU's NUMA node in perf-sanity	Clean cherry-pick	7874cdc1eb
#19847	6859934	Mike Iovine (via TRT-LLM agent)	Process-wide workspace caches keyed by Mapping with a per-mapping lock…	Clean cherry-pick	d6ffcd8fdd
#20152	6792615	Sunny Gai	Keep session-prefetch output off sys.stdout	Clean cherry-pick	4a9f074486
#20193	6918015	Stanley Sun	Fix spelling and capitalization in release/1.3 docs	Clean cherry-pick	faa919e6b1
#20156	6379406	Sunny Gai	Use coherent managed memory for EPLB when GDRCopy is unavailable	Clean cherry-pick	05330a38c7
#20195	6911393	Mike Iovine (via TRT-LLM agent)	Run the launcher test's python3 stand-in with -S so start-up hooks cannot…	Adapted: release-only waives dropped	96b7eb675c
#20264	6825983	Chenfei Zhang	Share TRTLLM-Gen FMHA cubins across isolated MPI ranks	Clean cherry-pick; the concurrent-rank cubin test records any worker exception	7a26fcc768
#20248	6912133	Asfiya Baig	Detect dead MPI pool manager thread in MpiPoolSession	Clean cherry-pick	9b11d2ae8c
#20415	6862662	Yuxian Qiu	Bound FP8 MLA page-table loads for partial tiles	Adapted: kernel moved by #19275; test uses FP8 device-scale inputs (#19960)	8abe0a09d9
#20381	—	Yuxian Qiu	Follow up PrimTS refresh backport review tests and docs	Adapted: fits main's metadata-owned plan cache; main's CPU-only Sage tests keep their marker	e2c8059083
#20588	6932241	Yuxian Qiu	Scope NCCL window leases in helix attention tests	Adapted: release-only waives dropped; main's nvbugs/6991647 helix waives (#21110) have since been removed (#21387), so these cases run with the port	4de3ac5112
#20344	6553508	Yibin Li	Reject conflicting LoRA adapter identities	Adapted: wraps main's _load_lora_adapter() -> None (#20043); updates main's test_lora_ownership.py fixture (#20605); the conflict test's fake request sets prompt_adapter_request (#21036)	c813b350a3
#20346	6857649	Yibin Li	Validate native transfer destinations before submission	Adapted: main-only disagg tests pass a valid WRITE request; the release-only bounce final-destination check and its test are dropped because main removed the Python KV bounce buffer (#20049)	b7e81daa0b
#20531	6384589	XingFei Xi	Mount a 640G /dev/shm for online-EPLB perf cases	Adapted: tests reuse main's submit fixtures	6819df5900
#20657	6944374	Lori Ren	Fix rank-asymmetric NCCL window allocation hang during PARALLEL autotuning	Clean cherry-pick	d6f5717e55
#20284	6885370	Shanshan Wang	Keep Nemotron-H SSM cache dtype unset	Clean cherry-pick	4d47c3e234
#20651	6887489	Yibin Li	Restrict coordinator HTTP cluster storage to loopback	Clean cherry-pick	066b491caf
#20794	6964933	Ruodi Lu	Use FlashInfer for Gemma4 NVFP4 perf on SM107	Adapted: updates main's SM121 comment	fd68065861
#20777	—	Maxim Gluhovskoi	Validate telemetry deployment IDs as UUIDs	Adapted: test patches _HEARTBEAT_STOP (renamed on main)	5b0b141f55

Notes:

• [https://nvbugs/6463987][fix] Cherry-pick #20103 to release/1.3: pin GB300 GLM-5 ctx workers to their GPU's NUMA node in perf-sanity #20136 and [https://nvbugs/6825983][perf] Cherry-pick #20257 to release/1.3: Share TRTLLM-Gen FMHA cubins across isolated MPI ranks #20264 were release cherry-picks of main PRs [https://nvbugs/6463987][fix] Pin GB300 GLM-5 ctx workers to their GPU's NUMA node in perf-sanity #20103 and [https://nvbugs/6825983][perf] Share TRTLLM-Gen FMHA cubins across isolated MPI ranks #20257, which were closed without merging, so the fixes are ported from the release PRs.
• [https://nvbugs/6877539][fix] Fold GQA group into SDPA query axis in MiniMax-M3 decode attention #20029, [https://nvbugs/6665906][fix] Restore Mistral Large 3 multimodal inference #19788 and [https://nvbugs/6562942][fix] Load Qwen3.5 packed INT4 checkpoints #19815 had earlier forward-port PRs ([https://nvbugs/6877539][fix] Fold GQA group into SDPA query axis in MiniMax-M3 decode attention #20115, [https://nvbugs/6665906][fix] Cherry-pick #19788: Restore Mistral Large 3 multimodal inference #20172, [https://nvbugs/6562942][fix] Cherry-pick #19815: Load Qwen3.5 packed INT4 checkpoints #20171) that were closed without merging.
• The main waive for nvbugs/6862662 (test_attention_mla) is kept; it can be removed once the [https://nvbugs/6862662][fix] Bound FP8 MLA page-table loads for partial tiles #20415 fix is confirmed on B200.
• Not included: release PRs that are only partly on main ([https://nvbugs/6783973][doc] Cherry-pick #19784 to release/1.3: stop listing supported SA speculation as a Kimi K3 limitation #19863, [https://nvbugs/6801229][fix] Fix NVFP4 gather test references and skip empty FP4 quantize launches #19761, [https://nvbugs/6818710][fix] Add helix_owned_new_tokens to the base update_helix_param hook #20018, [https://nvbugs/6892484][fix] Port of the 6791675 fix to release/1.3: RemoteTask gains sys_path, the… #20058, [TRTLLMINF-474][infra] Upgrade NGC PyTorch base image to 26.09 (#19679) #20146, [https://nvbugs/6784181][fix] Fix test_relative_path_validity: generated-doc allowlist and skip installed envs (cherry-pick #19405, #19403) #20601, [None][fix] Cherry-pick #20203, #20581, #20575 to release/1.3 #20689) and those already covered by open PRs to main.

Test Coverage

• Each commit carries its original PR's tests, including the new test_qwen3_5_int4_loading.py, test_fmha_v2_head_size_coverage.py, test_pixtral_head104_attention.py, fmhaHead104Test.cpp, test_allreduce_workspace_cache.py, test_flashinfer_cubin_pool.py, test_cute_dsl_mla_page_table.py, the LoRA identity tests in test_base_worker.py, the EPLB /dev/shm tests in test_perf_submit.py, and the coordinator, telemetry and MPI pool tests.
• Tests adapted for main: test_cute_dsl_mla_page_table.py (FP8 scale inputs), test_base_worker.py (no return value), test_retirement_deadline.py, test_kv_transfer_fail_stop.py and test_transfer_late_settlement.py (WRITE requests for the new pre-submission check), test_prims_ts_block_sparse.py (cpu_only markers, including on main's Sage tests), test_modeling_mistral.py (main's get_mm_item_metadata API and tuple fields), main's test_lora_ownership.py (adapter identity state in its worker fixture), and test_base_worker.py (prompt_adapter_request on the fake request). The release-only bounce test in test_transfer_ownership_regressions.py is dropped with the bounce code.
• Locally: all touched Python files compile with no new ruff F/E9 findings versus main, and the repo formatters pass. Before the first rebase, tests/unittest/scripts/test_perf_submit.py passed (127 tests). GPU tests were not run locally and rely on CI.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

🤖 Generated with Claude Code

[SimengLiu-nv]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #82315 [ run ] triggered by Bot. Commit: 5b0b141 Link to invocation

[tensorrt-cicd]

PR_Github #82315 [ run ] completed with state FAILURE. Commit: 5b0b141
/LLM/main/L0_MergeRequest_PR pipeline #67936 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[janbernloehr]

Additional validation for the #20284 forward-port in this PR: the NemotronHConfig default change fixes a Nemotron-Nano-9B-v2 FP8 startup regression on DGX Spark, tracked in NVBug 7029520.

The Core Models A/B run used the same TensorRT-LLM 1.4.0rc3.dev202611020007 image and Transformers 5.5.4, with trtllm-serve --tp_size 1 --pp_size 1 --max_batch_size 512 --max_num_tokens 8192 --max_seq_len 3000 --kv_cache_free_gpu_memory_fraction 0.9, block reuse disabled, and the SSM cache dtype left at auto.

Configuration	Result
Unmodified nightly	Executor initialization fails: V2 Mamba GPU cache quota got 50848800768 bytes, need at least 73640861696 bytes.
Same image, changing only mamba_ssm_cache_dtype: str = "float32" to mamba_ssm_cache_dtype: str | None = None in _torch/configs/nemotron_h.py	Server initializes and the ISL 2000 / OSL 128, concurrency 1 benchmark completes at 23.18 output tokens/s.

The checkpoint does not declare an SSM cache dtype. The registered config's float32 default makes auto select fp32 recurrent states instead of the model's bf16 dtype, doubling the minimum quota from 37,331,296,256 to 73,640,861,696 bytes. Leaving the field unset restores the intended fallback. This requirement increase is distinct from unified-memory page-cache effects on the available quota.

Evidence for NVIDIA maintainers: A/B pipeline, unmodified control, patched treatment.

Scope: one run per arm on separate DGX Spark hosts, using the same image and workload settings. Only the nemotron_h.py hunk was applied at runtime; this does not validate the full PR head, the trtllm-bench heuristic change, or the bf16/nvfp4 variants.

Thanks for carrying this fix to main.

Posted by Codex (GPT-6).

[SimengLiu-nv]

/bot run

[tensorrt-cicd]

PR_Github #82532 [ run ] triggered by Bot. Commit: 5b0b141 Link to invocation

[tensorrt-cicd]

PR_Github #82532 [ run ] completed with state FAILURE. Commit: 5b0b141
/LLM/main/L0_MergeRequest_PR pipeline #68126 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[SimengLiu-nv]

/bot skip --comment "Failure is caused by unrelated timeout. The CI before rebase was successfully as well."

[tensorrt-cicd]

PR_Github #82600 [ skip ] triggered by Bot. Commit: 5b0b141 Link to invocation

[tensorrt-cicd]

PR_Github #82600 [ skip ] completed with state SUCCESS. Commit: 5b0b141
Skipping testing for commit 5b0b141

Link to invocation