[TRTLLM-15766][refactor] Decouple decoder tests from private methods and split DecoderRunner by phase by lori-ren · Pull Request #20134 · NVIDIA/TensorRT-LLM

lori-ren · 2026-10-12T06:14:05Z

Dev Engineer Review

The refactor moves decoder input preparation, metadata, dummy-request creation, CUDA-graph handling, and warmup into dedicated components. DecoderRunner delegates those responsibilities and retains its forward path and public entry points. EncoderDecoderRunner supplies specialized components.

common.py adds shared token-padding and model-input helpers. PreparedInputs gains runtime_draft_len, defaulting to 0. Speculative metadata helpers move from speculative.py to metadata.py. LoraParamBuilder initializes model_config to None.

Key regression risks are component wiring, padding and graph flags, speculative metadata, warmup rank agreement and failure handling, and encoder-decoder cross-KV allocation and cleanup. The supplied material states that behavior changes are not intended. It provides no current review findings or complete test results. CI reports include SUCCESS, UNSTABLE, and FAILURE outcomes, but do not identify failed tests.

QA Engineer Review

Unit tests changed across decoder inputs, metadata, graph handling, warmup, no-KV-cache inputs, and encoder-decoder handling. The waiver entry in tests/integration/test_lists/waives.txt changed to test_prepare_inputs_with_helix_parallelism; its skip reason remains unchanged. No test functions or IDs beyond that waiver rename are identified in the supplied summaries.

Coverage includes padding and position layouts, dummy-request admission and cleanup, graph selection and fallback, speculative metadata, warmup rank agreement and OOM handling, and encoder-decoder cross-attention. The changed tests are under tests/unittest/, so integration-list membership is not applicable. The live list search confirms the Helix test appears in waives.txt; it found no matching entry for test_kvcm2_integration.py in CI or manual-QA lists. Coverage verdict: needs follow-up. No test execution results are supplied.

Per-File QA Perspective

Source files

tensorrt_llm/_torch/attention/backends/sparse/dsa/metadata.py: Comment references change to the input preparer; no runtime change is reported.
tensorrt_llm/_torch/models/modeling_multimodal_mixin.py: Comments identify the new input-preparation path; no runtime change is reported.
tensorrt_llm/_torch/models/modeling_multimodal_utils.py: The sync-free contract comment references the new preparation methods; no runtime change is reported.
tensorrt_llm/_torch/modules/mamba/mamba2_metadata.py: A comment identifies the new input-preparation path; no runtime change is reported.
tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py: Comments refer to the new graph and input components; no runtime change is reported.
tensorrt_llm/_torch/pyexecutor/engine/lora.py: model_config now initializes to None; verify its initialization and graph-manager replacement during resource rebuilds.
tensorrt_llm/_torch/pyexecutor/engine/multimodal.py: A docstring references the new metadata component; no runtime change is reported.
tensorrt_llm/_torch/pyexecutor/engine/runners/common.py: New helpers handle padding, graph eligibility, metadata updates, and model-input construction; verify padded and unpadded inputs.
tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/__init__.py: ExtraInputsCollector is imported from its new module; package exports are reported unchanged.
tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/dummy_requests.py: New factory paths create and release warmup and graph-capture requests; verify admission limits and cleanup on allocation failures.
tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/graphs.py: New graph management handles selection, capture, replay, and cleanup; verify shape selection, promotion gates, and eager fallback.
tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/metadata.py: New metadata setup and update paths handle attention, speculation, and rank counts; verify draft-length and synchronization rules.
tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/speculative.py: Speculative helpers are removed from this module and relocated to metadata.py; verify imports and callers use the new location.
tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/warmup.py: New warmup orchestration coordinates ranks, shapes, backend selection, and recoverable failures; verify distributed agreement and OOM handling.
tensorrt_llm/_torch/pyexecutor/engine/runners/encoder_decoder.py: New specialized components handle cross-attention, graph capture, and dummy requests; verify fast-path eligibility and cross-KV cleanup.
tensorrt_llm/_torch/pyexecutor/engine/runners/interface.py: PreparedInputs.runtime_draft_len defaults to 0; verify callers accept the additional field.
tensorrt_llm/_torch/pyexecutor/engine/runners/no_kv_cache.py: Shared helpers replace inline padding and input construction; verify no-KV-cache metadata and buffer behavior.
tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/runner.py: Runner responsibilities now delegate to the split components; verify forward, graph, warmup, and release paths preserve behavior.
tensorrt_llm/_torch/pyexecutor/guided_decoder.py: A comment points to the new preprocessing location; no runtime change is reported.
tensorrt_llm/_torch/pyexecutor/kv_cache/mamba_cache_manager.py: A docstring references the relocated draft-length mapping helper; implementation is reported unchanged.
tensorrt_llm/_torch/pyexecutor/llm_request.py: Comments reference the new input-preparation methods; no runtime change is reported.
tensorrt_llm/_torch/pyexecutor/py_executor.py: A comment references the new input-preparation path; no runtime change is reported in the supplied summary.
tensorrt_llm/_torch/pyexecutor/py_executor_creator.py: The helper import now comes from the decoder inputs module; verify the import path.
tensorrt_llm/_torch/pyexecutor/resource_manager.py: Comments reference the new input-preparation methods; no runtime change is reported.
tensorrt_llm/_torch/route_capture.py: A comment identifies the new input-preparation path; no runtime change is reported.
tensorrt_llm/_torch/speculative/dspark.py: A comment references the new dummy-request factory; no runtime change is reported.
tensorrt_llm/_torch/speculative/interface.py: A field comment references the relocated greedy-sampling helper; no runtime change is reported.

Test-code files

tests/unittest/_torch/attention/sparse/csa2/test_metadata.py: Exercises deferred metadata refresh through DecoderInputPreparer; integration-list membership is not applicable.
tests/unittest/_torch/executor/engine/test_common.py: Covers padded and unpadded tokens, position layouts, graph flags, and buffer updates; integration-list membership is not applicable.
tests/unittest/_torch/executor/engine/test_decoder_speculative.py: Updates imports for relocated metadata helpers; no coverage change is reported. Integration-list membership is not applicable.
tests/unittest/_torch/executor/engine/test_encoder_decoder.py: Covers specialized components, cross-KV cleanup and allocation failure, warmup ordering, and cross-attention; integration-list membership is not applicable.
tests/unittest/_torch/executor/engine/test_no_kv_cache.py: Updates the patch target for the shared helper; integration-list membership is not applicable.
tests/unittest/_torch/executor/engine/test_runners.py: Initializes the JIT warmup flag in a fixture; integration-list membership is not applicable.
tests/unittest/_torch/executor/kv_cache/test_kvcm2_integration.py: Covers dummy-request creation for graph warmup; no matching CI or manual-QA list entry was found, and integration-list membership is not applicable to this unit-test path.
tests/unittest/_torch/executor/test_context_only_warmup.py: Covers request bounds, admission failures, request phases, and graph selection; integration-list membership is not applicable.
tests/unittest/_torch/executor/test_distributed_warmup_oom.py: Covers rank agreement, shape selection, and recovery or fatal errors; integration-list membership is not applicable.
tests/unittest/_torch/executor/test_eager_workspace_engine.py: Covers workspace-reclaimer eligibility through DecoderWarmup; integration-list membership is not applicable.
tests/unittest/_torch/executor/test_gen_only_cached_tokens.py: Updates a test comment to identify the context-packing method; integration-list membership is not applicable.
tests/unittest/_torch/executor/test_py_executor.py: Updates a comment about draft-token preparation; integration-list membership is not applicable.
tests/unittest/_torch/executor/test_pytorch_model_engine.py: Covers split-component behavior, graph paths, metadata, and buffer ownership; integration-list membership is not applicable.
tests/unittest/_torch/executor/test_pytorch_model_engine_warmup.py: Covers warmup ordering, tuning, fallback, and graph timing; integration-list membership is not applicable.
tests/unittest/_torch/executor/test_remote_tail_adp_phase.py: Covers attention-DP and decode-rank-token helpers; integration-list membership is not applicable.
tests/unittest/_torch/executor/test_token_budget_fallback.py: Updates comments about input preparation; integration-list membership is not applicable.
tests/unittest/_torch/modeling/test_modeling_deepseekv41.py: Covers all-rank speculative token counts through the input preparer; integration-list membership is not applicable.
tests/unittest/_torch/speculative/test_capture_sampling_params.py: Updates the import source for capture sampling parameters; integration-list membership is not applicable.

Test-list file

tests/integration/test_lists/waives.txt: Renames the Helix-parallelism waiver entry and retains its skip reason. The updated test name is present in the waiver list.

Description

#19896 moved the decoder code out of model_engine.py unchanged, leaving DecoderRunner as a single class that owns every phase. This PR splits it into components. DecoderRunner keeps construction, the public entry points and the forward path. Each phase becomes a component that owns its state and gets its dependencies through its constructor, without a reference back to the runner:

input buffers, a frozen subclass of the engine's InputBuffers;
input preparation, split into context, generation, speculation and multimodal stages;
attention and speculative metadata;
warmup/capture dummy requests;
CUDA graph selection, capture and dispatch, the only owner of the graph runner state;
warmup, and the cross-rank warmup agreement.

The runner builds its components through overridable factories. EncoderDecoderRunner stays a subclass and overrides them to return component subclasses:

dummy requests that also hold cross-KV cache blocks;
an input preparer that adds the cross-attention inputs and the native fast path;
CUDA graphs that prefill cross-KV before capture and capture the mixed shapes.

Execution paths that a model family cannot take are declared as capability flags on the runner.

Method bodies move unchanged apart from attribute paths. The few deliberate reshapes are:

a padding and model-input tail in runners/common.py, shared by the decoder and no-KV-cache runners;
dummy-batch release owned by the dummy-request factory;
the prepared draft length returned in PreparedInputs.

No behaviour change is intended.

The tests no longer patch or call private runner methods by name:

forward tests drive a real runner;
graph selection is asserted through the runner's public seams;
component behaviour is tested on the components themselves.

Test Coverage

tests/unittest/_torch/executor/test_pytorch_model_engine.py
tests/unittest/_torch/executor/test_pytorch_model_engine_warmup.py
tests/unittest/_torch/executor/test_context_only_warmup.py
tests/unittest/_torch/executor/test_distributed_warmup_oom.py
tests/unittest/_torch/executor/test_eager_workspace_engine.py
tests/unittest/_torch/executor/test_remote_tail_adp_phase.py
tests/unittest/_torch/executor/engine/test_encoder_decoder.py
tests/unittest/_torch/executor/engine/test_common.py
tests/unittest/_torch/executor/test_py_executor.py

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Graph selection, promoted-context, overlap and MRoPE tests run the real runner through engine.forward, stubbing only the CUDA graph runner. Signed-off-by: Lori Ren <lorir@nvidia.com>

Check the graph lookup's arguments instead of patching the selector by module path, and stub release_graphs instead of the private graph release. Signed-off-by: Lori Ren <lorir@nvidia.com>

…ffers The decoder runner's persistent input buffers move into one dataclass that allocates them in the same order. Signed-off-by: Lori Ren <lorir@nvidia.com>

The distributed warmup agreement moves out of DecoderRunner into a class built from dist and mapping. Signed-off-by: Lori Ren <lorir@nvidia.com>

Attention and speculative metadata memos move into DecoderMetadata; the runner delegates attn_metadata and spec_metadata to it. Signed-off-by: Lori Ren <lorir@nvidia.com>

Warmup and capture dummy requests move into DummyRequestFactory, which takes the runner's dummy-request hooks as bound methods. Signed-off-by: Lori Ren <lorir@nvidia.com>

CUDA graph runners, capture, and no_cuda_graph move into DecoderCudaGraphs; the runner exposes the two graph runners as read-only properties. Signed-off-by: Lori Ren <lorir@nvidia.com>

Input preparation and the steady-gen cache move into DecoderInputPreparer; the runner delegates guided_decoder to it. Signed-off-by: Lori Ren <lorir@nvidia.com>

…paration prepare_inputs and the fast-path hooks return PreparedInputs instead of tuples; the runner resolves the tree draft length itself. Signed-off-by: Lori Ren <lorir@nvidia.com>

…uts hook Steady-gen reuse and context promotion turn off when the runner has an extra-inputs hook; the encoder-decoder runner overrides eager workspace reclamation instead of setting flags. Signed-off-by: Lori Ren <lorir@nvidia.com>

…er metadata DecoderMetadata.set_up_attn/set_up_spec become setup_attention/setup_speculative, and the speculative.py helpers move into metadata.py; resolve_runtime_draft_len moves to inputs.py. Signed-off-by: Lori Ren <lorir@nvidia.com>

Fold _prepare_tp_inputs into prepare_inputs and split it into DecoderInputPreparer stage methods sharing a per-pass HostInputs accumulator. Signed-off-by: Lori Ren <lorir@nvidia.com>

Move decoder warmup into a DecoderWarmup component built from the runner's components. Derive the encoder-decoder warmup differences from the extra-inputs hook. Signed-off-by: Lori Ren <lorir@nvidia.com>

Move encoder-decoder input preparation into its own component and share the padding and model-input tail with the decoder preparer. Signed-off-by: Lori Ren <lorir@nvidia.com>

…putPreparer The forward-time position and KV-length corrections now live with the input buffers they read. Signed-off-by: Lori Ren <lorir@nvidia.com>

Dummy batch release and mixed context rows move into DummyRequestFactory, and encoder-decoder capture moves into its own component. Signed-off-by: Lori Ren <lorir@nvidia.com>

…ted iter states out of DecoderRunner The greedy-sample group sync is a metadata function, and the preparer owns the input copy event and every iter_states write. Signed-off-by: Lori Ren <lorir@nvidia.com>

…erCudaGraphs LoRA graph choice, final-context promotion and the eager, breakable, capture and replay dispatch now live with the graph runners. Signed-off-by: Lori Ren <lorir@nvidia.com>

Comments added upstream still named DecoderRunner methods that moved into components. Signed-off-by: Lori Ren <lorir@nvidia.com>

Context warmups, context-row promotion, steady-generation reuse, eager workspace reclaim and deferred decode outputs each get their own flag again instead of following the extra-inputs hook. Signed-off-by: Lori Ren <lorir@nvidia.com>

… factories Encoder-decoder behaviour moves into component subclasses instead of runner hooks, and one LoRA builder holds the LoRA state. Signed-off-by: Lori Ren <lorir@nvidia.com>

…udaGraphs The runner selects, pads and runs graphs through the component, which also owns the two-pass warmup and capture. Signed-off-by: Lori Ren <lorir@nvidia.com>

… runner Warmup returns the eager workspace reclaimer and marks JIT warmup forwards through the forward hook. Signed-off-by: Lori Ren <lorir@nvidia.com>

…are the padding tail DecoderInputBuffers extends the frozen engine buffers, and the decoder and no-KV-cache runners pad through common helpers. Signed-off-by: Lori Ren <lorir@nvidia.com>

… signatures Preparation reports the step's draft length and keeps its lists in a frozen group; constructors are keyword-only and returns annotated. Signed-off-by: Lori Ren <lorir@nvidia.com>

Rank agreement only serves the warmup phase, so it lives with DecoderWarmup again. Signed-off-by: Lori Ren <lorir@nvidia.com>

lori-ren · 2026-10-12T08:40:22Z

/bot run

tensorrt-cicd · 2026-10-12T08:46:18Z

PR_Github #77244 [ run ] triggered by Bot. Commit: b50b0c6 Link to invocation

tensorrt-cicd · 2026-10-12T14:34:05Z

PR_Github #77244 [ run ] completed with state SUCCESS. Commit: b50b0c6
/LLM/main/L0_MergeRequest_PR pipeline #63676 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Either:

Wait for the PR to be fully approved — the label is added automatically once approval is complete. Having unresolved open comments is fine, or
If needed, ask a member of NVIDIA/trt-llm-ci-approvers to add the label manually.
Then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

lori-ren · 2026-10-16T23:23:23Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-16T23:30:10Z

PR_Github #78573 [ run ] triggered by Bot. Commit: 35d0ed3 Link to invocation

tensorrt-cicd · 2026-10-17T00:32:34Z

PR_Github #78573 [ run ] completed with state FAILURE. Commit: 35d0ed3

Link to invocation

lori-ren · 2026-10-17T00:52:26Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-17T00:59:09Z

PR_Github #78583 [ run ] triggered by Bot. Commit: 35d0ed3 Link to invocation

tensorrt-cicd · 2026-10-17T09:07:11Z

PR_Github #78583 [ run ] completed with state SUCCESS. Commit: 35d0ed3
/LLM/main/L0_MergeRequest_PR pipeline #64741 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lori-ren · 2026-10-17T09:24:28Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-17T09:32:15Z

PR_Github #78630 [ run ] triggered by Bot. Commit: 35d0ed3 Link to invocation

tensorrt-cicd · 2026-10-17T14:00:21Z

PR_Github #78630 [ run ] completed with state SUCCESS. Commit: 35d0ed3
/LLM/main/L0_MergeRequest_PR pipeline #64781 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Port the generation-only autotuner warmup for Helix CP into DecoderWarmup, and its test onto the component. Signed-off-by: Lori Ren <lorir@nvidia.com>

lori-ren · 2026-10-17T14:28:06Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-17T14:34:20Z

PR_Github #78651 [ run ] triggered by Bot. Commit: adc8594 Link to invocation

tensorrt-cicd · 2026-10-17T19:59:07Z

PR_Github #78651 [ run ] completed with state SUCCESS. Commit: adc8594
/LLM/main/L0_MergeRequest_PR pipeline #64798 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Keep the renamed helix waive alongside the new compile-bypass waives. Signed-off-by: Lori Ren <lorir@nvidia.com>

lori-ren · 2026-10-17T20:29:50Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-17T20:36:12Z

PR_Github #78666 [ run ] triggered by Bot. Commit: e774ce8 Link to invocation

tensorrt-cicd · 2026-10-18T03:20:43Z

PR_Github #78666 [ run ] completed with state FAILURE. Commit: e774ce8
/LLM/main/L0_MergeRequest_PR pipeline #64813 completed with status: 'UNSTABLE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

lori-ren · 2026-10-19T01:47:01Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-19T01:54:36Z

PR_Github #78747 [ run ] triggered by Bot. Commit: e774ce8 Link to invocation

tensorrt-cicd · 2026-10-19T04:35:58Z

PR_Github #78747 [ run ] completed with state SUCCESS. Commit: e774ce8
/LLM/main/L0_MergeRequest_PR pipeline #64883 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lori-ren · 2026-10-19T04:36:29Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-19T04:43:10Z

PR_Github #78797 [ run ] triggered by Bot. Commit: e774ce8 Link to invocation

tensorrt-cicd · 2026-10-19T08:08:08Z

PR_Github #78797 [ run ] completed with state SUCCESS. Commit: e774ce8
/LLM/main/L0_MergeRequest_PR pipeline #64922 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lori-ren · 2026-10-19T08:11:39Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-19T08:19:58Z

PR_Github #78853 [ run ] triggered by Bot. Commit: e774ce8 Link to invocation

tensorrt-cicd · 2026-10-19T09:42:41Z

PR_Github #78853 [ run ] completed with state SUCCESS. Commit: e774ce8
/LLM/main/L0_MergeRequest_PR pipeline #64968 completed with status: 'SUCCESS'

CI Report

Link to invocation

Resolve the conflict with NVIDIA#20134, which moved warmup out of DecoderRunner: the MiniMax-M3 MSA indexer prewarm now lives in DecoderWarmup (decoder/warmup.py). Signed-off-by: Zheyu Fu <zheyuf@nvidia.com>

lori-ren added 18 commits October 12, 2026 04:51

[TRTLLM-15766][test] Drive decoder forward tests through a real runner

d4c653a

Graph selection, promoted-context, overlap and MRoPE tests run the real runner through engine.forward, stubbing only the CUDA graph runner. Signed-off-by: Lori Ren <lorir@nvidia.com>

[TRTLLM-15766][test] Assert graph selection through public runner seams

2009efa

Check the graph lookup's arguments instead of patching the selector by module path, and stub release_graphs instead of the private graph release. Signed-off-by: Lori Ren <lorir@nvidia.com>

[TRTLLM-15766][refactor] Hold decoder input buffers in DecoderInputBu…

309402d

…ffers The decoder runner's persistent input buffers move into one dataclass that allocates them in the same order. Signed-off-by: Lori Ren <lorir@nvidia.com>

[TRTLLM-15766][refactor] Extract WarmupAgreement

e7fa68e

The distributed warmup agreement moves out of DecoderRunner into a class built from dist and mapping. Signed-off-by: Lori Ren <lorir@nvidia.com>

[TRTLLM-15766][refactor] Extract DecoderMetadata

76a9b94

Attention and speculative metadata memos move into DecoderMetadata; the runner delegates attn_metadata and spec_metadata to it. Signed-off-by: Lori Ren <lorir@nvidia.com>

[TRTLLM-15766][refactor] Extract DummyRequestFactory

ea1d445

Warmup and capture dummy requests move into DummyRequestFactory, which takes the runner's dummy-request hooks as bound methods. Signed-off-by: Lori Ren <lorir@nvidia.com>

[TRTLLM-15766][refactor] Extract DecoderCudaGraphs

383058a

CUDA graph runners, capture, and no_cuda_graph move into DecoderCudaGraphs; the runner exposes the two graph runners as read-only properties. Signed-off-by: Lori Ren <lorir@nvidia.com>

[TRTLLM-15766][refactor] Extract DecoderInputPreparer

4ab056d

Input preparation and the steady-gen cache move into DecoderInputPreparer; the runner delegates guided_decoder to it. Signed-off-by: Lori Ren <lorir@nvidia.com>

[TRTLLM-15766][refactor] Return PreparedInputs from decoder input pre…

33a35da

…paration prepare_inputs and the fast-path hooks return PreparedInputs instead of tuples; the runner resolves the tree draft length itself. Signed-off-by: Lori Ren <lorir@nvidia.com>

[TRTLLM-15766][refactor] Split decoder input preparation into stages

d343d88

Fold _prepare_tp_inputs into prepare_inputs and split it into DecoderInputPreparer stage methods sharing a per-pass HostInputs accumulator. Signed-off-by: Lori Ren <lorir@nvidia.com>

[TRTLLM-15766][refactor] Extract DecoderWarmup

f0b5c21

Move decoder warmup into a DecoderWarmup component built from the runner's components. Derive the encoder-decoder warmup differences from the extra-inputs hook. Signed-off-by: Lori Ren <lorir@nvidia.com>

[TRTLLM-15766][refactor] Extract EncoderDecoderInputPreparer

5466792

Move encoder-decoder input preparation into its own component and share the padding and model-input tail with the decoder preparer. Signed-off-by: Lori Ren <lorir@nvidia.com>

[TRTLLM-15766][refactor] Move input pre/postprocessing into DecoderIn…

02b8619

…putPreparer The forward-time position and KV-length corrections now live with the input buffers they read. Signed-off-by: Lori Ren <lorir@nvidia.com>

[TRTLLM-15766][refactor] Extract EncoderDecoderCudaGraphs

3843f0a

Dummy batch release and mixed context rows move into DummyRequestFactory, and encoder-decoder capture moves into its own component. Signed-off-by: Lori Ren <lorir@nvidia.com>

[TRTLLM-15766][refactor] Move greedy sync, input copy event and promo…

5dc705b

…ted iter states out of DecoderRunner The greedy-sample group sync is a metadata function, and the preparer owns the input copy event and every iter_states write. Signed-off-by: Lori Ren <lorir@nvidia.com>

[TRTLLM-15766][refactor] Move graph selection and dispatch into Decod…

17d4f18

…erCudaGraphs LoRA graph choice, final-context promotion and the eager, breakable, capture and replay dispatch now live with the graph runners. Signed-off-by: Lori Ren <lorir@nvidia.com>

github-actions Bot assigned lori-ren Oct 12, 2026

lori-ren added 8 commits October 12, 2026 07:21

[TRTLLM-15766][chore] Point comments at the decoder input preparer

5d2024f

Comments added upstream still named DecoderRunner methods that moved into components. Signed-off-by: Lori Ren <lorir@nvidia.com>

[TRTLLM-15766][refactor] Build decoder components through overridable…

82ff2fa

… factories Encoder-decoder behaviour moves into component subclasses instead of runner hooks, and one LoRA builder holds the LoRA state. Signed-off-by: Lori Ren <lorir@nvidia.com>

[TRTLLM-15766][refactor] Keep CUDA graph runner state inside DecoderC…

150bb4e

…udaGraphs The runner selects, pads and runs graphs through the component, which also owns the two-pass warmup and capture. Signed-off-by: Lori Ren <lorir@nvidia.com>

[TRTLLM-15766][refactor] Keep per-forward warmup state on the decoder…

8c1ee57

… runner Warmup returns the eager workspace reclaimer and marks JIT warmup forwards through the forward hook. Signed-off-by: Lori Ren <lorir@nvidia.com>

[TRTLLM-15766][refactor] Build decoder buffers on InputBuffers and sh…

ab80a4e

…are the padding tail DecoderInputBuffers extends the frozen engine buffers, and the decoder and no-KV-cache runners pad through common helpers. Signed-off-by: Lori Ren <lorir@nvidia.com>

[TRTLLM-15766][refactor] Tighten decoder prepared state and component…

d1cd7a9

… signatures Preparation reports the step's draft length and keeps its lists in a frozen group; constructors are keyword-only and returns annotated. Signed-off-by: Lori Ren <lorir@nvidia.com>

[TRTLLM-15766][refactor] Keep WarmupAgreement in the warmup module

b50b0c6

Rank agreement only serves the warmup phase, so it lives with DecoderWarmup again. Signed-off-by: Lori Ren <lorir@nvidia.com>

Merge branch 'main' into refactor/split-decoder-runner

adc8594

Port the generation-only autotuner warmup for Helix CP into DecoderWarmup, and its test onto the component. Signed-off-by: Lori Ren <lorir@nvidia.com>

Merge branch 'main' into refactor/split-decoder-runner

e774ce8

Keep the renamed helix waive alongside the new compile-bypass waives. Signed-off-by: Lori Ren <lorir@nvidia.com>

lori-ren merged commit a3766d6 into NVIDIA:main Oct 19, 2026
7 checks passed

Wanli-Jiang mentioned this pull request Oct 19, 2026

[TRTLLM-17033][feat] Qwen3.8-Flash-Next modeling_v2: CUTEDSL, MTP 1..7, video, FP8/NVFP4 KV, bf16 state, canonical QSA selection, breakable prefill graph, GB300 fast paths #20276

Open

1 task done

farazkh80 mentioned this pull request Oct 19, 2026

[None][feat] Locality domain support for the C++ KV cache manager v2 and two-stream attention #20229

Open

1 task done

Conversation

lori-ren commented Oct 12, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Per-File QA Perspective

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

lori-ren commented Oct 12, 2026

Uh oh!

tensorrt-cicd commented Oct 12, 2026

Uh oh!

tensorrt-cicd commented Oct 12, 2026

Uh oh!

lori-ren commented Oct 16, 2026

Uh oh!

tensorrt-cicd commented Oct 16, 2026

Uh oh!

tensorrt-cicd commented Oct 17, 2026

Uh oh!

lori-ren commented Oct 17, 2026

Uh oh!

tensorrt-cicd commented Oct 17, 2026

Uh oh!

tensorrt-cicd commented Oct 17, 2026

Uh oh!

lori-ren commented Oct 17, 2026

Uh oh!

tensorrt-cicd commented Oct 17, 2026

Uh oh!

tensorrt-cicd commented Oct 17, 2026

Uh oh!

lori-ren commented Oct 17, 2026

Uh oh!

tensorrt-cicd commented Oct 17, 2026

Uh oh!

tensorrt-cicd commented Oct 17, 2026

Uh oh!

lori-ren commented Oct 17, 2026

Uh oh!

tensorrt-cicd commented Oct 17, 2026

Uh oh!

tensorrt-cicd commented Oct 18, 2026

Uh oh!

lori-ren commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

lori-ren commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

lori-ren commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

lori-ren commented Oct 12, 2026 •

edited by coderabbitai Bot

Loading