[krishung5]

Description

Add audio URL, data-URI, and raw-byte inputs to OpenEngine Generate, with model/checkpoint capability gates, prompt-order preservation, caller UUID forwarding, and decoded-memory reservations.

This builds on image #20446 and video #20654. It does not depend on image performance #21037 or image routing #21125. Whisper disables multimodal hashing, so its conformance test does not claim UUID/cache-identity coverage.

Shared and user-visible behavior:

• Media HTTP fetching now keeps one aiohttp session per event loop. This affects trtllm-serve and every async media-loading caller, as well as OpenEngine. Live loops retain independent sessions; closed-loop entries are pruned and disposed on the next lookup. Explicit server shutdown closes the current loop's session.
• Audio decoding reserves storage before allocation and accepts short reads from estimated compressed-audio headers. It shrinks the owning buffer before committing the retained byte count.
• Processors declare supports_token_ids_input, defaulting to true. Whisper opts out. Its optional token-ID inference health probe reports that it is not applicable and that inference liveness was not tested. Ordinary lifecycle and executor health checks still apply; this is not an inference liveness guarantee.
• Gemma4 processors with missing configuration fail closed for audio discovery.

Validation

Published head b422f01:

• Upstream pipeline #66904 passed. A10-PyTorch-2 ran all five test_multimodal_conformance.py cases with no skips: two image, one video, and both real-weight Whisper transcription cases.
• A10 report: https://prod.blsm.nvidia.com/sw-tensorrt-llm-github-2/job/LLM/job/main/job/L0_Test-x86_64-Single-GPU/10089/testReport/
• Pipeline reported unrelated L40S-PyTorch-2 maintenance coverage reduction; the A10 cases above executed.

Follow-up adc04f7 (retained unchanged in main merge 16f4d69):

• 315 focused CPU tests passed with zero failures or skips, using the pipeline #66904 wheel plus the four changed Python modules. Targeted pre-commit checks and git diff --check passed.
• Added regression coverage for closed-loop session cleanup, missing Gemma4 processor configuration, the default token-ID capability, FLAC/MP3 bytes/base64/file decoding, and short-read owning-buffer accounting.
• No new GPU conformance run is claimed for the follow-up. Current-head upstream GPU CI still needs confirmation after publishing the fixes and main merge.

Related Issues

Relates to DIS-2971; audio passthrough is one part of the image/video/audio scope.

PR Checklist

• Reviewed the checklist; the change is backward compatible and includes regression coverage.

[github-actions]

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

[tensorrt-cicd]

PR_Github #81087 [ run ] completed with state SUCCESS. Commit: 6a07a16
/LLM/main/L0_MergeRequest_PR pipeline #66798 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

[krishung5]

/bot run

[tensorrt-cicd]

PR_Github #81211 [ run ] triggered by Bot. Commit: b422f01 Link to invocation

[tensorrt-cicd]

PR_Github #81211 [ run ] completed with state SUCCESS. Commit: b422f01
/LLM/main/L0_MergeRequest_PR pipeline #66904 completed with status: 'SUCCESS'

CI Report

⚠️ Resource Maintenance — Test Coverage Reduced:

Resource maintenance in LLM/main/L0_Test-x86_64-Single-GPU #10089: L40S-PyTorch-2 is temporarily paused due to Kubernetes agent provisioning and node issues.
Affected stages:
L40S-PyTorch-2

Link to invocation

[brnguyen2]

Approving — the comments below are optional touch-ups, not blockers.

Prior-round items check out in the code: Whisper declares supports_token_ids_input = False at [tensorrt_llm/_torch/models/modeling_whisper.py:603](https://github.com/NVIDIA/TensorRT-LLM/pull/20655/files#diff-86df3007091ba83b51cac2cf371b1549874ccf175ffd1987c1ea8b98502c8ae2R603), Control and Generate both go through token_ids_input_supported, the conformance request no longer carries a uuid, and the fixture candidates mirror tests/integration/defs/llmapi/test_llm_api_pytorch_whisper.py:90-109. Keeping the callable() guard with a fail-closed test is fine.

One merge condition remains, and it is the one from last round: the description still says the A10 test_multimodal_conformance.py stage has not been confirmed on the published head. Please post the green A10 run on the current head (b422f01c52 or later) before merging, since the Whisper fixture hard-fails rather than skips when the model or wav is missing.

Two behavior changes are not in the description and should be, since they touch shared or user-visible paths:

• tensorrt_llm/inputs/media_io.py now keeps one aiohttp session per event loop instead of one global session. This affects every media fetch caller, not just OpenEngine.
• Health with include_inference_probe now returns READY without running a probe when the model does not accept token-ID input. A wedged Whisper engine passes the probe. Defensible, because Whisper also rejects text-only prompts, but it belongs in the description.

Non-blocking follow-up: supports_token_ids_input is a duck-typed attribute only Whisper defines. Declaring the default True on InputProcessor in tensorrt_llm/inputs/registry.py would make it a documented contract for the next encoder-driven model.

[brnguyen2]

Approving — the comments below are optional touch-ups, not blockers.

The audio mapping, budget accounting and capability gating fit the existing image/video handler design, and the tests exercise the cleanup paths, not just the happy path.

Two items to note in the description or address:

• Shared media_io session change. The switch from one global aiohttp session to one session per event loop ([tensorrt_llm/inputs/media_io.py:144](https://github.com/NVIDIA/TensorRT-LLM/pull/20655/files#diff-511e18385bac6f8ee8cc4210aace0449fb98028c33feca4743c8ea47555487caR144)-162) also affects trtllm-serve and every async_load_* caller, not just OpenEngine. Please mention it in the description so reviewers outside gRPC see it. Also note that the old code closed the previous session when the loop changed, and the new code never does unless _close_aiohttp_session() runs on that loop (see inline).
• Health probe for prompt-only models. For Whisper, Health(include_inference_probe=True) now returns READY without touching the engine ([control.py:478](https://github.com/NVIDIA/TensorRT-LLM/pull/20655/files#diff-fea71f4b445357e541be13779ce5b1c404d44c5d0edbb3933e98766e873f8944R478)-484). That is reasonable given the probe is token-ID based, but it means a wedged Whisper engine reports healthy. If a text+audio probe isn't feasible, the probe message or docs should make clear that the probe gives no liveness signal for these models.

[krishung5]

@brnguyen2 Addressed the review follow-ups in adc04f7 and merged current main in 16f4d69. The PR description now covers per-loop sessions across shared media callers and the inference-probe limitation. InputProcessor declares the default token-ID capability, and the probe response and README explicitly state that inference liveness was not tested when the probe is inapplicable.

Confirmed all five A10 conformance cases, including both real-weight Whisper cases, passed without skips in pipeline #66904 on b422f01: https://prod.blsm.nvidia.com/sw-tensorrt-llm-github-2/job/LLM/job/main/job/L0_Test-x86_64-Single-GPU/10089/testReport/ . The follow-up passes 315 CPU tests against that pipeline's wheel, with zero skips; eight regression cases fail against the original implementation. Audio/OpenEngine files are unchanged by the main merge. Current-head upstream GPU CI still needs confirmation.

[krishung5]

/bot run

[tensorrt-cicd]

PR_Github #81281 [ run ] triggered by Bot. Commit: 16f4d69 Link to invocation

[tensorrt-cicd]

PR_Github #81281 [ run ] completed with state FAILURE. Commit: 16f4d69
/LLM/main/L0_MergeRequest_PR pipeline #66970 completed with status: 'FAILURE'

CI Report

⚠️ Resource Maintenance — Test Coverage Reduced:

Resource maintenance in LLM/main/L0_Test-x86_64-Single-GPU #4148: L40S-PyTorch-2 is temporarily paused due to Kubernetes agent provisioning and node issues.
Affected stages:
L40S-PyTorch-2

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[krishung5]

/bot run

[tensorrt-cicd]

PR_Github #81338 [ run ] triggered by Bot. Commit: 16f4d69 Link to invocation

[tensorrt-cicd]

PR_Github #81338 [ run ] completed with state FAILURE. Commit: 16f4d69
/LLM/main/L0_MergeRequest_PR pipeline #67021 completed with status: 'FAILURE'

CI Report

⚠️ Resource Maintenance — Test Coverage Reduced:

Resource maintenance in LLM/main/L0_Test-x86_64-Single-GPU #4159: L40S-PyTorch-2 is temporarily paused due to Kubernetes agent provisioning and node issues.
Affected stages:
L40S-PyTorch-2

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[krishung5]

/bot run

[tensorrt-cicd]

PR_Github #81354 [ run ] triggered by Bot. Commit: 16f4d69 Link to invocation

[tensorrt-cicd]

PR_Github #81354 [ run ] completed with state FAILURE. Commit: 16f4d69
/LLM/main/L0_MergeRequest_PR pipeline #67046 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[krishung5]

/bot run

[tensorrt-cicd]

PR_Github #81486 [ run ] triggered by Bot. Commit: 7435000 Link to invocation

[tensorrt-cicd]

PR_Github #81486 [ run ] completed with state SUCCESS. Commit: 7435000
/LLM/main/L0_MergeRequest_PR pipeline #67158 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation