Repository navigation
[https://nvbugs/6919405][fix] Cherry-pick #20252 to release/1.3: keep the 26.08 CUDA forward-compat driver on the 26.09 image - #20397
Conversation
…er on the 26.09 image The NGC PyTorch 26.09 base ships CUDA forward-compatibility driver 615.71.09 (26.08 shipped 615.65.02). Under it the unchanged trtllm-gen context FMHA cubin runs 7-13% slower per call on GB200, which costs aggr-ctx_only-gb200_deepseek-r1-fp4_128k8k_con1_ctx1_pp8_gen1_tep8_eplb0_mtp3_ccb-NIXL ~6% Total Token Throughput since NVIDIA#19679. Add a CUDA_COMPAT_TAG build arg (default 26.08-py3) and replace the base image's /usr/local/cuda-13.4/compat driver libraries with that tag's during the devel stage. The CUDA toolkit is untouched, and the base image's shell init still validates and selects the compat driver at container start. Targets built on nvcr.io/nvidia/cuda keep their own compat driver. Signed-off-by: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com> (cherry picked from commit 8085e85)
|
/bot run --stage-list "Build-Docker-Images" --disable-fail-fast |
|
PR_Github #78050 [ run ] triggered by Bot. Commit: |
|
Important Review skippedReview was skipped as selected files did not have any reviewable changes. ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
WalkthroughThe Docker build can use compatibility-driver directories from a selected CUDA image. It skips installation when the compatibility image tag matches the base image tag. Make targets set and pass the compatibility image tag. Five Docker image tag suffixes are also updated. ChangesCUDA compatibility driver installation
Priority: ⬆️ High Estimated code review effort: 3 (Moderate) | ~20 minutes Change: Bug fix Sequence Diagram(s)sequenceDiagram
participant DockerBuild
participant CompatImage
participant InstallScript
DockerBuild->>CompatImage: Mount /usr/local when tags differ
DockerBuild->>InstallScript: Pass mounted /usr/local
InstallScript->>DockerBuild: Replace compatibility-driver directories
Suggested reviewers: Merge Risk: ⚪ Minimal · up to
|
| Check name | Status | Explanation |
|---|---|---|
| Docstring Coverage |  |
No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1… |
| Linked Issues check | |
Check skipped because no linked issues were found for this pull request. |
| Out of Scope Changes check | |
Check skipped because no linked issues were found for this pull request. |
| Title check | |
The title follows the required ticket and type format and clearly identifies the CUDA compatibility-driver fix for the 26.09 image. |
| Description check | |
The description includes the required Description, Test Coverage, and PR Checklist sections. It explains the problem, solution, affected files, validation results, image provenance, and remaining CI w… |
✨ Finishing Touches
-
🧪 Generate unit tests (beta)
-
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
|
/bot run --stage-list "Build-Docker-Images" --disable-fail-fast |
|
PR_Github #78077 [ run ] triggered by Bot. Commit: |
|
PR_Github #78050 [ run ] completed with state |
…8 compat driver Point jenkins/current_image_tags.properties at the CI images built from this branch's Dockerfile change (commit 8e14707), retagged into the tensorrt-llm repository with scripts/rename_docker_images.py run against release/1.3's copy of the tags file. The SBSA tritondevel image comes from L0 build 246 and the other four from L0 build 251; each run pushed only part of the set. Signed-off-by: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com>
|
/bot run --disable-fail-fast |
|
PR_Github #78109 [ run ] triggered by Bot. Commit: |
|
PR_Github #78077 [ run ] completed with state |
|
PR_Github #78109 [ run ] completed with state
|
|
Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements. |
…lease-1.3 Signed-off-by: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com>
|
/bot run --disable-fail-fast |
|
PR_Github #78629 [ run ] triggered by Bot. Commit: |
|
/bot run --disable-fail-fast |
|
PR_Github #78629 [ run ] completed with state
|
|
PR_Github #78645 [ run ] triggered by Bot. Commit: |
|
PR_Github #78645 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #78650 [ run ] triggered by Bot. Commit: |
|
PR_Github #78650 [ run ] completed with state |
Cherry-pick #20252 (commit 8085e85) to release/1.3.
Description
#20146 brought the NGC PyTorch 26.09 base image to release/1.3. That image bundles CUDA forward-compatibility driver
libcuda.so.615.71.09(26.08 had615.65.02), and every process in the container loads it ahead of the host driver. Under 615.71.09 the trtllm-gen context FMHA cubin, which is unchanged, runs 7–13% slower per call on GB200 (https://nvbugs/6919405).release/1.3 now shows the same step as main.
ctx_only-gb200_deepseek-r1-fp4_128k8k_con1_ctx1_pp8_gen1_tep8_eplb0_mtp3_ccb-NIXL, post-merge Total Token Throughput:cd4cc1778f, first build with #20146; flagged as a regression)The change keeps the 26.09 base image and its CUDA 13.4 toolkit, and swaps only the compat driver libraries for the 26.08 ones:
docker/Dockerfile.multi: newCUDA_COMPAT_TAGbuild arg (default26.08-py3) and acuda_compatstage from${BASE_IMAGE}:${CUDA_COMPAT_TAG}. The devel stage runsinstall_cuda_compat.shfrom that stage's/usr/local, unlessCUDA_COMPAT_TAG == BASE_TAG.docker/common/install_cuda_compat.sh(new): replaceslib.real(andlib.real-nvgpuwhere present) under/usr/local/cuda-<ver>/compat, drops the per-host.checkedmarkers recorded against the old driver, then runsldconfigand checks the result. It refuses to run if the source image does not ship the same CUDAmajor.minor.docker/Makefile: passesCUDA_COMPAT_TAG. Therockylinux8/ubuntu22/ubuntu24targets, which build onnvcr.io/nvidia/cuda, set it to their ownBASE_TAGand so keep their compat driver.jenkins/current_image_tags.properties: points CI at the images built from this change (see Test Coverage).The cherry-pick applied cleanly. The release/1.3-only edits in these files (
NCCL_GIN_GPUNETIO_PATH, theurm.nvidia.compip mirror,CONAN_DIR) don't overlap it.Test Coverage
Measured on main's CI wheels on GB200 (oci-hsg); full evidence is in #20252 and on the NVBug. The decisive arm ran the regressed commit's wheel in the 26.09 container with only
/usr/local/cuda-13.4/compattaken from 26.08. Workers were verified to maplibcuda.so.615.65.02, and the run measured 5703.03 / 5704.28 tok/s against 5346.08 as shipped. That fully recovers the pre-regression 5682.42.I ran
install_cuda_compat.shinside the published 26.09 TRT-LLM image: the NGC shell-init compat check reportedCUDA Driver OKand torch mappedlibcuda.so.615.65.02. release/1.3 runs the same 26.09 base (pytorch-26.09-py3-…-202610130623-20146), same CUDA 13.4.1, same host driver.Images for this branch, built from commit 8e14707 and retagged into
sw-tensorrt-llm-docker-local/tensorrt-llmas…-202610150955-20397:scripts/rename_docker_images.py, run against release/1.3's copy ofjenkins/current_image_tags.properties. Notensorrt-llm-stagingtags are in the PR.Build-Docker-Imagesruns pushed only part of the CI image set; the missing images hitdocker logintimeouts to artifactory. So the SBSA tritondevel image comes from L0 Performance differs on different V100 #246, and the x86_64 tritondevel, SBSA ubuntu24 py312 and both Rocky 8 images come from L0 How can I use flan-t5-base? #251.install_cuda_compat.shlayer (BASE_TAG=26.09-py3 CUDA_COMPAT_TAG=26.08-py3). Thenvcr.io/nvidia/cuda-based images keep their own driver./opt/tensorrt_llm/bolt/profiles. No CI stage reads that directory.Still to do: CI on these images, then re-measure the case above on release/1.3.
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.🤖 Generated with Claude Code
Dev Engineer Review
CUDA_COMPAT_TAGdefaults to26.08-py3. Thedevelstage installs compatibility-driver libraries from that image when the tag differs fromBASE_TAG; it skips installation when the tags match.CUDA_COMPAT_TAGto builds. Rocky Linux 8 and Ubuntu 22/24 targets set it toBASE_TAG, preserving their configured compatibility drivers.install_cuda_compat.shreplaces compatibility-driver directories from the source/usr/localtree. It checks for driver libraries, updates linker configuration, and checks for expected libraries.QA Engineer Review
No test changes.
Per-File QA Perspective
docker/Dockerfile.multi: Verify that differing tags install compatibility-driver libraries and matching tags skip installation.docker/Makefile: Verify that the intended Rocky Linux 8 and Ubuntu targets pass theirBASE_TAG, and that other Docker builds remain unaffected.docker/common/install_cuda_compat.sh: Verify missing-library error paths, handling of optionallib.real-nvgpu, and linker updates in supported target images.jenkins/current_image_tags.properties: Verify that the updated image tags resolve to the intended CI-built images for each listed platform.