Fix linear bias broadcasting in the Dynamo converter#4393
Open
fs-eire wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This change fixes linear bias broadcasting for higher-rank inputs in the Dynamo converter.
Previously, the
aten.linearconverter passed a rank-1 bias directly to the generic elementwise-add converter. For static shapes, that path broadcast the bias to the full output shape—for example,[O]to[B, S, O]—before creating the TensorRT elementwise add.This change rank-aligns the operands first, preserving singleton dimensions—for example,
[O]to[1, 1, O]for a[B, S, O]linear output—and lets TensorRT perform the elementwise broadcast. This avoids unnecessary bias materialization and gives TensorRT/Myelin more opportunities to canonicalize and fuse related linear operations.The issue was identified while comparing ONNX-TensorRT and Torch-TensorRT AOT plans for the
black-forest-labs/FLUX.1-schnelltext encoder. Before this change, the three Q/K/V projections were combined into a token-major[1, 77, 2304]output, with their biases and layout transformations handled downstream. This produced three additionalAddReshTranReshMovematerializations per transformer block.With this change, TensorRT recognizes a projection-major QKV operation with:
[3, 768, 768][3, 1, 768][3, 77, 768]For the 12-block FLUX text encoder, this eliminates all 36 QKV
AddReshTranReshMovematerializations. In an isolated patch-only benchmark on B200 with TensorRT 11.3.0.1, BF16, and batch size 1, Torch-TensorRT AOT latency improved from 0.5284 ms to 0.5144 ms.A rank-3
aten.lineartest with bias is included to cover the corrected broadcasting behavior.No new dependencies are required, and there are no public API changes.
Additional attention-layout and mask optimizations are separate issues and are intentionally outside the scope of this change.
Type of change
Checklist: