Skip to content

[PyTorch] Normalize FA4 unlimited attention windows - #3570

Draft
yashaswikarnati wants to merge 1 commit into
NVIDIA:mainfrom
yashaswikarnati:codex/fa4-window-normalization
Draft

yashaswikarnati wants to merge 1 commit into
NVIDIA:mainfrom
yashaswikarnati:codex/fa4-window-normalization

Conversation

@yashaswikarnati

@yashaswikarnati yashaswikarnati commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Transformer Engine represents an unlimited attention window with -1. FlashAttention 4 expects None; passing (-1, 0) for causal attention can select an empty local window. Normalize the tuple for non-CP attention and the left/right kwargs for CP p2p, all-gather, and a2a forward and backward calls.

Tests

  • Added a unit test for the sentinel conversion, including None and finite windows.
  • Added a BSHD/THD causal forward and backward comparison against PyTorch SDPA while forcing FA4.
  • AGA 905211 passed the helper unit test against the matching internal TE extension.
  • AGA 906233 passed head-256 causal BSHD and THD FA4 b32 forward/backward comparisons with PyTorch SDPA using the cherry-picked TE commit in a full NGC-base image.
  • Python compilation, Ruff, git diff --check, DCO, pre-commit, and all reported GitHub CI checks passed. A separate head-128 CP suite case (AGA 906243) was stopped after 45 minutes of compilation without a test result; the target head-256 CP32 training path is running as AGA 906508.

The internal nemotron4 port is one cherry-picked commit on top of 1ac66e5a, proposed in draft MR !4.

Translate Transformer's -1 unlimited-window sentinel to FA4's None convention for direct and context-parallel calls. Cover the non-CP tuple argument and raw CP window kwargs with focused tests.

Signed-off-by: ykarnati <ykarnati@nvidia.com>
@github-actions github-actions Bot added the community-contribution PRs from external contributor outside the core maintainers, representing community-driven work. label Sep 28, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

attention community-contribution PRs from external contributor outside the core maintainers, representing community-driven work.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants