Skip to content

fix(hlo): SDPA explicit mask under GQA and onto a dynamic key length - #1303

Merged
michalharakal merged 1 commit into
developfrom
fix/sdpa-gqa-mask-broadcast
Sep 24, 2026
Merged

michalharakal merged 1 commit into
developfrom
fix/sdpa-gqa-mask-broadcast

Conversation

@michalharakal

Copy link
Copy Markdown
Contributor

Closes #1302.

What

AttentionOperationsConverter's explicit-mask path emitted invalid StableHLO:

  • Per-head mask under GQA ([b, H, Sq, Sk], nKV < H): was broadcast_in_dim dims = [0, 1, 3, 4] onto [b, nKV, nRep, Sq, Sk], mapping H onto nKV. Now reshaped into [nKV, nRep] in the h = kv * nRep + r order Q uses (dynamic_reshape when a dim is dynamic). A mask head dim other than 1, nKV or H is rejected with a clear message.
  • Mask onto a dynamic scores shape (the ? key length of KV-cache chunk graphs), with or without GQA: was a static broadcast_in_dim to a dynamic type. Now dynamic_broadcast_in_dim with the scores-shape operand the softmax already builds (hoisted so both share it), plus known_expanding_dimensions / known_nonexpanding_dimensions. IREE 3.11 needs those hints to lower it. Emitted in generic op syntax, since the hints are not in every StableHLO version's pretty form.

Static graphs emit exactly what they did before.

Verification

  • Four new SdpaGqaHloExportTest cases fail on develop and pass here. :skainet-compile:skainet-compile-hlo:jvmTest passes 240/240.
  • The emitted modules were compiled with iree-compile 3.11.0:
case vulkan-spirv valhall4 llvm-cpu arm32
GQA, per-head mask, static compiles compiles
GQA, head-shared mask, dynamic key length compiles compiles
MHA, head-shared mask, dynamic key length compiles compiles
GQA, per-head mask, dynamic key length valid StableHLO, IREE 3.11 does not lower dynamic_reshape same

The last row is an IREE limitation. It is noted in the CHANGELOG with the recommendation to use a head-shared mask when the key length is dynamic.

AttentionOperationsConverter's explicit-mask path emitted invalid StableHLO in two cases (#1302):

- A per-head mask [b, H, Sq, Sk] under grouped-query attention was broadcast with
  dims = [0, 1, 3, 4] onto the grouped scores [b, nKV, nRep, Sq, Sk], mapping H onto nKV. It is
  now viewed as [b, nKV, nRep, Sq, Sk] by a reshape in the h = kv * nRep + r order Q already uses,
  or a dynamic_reshape when a dim is dynamic. A mask head dim other than 1, nKV or H is rejected.
- A mask whose shape differs from a dynamic scores shape got a static broadcast_in_dim with a
  dynamic result type, with or without GQA. It now uses dynamic_broadcast_in_dim with the
  scores-shape operand the softmax builds (hoisted so both use it), stating
  known_expanding_dimensions / known_nonexpanding_dimensions. Without the hints a dynamic operand
  dim onto a dynamic result dim is ambiguous and IREE 3.11 fails to legalize it. The op is written
  in generic syntax because the hints are not in every StableHLO version's pretty form.

Static graphs emit exactly what they did before.

Tests: four new SdpaGqaHloExportTest cases (per-head static, head-shared dynamic, per-head dynamic,
non-GQA dynamic) fail on develop and pass here; :skainet-compile:skainet-compile-hlo:jvmTest
240/240. The emitted modules were compiled with iree-compile 3.11.0: the static per-head, dynamic
head-shared GQA and dynamic head-shared MHA cases compile for vulkan-spirv (valhall4) and llvm-cpu
(arm32). The dynamic per-head case is valid StableHLO but IREE 3.11 does not lower
dynamic_reshape; that limitation is noted in the changelog.

Closes #1302
@michalharakal

Copy link
Copy Markdown
Contributor Author

Compatibility check

skainet-iree-conformance. I ran scripts/run-conformance.sh with USE_SKAINET_SOURCE=1 and core built from this branch (4e586b24). skainet-compile-hlo resolved to the source project. The only change in the throwaway worktree was the Gradle wrapper, bumped to 9.7.1 because core needs 9.6 or newer.

  • 12/12 models pass export, compile, run and validate on IREE 3.11.0 llvm-cpu. That includes tiny-transformer (both oracles), rope-attention and gemma-decoder, the multi-query model with 2 query heads and 1 K/V head.
  • 33/33 ops compile, run and validate, including sdpa.
  • The gate passes.

Byte-identical output for every SDPA shape the suite uses. The converter emits the same bytes on develop and on this branch for these configurations:

config q k/v mask
ops/sdpa [1,2,4,8] [1,2,4,8] [1,2,4,4]
functiongemma-270m scaffold [1,1,4,64] [1,1,4,64] [1,1,4,4], causal
gemma-decoder, causal path [1,2,4,32] [1,1,4,32] none
gemma-decoder, sliding-mask path [1,2,4,32] [1,1,4,32] [1,1,4,4]
tiny-transformer / rope-attention [1,2,4,4] [1,2,4,4] none, causal

Existing dynamic-shape conventions are unaffected. Downstream KV-cache exports already rely on the dynamic-safe tracer and emitter from 0.38 (#891). The ones with a dynamic key length avoid this bug in one of two ways:

  • The chunk mask is exactly the scores shape, with K/V expanded to the query heads. The converter adds it directly, and this PR keeps that maskShape == scoresShape branch unchanged.
  • The masked attention is static, and only the unmasked self-attention of the decode step is dynamic.

This PR changes only the two cases that were invalid IR before: a per-head mask under GQA, and a mask that differs from a dynamic scores shape.

@michalharakal
michalharakal merged commit 9a3a755 into develop Sep 24, 2026
14 checks passed
@michalharakal
michalharakal deleted the fix/sdpa-gqa-mask-broadcast branch September 24, 2026 10:53
@michalharakal michalharakal mentioned this pull request Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SDPA StableHLO export: explicit mask is broadcast invalidly under GQA and onto a dynamic key length

2 participants