fix(hlo): SDPA explicit mask under GQA and onto a dynamic key length - #1303
Conversation
AttentionOperationsConverter's explicit-mask path emitted invalid StableHLO in two cases (#1302): - A per-head mask [b, H, Sq, Sk] under grouped-query attention was broadcast with dims = [0, 1, 3, 4] onto the grouped scores [b, nKV, nRep, Sq, Sk], mapping H onto nKV. It is now viewed as [b, nKV, nRep, Sq, Sk] by a reshape in the h = kv * nRep + r order Q already uses, or a dynamic_reshape when a dim is dynamic. A mask head dim other than 1, nKV or H is rejected. - A mask whose shape differs from a dynamic scores shape got a static broadcast_in_dim with a dynamic result type, with or without GQA. It now uses dynamic_broadcast_in_dim with the scores-shape operand the softmax builds (hoisted so both use it), stating known_expanding_dimensions / known_nonexpanding_dimensions. Without the hints a dynamic operand dim onto a dynamic result dim is ambiguous and IREE 3.11 fails to legalize it. The op is written in generic syntax because the hints are not in every StableHLO version's pretty form. Static graphs emit exactly what they did before. Tests: four new SdpaGqaHloExportTest cases (per-head static, head-shared dynamic, per-head dynamic, non-GQA dynamic) fail on develop and pass here; :skainet-compile:skainet-compile-hlo:jvmTest 240/240. The emitted modules were compiled with iree-compile 3.11.0: the static per-head, dynamic head-shared GQA and dynamic head-shared MHA cases compile for vulkan-spirv (valhall4) and llvm-cpu (arm32). The dynamic per-head case is valid StableHLO but IREE 3.11 does not lower dynamic_reshape; that limitation is noted in the changelog. Closes #1302
Compatibility checkskainet-iree-conformance. I ran
Byte-identical output for every SDPA shape the suite uses. The converter emits the same bytes on
Existing dynamic-shape conventions are unaffected. Downstream KV-cache exports already rely on the dynamic-safe tracer and emitter from 0.38 (#891). The ones with a dynamic key length avoid this bug in one of two ways:
This PR changes only the two cases that were invalid IR before: a per-head mask under GQA, and a mask that differs from a dynamic scores shape. |
Closes #1302.
What
AttentionOperationsConverter's explicit-mask path emitted invalid StableHLO:[b, H, Sq, Sk],nKV < H): wasbroadcast_in_dim dims = [0, 1, 3, 4]onto[b, nKV, nRep, Sq, Sk], mappingHontonKV. Now reshaped into[nKV, nRep]in theh = kv * nRep + rorder Q uses (dynamic_reshapewhen a dim is dynamic). A mask head dim other than 1,nKVorHis rejected with a clear message.?key length of KV-cache chunk graphs), with or without GQA: was a staticbroadcast_in_dimto a dynamic type. Nowdynamic_broadcast_in_dimwith the scores-shape operand the softmax already builds (hoisted so both share it), plusknown_expanding_dimensions/known_nonexpanding_dimensions. IREE 3.11 needs those hints to lower it. Emitted in generic op syntax, since the hints are not in every StableHLO version's pretty form.Static graphs emit exactly what they did before.
Verification
SdpaGqaHloExportTestcases fail ondevelopand pass here.:skainet-compile:skainet-compile-hlo:jvmTestpasses 240/240.iree-compile3.11.0:dynamic_reshapeThe last row is an IREE limitation. It is noted in the CHANGELOG with the recommendation to use a head-shared mask when the key length is dynamic.