[AMD] Tune MiniMax-M3 MXFP8 MI300X vLLM: async scheduling + big-prefill, fix conc256 EP8→EP1#1950
Closed
ZhengGong-amd wants to merge 5 commits into
Closed
Conversation
Enable AITER on MI300X/gfx942 for MiniMax-M3 MXFP8 via the single master toggle VLLM_ROCM_USE_AITER=1. The per-component AITER flags (_MOE, _LINEAR, _RMSNORM, _FP8BMM) default to True and are gated behind the master flag, so they are left at their defaults. VLLM_ROCM_USE_AITER_MHA defaults to True and is explicitly set to 0 to keep attention on TRITON_ATTN, since the MXFP8 checkpoint lacks calibrated q/prob scales for ROCm FP8 attention. Also set AMD-recommended numerically-inert MI300X runtime knobs: TORCH_BLAS_PREFER_HIPBLASLT=1, NCCL_MIN_NCHANNELS=112 (RCCL channels, raised above the ~32-64 default for TP8), GPU_MAX_HW_QUEUES=2 (HIP streams, capped below the default of 4). All changes are kernel-selection/runtime only; GSM8K holds ~0.95. Measured uplift (8xMI300X, 1k1k, total tok/s/gpu): +5.6..+10.8% across conc 4..256; conc 1-2 unchanged (latency-bound). Co-authored-by: Cursor <cursoragent@cursor.com>
Sync the branch with the latest upstream main (fork main force-synced to upstream). Resolve the perf-changelog.yaml conflict by taking main's version and re-appending the branch's own minimaxm3-fp8-mi300x-vllm AITER entry at the tail. The AITER target benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi300x.sh auto-merged cleanly (main's SemiAnalysisAI#1837 image/FP8-KV change was reverted by SemiAnalysisAI#1857, so main's net change to that file is zero); the AITER env exports are preserved. Co-authored-by: Cursor <cursoragent@cursor.com>
…P8->EP1 Stack the accuracy-safe scheduling levers found across the arbor tuning sessions on top of the AITER MI300X recipe: - --async-scheduling (overlap CPU input-prep with GPU decode) - --max-num-batched-tokens 16384 (amortize the per-step ~95 GB/rank BF16-emulated MoE weight read; halves prefill weight-reads vs the 8192 default) - amd-master.yaml: switch the 1k1k conc256 row from TP8/EP8 to TP8/EP1; the EP8 topology regressed high-concurrency throughput (434 vs 905 tok/s/gpu @ conc256) and EP1 matches the topology the AITER uplift was measured against. Both serve flags are token-for-token identical (scheduling only). Measured on 8xMI300X 1k1k vs the AITER baseline (total tok/s/gpu): conc256 434->905 (EP8->EP1 + levers, +108%), conc64 364->429 (+18%), conc128 585->628 (+7.3%); conc1-32 neutral. GSM8K exact-match 0.959. Co-authored-by: Cursor <cursoragent@cursor.com>
Condense the recipe header note and the amd-master.yaml search-space comment introduced in the previous commit; rationale/measurements live in the perf-changelog entry. Co-authored-by: Cursor <cursoragent@cursor.com>
ZhengGong-amd
requested review from
1am9trash,
billishyahao,
chunfangamd,
seungrokj and
yctseng0211
as code owners
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
…mbo-tuning # Conflicts: # perf-changelog.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacks two accuracy-safe scheduling levers on the
minimaxm3-fp8-mi300x-vllmrecipe and corrects a high-concurrency parallelism regression in its search space:benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi300x.sh— add two server flags:--async-scheduling— overlaps CPU input-prep with GPU decode (token-for-token identical).--max-num-batched-tokens 16384— amortizes the per-step ~95 GB/rank BF16-emulated MoE weight read (gfx942 has no native MXFP8 MoE GEMM, so MXFP8 experts are dequantized to BF16 and re-read every prefill step) over a larger prefill batch, halving prefill weight-reads vs the vLLM 8192 default..github/configs/amd-master.yaml— switch the 1k1kconc256search-space row fromTP8/EP8toTP8/EP1. The EP8 topology regressed high-concurrency throughput, and EP1 is the topology the prior AITER uplift was measured against.perf-changelog.yaml— append the required trigger entry forminimaxm3-fp8-mi300x-vllm.Both serve flags are scheduling-only (no numerics, no quantization, no reduction reassociation) and accuracy-safe. No image change; serving shape (block-size 128, BF16 KV,
TRITON_ATTN, MiniMax-M3 parsers, TP8) is otherwise unchanged. The MTP variant and the 8k1k EP8 rows are left untouched.Why
On the 1024/1024 closed-loop sweep the recipe was prefill-bound at high concurrency: synchronized
--ignore-eosprefill waves stole wall-clock from decode, and the default 8192 token budget needed multiple full MoE weight-reads to clear each wave. Raising the prefill budget plus async scheduling lifts the decode duty cycle. Separately, theconc256EP8 row was a regression — EP8 underperformed EP1 by >2× at conc256 on this shape.Validation
Measured on 8× MI300X (gfx942), TP8, random 1024-in/1024-out,
--request-rate inf --ignore-eos, vs the AITER baseline (minimaxm3-mi300x-aiter-tuning) measured back-to-back on the same host (total tok/s/gpu):bash -non the recipe — OK.generate_sweep_configs.py full-sweep --model-prefix minimaxm3 --framework vllm --runner-type mi300x— generates the 1k1k EP1 sweep (conc256 now EP1); no schema errors.python3 utils/validate_perf_changelog.py— final newline present, matrix generated.python -m pytest utils/matrix_logic/ -q— passing.PR Review Checklist
As a PR author, I have:
minimaxm3-fp8-mi300x-vllm) already exists; this PR only tunes it.--async-schedulingand--max-num-batched-tokensare upstream vLLM V1 scheduler flags; no custom code.Additional Details
minimaxm3-mi300x-aiter-tuning(the AITER-kernels enablement). If that PR has not merged, this PR's diff includes its commits; rebase/retarget once it lands.conc256+108%figure combines the EP8→EP1 topology fix with the two scheduling levers; theconc64/conc128gains (+18%/+7.3%) are pure scheduling-lever effects at fixed EP1.conc128/256EP8 rows are intentionally left unchanged pending 8k1k data.pr-link: .../pull/PENDINGplaceholders inperf-changelog.yamlwith this PR's URL after opening.