Add DSV4 GB300 1k1k STP disagg configs#1530
Conversation
|
Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers. If additional help is needed, PR authors can reach out to core maintainers over Slack. |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26176031180 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26176611225 |
2 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26176611225 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26176611225 |
yhyang201
force-pushed
the
add-dsv4-gb300-1k1k-stp-disagg-configs
branch
from
dbef1ac to
4f923af
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26302711476 |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
|
|
||
| name: "disagg-2p1d-dep12-conc18432" | ||
| slurm: | ||
| time_limit: 03:00:00 |
There was a problem hiding this comment.
Unquoted Slurm time limit
Medium Severity
slurm.time_limit is set as an unquoted 03:00:00, while every other new dynamo recipe in this change uses a quoted "03:00:00". YAML 1.1 parsers often coerce colon-separated values to sexagesimal integers (e.g. 10800), not an HH:MM:SS string, which can yield the wrong Slurm wall-clock limit or type errors downstream.
Reviewed by Cursor Bugbot for commit 1aaa3cc. Configure here.
yhyang201
force-pushed
the
add-dsv4-gb300-1k1k-stp-disagg-configs
branch
from
1aaa3cc to
e208645
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26974607644 |
yhyang201
force-pushed
the
add-dsv4-gb300-1k1k-stp-disagg-configs
branch
from
e208645 to
db553d8
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26974737242 |
yhyang201
force-pushed
the
add-dsv4-gb300-1k1k-stp-disagg-configs
branch
2 times, most recently
from
7f9d53a to
4abc217
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27957210028 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27970988080 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27978682829 |
yhyang201
force-pushed
the
add-dsv4-gb300-1k1k-stp-disagg-configs
branch
from
d10a7f4 to
f7f3601
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28015711920 |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 4 total unresolved issues (including 3 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 0c02a51. Configure here.
| max-running-requests: [128, 128, 128, 128] | ||
| cuda-graph-max-bs: [128, 128, 128, 128] | ||
| benchmark: | ||
| concurrencies: ["1x8x32x64x128", "1x8x32x64x128x256", "1x8x32x64x128x256x512", "1x8x32x64x128x256x512x1024"] |
There was a problem hiding this comment.
Low-latency recipes not wired
Medium Severity
The PR adds disagg-low-latency-dep4.yaml and disagg-low-latency-tp4.yaml (zip-override sweeps), but nvidia-master.yaml only registers seven dsv4-fp4-gb300-dynamo-sglang-1k1k-stp search-space entries. Nothing references these recipe files, so the low-latency dep4/tp4 configs described in the PR are not runnable from the benchmark matrix.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 0c02a51. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28015996913 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28017719329 |
…o source install
The five dynamo.hash recipes build dynamo from source, whose install runs
`apt-get install -y libclang-dev` (to maturin-build the Rust bindings).
Backend prefill/decode workers run the container as the unprivileged host
user, so dpkg refuses ("requested operation requires superuser privilege")
and apt-get exits 100 — which the install hides behind `> /dev/null 2>&1`,
leaving only apt-get update keyring warnings + exit 100 in the worker log
(root cause of the all-red multi-node 1k1k sweep).
Add top-level `srun_options: {container-remap-root: ""}` so workers run as
remapped root (pyxis maps container-root back to the calling user on the
host, so files on mounted dirs stay caller-owned). The dynamo frontend
already runs with this flag; this brings the workers in line.
The 1p4d/1p6d/low-latency recipes don't source-build (no dynamo.hash), so
they're left unchanged.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28017719329 |
… OOM on weight load A decode rank hung at "Load weight begin" and never finished, tripping the cross-rank load barrier (scheduler SIGQUIT, exit -3) and failing the job — a host-memory stall during the DSV4-Pro shard load, not GPU OOM (15/16 ranks loaded fine with ~194 GB GPU free). The recipes set job-level `sbatch --mem=0` but not the step-level `srun --mem=0`, so each worker srun step runs under a constrained memory cgroup that the job-level setting doesn't lift. The minimax GB300 recipes carry `srun_options: mem: "0"` for exactly this reason. Add it to the five source-install recipes (alongside the existing container-remap-root).
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28075373686 |
…ecipes Extend the host-RAM fix to the 1p4d/1p6d/low-latency recipes, which had no sbatch_directives or srun_options at all (not even job-level mem=0). They load the same DSV4-Pro weights on GB300, so they're equally exposed to the per-step memory cgroup stall/OOM seen on the DEP recipes (the 1p4d conc-64 "not healthy in 1800s" timeout is consistent with this). Add job-level `sbatch --mem=0` + `cpus-per-task: 144` and step-level `srun --mem=0`, matching the minimax GB300 recipes and the other dsv4 configs. For the base:-templated low-latency recipes the directives go under `base:` so they apply to all zip-expanded variants. No container-remap-root here: these use the PyPI dynamo install (no source build), so they don't need root.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28079770703 |
…ti-node load deadlock Root cause (captured live with py-spy on a hung rank): during decode-engine init, one rank's weight load wedges — every ThreadPoolExecutor worker stuck in fused_moe_triton/layer.py _load_w13/_load_w2, main thread blocked forever in concurrent.futures.as_completed. The other ranks finish loading and time out at the post-load torch monitored_barrier, surfacing as "Rank 0 scheduler died during initialization (exit code -3)" and an all-red multi-node sweep. Ruled out (SSH/node forensics): disk I/O (17 GB/s RAID0, no D-state), host memory (>200 GB free), GPU (0 ECC, no throttle), bad nodes (different node each run). deepseek_weight_loader.do_load_weights fans expert-weight H2D copies across an unbounded ThreadPoolExecutor (up to 32 threads on 144-CPU nodes); the deadlock is in concurrent CUDA H2D below the Python layer (no SGLang lock, no env/arg knob). Until the upstream threaded-loader race is fixed, serialize: a setup_script (run in each worker container before engine start, same mechanism as minimax-m3) patches should_async_load() -> False so loads run inline. Validated in-container: patch applies, should_async_load(cpu)=False.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28100157389 |
2 participants
Summary
Port 9 non-MTP disagg configs from NVIDIA/srt-slurm#161:
Reference: NVIDIA/srt-slurm#161
Note
Low Risk
Benchmark and launcher wiring only (YAML recipes and GB300 Slurm overlay); no changes to inference runtime, auth, or data paths.
Overview
Adds DeepSeek-V4-Pro GB300 1k/1k STP (non-MTP) disaggregated benchmark coverage ported from NVIDIA/srt-slurm#161.
Registers a new master config key
dsv4-fp4-gb300-dynamo-sglang-1k1k-stpwith seven fixed-seq-len scenarios (1P1D dep8/dep16, 1P4D/1P6D TP4 decode, 2P1D dep12/dep16/dep48), each pointing at localrecipes/sglang/deepseek-v4/1k1k/*.yamlfiles. Recipes mix Dynamo + megamoe high-concurrency shapes (custom InferenceX conc sweeps) and SGLang-native low-latency DEP4/TP4 layouts (sa-bench or zip-override sweeps).launch_gb300-nv.shnow overlaysbenchmarks/multi_node/srt-slurm-recipes/sglang/deepseek-v4into srt-slurm whendynamo-sglang+dsv4, and extendssrtslurm.yamlwithdsv4-promodel path and extra container squash aliases.perf-changelog.yamldocuments the new config key.Reviewed by Cursor Bugbot for commit 33bd556. Bugbot is set up for automated code reviews on this repo. Configure here.