Repository navigation
Add RTX 6000D (sm120) variants for five models - #20
Merged
Merged
Conversation
Every variant published so far targets H100, B300 or H800, and every one of
them is single-node. A cluster of eight-GPU RTX 6000D nodes therefore has no
entry it can deploy: preflight matches requires.gpuProduct against the node
label, and four of these five models do not fit on one node at all.
Three models already had an entry and gain a variant, in a new version file --
a published version is immutable, so its existing variants are carried forward
unchanged:
kimi-k3 1.0.3 vllm-tp8-pp4-6000d TP8 x PP4 over four nodes
glm5.3 1.0.2 vllm-tp8-pp2-6000d TP8 x PP2 over two nodes
deepseek-v4-flash 1.0.1 vllm-tp8-6000d TP8 on one node
Two had none at all:
glm5.2 1.0.0 vllm-tp8-pp2-6000d TP8 x PP2 over two nodes
deepseek-v4-pro 1.0.0 vllm-tp8-pp2-6000d TP8 x PP2 over two nodes
All five are vLLM. sglang cannot serve these weights on sm120: the GLM pair
fails in the weight loader, and for Kimi K3 the default MoE runner upcasts the
4-bit experts to bf16 and will not fit, while flashinfer_mxfp4 loads and serves
but computes wrong answers silently. vLLM's Marlin path keeps them 4-bit.
Values are transcribed from running deployments. Three details are load-bearing
and easy to lose:
- hostNetwork on every multi-node variant. RoCE GIDs belong to the host
netdev and a pod netns does not see them; without it NCCL finds no usable
device and the group hangs at rendezvous rather than failing.
- --no-enable-flashinfer-autotune wherever PP > 1.
- gpu-memory-utilization 0.85 for the GLM pair, not 0.90. Their KV pool
saturates at 24 concurrent requests and TTFT goes from 12s to 67s past
that, so a larger allocator buys nothing.
Per-cluster wiring is deliberately left out: cart, modelRoute, scaler,
serviceMonitor, and tolerations, the last because the charts have added the GPU
toleration by default since vllm-0.7.6. model.gpus is omitted too, since the
schema renders it from requires.gpus.
Both said zai-org. The NVFP4 builds are quantized with modelopt and published under nvidia: zai-org/GLM-5.3-NVFP4 does not exist, and neither did the GLM-5.2-NVFP4 id added alongside it in this branch. The mistake is easy to make and hard to catch, because huggingface.co answers 401 for a gated repo and for one that does not exist alike -- API and HTML page both -- so fetching an id and reading the status cannot tell the two apart. The search endpoint does work unauthenticated and enumerates real repos, which is what these two were confirmed against. Two more ids in the catalog do not resolve either, in models this branch does not touch, so they are left alone rather than guessed at: glm5.1 modelforge/Qwen3.6-35B-A3B-793303-glm-5 gpt-oss-120b openai-mirror/gpt-oss-120b
huskcass
reviewed
Oct 8, 2026
huskcass
left a comment
Collaborator
There was a problem hiding this comment.
helm values render check ✅
Validated every variant's values: with helm template against the sglang chart — both the pinned version and the latest published (0.8.10). All variants render cleanly. (Checked with the deploy-time modelRoute.nginx.outputConfigMap set, which swiss injects since the catalog schema rejects per-deploy values by design.)
chart.version already uses a semver range — no pinning to flag. 👍
servedName was DeepSeek-V4-Flash. Every entry in this catalog that was added with a lowercase served name keeps one -- kimi-k3, kimi-k2.5, glm-5.3 -- while the DeepSeek and most other families carry mixed case, so there is no single convention to follow and lowercase is the one being settled on. Here it also makes the field agree with `name`, which is what the schema says servedName defaults to, so 1.0.1 no longer advertises a name that differs from its own model id for no reason. This does change what callers of 1.0.1 ask for, relative to 1.0.0. That is the point of the new version, and 1.0.0 is left as published. The other fifteen entries with mixed-case served names are untouched: each would need a new version file of its own, which belongs in a separate change.
aceforeverd
approved these changes
Oct 8, 2026
zhanghaohit
added a commit
that referenced
this pull request
Oct 9, 2026
PR #20 said sglang cannot serve any of these weights on sm120. That holds for the GLM pair and is wrong for the other three. This adds the sglang variants, measured on the same hardware, and corrects the sentence in kimi-k3's vLLM variant description that claimed vLLM was the only engine. kimi-k3 1.0.5 sglang-tp8-pp4-6000d TP8 x PP4 over four nodes deepseek-v4-pro 1.0.1 sglang-tp8-pp2-6000d TP8 x PP2 over two nodes deepseek-v4-flash 1.0.2 sglang-tp8-6000d TP8 on one node sglang-tp4-6000d TP4 on four GPUs vllm-tp4-6000d TP4 on four GPUs A published version is immutable, so each model gets a new version file that carries its existing variants forward unchanged. For Kimi K3 the difference is one flag. All four MoE runners were tried: the default upcasts the MXFP4 experts to bf16 and needs ~6000 GiB; triton_kernel fails in the weight loader, transposing before the padding compensation; flashinfer_mxfp4 loads, serves and computes wrong answers; marlin is correct, at 47.81 GiB of weights per card and max_total_num_tokens 1,237,195. flashinfer_mxfp4 is why these variants are backed by generated text rather than by readiness. It keeps the weights 4-bit, prints `fired up and ready to roll`, and serves 200s with a normal finish_reason -- while answering "2 + 2 =" with " 5". The raw /v1/completions text is equally wrong, which rules out the chat template and the parsers. A variant accepted on container readiness would be a service that silently answers incorrectly. DeepSeek V4 Flash does not need eight cards. At TP4: 38.57 GiB of weights per card and max_total_num_tokens 4,002,048, against 6,524,672 at TP8 -- half the cards keep 61% of the pool, because the weights are a fixed cost that does not shrink with the card count. Those two numbers imply ~34 and ~68 KiB per token, so neither extrapolates to another TP width, and the descriptions say so. vLLM reaches the same shape: 33.07 GiB left for KV per card and a 3,447,225-token pool. The GLM pair stays vLLM-only. sglang selects the ModelOpt loader but its MoE weights still fall through to the generic group-weight branch and fail with `tensor a (3072) must match tensor b (6144)` -- 6144 = 2 x 3072 being the fp4 packing factor, so NVFP4 packing is not wired into this architecture's MoE weight creation. --moe-runner-backend cannot help: it selects a kernel for apply(), and execution never gets there. Verified by starting each shape on idle nodes and reading generated answers to four questions with one unambiguous answer each; nothing is inferred from a sibling model. These ran under docker with the flags in the files, not under the chart, so the chart values are a translation of the vLLM variants' RoCE plumbing rather than a running configuration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every variant published so far targets H100, B300 or H800, and all 36 of them
are single-node. A cluster of eight-GPU RTX 6000D nodes therefore has nothing
it can deploy: preflight matches
requires.gpuProductagainst the node label,and four of these five models do not fit on one node anyway.
What this adds
Three models already had an entry and gain a variant. A published version file
is immutable, so each goes into a new version file that carries its
existing variants forward unchanged:
kimi-k3vllm-tp8-pp4-6000dglm5.3vllm-tp8-pp2-6000ddeepseek-v4-flashvllm-tp8-6000dTwo had no entry at all:
glm5.2vllm-tp8-pp2-6000ddeepseek-v4-provllm-tp8-pp2-6000dWhy all five are vLLM
sglang cannot serve these weights on sm120. The GLM pair fails inside the
weight loader, which no
--moe-runner-backendchoice reaches. For Kimi K3 thedefault MoE runner upcasts the 4-bit experts to bf16 and does not fit, and
flashinfer_mxfp4is worse than a failure: it loads, serves, reports healthy,and computes wrong answers. vLLM's Marlin path keeps the experts 4-bit.
Details that are load-bearing
Values are transcribed from running deployments rather than written from
scratch, and each variant was diffed key by key against the release it came
from: no value differs, and nothing is asserted that the release does not set.
Three of them are easy to lose and expensive to rediscover:
hostNetwork: trueon every multi-node variant — and on none of thesingle-node ones, which do not need it. RoCE GIDs belong to the host netdev
and a pod netns cannot see them; without it NCCL finds no usable device and
the group hangs at rendezvous instead of failing, so the symptom is a stuck
rollout rather than an error. It is also what keeps the host port from
colliding: only whole-node variants take it, and
requires.gpus: 8leavesroom for one pod per node.
--no-enable-flashinfer-autotunewherever PP > 1.gpu-memory-utilization: 0.85for the GLM pair, not 0.90. Their KV poolsaturates at 24 concurrent requests, and past that TTFT goes from 12s to 67s;
a larger allocator buys nothing.
Deliberately not included
Per-cluster wiring the catalog does not own:
cart,modelRoute,scaler,serviceMonitor, andtolerations— the last because the charts have addedthe GPU toleration by default since
vllm-0.7.6.model.gpusis omitted too,since the schema renders it from
requires.gpus.Probes and
progressDeadlineSecondsare left to the chart, whose defaultstartup probe is
30s × 180— the same 90 minutes the B300 entry spells out,so a cold multi-node weight load has the same headroom either way.
Second commit: the GLM NVFP4 repo ids
glm5.3'ssource.hfsaidzai-org/GLM-5.3-NVFP4, which does not exist, andthe
glm5.2entry added here first repeated the same mistake. The NVFP4 buildsare quantized with modelopt and published under
nvidia, so both now readnvidia/GLM-5.{2,3}-NVFP4.Every
source.hfthis PR touches is confirmed by fetchingconfig.jsonfromthe Hugging Face repo and comparing it byte for byte with the weights the
engine actually loaded:
source.hfkimi-k3moonshotai/Kimi-K3e0b7be1d…glm5.2nvidia/GLM-5.2-NVFP4dd035d69…glm5.3nvidia/GLM-5.3-NVFP4e223f9a8…deepseek-v4-flashdeepseek-ai/DeepSeek-V4-Flash7f34d488…deepseek-v4-prodeepseek-ai/DeepSeek-V4-Pro1285e9e3…That comparison is worth preferring to "does the id resolve", for two reasons.
huggingface.co answers
401for a gated repo and for one that does not existalike — API and HTML page both — so a status code cannot tell them apart. And
repos of the same model can differ only in which snapshot they hold:
DeepSeek-V4-ProandDeepSeek-V4-Pro-0813agree on layer count, hidden size,quantization and expert count, and differ in
config.jsonand in shard count.Deliberately mismatched pairs were run through the same check and reported as
mismatches, so it is not passing vacuously.
Checking all 22 entries leaves two that still do not resolve, in models this
branch does not touch. They are left alone rather than guessed at:
source.hfglm5.1modelforge/Qwen3.6-35B-A3B-793303-glm-5gpt-oss-120bopenai-mirror/gpt-oss-120bopenai/gpt-oss-120bexistsThird commit: lowercase served names
Served names in this catalog are split by family —
kimi-k3,kimi-k2.5andglm-5.3are lowercase, while the DeepSeek entries and most others are mixedcase — so there is no single convention to inherit. Lowercase is the one being
settled on, which makes
deepseek-v4-flash1.0.1 servedeepseek-v4-flashrather than
DeepSeek-V4-Flash, and agree with its ownname, which is whatthe schema says
servedNamedefaults to. The other four entries here werealready lowercase.
Two consequences worth stating plainly. Callers of 1.0.1 ask for a different
name than callers of 1.0.0 — that is what the new version is for, and 1.0.0 is
left as published. And the fifteen other mixed-case entries are untouched:
changing each needs a new version file of its own, so that belongs in a
separate change.
Open questions for review
docker.ioreference in the catalog. Every otherentry names
harbor.4pd.io. These sayvllm/vllm-openai:v0.30.0, which isthe version the running engines report, but the image they run is a registry
copy and was not proven identical to the upstream tag. If the consuming
clusters cannot pull Docker Hub, this wants a real tag in the usual registry
instead.
kimi-k3keepsdefault: trueonsglang-tp8-b300. On a 6000D clustera deploy that does not name a variant selects one that cannot schedule; the
schema has no per-hardware default, so callers must name
vllm-tp8-pp4-6000d.--max-num-seqs=256for the GLM pair is what the deployments run, but itis an order of magnitude above the 24 concurrent requests their KV pool
sustains, so the TTFT cliff above is reachable through it. Left as found,
since changing it is a tuning decision rather than a transcription one.
Checks
npm run validate:schemaandnpm run test:libpass, andbuild-index.jsbuilds — the index comes out with all five new variants and every carried-over
variant intact. Each new version file was also diffed against the one it
supersedes: identical apart from the version line, the appended variant, and in
one case the served name.
Note that
npm test(hack/validate.sh) fails onmasteralready, beforethis branch:
models/gemma-4-31b-it/metadata.yamldeclaresgoogle-gemma-4-31b-it, so its directory-name check stops there. Because thatcheck runs first and the script is
set -e, the later ones never execute, sothey were re-run by hand against these files — filename/version agreement,
directory name, duplicate variant ids, at most one default, and
lws.size == requires.nodes— and all pass. CI runs onlyvalidate:schemaandtest:lib, so it is unaffected either way.