Skip to content

Add RTX 6000D (sm120) variants for five models - #20

Merged
zhanghaohit merged 3 commits into
masterfrom
feat/rtx-6000d-variants
Oct 8, 2026
Merged

zhanghaohit merged 3 commits into
masterfrom
feat/rtx-6000d-variants

Conversation

@zhanghaohit

@zhanghaohit zhanghaohit commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Every variant published so far targets H100, B300 or H800, and all 36 of them
are single-node. A cluster of eight-GPU RTX 6000D nodes therefore has nothing
it can deploy: preflight matches requires.gpuProduct against the node label,
and four of these five models do not fit on one node anyway.

What this adds

Three models already had an entry and gain a variant. A published version file
is immutable, so each goes into a new version file that carries its
existing variants forward unchanged:

model new version variant shape
kimi-k3 1.0.3 vllm-tp8-pp4-6000d TP8 × PP4, four nodes
glm5.3 1.0.2 vllm-tp8-pp2-6000d TP8 × PP2, two nodes
deepseek-v4-flash 1.0.1 vllm-tp8-6000d TP8, one node

Two had no entry at all:

model version variant shape
glm5.2 1.0.0 vllm-tp8-pp2-6000d TP8 × PP2, two nodes
deepseek-v4-pro 1.0.0 vllm-tp8-pp2-6000d TP8 × PP2, two nodes

Why all five are vLLM

sglang cannot serve these weights on sm120. The GLM pair fails inside the
weight loader, which no --moe-runner-backend choice reaches. For Kimi K3 the
default MoE runner upcasts the 4-bit experts to bf16 and does not fit, and
flashinfer_mxfp4 is worse than a failure: it loads, serves, reports healthy,
and computes wrong answers. vLLM's Marlin path keeps the experts 4-bit.

Details that are load-bearing

Values are transcribed from running deployments rather than written from
scratch, and each variant was diffed key by key against the release it came
from: no value differs, and nothing is asserted that the release does not set.
Three of them are easy to lose and expensive to rediscover:

  • hostNetwork: true on every multi-node variant — and on none of the
    single-node ones, which do not need it. RoCE GIDs belong to the host netdev
    and a pod netns cannot see them; without it NCCL finds no usable device and
    the group hangs at rendezvous instead of failing, so the symptom is a stuck
    rollout rather than an error. It is also what keeps the host port from
    colliding: only whole-node variants take it, and requires.gpus: 8 leaves
    room for one pod per node.
  • --no-enable-flashinfer-autotune wherever PP > 1.
  • gpu-memory-utilization: 0.85 for the GLM pair, not 0.90. Their KV pool
    saturates at 24 concurrent requests, and past that TTFT goes from 12s to 67s;
    a larger allocator buys nothing.

Deliberately not included

Per-cluster wiring the catalog does not own: cart, modelRoute, scaler,
serviceMonitor, and tolerations — the last because the charts have added
the GPU toleration by default since vllm-0.7.6. model.gpus is omitted too,
since the schema renders it from requires.gpus.

Probes and progressDeadlineSeconds are left to the chart, whose default
startup probe is 30s × 180 — the same 90 minutes the B300 entry spells out,
so a cold multi-node weight load has the same headroom either way.

Second commit: the GLM NVFP4 repo ids

glm5.3's source.hf said zai-org/GLM-5.3-NVFP4, which does not exist, and
the glm5.2 entry added here first repeated the same mistake. The NVFP4 builds
are quantized with modelopt and published under nvidia, so both now read
nvidia/GLM-5.{2,3}-NVFP4.

Every source.hf this PR touches is confirmed by fetching config.json from
the Hugging Face repo and comparing it byte for byte with the weights the
engine actually loaded:

model source.hf config.json md5
kimi-k3 moonshotai/Kimi-K3 e0b7be1d…
glm5.2 nvidia/GLM-5.2-NVFP4 dd035d69…
glm5.3 nvidia/GLM-5.3-NVFP4 e223f9a8…
deepseek-v4-flash deepseek-ai/DeepSeek-V4-Flash 7f34d488…
deepseek-v4-pro deepseek-ai/DeepSeek-V4-Pro 1285e9e3…

That comparison is worth preferring to "does the id resolve", for two reasons.
huggingface.co answers 401 for a gated repo and for one that does not exist
alike — API and HTML page both — so a status code cannot tell them apart. And
repos of the same model can differ only in which snapshot they hold:
DeepSeek-V4-Pro and DeepSeek-V4-Pro-0813 agree on layer count, hidden size,
quantization and expert count, and differ in config.json and in shard count.
Deliberately mismatched pairs were run through the same check and reported as
mismatches, so it is not passing vacuously.

Checking all 22 entries leaves two that still do not resolve, in models this
branch does not touch. They are left alone rather than guessed at:

model source.hf note
glm5.1 modelforge/Qwen3.6-35B-A3B-793303-glm-5 points at a Qwen repo
gpt-oss-120b openai-mirror/gpt-oss-120b only openai/gpt-oss-120b exists

Third commit: lowercase served names

Served names in this catalog are split by family — kimi-k3, kimi-k2.5 and
glm-5.3 are lowercase, while the DeepSeek entries and most others are mixed
case — so there is no single convention to inherit. Lowercase is the one being
settled on, which makes deepseek-v4-flash 1.0.1 serve deepseek-v4-flash
rather than DeepSeek-V4-Flash, and agree with its own name, which is what
the schema says servedName defaults to. The other four entries here were
already lowercase.

Two consequences worth stating plainly. Callers of 1.0.1 ask for a different
name than callers of 1.0.0 — that is what the new version is for, and 1.0.0 is
left as published. And the fifteen other mixed-case entries are untouched:
changing each needs a new version file of its own, so that belongs in a
separate change.

Open questions for review

  • The image is the only docker.io reference in the catalog. Every other
    entry names harbor.4pd.io. These say vllm/vllm-openai:v0.30.0, which is
    the version the running engines report, but the image they run is a registry
    copy and was not proven identical to the upstream tag. If the consuming
    clusters cannot pull Docker Hub, this wants a real tag in the usual registry
    instead.
  • kimi-k3 keeps default: true on sglang-tp8-b300. On a 6000D cluster
    a deploy that does not name a variant selects one that cannot schedule; the
    schema has no per-hardware default, so callers must name
    vllm-tp8-pp4-6000d.
  • --max-num-seqs=256 for the GLM pair is what the deployments run, but it
    is an order of magnitude above the 24 concurrent requests their KV pool
    sustains, so the TTFT cliff above is reachable through it. Left as found,
    since changing it is a tuning decision rather than a transcription one.

Checks

npm run validate:schema and npm run test:lib pass, and build-index.js
builds — the index comes out with all five new variants and every carried-over
variant intact. Each new version file was also diffed against the one it
supersedes: identical apart from the version line, the appended variant, and in
one case the served name.

Note that npm test (hack/validate.sh) fails on master already, before
this branch: models/gemma-4-31b-it/metadata.yaml declares
google-gemma-4-31b-it, so its directory-name check stops there. Because that
check runs first and the script is set -e, the later ones never execute, so
they were re-run by hand against these files — filename/version agreement,
directory name, duplicate variant ids, at most one default, and
lws.size == requires.nodes — and all pass. CI runs only validate:schema and
test:lib, so it is unaffected either way.

Every variant published so far targets H100, B300 or H800, and every one of
them is single-node. A cluster of eight-GPU RTX 6000D nodes therefore has no
entry it can deploy: preflight matches requires.gpuProduct against the node
label, and four of these five models do not fit on one node at all.

Three models already had an entry and gain a variant, in a new version file --
a published version is immutable, so its existing variants are carried forward
unchanged:

  kimi-k3           1.0.3  vllm-tp8-pp4-6000d   TP8 x PP4 over four nodes
  glm5.3            1.0.2  vllm-tp8-pp2-6000d   TP8 x PP2 over two nodes
  deepseek-v4-flash 1.0.1  vllm-tp8-6000d       TP8 on one node

Two had none at all:

  glm5.2            1.0.0  vllm-tp8-pp2-6000d   TP8 x PP2 over two nodes
  deepseek-v4-pro   1.0.0  vllm-tp8-pp2-6000d   TP8 x PP2 over two nodes

All five are vLLM. sglang cannot serve these weights on sm120: the GLM pair
fails in the weight loader, and for Kimi K3 the default MoE runner upcasts the
4-bit experts to bf16 and will not fit, while flashinfer_mxfp4 loads and serves
but computes wrong answers silently. vLLM's Marlin path keeps them 4-bit.

Values are transcribed from running deployments. Three details are load-bearing
and easy to lose:

  - hostNetwork on every multi-node variant. RoCE GIDs belong to the host
    netdev and a pod netns does not see them; without it NCCL finds no usable
    device and the group hangs at rendezvous rather than failing.
  - --no-enable-flashinfer-autotune wherever PP > 1.
  - gpu-memory-utilization 0.85 for the GLM pair, not 0.90. Their KV pool
    saturates at 24 concurrent requests and TTFT goes from 12s to 67s past
    that, so a larger allocator buys nothing.

Per-cluster wiring is deliberately left out: cart, modelRoute, scaler,
serviceMonitor, and tolerations, the last because the charts have added the GPU
toleration by default since vllm-0.7.6. model.gpus is omitted too, since the
schema renders it from requires.gpus.
Both said zai-org. The NVFP4 builds are quantized with modelopt and published
under nvidia: zai-org/GLM-5.3-NVFP4 does not exist, and neither did the
GLM-5.2-NVFP4 id added alongside it in this branch.

The mistake is easy to make and hard to catch, because huggingface.co answers
401 for a gated repo and for one that does not exist alike -- API and HTML page
both -- so fetching an id and reading the status cannot tell the two apart. The
search endpoint does work unauthenticated and enumerates real repos, which is
what these two were confirmed against.

Two more ids in the catalog do not resolve either, in models this branch does
not touch, so they are left alone rather than guessed at:

  glm5.1        modelforge/Qwen3.6-35B-A3B-793303-glm-5
  gpt-oss-120b  openai-mirror/gpt-oss-120b

@huskcass huskcass left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

helm values render check ✅

Validated every variant's values: with helm template against the sglang chart — both the pinned version and the latest published (0.8.10). All variants render cleanly. (Checked with the deploy-time modelRoute.nginx.outputConfigMap set, which swiss injects since the catalog schema rejects per-deploy values by design.)

chart.version already uses a semver range — no pinning to flag. 👍

servedName was DeepSeek-V4-Flash. Every entry in this catalog that was added
with a lowercase served name keeps one -- kimi-k3, kimi-k2.5, glm-5.3 -- while
the DeepSeek and most other families carry mixed case, so there is no single
convention to follow and lowercase is the one being settled on.

Here it also makes the field agree with `name`, which is what the schema says
servedName defaults to, so 1.0.1 no longer advertises a name that differs from
its own model id for no reason.

This does change what callers of 1.0.1 ask for, relative to 1.0.0. That is the
point of the new version, and 1.0.0 is left as published.

The other fifteen entries with mixed-case served names are untouched: each
would need a new version file of its own, which belongs in a separate change.
@zhanghaohit
zhanghaohit requested a review from huskcass October 8, 2026 08:44
@zhanghaohit
zhanghaohit merged commit f942861 into master Oct 8, 2026
3 checks passed
@aceforeverd
aceforeverd deleted the feat/rtx-6000d-variants branch October 8, 2026 08:48
zhanghaohit added a commit that referenced this pull request Oct 9, 2026
PR #20 said sglang cannot serve any of these weights on sm120. That holds for
the GLM pair and is wrong for the other three. This adds the sglang variants,
measured on the same hardware, and corrects the sentence in kimi-k3's vLLM
variant description that claimed vLLM was the only engine.

  kimi-k3            1.0.5  sglang-tp8-pp4-6000d   TP8 x PP4 over four nodes
  deepseek-v4-pro    1.0.1  sglang-tp8-pp2-6000d   TP8 x PP2 over two nodes
  deepseek-v4-flash  1.0.2  sglang-tp8-6000d       TP8 on one node
                            sglang-tp4-6000d       TP4 on four GPUs
                            vllm-tp4-6000d         TP4 on four GPUs

A published version is immutable, so each model gets a new version file that
carries its existing variants forward unchanged.

For Kimi K3 the difference is one flag. All four MoE runners were tried: the
default upcasts the MXFP4 experts to bf16 and needs ~6000 GiB; triton_kernel
fails in the weight loader, transposing before the padding compensation;
flashinfer_mxfp4 loads, serves and computes wrong answers; marlin is correct,
at 47.81 GiB of weights per card and max_total_num_tokens 1,237,195.

flashinfer_mxfp4 is why these variants are backed by generated text rather than
by readiness. It keeps the weights 4-bit, prints `fired up and ready to roll`,
and serves 200s with a normal finish_reason -- while answering "2 + 2 =" with
" 5". The raw /v1/completions text is equally wrong, which rules out the chat
template and the parsers. A variant accepted on container readiness would be a
service that silently answers incorrectly.

DeepSeek V4 Flash does not need eight cards. At TP4: 38.57 GiB of weights per
card and max_total_num_tokens 4,002,048, against 6,524,672 at TP8 -- half the
cards keep 61% of the pool, because the weights are a fixed cost that does not
shrink with the card count. Those two numbers imply ~34 and ~68 KiB per token,
so neither extrapolates to another TP width, and the descriptions say so. vLLM
reaches the same shape: 33.07 GiB left for KV per card and a 3,447,225-token
pool.

The GLM pair stays vLLM-only. sglang selects the ModelOpt loader but its MoE
weights still fall through to the generic group-weight branch and fail with
`tensor a (3072) must match tensor b (6144)` -- 6144 = 2 x 3072 being the fp4
packing factor, so NVFP4 packing is not wired into this architecture's MoE
weight creation. --moe-runner-backend cannot help: it selects a kernel for
apply(), and execution never gets there.

Verified by starting each shape on idle nodes and reading generated answers to
four questions with one unambiguous answer each; nothing is inferred from a
sibling model. These ran under docker with the flags in the files, not under
the chart, so the chart values are a translation of the vLLM variants' RoCE
plumbing rather than a running configuration.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants