Native vision (image input) serving of DeepSeek-V4-Flash-Vision-Exp on a pair of Jetson AGX Thor T5000 with vLLM TP=2 and DSpark speculative decoding intact.
Adapted from tonyd2wild's GB10 (DGX Spark) vision port. Verified working end-to-end on Thor 2026-09-01: text + vision discriminating tests all correct, full benchmark matrix run, one new concurrency bug found and documented.
Companion repo: SonicBotMan/dsv4-dual-thor-tuning (text-only DSV4-Flash tuning on the same hardware). This repo assumes that deployment as the baseline.
| test | result |
|---|---|
| red-left / blue-right (336×168) | ✅ "Red on the left, blue on the right." |
| green-top / yellow-bottom (168×336) | ✅ correct |
| 4 fresh color/orientation combos | ✅ 4/4 correct, 1.0–1.5 s each |
| count-to-100 / QA needle | ✅ / PASS ("74291") |
| solo decode 400 tok ×3 median | 19.7 tok/s (text-only prod: 23.9–25.3 → −22% vision tax) |
| count-to-300 | 37.6 tok/s |
| prefill 9 620 tok | 11.65 s TTFT = 826 tok/s (text-only parity: 820) |
| text c=4 aggregate | 46.8 tok/s (text-only prod: 39.4 → +19%) |
| DSpark acceptance | ≈1.47 tok/step (−6% vs text-only) |
| weights load | 48/48 shards, 80.27 GiB, 242 s |
vision+vision concurrent (c=2) fails 100% reproducibly (3/3 rounds) — each round exactly one request returns:
{"error": {"message": "Already borrowed", "type": "InternalServerError", "code": 500}}- text + vision mixed concurrent: both OK
- text-only c=2 / c=4: OK
- suspect: shared state in the vision forward path (
lru_cacheRoPE cos/sin tables inds4v_vision.py, or shared processor state inds4v_mm.py) - not in upstream's issue list (their vision traffic was single-stream)
- workaround: serialize vision requests (semaphore=1); text endpoints unaffected
- surviving answers are always correct; single-stream vision is fully reliable
The upstream port targets vLLM 0.21.1rc1 on GB10 (sm_121a); our Thor build is an
SM110 line (0.0.0+9c9fef7f01) with different module layout:
| difference | consequence |
|---|---|
standalone draft module models/deepseek_v4/nvidia/dspark.py |
upstream's patch 4 fix doesn't apply; our 5th patch: draft loader must skip bias_vl (KeyError otherwise) |
different __init__ structure (no HCHeadOp()) |
init anchor moved to the _mtp_hidden_buffer else-branch |
ForCausalLM inherits SupportsEagle3, DeepseekV4MixtureOfExperts |
mm-class anchor rewritten with 5 bases |
two embed_input_ids defs (Model + CausalLM) |
embed anchor disambiguated via extract_moe_parameters |
import anchor _env_flag absent |
anchored on _use_sequence_parallel |
KV dtype: no nvfp4_ds_mla |
we serve fp8 KV at 13 GiB instead of NVFP4 KV |
4 of 8 upstream patch anchors match verbatim; the rest are Thor-adapted in
patches/patch_vision_thor.py.
patches/
patch_vision_thor.py # 8-anchor idempotent patcher for OUR model.py (Thor anchors)
patch_registry_thor.py # registers DeepseekV4VForConditionalGeneration multimodal alias
patch_draft_skip_bias_vl.py # 5th patch: draft loader skips bias_vl (Thor-only fix)
ds4v_vision.py # ViT + Aligner (verbatim numerics from checkpoint reference)
ds4v_mm.py # multimodal plumbing (processor, geometry, block builder)
launchers/
restart-rank0.sh # head node (serves :19040)
restart-rank1.sh # worker node (--headless; START THIS ONE FIRST)
scripts/
check_anchors.py # dry-run: verify all patch anchors exist exactly once
benchmarks/
bench_s1_solo_prefill.sh # T1 solo decode + T2 prefill
bench_s2_concurrency_vision.sh # T3 concurrency + T4 vision load + T5 needle
bench_t4_vision_load.sh # standalone vision-load runner (self-contained requests)
docs/
PORTING.md # exact anchor diffs + 12 upstream pitfalls + 6 Thor boot pitfalls
BENCHMARKS.md # full methodology + numbers + concurrency bug writeup
# 0. download weights (HF direct often unreachable from CN networks; mirror works)
BASE=https://hf-mirror.com/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp/resolve/main
# ... 48 shards + config/tokenizer files -> /home/nvidia/dsv4f-vision-model (167.8 GB)
# 1. stage patched vLLM files on BOTH nodes at /var/tmp/ds4v/
# (model.py + registry.py extracted from YOUR image, then patched — see docs/PORTING.md)
python3 patches/patch_vision_thor.py /var/tmp/ds4v/model.py
python3 patches/patch_registry_thor.py /var/tmp/ds4v/registry.py
python3 patches/patch_draft_skip_bias_vl.py /var/tmp/ds4v/draft_dspark.py
cp patches/ds4v_vision.py patches/ds4v_mm.py /var/tmp/ds4v/
md5sum /var/tmp/ds4v/* # compare across nodes!
# 2. boot (worker first!)
ssh rank1 'bash launchers/restart-rank1.sh' # 139
sleep 30
ssh rank0 'bash launchers/restart-rank0.sh' # 179
# engine ready in ~9 min (weight load 242 s + JIT warmup + graph capture)
# 3. probe
curl http://<rank0>:19040/healthThe vision checkpoint defaults to thinking ON — a plain request returns
content: null with all tokens spent on reasoning. Either send:
{"chat_template_kwargs": {"thinking": false}, ...}or allow enough max_tokens for thinking to finish.
{"model": "deepseek-v4-flash-vision",
"messages": [{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},
{"type": "text", "text": "What colors and which sides?"}]}],
"max_tokens": 120, "temperature": 0,
"chat_template_kwargs": {"thinking": false}}- Bidirectional attention within image spans is not implemented — expect degradation on OCR / charts / dense documents. Smoke tests still pass.
bias_vlis loaded but not applied — image tokens route through the text gate bias (likely whybias_vlexists). Proper fix needs modality-aware MoE gating.
Stop the vision containers on both nodes, then re-run the text-only
restart-pcache.sh from the companion repo (rank1 first, 30 s, then rank0).
Full procedure in docs/PORTING.md.
- tonyd2wild/DeepSeek-v4-Flash-Vision-Exp-DSpark-1M-NVFP4-KV-2x-DGX-Spark — the GB10 port this Thor adaptation is based on; their patches 1–5 and vision-exp branch did the heavy lifting
- DeepSeek — the Vision-Exp checkpoint and reference implementation
- vLLM community — DSpark speculative decoding (#46995) and the DeepSeek-V4 stack
MIT (matches the checkpoint license and upstream port).