DeepSeek-V4-Flash-DSpark

[lobanov]

DeepSeek released DeepSeek-V4-Flash-DSpark variant of the V4 Flash model with built-in speculative decoding and described the methodology in a companion paper. Given the focus on performance of running the model locally, I think it makes sense to explore how DSpark can be integrated into ds4.

It seems that DSpark provides considerable improvement in tok/s performance with fixed concurrency over previous generation of speculative decoding (MTP):

[lobanov]

Initial quick benchmarking shows that it might be harder to achieve than it seems on the face of it with a fully-local engine.

Generation baseline on M5 Max on short-ish prompts is 28 ms per token. With current 2-token MTP enabled it's 2ms for prediction and 33ms for fast verification pass. That in practice only gains on average 6-7% because predictions are not always correct. When --quality is enabled, a different slower exact verifier kernel is used, which verifies two tokens in 67ms. In practice this gives 28% reduction of inference speed over the baseline. Under realistic assumptions of prediction quality and assuming exact verification cost scales linearly with number of tokens there is no gain.

Three avenues to explore:

1. Can exact verifier be made to scale sublinearly with increased number of predicted tokens? Very hard, especially on Metal. It's a research project.
2. Use a different approach for batch verification altogether, e.g. distribution-exact not logit-exact. Research project again, but perhaps easier.
3. Let go of exact verification and use existing batched verifier. It's algorithmically identical, but numerically different, so model precision can suffer. By how much is hard to tell, it would need benchmarking. This is more practical, because we are accepting some quality loss with quantization anyway, and this may allow to trade speed for quality.

[daaain]

This sounds incredible! So you already have a working implementation @lobanov?

I've been discussing it with Claude after cloning both repos and sounds like batching could be a way to go (there's a bit at the end in response to your comment):

---

DSpark on ds4 (DwarfStar): feasibility, tensor map & RAM budget

Status: investigation only, no code written yet.
Target: DeepSeek-V4-Flash + DSpark draft, Metal backend, Apple-silicon single-user (reference machine: 128 GB M3 Max, asymmetric q2).
Source artefacts: DeepSeek released the DSpark checkpoints and the DeepSpec training repo on 27 Jun 2026, MIT-licensed. All claims below are traceable to the files listed under Evidence at the bottom.

---

TL;DR

• The weights exist and fit the use case. deepseek-ai/DeepSeek-V4-Flash-DSpark is the base checkpoint + a draft module under the mtp.* namespace. MIT.
• It's a bolt-on, not a requant. The base 43 layers, embedding and LM head are frozen and shared by reference. The existing q2 Flash GGUF is untouched. We only convert the new mtp.* tensors.
• The draft is 3 full MoE layers (not dense, not tiny) + a vanilla Markov head + a 1-logit confidence head. That's where the RAM cost lives.
• The draft backbone is the same V4 layer type we already run (HC + MoE), but with uncompressed sliding-window attention (no compressor, no indexer) — which makes the Metal path simpler than feared.
• RAM is the real constraint, not correctness. Full draft at q8 ≈ ~20 GB; at q4 ≈ ~11 GB; at iq2 ≈ ~6 GB. We already resident-load MTP-1 (~1 layer), so the incremental cost is ~2 layers. Recommendation: ship behind --draft dspark, default the draft to q4, reclaim RAM by streaming more base experts (never the draft).
• The 6–7% MTP-1 ceiling is an acceptance artefact, not a verifier wall. Don't fight the exact verifier (it's slower-than-baseline by construction); use the lossless batched verifier + rejection sampling and measure its drift against the q2 noise floor. The economics are bandwidth-bound, so they largely hold on M3 Max too. See Verifier economics below.

---

Resolved questions (all statically verifiable)

#	Question	Answer	Evidence
1	Backbone: plain transformer or our HC layer?	HC + MoE V4 Block — DSparkBlock(Block). Reuses our layer machinery.	model.py: class DSparkBlock(Block)
2	Draft depth	3 stages (n_mtp_layers: 3) vs base model's 1 (num_nextn_predict_layers: 1) and the paper's 5.	inference/config.json; base config.json
3	Block size γ	5	dspark_block_size: 5
4	Markov head variant	Vanilla — markov_w1 (V×256) + markov_w2 (256×V) only. No gate, no RNN.	model.py: class DSparkMarkovHead
5	Markov rank r	256	dspark_markov_rank: 256
6	Confidence head	Linear(dim + 256 → 1) + sigmoid; scalar per position.	model.py: class DSparkConfidenceHead
7	Draft FFN: MoE or dense?	Full MoE — 256 routed experts (fp4) + 1 shared, per stage.	DSparkBlock inherits Block.ffn = MoE(...)
8	Draft attention	Uncompressed sliding-window, compress_ratio == 0 asserted. No compressor, no indexer.	model.py: class DSparkAttention (assert self.compress_ratio == 0)
9	Intra-block attention	Bidirectional within the γ block, expressed as an index list (not a triangular mask): every block position sees the window + all block positions.	get_dspark_topk_idxs
10	KV injection source	Mean-pooled hidden states of target layers [40, 41, 42] (pooled over the hc copies).	dspark_target_layer_ids: [40,41,42]; model.py: if i in self.target_layer_ids: main_hiddens.append(h.mean(dim=2))
11	Injection wiring	main_x = main_norm(main_proj(concat(h40,h41,h42))), main_proj: Linear(12288→4096); enters each draft layer as one KV row in its window.	model.py: DSparkBlock.forward_embed
12	Block input tokens	[anchor, noise, noise, noise, noise], noise id 128799.	dspark_noise_token_id: 128799; forward_embed
13	Embedding / LM head	Shared & frozen from the base model (tied), no separate weights.	model.py: mtp[-1].embed = self.embed; mtp[-1].head = self.head
14	Base precision	fp8 e4m3 (ue8m0 scales, 128×128 blocks); experts fp4.	base config.json quantization_config; inference/config.json

Correction to an earlier note: an earlier read suggested the DSpark config altered the base compression schedule for layers 40–42. It does not. The three extra 0 entries in DSpark's compress_ratios are the draft layers themselves (indices 43/44/45), which run uncompressed. The base 0–42 schedule is unchanged.

---

Loader spec — mtp.* tensor map

Three stages: mtp.0, mtp.1, mtp.2.

Every stage (maps onto our existing ds4_layer_weights):
attn.{wq_a, q_norm, wq_b, wkv, kv_norm, wo_a, wo_b, attn_sink},
ffn.{gate.weight, gate.bias, experts.{0..255}.{w1,w2,w3}, shared_experts.{w1,w2,w3}},
attn_norm, ffn_norm, hc_attn_fn/base/scale, hc_ffn_fn/base/scale.

Stage 0 only: mtp.0.main_proj (Linear 12288→4096), mtp.0.main_norm (RMSNorm 4096).

Last stage (mtp.2) only: norm, markov_head.markov_w1 (V×256), markov_head.markov_w2 (256×V), confidence_head.proj (Linear 4352→1), hc_head_fn/base/scale.

Not present (shared from base): token embedding, LM head.

Forward deltas vs our current MTP-1 path

This is a new draft subgraph, not a parameter swap into the existing MTP path:

• Input projection differs. MTP-1 uses e_proj/h_proj (embed + previous hidden). DSpark uses main_proj/main_norm over the concatenation of three target-layer hiddens, plus noise-token embeddings for the block. → needs a capture hook on layers 40/41/42 during the main forward.
• Parallel bidirectional block of γ=5 vs the current recursive single-token chain. Reuse the prefill multi-row path; supply the get_dspark_topk_idxs-equivalent visibility (window + all block positions). Since γ=5 ≪ window 128, compute the block densely; never touch the compressed/indexer path.
• Output head is new: hc_head collapse → shared LM head → per-position Markov bias loop (r=256) → confidence sigmoid → confidence prefix-cut. The cut generalises our existing mtp_margin static threshold.

Verify/rollback reuses what we already have: spec_frontier_snapshot/restore, mtp_n_raw, DS4_MTP_KEEP_ACCEPTED (equivalent to the reference's past_key_values.crop(start)).

---

Quantisation: do we requantise the FFN? No.

• The base model is inert. The existing q2 Flash GGUF stays byte-for-byte. We convert only the mtp.* tensors.
• In the checkpoint, the draft's experts are fp4 (same family as base), projections fp8, small heads bf16/fp32. We pick the GGUF quant for the draft independently of the base.
• Acceptance caveat (affects speed, never correctness): the draft was trained against the fp8 target's layer-40/41/42 hiddens. Running the base at q2 perturbs those hiddens → lower accepted length τ. Spec decoding stays lossless regardless. Mitigation that costs nothing: pair DSpark with the existing q2-q4-imatrix GGUF (last 6 layers at q4) so the injection sources are higher precision. The shared head is already kept high (OutQ8).

---

RAM budget (the part that matters for a usable laptop)

The draft is three full V4 MoE layers. Per layer ≈ 6.5 B params (256 routed experts dominate: 256 × 3 × 4096 × 2048 ≈ 6.44 B). Three layers ≈ ~19.7 B params. Stored size by quant:

Draft quant	Stored size (3 layers)	Notes
q8	~20 GB	the "19 GB" figure; highest τ
q4 / fp4	~11 GB	recommended default balance
iq2 (match base experts)	~6 GB	tightest; measure τ loss

Markov + confidence + norms are negligible (~70 MB total).

Reframe the increment. We already resident-load the base model's MTP-1 (1 layer ≈ 6.5 B). The honest incremental cost of DSpark over what ds4 loads today is ~2 extra layers:

Scenario	Draft RAM	Δ over current MTP-1 (≈1 layer)
DSpark @ q8	~20 GB	~ +13 GB
DSpark @ q4	~11 GB	~ +5 GB
DSpark @ iq2	~6 GB	~ +0 GB (≈ replaces MTP-1 footprint)

So at iq2 the draft is roughly RAM-neutral versus the MTP-1 we already carry; q4 is a ~5 GB increment; only q8 costs the full headroom.

MoE means stored ≠ bandwidth. Only ~6 of 256 experts fire per position, so keeping the draft at higher precision costs RAM but only a modest bandwidth hit per draft step (the δ in the speedup model). The trade is "spend resident RAM to protect τ," not "spend bandwidth."

KV cache headroom is mostly safe. V4's compressed KV is small by design, and the draft's own KV is its 128-row sliding window × 3 layers — negligible. The squeeze is the draft weights, not draft KV.

Levers to stay laptop-usable

1. Make it opt-in. Gate behind --draft dspark. RAM-constrained or background-only users keep MTP-1 and pay nothing.
2. Default the draft to q4 (~11 GB, ~+5 GB over MTP-1). Offer iq2 (~RAM-neutral) for tight setups and q8 only when RAM is flush. Treat the draft-quant-vs-τ curve as a measured knob, not a guess.
3. Reclaim RAM from the base, not the draft. ds4 already streams base experts (--ssd-streaming --ssd-streaming-cache-experts NGB). Stream more of the cold base experts to free room for an in-RAM draft. The base is amortised over τ accepted tokens, so its streaming penalty is diluted by the speedup. Do not SSD-stream the draft — it's on the latency-critical path (the T_draft term); paging draft experts per block would erase the win.
4. Wire a hard memory ceiling. Keep the draft buffers inside whatever iogpu/wired-limit budget we set, so enabling DSpark can't tip the machine into swap and make it unusable.

---

Verifier economics & the speedup ceiling

This section responds to a maintainer benchmark showing MTP-1 gaining only 6–7% on the fast path and losing 28% with --quality (exact verifier). The conclusion: the ceiling is acceptance rate, not the verifier — and the verifier choice has a clean, lossless answer.

• The 6–7% is an acceptance-rate artefact. The fast path reproduces exactly: 2 ms draft + 33 ms verify(2) = 35 ms ÷ τ≈1.33 ≈ 26.3 ms vs 28 ms baseline ≈ 6.5%. The bottleneck is τ — and τ is precisely what DSpark's semi-autoregressive draft + confidence trim raise. MTP-1 drafts one token of low acceptance; that's the wall, not the kernel.
• The exact verifier is underwater by construction. 67 ms / 2 tokens = 33.5 ms/token, already above the 28 ms sequential baseline. Even perfect prediction (τ = γ+1) floors per-token cost above baseline, so no acceptance rate rescues it. Avenue 1 (sublinear exact verify on Metal) chases a kernel that loses at τ→∞. Drop it.
• The fast/batched verifier is bandwidth-bound — which is why speculation can win. The benchmark shows it: the 2nd verified token adds only ~4 ms to a ~29 ms pass — nearly free, because weights stream once regardless of γ. That sublinearity is the speculative thesis. The exact kernel is ~linear (compute-bound) and cannot amortise.
• "Exact" vs "fast" is reference-drift, not lossless-vs-lossy. Both are lossless: rejection sampling (Chen et al. 2023 / Leviathan et al. 2023) yields exact samples from whatever logits the verifier produces. The batched verifier is distribution-exact w.r.t. its own numerics; it only drifts from the decode kernel ds4 validates against official DeepSeek logits, via fp accumulation order. So avenues 2 and 3 collapse into one practical answer — batched verifier + rejection sampling — and the real question is "how far does batched drift from the reference vs how far q2 already drifts." Almost certainly the latter dominates by orders of magnitude (rounding-order vs multi-bit). Measure it with gguf-tools/quality-testing: score "q2 + batched-verify spec" against official continuations next to "q2 + exact." If batched sits inside the q2 noise floor, ship it.

Where DSpark clears break-even (M5 Max, fast path, γ=5; assumes ~6 ms draft, ~48 ms verify(5)):

τ (accepted)	per-token	vs 28 ms baseline
2	~27 ms	~break-even
3	~18 ms	~35–40% faster
4	~13.5 ms	~2×

Break-even is τ≈2; real wins need τ≥3. The confidence prefix-cut also bounds wasted verification (the "predictions aren't always correct" loss) by not verifying the low-survival suffix — biggest help on chat.

M3 Max vs M5 Max: does slower prompt processing change the picture?

Short answer: predominantly bandwidth-bound, so the speculative speedup ratio is largely preserved on M3 Max — most of the % win survives on a slower absolute baseline. M3's weaker compute bites only where the work is compute-bound, and those are mostly the parts we're abandoning or that DSpark already manages.

From ds4's own bench (q2, short context): M3 Max decode 37.5 ms/tok, M5 Max 29.2 ms/tok — a 1.28× gap, tracking memory bandwidth (~400 GB/s vs higher). Prefill gap is wider: 1.49× short, 1.85× at ~12 k. M5's compute advantage exceeds its bandwidth advantage, and that asymmetry is the whole story:

• Per-token economics scale with bandwidth. Draft and fast-verify both stream weights once, both ~1.28× the baseline. Speedup is a ratio (τ ÷ cycle/baseline); numerator and denominator scale together → % speedup is roughly bandwidth-invariant. Worked at τ=3: ~1.65× on M5 vs ~1.57× on M3 — close.
• The marginal verify-token is compute, and M3 has less of it. The ~4 ms/extra-token on M5 becomes ~6–7 ms on M3 (compute scales 1.5–1.85×). The win erodes modestly, and the erosion grows with γ and context — larger blocks / long context push the verify pass toward compute-bound sooner on M3. Mitigation: the confidence trim shrinks effective γ exactly when acceptance is low, capping compute-bound verify width. The trim is more valuable on M3 Max.
• The exact verifier is even more hopeless on M3. Being compute-bound, it scales with M3's worse prefill throughput — the −28% seen on M5 would be a deeper slowdown on M3. Stronger reason to use the batched verifier.
• Prompt prefill is slower on M3 (1.49–1.85×) but that's a one-time per-request cost, orthogonal to per-token decode.

Verdict: RAM-bandwidth-bottlenecked in the decode/fast-verify regime, so DSpark's value proposition survives on M3 Max with a slightly thinner margin. The compounding risk to watch is q2-lowered τ × M3 compute erosion on chat specifically — if q2 drops chat τ below ~2.5, M3 could be break-even there while code/math (higher τ) stay clearly positive. One more reason τ-under-q2 is the first number to measure.

---

Still needs a runtime check (not blocking the loader)

• τ under q2 base. The acceptance-rate hit from q2-perturbed injection hiddens is the one number neither the paper (GPU/fp8) nor static analysis gives us. Produced by the step-3 isolated-backbone logit check + a short A/B vs MTP-1.
• Non-causal block mask inside our window-attention kernel. Expressible (the kernel already takes explicit allowed_masks for the indexer), but confirm the γ=5 dense block composes cleanly with the raw window.
• safetensors index only needed to locate shards / confirm stored dtypes for the converter — no architectural unknowns remain.

---

Suggested sequencing (each step independently verifiable)

1. Converter: mtp.* safetensors → ds4 GGUF (ds4_dspark_weights struct). Verify: tensor round-trip.
2. Draft backbone in Metal, validated in isolation against the PyTorch reference logits. Verify: max-abs logit diff < 1e-2 @ fp16.
3. Markov + confidence heads. Verify: identical draft tokens / prefix-cut at temp 0.
4. Wire into the speculative loop behind --draft dspark, reusing frontier/rollback. Verify: token-identical to non-spec greedy decode (spec decoding is lossless — any divergence is a bug; free oracle).
5. Bench with speed-bench + DS4_MTP_TIMING, A/B vs MTP-1 at short and ~12 k context, code/math vs chat. Sweep draft quant (q8/q4/iq2) for the RAM-vs-τ curve.

---

Evidence sources

• Draft module reference (authoritative for structure & tensor names):
  inference/model.py — classes DSparkBlock, DSparkAttention, DSparkMarkovHead, DSparkConfidenceHead, Transformer.forward_spec, get_dspark_topk_idxs.
  huggingface.co/deepseek-ai/DeepSeek-V4-Flash-DSpark → inference/
• Hyperparameters: inference/config.json (keys: n_mtp_layers, dspark_block_size, dspark_markov_rank, dspark_target_layer_ids, dspark_noise_token_id, compress_ratios, expert_dtype, dtype).
• Base architecture / quant: deepseek-ai/DeepSeek-V4-Flash/config.json (num_nextn_predict_layers: 1, quantization_config, compress_ratios).
• Algorithm reference (training-side, qwen3/gemma4 only — not V4): github.com/deepseek-ai/DeepSpec, deepspec/modeling/dspark/markov_head.py, eval/dspark/draft_ops.py, eval/dspark/confidence_head.py. Useful for the loss/calibration story; the V4 module itself is in the HF inference/ code above.
• Paper: DSpark, arXiv 2606.19348 (semi-autoregressive generation §3.1; confidence-scheduled verification §3.2 — note the hardware-aware scheduler is a multi-request serving optimisation and degenerates to a single-request prefix-cut for our use case).
• ds4 anchors for reuse: ds4.c — metal_graph_eval_mtp_draft_from_hc, ds4_session_eval_speculative_argmax, ds4_mtp_weights, spec_frontier_*, DS4_MTP_KEEP_ACCEPTED, mtp_margin. Bench: speed-bench/. Quant tooling: gguf-tools/.

All checkpoint files dated 27 Jun 2026, MIT-licensed. Figures (per-layer ≈6.5 B params; q8≈20 GB / q4≈11 GB / iq2≈6 GB) are derived from the config dims and standard bpw; treat as estimates pending the safetensors index and a real conversion.

[lobanov]

Thanks @daaain, Claude is spot on. I don't have an implementation yet, only a benchmarking rig. Rejection sampling with batch verifier is what I also got to after some deliberation. I'll try to get it to work.

I've been pondering building a DFlash+DDTree spec decoding for ds4 for some time, but what stopped me (aside from limited spare time) is lack of credible DFlash drafter for the model. Training one would take weeks on a MacBook, or $100s of cloud compute. This is no longer the case as DSpark model variant ships with one now, so no excuses!