fix(dflash): let the drafter see its trained context window (deep-context accept) - #740
Open
dusterbloom wants to merge 8 commits into
Open
dusterbloom wants to merge 8 commits into
dusterbloom wants to merge 8 commits into
Conversation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 10 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 4 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Graffioh
approved these changes
Sep 18, 2026
Port of eafc5de (never reached main). The feature-mirror cap was min(draft_ctx_max=4096, max_ctx, target_feat_cap) and the draft window was sub-floored at max(2048, draft_ctx_max), so beyond ~4K committed the drafter saw only a shallow suffix and spec-decode accept collapsed (~0% at 8K+). - mirror cap = min(max_ctx, DFLASH_DRAFTER_TRAINED_CTX=40960), the drafter's trained SWA window (constant in dflash_feature_ring.h) - drop the 2048/draft_ctx_max sub-floor in the qwen35 + qwen35moe spec loops and the shared helper: draft_ctx = min(committed, ring_cap) - default DFLASH_FEATURE_DTYPE to q4_0 (accept-neutral) so the deep ring is affordable; f32/f16/bf16/q8_0 still selectable Verification (accept at 8K/16K on RTX 3090) pending a build.
Adversarial review of the port found two blockers: - The mirror was raised to 40960 but the target feature ring (the mirror's source) stays 4096 slots, so restore/backfill for positions older than 4096 maps onto other tokens' features (aliased, silent). Raise TARGET_FEAT_CAP_DEFAULT to the same trained window so source and mirror span the same range. - copy_host_capture_slice_to_draft_ring wrote hidden*sizeof(float) bytes per row with F32 offsets regardless of storage_type; with the q4_0 default that is ~4x past the row and OOB for the last slot (layer-split host capture). Convert each row to the ring storage type and offset by ggml_row_size.
M2: one dflash_drafter_window() now derives the effective window (trained 40960, clamped by max_ctx) for all three feature producers — the target ring, the single-GPU mirror, and the layer-split mirror — so they cannot desync, and DFLASH_FEAT_RING_CAP is restored as the memory valve (lower clamps both rings). The layer-split path no longer uses the shallow draft_ctx_max window. M1: copy_feature_ring_range_to_tensor dequantises the whole contiguous run in one ggml_get_to_fp32_cuda call instead of one launch per row (the q4_0 mirror can span 40960 rows).
Adds test_dflash_window.cpp (trained cap, max_ctx clamp, DFLASH_FEAT_RING_CAP valve) and documents DFLASH_FEAT_RING_CAP / DFLASH_FEATURE_DTYPE in ENVIRONMENT.md.
…mate Address cubic review on the PR: - dflash_spec_decode: draft_ctx_max is a REMOTE-only cap (non-positive = no cap); the local draft reads the full ring, so the shared 4096 default no longer starves the deep window on layer-split. - default DFLASH_FEATURE_DTYPE back to f32 (lossless; the IPC draft ring is F32-only) and document q4_0 as opt-in. - concurrent_fixed_cache_bytes uses dflash_drafter_window(max_ctx) so the auto-memory estimate matches the enlarged target ring. - ENVIRONMENT inventory entry for DFLASH_FEAT_RING_CAP.
dusterbloom
force-pushed
the
fix/draft-ctx-floor
branch
from
September 21, 2026 10:05
e413965 to
2c18559
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR — The drafter's feature window was capped at
draft_ctx_max(default 4096) and its source ring was only 4096 slots, so spec-decode accept collapsed once the committed prefix exceeded ~4K. This sets the window to the drafter's trained 40960, with one shared helper feeding the target feature ring, the single-GPU mirror and the layer-split mirror, and restoresDFLASH_FEAT_RING_CAPas the memory valve. The F32 feature default is kept (lossless; the IPC draft ring is F32-only) andq4_0is an opt-in. Two latent overflow/aliasing bugs found in adversarial review of the same path are fixed too.What
Port of
eafc5de51("fix feature-mirror cap to drafter trained window, "), which never reachedmain. On currentmain:qwen35_backend.cpp:661capped the mirror atmin(draft_ctx_max=4096, max_ctx, target_feat_cap);max(2048, draft_ctx_max)in the qwen35/qwen35moe spec loops and the shared helper.Beyond ~4K committed the drafter saw only a shallow suffix, so accept decayed to ~0%.
Why
avg_commitis the whole point of spec decode, and it is gated by how much context the drafter can read. A 4K window on a 16-40K agentic prefix makes the drafter blind to the tokens that matter, and the adaptive controller correctly floors it to plain decode. The window should be the drafter's trained sliding window (named constantDFLASH_DRAFTER_TRAINED_CTX = 40960), not an unrelated 4096 default.Changes
common/dflash_feature_ring.h:DFLASH_DRAFTER_TRAINED_CTX = 40960+dflash_drafter_window(max_ctx)=min(max_ctx, trained), lowered byDFLASH_FEAT_RING_CAPif set. All three feature producers use it (target ring, mirror, layer-split mirror) so their windows cannot desync.qwen35_backend.cpp,qwen35moe_backend.cpp,common/dflash_spec_decode.cpp:draft_ctx = min(committed, ring_cap)— no 2048/draft_ctx_maxsub-floor.qwen35_target_graph.cpp: target feature ring cap follows the same helper (was 4096).common/dflash_feature_ring.cpp:copy_host_capture_slice_to_draft_ringnow converts to the ring storage type instead of writing F32-width rows (the opt-in q4_0 ring would otherwise overflow the row on the layer-split host-capture path);DFLASH_FEATURE_DTYPEstays f32 by default (q4_0 is opt-in).copy_feature_ring_range_to_tensor: quantised mirror runs dequantise in one bulk call instead of one launch per row.Before / After
RTX 3090 (24 GB), Qwen3.6-35B-A3B UD-Q3_K_XL +
qwen3.6-35b-a3b-dflash-new-f16-fixed, temp 0, greedy,--max-ctx 32768 --kvflash 8192:main)Peak VRAM 21.6 GiB (f32 default), no OOM;
[dflash-feature] mirror dtype=f32 cap=32768. Honest note: the after column is measured on the 3090; the before column is the documented 4K-floor failure mode (verified by inspection + the historical commit's own 8K accept 0%->44.8% measurement), not a fresh A/B — no baseline binary was built.Tests
test_dflash_window.cppcoversdflash_drafter_window: trained cap,max_ctxclamp, and theDFLASH_FEAT_RING_CAPvalve (unset/0/negative/high all fall back to the trained window). Runs intest_server_unit.Caveats
--draft-ctx-maxremains the remote-draft / layer-split draft-context knob; the local feature window is now governed by the trained constant +DFLASH_FEAT_RING_CAP.avg_commit ≈ 1.0(little net speedup); the fix restores depth-stable accept, it does not by itself make the drafter profitable on every workload.