Skip to content

Commit 8b4d41c

Browse files
Xiaodong Yeclaude
andcommitted
MUSA-0203: in-place query_start_loc update for captured loop replay
Add PR vllm-project#34880's matching in-place hunk for the captured replay path. Without it the propose() loop reassigns: common_attn_metadata.query_start_loc = self.arange[: batch_size + 1] which creates a fresh tensor each call. The captured FA kernel baked in the pre-capture pointer at boot time and reads stale data at replay -> draft tokens past position 0 collapse (Pos 1-4 fell to 7/3/1/0.6% on yeahdongcn70 cookbook). PR vllm-project#34880 changes to: common_attn_metadata.query_start_loc[: batch_size + 1] = self.arange[...] which is an in-place write into the persistent buffer the captured graph already references. Same fix for query_start_loc_cpu. Test: reboot with VLLM_MUSA_DRAFT_FULL_WRAP=1 and rerun cookbook bench. Expected: accept rate recovers, Pos 1-4 distribution matches eager. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1 parent afd11cd commit 8b4d41c

1 file changed

Lines changed: 22 additions & 0 deletions

File tree

vllm_musa/patches/vllm__v1__spec_decode__llm_base_proposer.patch.py

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -192,6 +192,27 @@
192192
batch_descriptor=batch_desc,
193193
slot_mapping=self._get_slot_mapping(input_batch_size),"""
194194

195+
# ---- Hunk 9b: propose() loop — in-place query_start_loc/query_start_loc_cpu ----
196+
# PR #34880 changes reassignment to in-place write so the captured loop graph's
197+
# data_ptr stays valid at replay. Reassignment created a fresh tensor each call
198+
# and the captured FA kernel read from the stale pre-capture pointer, which is
199+
# the root cause of Position 1-4 accept-rate collapse on Step 4.
200+
_OLD_QSL_REASSIGN = """ common_attn_metadata.query_start_loc = self.arange[: batch_size + 1]
201+
common_attn_metadata.query_start_loc_cpu = torch.from_numpy(
202+
self.token_arange_np[: batch_size + 1]
203+
).clone()"""
204+
205+
_NEW_QSL_REASSIGN = """ # MUSA-0203 / PR #34880: in-place write keeps the captured graph's
206+
# data_ptr valid across replays (the previous reassignment created a
207+
# fresh tensor each call, stranding the captured FA kernel on a stale
208+
# pointer and collapsing accept rate at positions > 0).
209+
common_attn_metadata.query_start_loc[: batch_size + 1] = self.arange[
210+
: batch_size + 1
211+
]
212+
common_attn_metadata.query_start_loc_cpu[: batch_size + 1] = torch.from_numpy(
213+
self.token_arange_np[: batch_size + 1]
214+
).clone()"""
215+
195216
# ---- Hunk 11: full dummy_run rewrite — accept common_attn_metadata and
196217
# capture FULL-mode entries during boot. Without this, CUDAGraphWrapper
197218
# tries to capture at request time and trips validate_cudagraph_capturing_enabled.
@@ -360,6 +381,7 @@
360381
(_OLD_FFC_FIRST, _NEW_FFC_FIRST),
361382
(_OLD_PROPOSE_LOOP_DET, _NEW_PROPOSE_LOOP_DET),
362383
(_OLD_FFC_LOOP, _NEW_FFC_LOOP),
384+
(_OLD_QSL_REASSIGN, _NEW_QSL_REASSIGN),
363385
(_OLD_DUMMY_RUN, _NEW_DUMMY_RUN),
364386
(_OLD_ISINSTANCE, _NEW_ISINSTANCE),
365387
]

0 commit comments

Comments
 (0)