[Model] Qwen3-ASR realtime: SDK-style streaming with cross-segment context

[TheCodeWrangler]

Summary

The Qwen3-ASR realtime WebSocket endpoint (/v1/realtime) currently transcribes each audio segment in complete isolation — no cross-segment context, no output post-processing, and the input_stream feedback mechanism accepted but never consumed. This produces significantly degraded transcription compared to the REST batch endpoint, with repeated content at segment boundaries and raw language English<asr_text> format leaking to clients.

This PR rewrites the realtime streaming logic to match the official Qwen3-ASR SDK streaming approach, where each inference step sends all accumulated audio plus a text prefix of previously-decoded output (with a small rollback to let the model re-decide boundary tokens).

Addresses #35767.

What changed

Model layer (qwen3_asr_realtime.py):

• buffer_realtime_audio now accumulates all audio (not fixed 5s segments) and re-infers over the full buffer each step
• Wires up the input_stream feedback loop: generated tokens flow back via _collect_generation → _rollback_prefix → next segment's prompt prefix
• Prefix rollback drops the trailing N tokens before feeding back, so the model can correct word boundaries and punctuation at segment junctions
• _cap_prefix_tokens prevents unbounded prefix growth

Connection layer (connection.py):

• Implements holdback: only the "stable" portion of each segment (minus trailing rollback tokens) is sent as a transcription.delta — the uncertain tail is held back until the next segment confirms it or stream end flushes it
• Detects segment boundaries via prompt_token_ids length changes and EOS tokens
• Passes full prefix strings (not just lengths) through a shared deque so delta deduplication tokenizes identically to the model's rollback

Session configuration (protocol.py, serving.py):

• Adds RealtimeSessionConfig dataclass with per-session tunable parameters
• Clients can set segment_duration_s, rollback_tokens, unfixed_chunks, max_prefix_tokens, max_audio_s, and realtime_max_tokens via session.update

Engine fix (async_llm.py):

• AsyncLLM.generate() no longer terminates the overall stream when an individual segment finishes with finished=True — only the STREAM_FINISHED sentinel ends the generation loop

Bug fix (qwen3_asr.py):

• Guards mm_options.get("audio") against None to prevent AttributeError when mm_options is not provided

Streaming architecture

Client (WebSocket)          Connection              Model (buffer_realtime_audio)
─────────────────          ──────────              ────────────────────────────
audio chunks ──────────►  audio_queue  ──────────►  Qwen3ASRRealtimeBuffer
                                                         │
                           ◄─── StreamingInput ◄────── yield TokensPrompt(all_audio + prefix)
                                                         │
engine.generate() ◄────────────────────────────────────  │
     │                                                   │
     ├── output tokens ──► input_stream.put(tok_ids) ──► _collect_generation()
     │                         │                              │
     │                    holdback_rollback()            _rollback_prefix()
     │                         │                              │
     │                    delta to client              prefix for next segment
     │                                                        │
     └── segment finished ─► input_stream.put([]) ──────────► │ (sentinel → next iteration)

Session configuration parameters

All parameters are optional and have sensible defaults. They can be set per-session via session.update:

• segment_duration_s (default: 2.0) — seconds of new audio to accumulate before triggering re-inference
• rollback_tokens (default: 5) — tokens dropped from the end of the previous transcription before it becomes the next segment's prefix; higher values give the model more room to revise boundaries but increase latency
• unfixed_chunks (default: 2) — initial segments that skip prefix rollback entirely (the model has too little context to benefit from it)
• max_prefix_tokens (default: 1024) — hard cap on prefix length in tokens to bound prompt size
• max_audio_s (default: 300) — max accumulated audio duration before trimming oldest samples
• realtime_max_tokens (default: 128) — max generation tokens per segment
• language — ISO-639-1 code (e.g. "en") for the language {Lang}<asr_text> prompt prefix
• prompt — free-text context hint injected as a system message (e.g. "user talking about their org/organization")

Test plan

• Manual WebSocket streaming test with 30s audio clip at various speeds (1x, 2x, 5x, 15x realtime)
• Verify deltas concatenate cleanly without character-level artifacts
• Verify session.update parameters propagate through and affect streaming behavior
• Verify REST /v1/audio/transcriptions endpoint is unaffected
• Verify the mm_options null-check fix doesn't regress dummy input generation

Made with Cursor

[gemini-code-assist]

Code Review

This pull request introduces a significant improvement to the Qwen3-ASR realtime streaming functionality by adopting the SDK-style cross-segment context approach, which should substantially enhance transcription quality. However, a high-severity prompt injection vulnerability was identified in the prompt construction logic, as untrusted user input from the session configuration is directly embedded into the model's prompt without sanitization, allowing for potential manipulation of the model's behavior via ChatML delimiters. Furthermore, the review also identified high-severity issues related to error handling that could mask bugs and an inconsistent default value for segment duration. Addressing these points will be crucial for the robustness, correctness, and security of this new implementation.

[oktrained]

Hello, when will I be able to test the realtime transcription? The state-maintained approach looks like a good one, as it matches the implementation used in the Qwen realtime demo. I also observed the same thing as you the current realtime endpoint stateless does not produce correct output.

[TheCodeWrangler]

Hello, when will I be able to test the realtime transcription? The state-maintained approach looks like a good one, as it matches the implementation used in the Qwen realtime demo. I also observed the same thing as you the current realtime endpoint stateless does not produce correct output.

The implementation from qwen directly is just monotonically increasing the audio window sent to ASR and then on time window T+1 it uses a prefix of the output of T but strips the last 5 tokens.

This will not work for realtime in use cases where audio can be arbitrarily long. For that reason we cannot just package the implementation from the qwen3 asr demo.

A custom wrapping implementation would be needed, but I think the vllm community needs broader input into how that should be managed within vllm.

[lifeiteng]

@TheCodeWrangler

Could you provide a quick update on the current status? Specifically, are there any remaining blockers or specific areas where the community could help with testing?

Looking forward to seeing this merged!

[TheCodeWrangler]

Closing this draft to focus attention on the smaller-scoped Qwen3-ASR work in #35415 and #36018. The cross-segment streaming approach here still seems right to me and the rebased code lives on TheCodeWrangler/vllm:fix/qwen3-asr-realtime-improvements if anyone wants to pick it up. Thanks @lifeiteng @oktrained for the interest along the way.