[Feature] Add response_prefix parameter to audio transcription/translation endpoints

[TheCodeWrangler]

Summary

• Adds an optional response_prefix field to both /v1/audio/transcriptions and /v1/audio/translations endpoints
• When provided, the text is injected into the assistant turn of the prompt so the model continues generation from that point — as if it had already produced those words
• Enables callers to implement streaming-quality transcription over REST without WebSocket infrastructure

This implements Option A from RFC #35908.

How it works

For Qwen3-ASR, response_prefix is appended after the <asr_text> tag in the assistant turn, matching the prompt shape used by the Qwen3-ASR SDK's streaming_transcribe:

<|im_start|>assistant
language English<asr_text>{response_prefix}

A caller implements the streaming loop as:

1. POST audio[0:2s], response_prefix=""               → T1
2. POST audio[0:4s], response_prefix=rollback(T1)     → T2
3. POST audio[0:6s], response_prefix=rollback(T2)     → T3
...

The caller handles audio accumulation, rollback, and prefix capping — the server just runs a single transcription with the given prompt.

Changes

File	Change
vllm/config/speech_to_text.py	Add response_prefix: str = "" to SpeechToTextParams
vllm/entrypoints/speech_to_text/transcription/protocol.py	Add response_prefix to TranscriptionRequest and pass it through
vllm/entrypoints/speech_to_text/translation/protocol.py	Add response_prefix to TranslationRequest and pass it through
vllm/model_executor/models/qwen3_asr.py	Sanitize response_prefix and append after <asr_text> tag in the assistant turn

response_prefix rides on the existing SpeechToTextParams dataclass plumbing (the refactor from #36268), so no SupportsTranscription interface change is needed and other ASR backends pick up the new field as a no-op default "" automatically.

Companion PR

This sits on top of the same Qwen3-ASR prompt-handling surface as #35415 (prompt / request_prompt). The prompt sanitizer (_sanitize_transcription_user_text) is shared between the two — response_prefix is sanitized identically so a malicious prefix cannot escape the assistant role and inject control tokens. After #35415 lands, this PR rebases to a much smaller diff.

Related

• RFC: [RFC]: Model-specific realtime streaming abstraction #35908
• Original streaming issue: [Enhancement]: Qwen3-ASR realtime endpoint produces degraded output — stateless segments, no cross-segment context, raw format leaks #35767
• Qwen ASR initial release examples of "streaming" rely heavily on prefixing the prompt of audio as it accumulates [ref]. This allows for a similar implementation to be achieved while hosting in vllm.

Test plan

• Pre-commit hooks pass (ruff, mypy, typos, etc.)
• DCO sign-offs match author email on every commit
• Built Docker dev image with patched files and started Qwen3-ASR-1.7B server
• Verified baseline transcription still works (no response_prefix)
• Verified response_prefix is accepted and changes model output
• Sanitizer test added on companion PR feat(qwen3-asr): support prompt parameter in v1/audio/transcriptions #35415 covers the strip path used here too
• Upstream CI (pre-run-check is the label gate, not a code failure)

Notes

• AI assistance (Cursor) was used; every line was reviewed and tested by the human submitter before push.

[gemini-code-assist]

Code Review

This pull request introduces a response_prefix parameter to the audio transcription and translation endpoints, enabling callers to guide the model's generation and implement streaming-like functionality. The changes are well-structured, updating the API protocol, plumbing the new parameter through the serving layer, and modifying the SupportsTranscription interface. The core logic is implemented for the Qwen3-ASR model, while other models are updated for interface compatibility. Additionally, the request_prompt is now correctly used as a system message for Qwen3-ASR. However, the implementation in the Qwen3-ASR model is vulnerable to prompt injection as it directly concatenates user-supplied strings into a structured prompt without sanitizing model-specific control tokens. This could allow an attacker to manipulate the model's behavior by injecting arbitrary turns into the prompt.

[TheCodeWrangler]

tested locally.

Results looked the way I expected. The response picks up from where the response prefix finishes (even if the response prefix started with words prior to the audio segment)

The prompt guided the model to corrected transcription of "phonograph"

curl -s -X POST http://localhost:8000/v1/audio/transcriptions   -F "file=@mary_had_lamb.ogg"   -F "model=Qwen/Qwen3-ASR-1.7B"   -F "language=en"   -F "response_prefix=A bunch of text that may have occurred prior to this audio starting. One of the first wo
rds" -F "prompt=A phonograph plays audio" | python3 -m json.tool

{
    "text": " I spoke in the original phonograph, a little piece of practical poetry. Mary had a little lamb, its fleece was white as snow, and everywhere that Mary went, the lamb was sure to go.",
    "usage": {
        "type": "duration",
        "seconds": 16
    }
}

[mergify]

Hi @TheCodeWrangler, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[mergify]

Hi @TheCodeWrangler, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[mergify]

Hi @TheCodeWrangler, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[TheCodeWrangler]

Update (prompt injection): prompt and response_prefix are sanitized in qwen3_asr.py via _sanitize_transcription_user_text(), which strips any <|...|>-shaped ChatML fragments and the <asr_text> tag before embedding user text in the system or assistant turn. Latest push tightens this to a regex over all such tokens rather than a fixed short list. Same trust model as other user-controlled OpenAI API strings.

[DarkLight1337]

@Isotr0py @NickLucche what do you think of this? I haven't used audio models much so couldn't comment on the use case

[TheCodeWrangler]

@Isotr0py @NickLucche what do you think of this? I haven't used audio models much so couldn't comment on the use case

Thanks for pinging the right reviewers!