[Feature Request] Allow VAD to run on a different device than the ASR model (Apple Silicon MPS regression: VAD 5x slower than CPU) #3701

Activity

1. LauraGPT commented on Sep 12, 2026

   

   LauraGPT

   on Sep 12, 2026

   Collaborator

   More actions

   Thanks for the detailed report and the offer to contribute a PR. I recommend honoring the existing nested device options rather than adding a second family of vad_device/punc_device/spk_device arguments. The proposed interface is:

   # Proposed behavior, not supported correctly by current main yet.
   model = AutoModel(
       model=asr,
       device="mps",
       vad_model=vad,
       vad_kwargs={"device": "cpu"},
       punc_model=punc,
       punc_kwargs={"device": "cpu"},
   )

   I checked current main 486b4b7ceb27b72db24a84c6e1be7cf6c5be6609. A CPU-only configuration probe using the actual AutoModel orchestration and stand-in models confirmed three relevant paths:

   • Construction overwrites an explicit device="cpu" in all three submodel dictionaries. It also modifies the caller's dictionaries in place.
   • Even after establishing a CPU VAD baseline, a shared generate(device="cuda:0") config changes the device passed to VAD back to cuda:0; the next call without that override resets to CPU. This is the deep_update path in both inference_with_vad and inference, so changing only the constructor assignments would not cover it.
   • The speaker-embedding call at auto_model.py:982 receives the ASR kwargs, not self.spk_kwargs. In the probe it received the ASR device and lost a speaker-only sentinel setting, despite a CPU speaker baseline.

   For a PR, please make the contract consistent across VAD, punctuation and speaker models:

   1. An explicitly supplied submodel device takes precedence; when omitted, retain the existing inheritance from the resolved ASR device. Copy caller-owned config dictionaries before filling defaults.
   2. Treat the selected submodel device as model placement, not an ordinary shared per-call option. Preserve it through runtime config merging and repeated-call resets, keeping weights and input features on the same device. Do not imply that changing a runtime device string moves already-loaded weights; runtime migration would need a separate explicit contract.
   3. Pass the speaker model its own resolved config. Cover punctuation both with and without VAD, since both routes merge shared runtime options.
   4. Add regressions for explicit overrides, omitted-device inheritance, unchanged caller dictionaries, shared runtime options, two successive calls, and the actual per-model inference boundaries. Please also exercise unavailable-device fallback without claiming hardware execution from mocked availability.

   A PR covering this would be welcome. Please include an actual mixed-device smoke test where hardware is available, checking model/feature device agreement and successful output. Your original Apple M2 audio is particularly valuable for validating the reported performance after the fix. My probe used CPU tensors only (cuda:0 was a configuration label), so I have not independently reproduced the MPS timing figures or a hardware device-mismatch crash. No ComputeScores monkeypatch should be required by the final public API.
2. LauraGPT commented on Sep 16, 2026

   

   LauraGPT

   on Sep 16, 2026

   Collaborator

   More actions

   Update: the nested-device fix has now been merged in PR #3706, at commit 4d8e087. The earlier comment's “not supported correctly by current main yet” note describes the old base and no longer applies to this merged source.

   The public configuration is documented in the merged English and Chinese API guides:

   • An explicit nested device in vad_kwargs, punc_kwargs, or spk_kwargs is preserved; omitted values inherit the resolved ASR device.
   • Shared per-call options do not migrate already-loaded models.
   • Caller-owned submodel dictionaries are copied, and the speaker model receives its own configuration.

   I checked that all six merged PR files match the final candidate. Compared with the previously reviewed eee63614, the only follow-up change is an optional-clustering test-fixture adjustment; the production AutoModel source is unchanged.

   This is a source merge, not a new PyPI release announcement. The original M2/121-minute performance case has not been independently retested, and the author's M4 measurements remain workload-specific. Keeping this issue open for original-environment confirmation; no internal VAD replacement or ComputeScores patch should be necessary for the documented placement configuration.
3. LauraGPT commented on Sep 21, 2026

   

   LauraGPT

   on Sep 21, 2026

   Last edited by LauraGPT

   Collaborator

   More actions

   Release availability update: the mixed-device placement fix from #3706 is now included in PyPI 1.4.16, uploaded September 18. A source checkout is no longer required just to obtain this fix:

   python -m pip install "funasr==1.4.16"

   I verified the actual published wheel against PyPI's SHA-256; its funasr/auto/auto_model.py is byte-for-byte identical to the #3706 merged implementation. The matching English and Chinese API version-note correction is now merged in PR #3724, after all PR CI checks passed. Please still follow the installation guide for dependencies and record the imported funasr.__file__ when checking an existing environment.

   This supersedes only the release-availability caveat in my September 16 comment. It does not establish the original M2/121-minute performance result, which has not been independently retested; keeping this issue open for that confirmation.