[recipe] qwen2audio & kimiaudio sft on wenetspeech#96
Conversation
WalkthroughThis update introduces support for finetuning and evaluating the Qwen2-Audio-7B and Kimi-Audio-7B models on ASR tasks. It adds new configuration files, expands documentation with results and observations, generalizes the training/evaluation script for multiple models, and makes minor code adjustments for output consistency and device compatibility. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant Script
participant ModelConfig
participant Trainer
participant Inference
participant Evaluator
User->>Script: Select model (Qwen2/Kimi)
Script->>ModelConfig: Load model config JSON
Script->>Trainer: Start training with config
Trainer->>Script: Save checkpoints
Script->>Inference: Run inference on test sets
Inference->>Evaluator: Generate transcriptions
Evaluator->>Script: Compute and report WER
Possibly related PRs
Suggested reviewers
Poem
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (1)
examples/audio/sft/asr/wenetspeech/config/Kimi-Audio-7B-Instruct.json (1)
34-37: Same sliding-window inconsistency as base Kimi configSee comment on
Kimi-Audio-7B.json. Align the two fields for clarity.
🧹 Nitpick comments (5)
README.md (1)
15-18: Fix formatting issues in the Latest News sectionStatic analysis identified several formatting issues that should be addressed:
Apply these fixes:
## Latest News 🔥 -- [2025/07/07] We support finetuning `Qwen2-Audio-7B` & `Kimi-Audio-7B` on ASR task! See [WeneSpeech results](examples/audio/sft//asr/README.md) for details. +- [2025/07/07] We support finetuning `Qwen2-Audio-7B` & `Kimi-Audio-7B` on ASR tasks! See [WenetSpeech results](examples/audio/sft/asr/README.md) for details.Changes made:
- "task" → "tasks" (plural form)
- "WeneSpeech" → "WenetSpeech" (correct spelling)
- Fixed double slash in path: "//asr" → "/asr"
examples/audio/sft/asr/README.md (2)
6-6: Fix minor punctuation/formatting of external link note“NOTE:” is followed by two spaces before the sentence continues. Markdown renders fine, but remove the extra spaces for consistency and to avoid triggering linters.
22-26: Correct typos & agreement in the observation listSeveral small language glitches slipped in (e.g. “Finetuning … get”, “benifits”, missing article). Tighten wording – it improves credibility of the experimental write-up.
-1. Comparing 0 & 1, Finetuning Qwen2-Audio on wenetspeech training set get much better results. -2. Comparing 1 & 2, Pretraining benifits a lot on downstream task. +1. Comparing 0 & 1, finetuning Qwen2-Audio on the WenetSpeech training set **gets** much better results. +2. Comparing 1 & 2, pre-training **benefits** downstream tasks substantially.(Apply similar grammar fixes to items 3-5.)
examples/audio/sft/asr/wenetspeech/run.sh (1)
254-280: Huge positional-parameter block is unquoted – can break on whitespaceInside the
if [ "${model_type}" = "touch_audio" ]; then … fisubstitution, the multi-lineechocontains many unquoted expansions (${stack}${stride}). If either variable is empty, trailing spaces collapse and options shift.Recommend switching to a here-document or quoting the whole string:
cat <<EOF --lr_scheduler_lr_min 0.0 \ --audiofeat_stack_length ${stack} \ --audiofeat_stride_length ${stride} \ … EOFexamples/audio/sft/asr/wenetspeech/config/Kimi-Audio-7B.json (1)
34-37:sliding_windowset butuse_sliding_windowis falseYou specify
"sliding_window": 4096while"use_sliding_window": false.
Either drop the size or enable the feature; keeping both is confusing and may be ignored by downstream libraries.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (10)
README.md(1 hunks)examples/audio/sft/asr/README.md(1 hunks)examples/audio/sft/asr/wenetspeech/config/Kimi-Audio-7B-Instruct.json(1 hunks)examples/audio/sft/asr/wenetspeech/config/Kimi-Audio-7B.json(1 hunks)examples/audio/sft/asr/wenetspeech/config/Qwen2-Audio-7B.json(1 hunks)examples/audio/sft/asr/wenetspeech/run.sh(4 hunks)touchnet/models/kimi_audio/modeling_kimi_audio.py(1 hunks)touchnet/models/kimi_audio/processing_kimi_audio.py(1 hunks)touchnet/models/qwen2_audio/__init__.py(3 hunks)touchnet/utils/metrics.py(1 hunks)
🧰 Additional context used
🪛 LanguageTool
examples/audio/sft/asr/README.md
[grammar] ~6-~6: There might be a problem here.
Context: ...used the same WER calculation method as SpeechIO. ## WenetSpeech | exp id | model | note | instruct...
(QB_NEW_EN_MERGED_MATCH)
[grammar] ~22-~22: Insert the missing word
Context: ...5 del, 8927 sub ] | 1. Comparing 0 & 1, Finetuning Qwen2-Audio on wenetspeech tr...
(QB_NEW_EN_OTHER_ERROR_IDS_32)
[grammar] ~22-~22: Use articles correctly
Context: ...Comparing 0 & 1, Finetuning Qwen2-Audio on wenetspeech training set get much better...
(QB_NEW_EN_OTHER_ERROR_IDS_11)
[grammar] ~22-~22: Make sure you are using the right part of speech
Context: ...Qwen2-Audio on wenetspeech training set get much better results. 2. Comparing 1 & 2...
(QB_NEW_EN_OTHER_ERROR_IDS_21)
[grammar] ~23-~23: Ensure spelling is correct
Context: ...uch better results. 2. Comparing 1 & 2, Pretraining benifits a lot on downstream task. 3. Comparing ...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
[grammar] ~23-~23: Make sure to use plural and singular nouns correctly
Context: ...retraining benifits a lot on downstream task. 3. Comparing 3 & 4, Kimi-Audio-7B-Inst...
(QB_NEW_EN_OTHER_ERROR_IDS_10)
[style] ~24-~24: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...g benifits a lot on downstream task. 3. Comparing 3 & 4, Kimi-Audio-7B-Instruct is highly...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
[uncategorized] ~26-~26: Do not mix variants of the same word (‘pre-train’ and ‘pretrain’) within a single text.
Context: ...an order-of-magnitude difference in the pre-training data between Qwen2-Audio-7B and Kimi-Au...
(EN_WORD_COHERENCY)
README.md
[grammar] ~17-~17: Insert the missing word
Context: ...> ## Latest News 🔥 - [2025/07/07] We support finetuning Qwen2-Audio-7B & `Kimi-Audi...
(QB_NEW_EN_OTHER_ERROR_IDS_32)
[grammar] ~17-~17: Make sure to use plural and singular nouns correctly
Context: ...wen2-Audio-7B&Kimi-Audio-7B` on ASR task! See [WeneSpeech results](examples/audio...
(QB_NEW_EN_OTHER_ERROR_IDS_10)
[grammar] ~17-~17: Use correct spacing
Context: ...(examples/audio/sft//asr/README.md) for details. ## Overview 👆 touchnet is highly motiv...
(QB_NEW_EN_OTHER_ERROR_IDS_5)
🪛 Shellcheck (0.10.0)
examples/audio/sft/asr/wenetspeech/run.sh
[warning] 125-125: Quote this to prevent word splitting.
(SC2046)
[warning] 205-205: Quote this to prevent word splitting.
(SC2046)
[warning] 254-280: Quote this to prevent word splitting.
(SC2046)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: unit-test (ubuntu-22.04, 3.10.16)
🔇 Additional comments (6)
touchnet/models/kimi_audio/processing_kimi_audio.py (1)
66-66: LGTM - Instruction format improvementAdding the colon to the instruction string aligns with standard prompt formatting conventions and improves consistency.
touchnet/utils/metrics.py (1)
93-93: LGTM - Device support expansionAdding support for the "H20Z" device variant with the same peak FLOPS as H200 is appropriate.
touchnet/models/qwen2_audio/__init__.py (1)
198-198: Attribute rename verified across codebaseSearch results show no remaining
audio_token_idreferences and only three uses ofaudio_token_indexintouchnet/models/qwen2_audio/__init__.py:• Line 198
• Line 207
• Line 226Please ensure your configuration class defines
audio_token_index(replacing the oldaudio_token_id) to prevent runtime errors.touchnet/models/kimi_audio/modeling_kimi_audio.py (1)
1069-1081: Ignore audio_logits removal – no downstream usage detectedA search across the repository shows that
audio_logitsonly appears in internal modeling methods (e.g.,sample_audio_logits) and is never unpacked or referenced by any calling or inference code. Removing it from theforwardoutput does not break any existing code paths.• All
rg "audio_logits"hits are inmodeling_kimi_audio.py(lines around 748, 1067, 1178–1184).
•inference_kimi_audio.pyinstantiates the model and calls.generate()orsample_audio_logits, but never unpacksaudio_logitsfromforward.
• No other modules referenceaudio_logitsin output tuples or expect two logits inforwardresponse.You can proceed with ASR-only support and reintroduce
audio_logitslater for TTS without backward-compatibility concerns.Likely an incorrect or invalid review comment.
examples/audio/sft/asr/wenetspeech/run.sh (1)
185-187:--local-ranks-filtermismatchesnproc_per_nodeYou launch all GPUs with
torchrun --nproc_per_node=$num_gpus, yet immediately filter to"0,1,2,3,4,5,6,7".
On machines with fewer than 8 GPUs (or whenCUDA_VISIBLE_DEVICEShides some), this silently drops ranks and hangs. Either:
- Drop
--local-ranks-filter, or- Derive the list dynamically from
$gpu_list.[further verification recommended]
examples/audio/sft/asr/wenetspeech/config/Qwen2-Audio-7B.json (1)
24-25: Confirmaudio_token_indexwithin vocabulary range
audio_token_index=151646<vocab_size=156032– OK.
Just a quick sanity check; no action needed.
1 participant
Summary by CodeRabbit
New Features
Documentation
Bug Fixes
Style
Refactor