Skip to content

Add VoxMel TTS and voice cloning - #1353

Merged
0xShug0 merged 2 commits into
mainfrom
feature/voxmel
Nov 20, 2026
Merged

0xShug0 merged 2 commits into
mainfrom
feature/voxmel

Conversation

@0xShug0

@0xShug0 0xShug0 commented Nov 20, 2026 •

Copy link
Copy Markdown
Owner

Adds the voxmel_tts family for VoxMel 0.6B English/Mandarin TTS and reference-audio voice cloning. Generation is offline, with framework text chunking for long text. F16 and Q8 packages include the tokenizer, frontend constants and F32 Vocos; F16 is the default and the spec status is supported.

Reuses the existing causal decoder, Transformer/FFN modules, device denoiser, Vocos, RNG, text chunker and spec-v1 loader. Graphs are session-owned with bounded retention. No shared framework or GGML implementation changes.

Performance

RTX 5090 / Ryzen 9 7900X, eight threads, Debug build. Official English and Mandarin cloning examples, seed 2026, default 32 sampling steps. Results are medians of three measured requests after one warmup in the same session. All runs were sequential. Generation time excludes model loading and HTTP transport; process peak memory includes loading, warmup and measured requests.

Matched-FP32 CUDA comparison: model and vocoder weights are F32, C++ attention is eager, and Python uses its official default uncompiled inference path. Wall-time speedup is calculated from generation latency; RTF uses each output's actual duration.

Example Python RTF C++ RTF Wall-time speedup Peak VRAM Python → C++
English 0.76793 0.29154 2.70× 4,298 → 3,728 MiB
Mandarin 0.74221 0.29077 2.60× 4,828 → 4,008 MiB

Native release packages use F16/Q8 model weights, F32 Vocos and default auto attention. These results are separate from the matched-FP32 comparison above. Memory columns show the maximum process peak across the two examples; RAM is peak RSS.

Backend Package English RTF Mandarin RTF Peak VRAM (MiB) Peak RAM (MiB)
CUDA F16 0.22520 0.21643 2,436 1,177
CUDA Q8 0.20800 0.19946 2,020 990
Vulkan F16 0.47630 0.46758 2,198 1,156
Vulkan Q8 0.47896 0.46394 1,810 906
CPU F16 1.65270 1.51210 0 2,259
CPU Q8 1.25459 1.14775 0 1,835

RTF below one is faster than real time. CPU remains slower than real time on this machine. The model is offline-only; no streaming TTFA is reported.

Validation

  • Independent FP32 replay covers exact tokenizer IDs, prefix embeddings, 206 teacher-forced AR steps and matching stop decisions, flow inputs/noise, residual contexts and reconstruction.
  • CPU propagated waveform cosine against Python: 0.9999944. CUDA: 0.9998667, with 0.9999907 when Vocos receives the identical Python Mel. The documented CUDA drift is accepted; the strict 0.99999 waveform test gate remains unchanged.
  • CUDA, Vulkan and CPU English/Mandarin outputs match checked ASR transcripts. All eight measured CPU/Vulkan F16/Q8 outputs have 0% English WER / Mandarin CER after punctuation normalization. F16/Q8 CUDA and unconditioned TTS outputs also transcribe correctly.
  • Seed A/B/A, reference/shape changes, guidance changes and real multi-chunk generation return byte-identical original outputs. FP32 mixed-shape CUDA reuse peaks at 3,812 MiB without an upward resident-memory trend.
  • Debug C++17 build, spec checks and loader/catalog synchronization pass. Family, source paths, option prefixes and embedded GGUF specs consistently use voxmel_tts.

Includes conversion and independent replay tests, usage documentation and license attribution. Weight files and generated fixtures are not committed. The F16/Q8 packages target audio-cpp/audio.cpp-gguf.

@0xShug0 0xShug0 added the new model Request for new model support label Nov 20, 2026
@0xShug0
0xShug0 merged commit 0906d8f into main Nov 20, 2026
7 checks passed
@0xShug0
0xShug0 deleted the feature/voxmel branch November 20, 2026 19:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

new model Request for new model support

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant