Training model matrix¶
Use the model guides for one focused inference, data preparation, and training page per registered model.
This table describes what the current VoiceHub integration can fine-tune. It does not describe whether a model was trainable in its original research repository.
| Status | Meaning |
|---|---|
| End-to-end / raw data | VoiceHub provides the model-specific objective, dataset/collator, and text/audio or conversation preprocessing needed to start from ordinary training records. |
| Preprocessed recipe | VoiceHub provides the verified differentiable objective and training route, but the caller must supply model-ready tokens, codec codes, features, or source-shaped batches. |
| Specialized / partial | A component/checkpoint variant is covered, or an explicitly experimental reconstructed objective is available, but the complete author-verified family recipe is not. |
| Unavailable | The integrated runtime has no verified training graph, or the declared custom recipe is intentionally blocked until its model-specific adapter is implemented. |
native and preprocessed in the Python training specification are capability
checks used by the trainer. The table uses the more practical categories above:
a native scalar loss is still a preprocessed recipe when VoiceHub does not
provide raw-data preparation.
Dataset readiness is independently inspectable through
get_tts_dataset_spec(model_type).readiness for TTS or
get_asr_dataset_spec(model_type).readiness for ASR. Shared multi-codebook,
diffusion/flow, VITS, CTC, sequence, and transducer objective primitives make
specialized adapters easier to implement; they do not upgrade a model in this
table unless the graph, objective, optimizer schedule, checkpoint topology,
and data boundary required for the claimed support level are integrated and
tested. Complete raw-data preparation is required only for End-to-end / raw
data status.
Model-by-model support¶
Model (model_type) |
Status | Fine-tuning boundary |
|---|---|---|
Orpheus-TTS (orpheustts) |
End-to-end / raw data | Text plus 24 kHz audio, or precomputed SNAC codes, is framed as the author-style causal codec sequence. SNAC remains frozen; train the unquantized language model. |
Dia (dia) |
End-to-end / raw data | VoiceHub owns the byte tokenizer, encoder/decoder graph, nine-codebook delay protocol, teacher-forced cross-entropy, native DAC target path, strict Safetensors loader, and portable export. nari-labs/Dia-1.6B-0626 supports full fine-tuning; the original pickle/JAX Dia-1.6B layout is rejected rather than executed. |
Vui (vui) |
VoiceHub-native preprocessed recipe / reconstructed | VoiceHub owns the pinned 100M Vui/Fluac graph, byte tokenizer, delayed-codebook teacher-forcing path, and coherent model-plus-codec artifact resolver. Callers provide text token IDs and Fluac codec IDs; schema-generated masks and lengths prevent padded text or audio frames from entering the loss, and the codec stays frozen. Export writes versioned config.json, model.safetensors, and codec.safetensors files with strict graph and component metadata; a fresh Vui wrapper reloads the complete standalone artifact without the external Safetensors package. The released repository did not publish its dataset, optimizer, or training loop, so the objective is architecture-consistent rather than author-verified. |
Chatterbox (chatterbox) |
End-to-end / raw data, separate native jobs | VoiceHub owns the T3, S3Gen, S3Tokenizer, voice-encoder, audio frontend, and Perth graphs. Select training_component="language_model" for shifted T3 text/speech-token cross-entropy or "flow" for the published S3Gen conditional flow objective; the two components deliberately use separate optimizer jobs. Both accept raw PCM audio, while community-compatible precomputed tensors remain supported. Full training and native T3 LoRA export complete inference-reloadable Safetensors artifacts. The released repository did not publish one complete end-to-end optimizer recipe, so VoiceHub records the reviewed community recipe and its causal-shift, prompt-mask, padded-key, and frozen-frontend corrections explicitly. |
Kokoro (kokoro) |
Specialized / partial | enable_preprocessed_training=True activates VoiceHub's native PL-BERT, duration/prosody, text-encoder, and iSTFTNet objectives. Duration batches require phoneme IDs (or caller-supplied phonemes), a 256-D style vector, and integer durations. Acoustic batches additionally require an aligned waveform; audio_lengths masks padded samples, while a dense alignment and F0/energy targets are optional. This is a reconstructed released-graph recipe, not the unpublished raw-audio StyleTTS2 alignment, style-encoder/diffusion, discriminator, or optimizer recipe. |
Echo-TTS (echo) |
VoiceHub-native preprocessed recipe / experimental | VoiceHub owns the Echo DiT, Euler/CFG samplers, byte tokenizer, Fish-S1-DAC graph, strict Safetensors loaders, and an architecture-consistent rectified-flow velocity objective. Callers provide source-shaped codec latents and masks; the Fish codec remains frozen. The public repository is inference-only, so this is not an author-verified reproduction of the original data or optimization recipe. |
ConversationTTS (conversationtts) |
End-to-end / raw data | VoiceHub owns the pinned Llama 3.2 backbone/depth-decoder graph, Llama 3 byte-BPE tokenizer, Mimi codec, exact 33-stream text/audio framing, and published codebook-zero plus residual masked cross-entropies. Records may provide raw text and 24 kHz audio or precomputed text IDs and 32-codebook Mimi codes. Mimi is frozen, padding uses the published text/audio IDs, and trained weights export to strict inference-reloadable Safetensors. The original 9.3 GB PyTorch checkpoint is accepted only through weights_only=True; normal reload never executes pickle. Source and checkpoint use remain subject to CC BY-NC 4.0. |
LLaSA (llasa) |
End-to-end / raw data | VoiceHub's native byte-BPE tokenizer and Llama 3.2 graph convert text plus 16 kHz audio, or precomputed XCodec2 codes, into completion-only codec-LM labels. The complete native XCodec2 graph is pinned and frozen. Export writes a fresh-inference Safetensors bundle. The checkpoint is CC BY-NC 4.0. |
CosyVoice 3 (cosyvoice) |
VoiceHub-native raw-audio/preprocessed recipe / separate component jobs | VoiceHub owns the Qwen2 speech-token LM, conditional flow matcher, HiFT generator, training-only HiFi-GAN discriminator, byte-BPE tokenizer, and native frozen S3Tokenizer encoder/FSQ. Select training_component="llm", "flow", "hifigan_generator", or "hifigan_discriminator"; each uses its released objective and optimizer route. LM rows accept raw speech_audio/audio/waveform with a source rate, a PCM WAVE audio_path, or precomputed speech_tokens; the native tokenizer prepares raw targets at 16 kHz. CAMPPlus embeddings remain an explicit conditioning boundary. Only CosyVoice 3 is checkpoint-compatible; CosyVoice 1/2, streaming, JIT, TensorRT, and vLLM are not claimed. |
F5-TTS (f5tts) |
Preprocessed recipe / full released flow graph | VoiceHub owns the complete F5-TTS v1 DiT, HTK log-mel frontend, CFG flow objective, Euler/midpoint sampler, strict checkpoint loader, and Vocos decoder. Fine-tuning accepts raw waveform or [batch, frames, 100] mel input plus vocabulary IDs and optional frame lengths; Vocos stays frozen. EMA export writes a fresh-inference Safetensors artifact with its vocabulary. Raw Chinese requires caller-supplied pinyin-with-tone tokens or an explicit native normalizer because VoiceHub does not hide a non-equivalent jieba/pypinyin dependency. Source is MIT; released SWivid/F5-TTS weights are CC-BY-NC-4.0. |
GPT-SoVITS (gptsovits) |
VoiceHub-native preprocessed staged recipe for V1, V2, V2Pro, and V2ProPlus | VoiceHub owns the four audited classic-S2 families: their S1 autoregressive semantic graph, variant-exact S2 VITS/GAN generator, frozen residual quantizer, and classic or seven-period Pro discriminator. Three independently routed phases optimize S1 summed cross-entropy, the S2 generator's VITS/LSGAN/mel/KL/codebook objective, and S2 discriminator LSGAN. Callers provide variant-exact phoneme IDs, 1,024-D Chinese-RoBERTa features, 768-D CN-HuBERT features, 1,025-bin spectrograms, and aligned 32 kHz audio; Pro variants additionally require a prepared 20,480-D ERes2NetV2 speaker-verification embedding. Raw multilingual preprocessing is not approximated. All 12 published S1/generator/discriminator inventories are revision- and digest-pinned; staged variant-tagged Safetensors support fresh inference reload. V3, V4, and LoRA fail closed because their flow-matching/vocoder or PEFT-merge graphs are distinct. |
MeloTTS (melotts) |
VoiceHub-native preprocessed full-VITS recipe | VoiceHub owns the checkpoint-exact multilingual VITS2 generator and recovers the published duration, KL, mel, LSGAN, feature-matching, waveform-discriminator, duration-discriminator, and monotonic-alignment-noise schedule. Three optimizer phases update the complete seven-component generator, a fresh MPD, and a fresh duration discriminator. Callers provide exact phone, tone, language, 1,024-channel BERT, 768-channel Japanese-BERT, magnitude spectrogram, speaker-ID, and aligned waveform tensors; raw multilingual G2P/BERT extraction is not approximated. Official .pth releases require an explicit digest-pinned weights_only=True import, while steady-state training and fresh inference use Safetensors. Discriminator and optimizer state are training-only, so an inference export is a weight warm start rather than an author-resumable training checkpoint. |
OpenVoice V2 (openvoice) |
Specialized / reconstructed raw-audio recipe | VoiceHub owns the checkpoint-exact 486-tensor reference encoder, posterior encoder, flow, HiFi-GAN decoder, 22.05 kHz magnitude-STFT frontend, strict loader, and Safetensors export. With enable_reconstructed_finetuning=True, paired source/target waveforms with matching linguistic content optimize a length-masked smooth-L1 conversion objective; optional source/target references stay inside autograd so the complete converter can update. The public OpenVoice repository does not release its original converter training loop, discriminator, dataset, or loss. This path is therefore an explicit VoiceHub reconstruction—not upstream training parity—and no quality or speaker-similarity improvement is claimed. |
OuteTTS (outetts) |
Preprocessed recipe / author-verified LM objective | VoiceHub owns the Llama/Qwen causal graph, byte-BPE tokenizer, exact V3 prompt protocol, strict Safetensors loader, and frozen 24 kHz DAC. Fine-tuning accepts validated V3 speaker profiles with word timestamps, aligned c1/c2 codes, and acoustic features, or prepared input_ids/labels; completion-only causal cross-entropy updates the full language model. Raw audio fails closed because the author pipeline's word alignment and feature extraction cannot be reconstructed equivalently. Native exports include the LM, tokenizer, DAC, default speaker, and integrity manifest. GGUF, llama.cpp, EXL2, vLLM, quantized, batch, guided, server, and streaming paths are not training backends. The default 1B weights are CC-BY-NC-SA-4.0. |
Parler-TTS (parlertts) |
Native raw-audio recipe | VoiceHub owns the Parler decoder, FLAN-T5 encoder, tokenizer, delayed-codebook objective, and shared DAC runtime. Rows may provide raw description, text, audio_values, and lengths, or precomputed audio_codes. DAC is always frozen; T5, the decoder, and prompt embeddings train by default, with the source-compatible freeze_text_encoder=True memory-saving mode. Portable exports reload without Transformers. |
StyleTTS 2 (styletts2) |
VoiceHub-native preprocessed recipe / reconstructed | VoiceHub owns the released HiFi-GAN multispeaker/finetune and iSTFTNet single-speaker graphs, exact 178-symbol/PL-BERT namespace, native mel frontend, style diffusion, duration/F0/noise objectives, mel spectral convergence, and separate fresh MPD/MSD adversarial, feature-matching, and TPRLS phases. Callers provide checkpoint-compatible phoneme IDs, monotonic alignments, normalized and reference mels, F0/noise targets, and aligned 24 kHz audio with lengths. Raw text fails closed unless explicit phonemes or IDs are supplied. Export writes the deployable generator components to strict, fresh-inference Safetensors. The release does not include discriminator or optimizer state; VoiceHub does not claim author-resume semantics and intentionally excludes the WavLM objective until that exact graph and targets are available. |
MOSS-TTS (mosstts) |
End-to-end / raw data | VoiceHub owns all four published semantic graphs (Delay, Local, Local v1.5, and Realtime), the Qwen byte-BPE tokenizer and prompt processors, MOSS Audio Tokenizer v1/v2, strict Safetensors loading, and portable export. A record contains text and exactly one of raw audio/waveform/audio_path or pre-encoded [frames, n_vq] speech_tokens; optional reference audio or codes follow the active variant's protocol. The frozen native codec lazily creates targets for raw records. Fine-tuning updates the complete semantic graph with its release-specific multichannel next-token cross-entropy, including Local v1.5's binary text-control head and Realtime's 16-codebook schedule. Codec weights are deliberately outside this phase. “Realtime” is the model family name; VoiceHub does not claim incremental transport streaming. |
Qwen3-TTS (qwen3tts) |
VoiceHub-native preprocessed full SFT and LoRA | VoiceHub owns the Qwen3 talker, residual code predictor, speaker encoder, text tokenizer, and checkpoint-exact 12 Hz Mimi-derived speech encoder, semantic/acoustic residual quantizers, and decoder. It implements the official Base single-speaker SFT objective; the registered training default is Qwen/Qwen3-TTS-12Hz-1.7B-Base, distinct from the inference-oriented default. Full SFT is the default. Opt-in native LoRA freezes the complete base graph and adapts talker plus residual-predictor attention/MLP projections; it exports strict adapter-only Safetensors and a clean merged CustomVoice runtime. Training records require text, 16-codebook target audio_codes, and 24 kHz reference audio, while raw reference audio can be encoded natively for ICL inference. CustomVoice and VoiceDesign are export/inference targets, not SFT starting checkpoints. |
Irodori-TTS (irodoritts) |
End-to-end / raw data | VoiceHub owns the RF-DiT, duration predictor, unigram/byte-fallback tokenizer, frozen Semantic-DACVAE, released rectified-flow objective, duration objective, strict Safetensors loader, and export path for v2, v3, and VoiceDesign checkpoints. Training can use raw 48 kHz audio or pre-encoded continuous latents and can optimize flow, duration, or their weighted joint objective. SilentCipher watermark parity is not claimed. |
Zonos 1 (zonos) |
VoiceHub-native raw-audio/preprocessed recipe / reconstructed | VoiceHub owns the exact dense Transformer graph, conditioning stack, delay protocol, native frozen DAC path, strict Safetensors loader, and inference-reloadable export. Training accepts text plus raw waveform (with precomputed eSpeak-compatible phonemes or an injected frontend), or model-ready prefix conditioning and DAC codes. Zyphra did not publish the original data pipeline, optimizer, or objective, so the tested delayed-codebook causal cross-entropy is explicitly reconstructed. The distinct Mamba-2 hybrid checkpoint fails closed until that graph is implemented. |
ZONOS2 (zonos2) |
VoiceHub-native raw-data recipe / reconstructed | VoiceHub owns the dense/MoE acoustic language model, KV cache, UTF-8 prompting, speaker encoder, DAC integration, strict Safetensors loader, and full-model gradient path. Batches may provide raw text plus waveform values—with per-example lengths and sample rates—or cached delayed DAC codes. Zyphra has not published the original data pipeline, optimizer recipe, or loss implementation, so VoiceHub's delayed-codebook causal cross-entropy is explicitly marked reconstructed rather than author-verified. |
VoxCPM2 (voxcpm) |
VoiceHub-native raw-audio/preprocessed full SFT and LoRA | VoiceHub owns the exact 577-tensor MiniCPM/flow graph, SentencePiece-BPE tokenizer, 312-tensor AudioVAE V2 codec, published conditional-flow-matching and stop-token objectives, and official LoRA target topology. Records may provide text plus 16 kHz waveform data or pre-encoded latent patches. AudioVAE V2 is always frozen. Full-model and adapter training export a strict, fresh-inference Safetensors runtime; LoRA exports also retain an unmerged adapter artifact for reuse. Upstream publishes the codec only as a legacy archive, so its one-time digest-pinned weights_only=True conversion requires explicit trust. Streaming, external denoising, and legacy VoxCPM1 checkpoints fail closed because they are outside the verified native contract. |
OmniVoice (omnivoice) |
End-to-end / raw data | VoiceHub owns the complete bidirectional Qwen3 graph, text tokenizer, frozen Higgs Audio v2 codec, padding/packing collators, iterative masked-token generation, and the published independently normalized eight-codebook masked cross-entropy. Records may contain raw 24 kHz mono audio or pre-encoded [8, frames] codec IDs. Full-model training, gradient checkpointing, strict Safetensors reload, and portable export are supported. |
Higgs Audio (higgstts) |
End-to-end / raw data | VoiceHub owns the dual-FFN decoder, Llama 3 byte-BPE tokenizer, delayed eight-codebook protocol, native frozen 201M-parameter HuBERT/DAC audio tokenizer, strict Safetensors loading, and portable export. ChatML text/audio records optimize the source-authored sum of delayed-codebook causal cross-entropies plus optional text cross-entropy. Boson AI does not publish a complete optimizer or schedule recipe, so VoiceHub owns that orchestration without claiming author-resume parity. |
XTTS v2 (xtts) |
VoiceHub-native raw-audio/preprocessed GPT recipe | VoiceHub owns the complete autoregressive GPT, BPE tokenizer, Perceiver conditioning, speaker encoder, HiFi-GAN decoder, and separately published full DVAE encoder/codebook/decoder. Fine-tuning accepts raw waveform or precomputed DVAE acoustic codes plus conditioning mels; raw audio is encoded through the frozen DVAE under no_grad, while offline codes remain the highest-throughput option. The speaker encoder and waveform decoder stay frozen, and no vocoder/adversarial phase is claimed. Coqui's legacy model, DVAE, and mel-stat artifacts require explicit digest-checked one-time conversion; see the XTTS DVAE guide. |
VibeVoice (vibevoice) |
VoiceHub-native preprocessed recipe | VoiceHub owns the exact non-streaming microsoft/VibeVoice-1.5B language model, continuous acoustic/semantic codecs, connector, diffusion head, DPM solver, strict three-shard Safetensors adapter, and portable export. The author recipe freezes both codecs and trains the LM, connectors, and diffusion head from explicit token, speech-latent, and acoustic-mask tensors. Microsoft has not published a raw-data serializer for this recipe. The realtime 0.5B graph exposes verified low-level stages only; unified fine-tuning and the unverified cached-prompt waveform loop fail closed. |
Fish Speech S2 (fishtts) |
VoiceHub-native preprocessed full semantic FT | VoiceHub owns the exact 36-layer slow and 4-layer fast DualAR graph, byte-BPE protocol, ModifiedDAC runtime, strict checkpoint adapter, and source-aligned base-token plus residual-codebook objectives. Training accepts validated pretokenized [num_codebooks + 1, time] inputs and labels; legacy protobuf data must be converted before entering VoiceHub. The codec is a frozen offline tokenizer, not a trainable phase. Export writes semantic and codec Safetensors that reload in a fresh inference process. Fish fine-tunes remain non-commercial derivative works under the Fish Audio Research License. |
Sesame CSM (csm) |
End-to-end / raw data | VoiceHub owns the checkpoint-compatible model, byte-BPE tokenizer, processor, and source-faithful two-level backbone/depth-decoder objective. Conversation or text/audio records use the separately pinned, frozen native Mimi encoder; pre-encoded Mimi codes bypass codec preprocessing. |
NeuTTS (neutts) |
End-to-end / raw data for NeuTTS-Air | VoiceHub owns the Qwen2/Qwen3/Llama causal backbones, byte-BPE tokenizers, exact speech-token protocol, and full native NeuCodec graph. Text plus raw 16 kHz audio, or precomputed NeuCodec codes, becomes completion-only labels; the LM trains while NeuCodec remains frozen, and export includes LM, tokenizer, and codec for fresh reload. The author-verified recipe is registered only for neuphonic/neutts-air; Nano, multilingual Nano, and 2E remain native inference targets but fail closed for training. GGUF, ONNX, pickle, and distilled-codec artifacts are rejected. Phoneme variants require explicit phonemes or an injected phonemizer. |
Supertonic (supertonic) |
Preprocessed recipe / reconstructed | VoiceHub executes the four reviewed release graphs with differentiable PyTorch operations and trains duration, text, one-step flow, and vocoder losses. Supply released-format style tensors plus duration and/or latent targets. Supertone does not publish its audio/style encoders, optimizer configuration, dataset pipeline, or complete author recipe, so this is not presented as end-to-end raw-audio fine-tuning. |
Inflect (inflecttts) |
VoiceHub-native preprocessed full-VITS warm start / reconstructed | VoiceHub owns the exact Micro/Nano v2 generator graph and validates the released 410-tensor inventories. With enable_native_finetuning=True, it restores the deployable generator, freshly initializes the omitted posterior encoder and optional multi-period discriminator, then runs separate generator and discriminator phases with mel, duration, KL, adversarial, and feature-matching losses. Callers provide checkpoint-compatible English phoneme IDs, a 513-bin linear magnitude spectrogram, lengths, and aligned 24 kHz audio. This is real full-graph fine-tuning and exports an inference-reloadable Safetensors artifact, but it is a warm start—not an author-resumable checkpoint or a claim that the unpublished data/optimizer recipe was recovered. |
Bark (bark) |
VoiceHub-native preprocessed recipe | VoiceHub owns the semantic, coarse, and fine Transformer stages, multilingual WordPiece tokenizer, shared Encodec graph, generation protocol, and stage-aligned cross-entropies. The caller supplies semantic/coarse/fine token IDs, labels, masks, and the fine-codebook index; Encodec stays frozen and raw-audio tokenization remains an offline dataset step. The official release publishes only a legacy archive, so importing it requires an explicit digest-pinned weights_only=True conversion. Steady-state training and fresh-runtime export use Safetensors and never depend on Transformers. |
SpeechT5 (speecht5) |
End-to-end / raw data | VoiceHub owns the complete text-to-spectrogram graph, SentencePiece tokenizer, 16 kHz log-mel frontend, pre/postnet L1 losses, weighted stop-token BCE, and guided cross-attention loss. Mixed-rate text/audio records and optional speaker embeddings are processed natively; HiFi-GAN remains frozen. Export includes the acoustic model, processor assets, and vocoder in an inference-reloadable Safetensors bundle. |
VITS / MMS-TTS (vits) |
VoiceHub-native full adversarial recipe with an explicit acoustic-config boundary | enable_native_adversarial_training=True activates VoiceHub's native raw-waveform frontend, posterior encoder, MAS or supervised durations, flow, decoder, scale-plus-five-period discriminator, and independently optimized discriminator/generator phases. The generator combines source-style mel reconstruction, duration, KL, feature matching, and least-squares adversarial losses; detached fake audio and phase-local freezing preserve the GAN gradient boundary. MMS-TTS snapshots do not publish their source FFT, hop, window, mel, and segment settings, so callers must supply an exact training_acoustic_config instead of letting VoiceHub guess. enable_native_generator_training=True remains as an explicitly partial, precomputed-spectrogram warm-start route. Inference exports contain the generator; exact trainer resumes additionally retain discriminator and optimizer state. |
Native ASR dispatcher (asr_transformers) |
End-to-end / raw data | Dispatches strict Safetensors checkpoints to the verified VoiceHub Whisper, Wav2Vec2 CTC, HuBERT CTC, WavLM CTC, or Moonshine graph and preserves each graph's native objective and processor. RNN-T, TDT, and prompted audio-language models require dedicated model types. |
Whisper ASR (asr_whisper) |
End-to-end / raw data | Defaults to Whisper large-v3-turbo and uses the native processor plus teacher-forced sequence-to-sequence loss. |
Faster Whisper compatibility (asr_faster_whisper) |
End-to-end / raw data | Resolves CTranslate2-style model aliases to canonical Whisper Safetensors and fine-tunes the same VoiceHub-native teacher-forced graph; CTranslate2 is not used during training or export. |
WhisperX compatibility (asr_whisperx) |
End-to-end / raw data | Fine-tunes the VoiceHub-native Whisper transcription graph. Word timing is a separate, frozen native Wav2Vec2 CTC alignment pass; fine-tune that checkpoint independently through asr_wav2vec2. |
OpenAI Whisper compatibility (asr_openai_whisper) |
End-to-end / raw data | Resolves official OpenAI aliases to canonical Whisper Safetensors and uses the VoiceHub-native Whisper processor, objective, and portable export without the openai-whisper runtime. |
Tiron (asr_tiron) |
End-to-end / raw data | Fine-tunes VoiceHub's native Whisper graph from strict Safetensors. Inline targets are validated against the published speaker_blocks grammar before teacher forcing, preserving all speaker and 20 ms timestamp tokens. Whole-meeting chunking and cross-window speaker linking are separate architectures, not hidden training phases. |
Qwen3-ASR (asr_qwen3) |
End-to-end / raw data | Implements the official supervised collator with VoiceHub-owned audio and text processing: language metadata plus <asr_text> are part of the assistant completion, while system/user/audio prompt and padding positions are masked. Full audio-tower/decoder backward, native LoRA, strict Safetensors export, and fresh-runtime reload are supported. |
VibeVoice-ASR-HF (asr_vibevoice) |
VoiceHub-native end-to-end / raw data | VoiceHub owns the exact continuous acoustic and semantic encoders, multimodal projector, Qwen2 decoder, byte-BPE tokenizer, prompt renderer, causal objective, strict eight-shard Safetensors adapter, and portable export. Raw 24 kHz audio plus speaker/timestamp segments becomes assistant-completion-only labels. The published recipe keeps both speech encoders frozen while training the projector, language model, and LM head. BitNet/GGML serving artifacts are rejected. |
Granite Speech 4.1 (asr_granite_speech) |
VoiceHub-native end-to-end / raw data | VoiceHub owns the exact 954-tensor Conformer/Q-Former/Granite graph, HTK log-mel frontend, byte-BPE tokenizer, completion-only causal objective, and strict sharded-Safetensors lifecycle. Its collator reproduces IBM's prompt/audio plus transcript/EOS concatenation and masks prompt and target-padding labels with -100. The default source-compatible recipe trains the projector and VoiceHub-native language-model LoRA while freezing the encoder and dense language-model weights; the complete graph retains a tested backward path. Export merges LoRA into the official namespace and writes a fresh-inference, pickle-free artifact. |
Parakeet TDT v3 (asr_parakeet_tdt) |
VoiceHub-native end-to-end / raw data | VoiceHub owns the exact 723-tensor FastConformer, LSTM prediction network, TDT joint head, 16 kHz log-mel frontend, tokenizer, anti-diagonal TDT objective, greedy duration decoder, strict Safetensors lifecycle, and full-graph gradient-checkpointed backward. The raw-audio collator creates contiguous right-padded targets and exact [blank] + labels decoder inputs. Export validates the complete namespace and writes an inference-reloadable artifact. CTC/RNN-T checkpoints, beam search, hotwords, stateful streaming, translation, and forced language remain separate unsupported contracts. |
Nemotron 3.5 ASR (asr_nemotron) |
VoiceHub-native end-to-end / raw data | VoiceHub owns the pinned 655-tensor FastConformer, prompt projector, LSTM prediction network, RNN-T joint, 16 kHz log-mel frontend, bounded tokenizer, greedy decoder, and strict single-file Safetensors lifecycle. Joint processing creates the language prompt, blank-prefixed decoder inputs, transcript targets, and target lengths used by the exact differentiable RNN-T objective. Full-model backward, gradient checkpointing, and portable export/reload are supported. Beam search, hotword bias, pickle, sharded checkpoints, and corpus-level accuracy claims are outside the verified contract. |
Cohere Transcribe (asr_cohere) |
VoiceHub-native end-to-end / raw data | VoiceHub owns the exact 2,152-tensor FastConformer/decoder graph, 128-bin log-mel frontend, byte-fallback BPE tokenizer, 14 language prompts, strict gated Safetensors loader, and export lifecycle. Training routes language and punctuation through the processor, combines the decoder prompt with teacher-forced transcript tokens, masks prompt/padding targets, and backpropagates through the complete graph with optional gradient checkpointing. Long recordings must be pre-segmented with one aligned transcript per segment because inference-time quiet-boundary reassembly cannot infer transcript-to-chunk alignment. |
MedASR (asr_medasr) |
VoiceHub-native end-to-end / raw data | VoiceHub owns the exact 368-tensor LASR graph, 128-bin log-mel frontend, 512-piece Unigram tokenizer, greedy decoder, strict gated Safetensors loader, and CTC objective. Raw 16 kHz audio plus transcripts fine-tunes the complete 17-layer Conformer and CTC head, with gradient checkpointing and portable export/reload. The recorded source recipe uses AdamW at 3e-5 with 300 warmup steps. Access, fine-tuning, and deployment remain governed by Google's Health AI Developer Foundations terms; no clinical-performance claim is made. |
Wav2Vec2 ASR (asr_wav2vec2) |
End-to-end / raw data | Uses VoiceHub's native Wav2Vec2 graph, waveform processor, character tokenizer, and CTC loss with correctly padded labels. |
HuBERT ASR (asr_hubert) |
End-to-end / raw data | Uses VoiceHub's native HuBERT stable-layer-norm graph, learned SpecAugment mask embedding, waveform processor, character tokenizer, and CTC loss. Safetensors are loaded strictly; adapters and non-CTC heads are rejected. |
WavLM ASR (asr_wavlm) |
End-to-end / raw data | Uses VoiceHub's native WavLM graph, including bucketed gated relative-position attention, the learned SpecAugment mask embedding, waveform normalization, character tokenization, and CTC loss. Checkpoints are Safetensors-only; adapters and non-CTC heads are rejected. |
Moonshine ASR (asr_moonshine) |
End-to-end / raw data | Uses VoiceHub's native raw-waveform convolutional encoder, rotary encoder-decoder, SentencePiece BPE with byte fallback, padding-aware processor, and teacher-forced cross-entropy. Tiny/base Safetensors load through a strict complete-inventory adapter; repository code and pickle are rejected. |
NeMo QuartzNet ASR (asr_nemo) |
End-to-end / raw data | Runs the audited 18-block QuartzNet15x5 graph entirely in VoiceHub, including the 16 kHz log-mel frontend, per-feature normalization, strict 28-character tokenizer, rectangular spectrogram cutout, and NeMo-compatible mean-batch CTC loss. The exact NGC archive is SHA-256 pinned and converted once with PyTorch's restricted weights reader; steady-state training and export use Safetensors. Parakeet TDT, RNN-T, Canary, Citrinet, Conformer, and FastConformer remain explicitly unsupported distinct graph families. |
SpeechBrain CRDNN ASR (asr_speechbrain) |
End-to-end / raw data | Runs the exact 16 kHz/40-bin frontend, global CMVN, CRDNN encoder, location-aware attention decoder, and frozen RNNLM in VoiceHub. Transcripts are whitespace-normalized and uppercased for the released 1,000-piece unigram vocabulary. Epochs 1–5 optimize the released 0.5 CTC plus 0.5 label-smoothed sequence objective; later epochs use sequence NLL. The adapter preserves Adadelta (lr=1, rho=0.95, eps=1e-8), the validation beam without RNNLM fusion, corpus WER, and NewBob scheduling. OpenRIR corruption and speed perturbation remain explicit dataset transforms. |
SenseVoiceSmall (asr_funasr) |
End-to-end / raw data | Runs the audited SenseVoiceSmall SANM-CTC graph, Kaldi-compatible frontend, bounded SentencePiece-unigram reader, and rich control-token decoder entirely in VoiceHub. Raw 16 kHz audio and transcripts are converted into language, emotion, audio-event, inverse-text-normalization, and text targets. Training combines CTC with the four-query control-token objective and preserves the released AdamW/WarmupLR defaults. The 917-tensor release is validated strictly; the original hash-pinned pickle requires one explicit trust-gated conversion, while normal training, export, and reload use Safetensors. Other FunASR registry architectures are rejected. |
ESPnet LibriSpeech Transformer (asr_espnet) |
End-to-end / raw or cached-feature data | Runs the audited Transformer-e18 frontend, encoder/decoder, CTC head, tokenizer, and hybrid objective entirely in VoiceHub. Raw waveforms or cached frontend features preserve the published 0.3 CTC / 0.7 label-smoothed attention weighting, SpecAugment, Adam/WarmupLR settings, lazy dataset fingerprints, and portable export. |
WeNet GigaSpeech U2++ (asr_wenet) |
End-to-end / raw data | Runs the exact 20210728 conv2d6 Conformer, CTC head, and bidirectional Transformer decoder in VoiceHub. Training preserves 0.3 CTC weighting, 0.3 reverse weighting, 0.1 label smoothing, Kaldi fbank/CMVN, SpecAugment, Adam 1e-3, 80k warmup steps, and gradient clipping at 5.0. The legacy pickle crosses an explicit, hash-pinned, weights_only=True conversion boundary; steady-state training and export are Safetensors-only. |
SeamlessM4T v2 ASR (asr_seamless_m4t_v2) |
VoiceHub-native end-to-end / raw data | VoiceHub owns the exact 1,429-persisted-tensor S2T subset: 16 kHz stacked Kaldi-style frontend, 24-layer relative-key Conformer, adapter, 24-layer text decoder, native SentencePiece BPE, 98 language prompts, strict two-shard Safetensors projection, and portable export. Raw audio and transcripts become target-language-prefixed teacher-forced labels; cross-entropy updates the complete graph with optional gradient checkpointing. The verified generation path is greedy recognition only—translation mode, beam search, sampling, timestamps, hotwords, and streaming fail closed. The checkpoint and derivatives are non-commercial under CC-BY-NC-4.0, and no WER or accuracy-improvement claim is made. |
Native Wav2Vec2 VAD (vad_transformers) |
End-to-end / raw data | Dispatches supported Wav2Vec2 audio- and frame-classification Safetensors into VoiceHub's native graph. Clip and aligned frame objectives are differentiable; arbitrary remote-code classifiers are rejected. |
Silero VAD (vad_silero) |
End-to-end / raw data | Runs the released recurrent convolutional architecture inside VoiceHub and supports the official audio_path/speech_ts records or aligned frame labels. The default frame-BCE recipe preserves sequence-local state and optimizes the decoder; exported artifacts are strict Safetensors. |
FSMN VAD (vad_funasr) |
End-to-end / raw data | Runs VoiceHub's native 16 kHz Kaldi-compatible fbank/LFR/CMVN frontend and four-layer FSMN graph. Records may provide speech timestamp segments or aligned binary/PDF frame labels. Binary targets optimize grouped speech/silence NLL; 248-class PDF targets use cross-entropy. The original private data and PDF-label recipe are unpublished, so VoiceHub does not claim to reproduce them. Training exports strict Safetensors artifacts. |
MarbleNet Frame-VAD (vad_nemo) |
End-to-end / raw data | Runs VoiceHub's native 16 kHz log-mel frontend and exact six-block multilingual MarbleNet graph. Records may provide timestamp segments or aligned binary frame labels. Training uses frame cross-entropy and preserves the published SGD, warmup/hold polynomial decay, and augmentation parameters. Official .nemo weights cross an explicit one-time restricted conversion boundary; training exports strict Safetensors. |
WebRTC VAD (vad_webrtc) |
Not applicable | VoiceHub owns the pinned fixed-point filterbank, resamplers, adaptive GMM, and hangover logic. The algorithm updates stream statistics online but has no differentiable parameters, optimizer state, or checkpoint to fine-tune. |
Auditok VAD (vad_auditok) |
Unavailable | Deterministic energy detector with no trainable graph. |
Sherpa-compatible VAD (vad_sherpa_onnx) |
End-to-end / raw data | Dispatches to the verified native Silero graph or the exact native TEN graph. TEN ONNX weights cross a strict, acknowledged one-time conversion into Safetensors; no ONNX runtime is used. Silero retains its official decoder-BCE recipe. TEN accepts interval or aligned frame targets with masked window BCE, but the source training recipe is unpublished, so VoiceHub explicitly labels that recipe reconstructed rather than source-equivalent. |
PyanNet segmentation (vad_pyannote) |
End-to-end / frame labels | Uses VoiceHub's native SincNet/LSTM graph and multi-label BCE. Official Lightning weights cross an explicit one-time restricted conversion boundary; runtime, training, and export are Safetensors-only. |
PyanNet powerset segmentation (vad_pyannote_segmentation) |
End-to-end / frame labels | Uses the native seven-class powerset graph and categorical cross-entropy. Targets are class IDs aligned to the graph's output frames. |
PyanNet Brouhaha (vad_pyannote_brouhaha) |
End-to-end / frame labels | Uses native VAD BCE plus speech-masked SNR MSE and C50 MSE, including finite all-silence batches and portable Safetensors export. |
Safetensors, GGUF, and resume semantics¶
A file format is not a training capability:
- Safetensors can be a safe weight warm start when it belongs to the unfused VoiceHub-native PyTorch graph expected by the adapter. It does not by itself contain optimizer, scheduler, scaler, sampler, RNG, or recipe state.
- A VoiceHub checkpoint is the exact-resume artifact. It stores model
components together with optimizer/scheduler and trainer runtime state.
Model-specific adapters may also write
native_export/. Its recipe manifest states whether that directory is a complete inference export or only a component weight warm start; a safetensors filename alone is not evidence that the full upstream loader topology is present. - GGUF, ONNX, JIT, TensorRT, vLLM, and other fused or serving-only artifacts are not generic gradient-bearing checkpoints. If a repository publishes both safetensors and GGUF, select the compatible unquantized safetensors/source checkpoint for fine-tuning. The base adapter rejects recognizable serving, compiled, and quantized artifacts before loading and validates the resolved training graph again afterward; family-specific loaders add stricter checks.
- Quantized adapter training is a separate capability. It requires a PEFT/LoRA-aware adapter and must not be inferred from support for full-precision fine-tuning.
Before a run¶
The training workflow and companion notebooks demonstrate these checks with Dia's raw text/audio training path.
Verify that:
- the selected checkpoint variant uses the differentiable backend named in the table;
- one small batch returns a finite scalar loss with
requires_grad=True; - the intended parameters receive gradients and frozen codecs/vocoders do not;
- every adversarial or auxiliary phase has its source training graph, loss, detach boundary, optimizer, and update cadence;
- saving and resuming restores the same component and optimizer topology.
See the trainer architecture for phase scheduling, data collation, optimizer routing, strategy integration, and checkpoint semantics.