Cosyvoice
Multi-lingual large voice generation model, providing inference, training and deployment full-stack ability.
Last verified:
What is Cosyvoice?
CosyVoice 3 is an advanced text-to-speech (TTS) system based on large language models (LLM) designed for zero-shot multilingual speech synthesis in real-world scenarios. It surpasses its predecessor CosyVoice 2 in content consistency, speaker similarity, and prosody naturalness. The model integrates an LLM with a chunk-aware flow matching model to achieve low-latency bi-streaming speech synthesis with human-parity quality.
Key features include a novel speech tokenizer developed through supervised multi-task training (covering automatic speech recognition, speech emotion recognition, language identification, audio event detection, and speaker analysis), a new differentiable reward model for post-training, dataset scaling from 10,000 to 1 million hours covering 9 languages and 18 Chinese dialects, and model size scaling from 0.5B to 1.5B parameters. The system supports zero-shot in-context generation, mixed-lingual generation, emotionally expressive voice generation, Chinese dialect voice generation, cross-lingual generation, instructed voice generation, and hotfix capability for pronunciation correction.
CosyVoice 3 is designed for researchers, developers, and applications requiring high-quality multilingual TTS with zero-shot voice cloning capabilities. It is particularly useful for content creators, audiobook producers, accessibility applications, voice assistants, and anyone needing natural-sounding speech synthesis across multiple languages and languages with emotional expression control.
The model provides target speaker fine-tune models for both Chinese/English and minority languages, with in-ability transfer for styles like sad, surprised, angry, fearful, fast, slow, Peppa pig voice, and robot voice.
Cosyvoice pricing
Pricing model: Freemium
The website does not mention any commercial pricing, free tier, or paid plans. CosyVoice 3 is provided for academic purposes only to demonstrate technical capabilities. The model appears to be open-source based on GitHub repository references, but no explicit licensing or monetization details are provided on this page.
Cosyvoice pros
- Zero-shot multilingual voice cloning without training data
- Supports 9 languages and 18 Chinese dialects
- Improved content consistency over CosyVoice 2
- Enhanced speaker similarity in voice cloning
- More natural prosody with novel speech tokenizer
- Emotionally expressive voice generation (happy, sad, fearful, angry, surprised)
- Mixed-lingual in-context generation capability
- Cross-lingual generation across multiple languages
- Instructed voice generation with style control
- Hotfix capability for pronunciation correction
- Low-latency bi-streaming speech synthesis
- Human-parity speech quality
- 1.5B parameter model with enhanced capacity
- Trained on 1 million hours of diverse data
- Target speaker fine-tune models available
- Post-training with differentiable reward model
- Fine-grained control including breath and轻声
- Audio event detection in tokenizer training
- Supports German, Spanish, French, Italian, Russian previously unsupported languages
Cosyvoice cons
- Only academic release, no commercial API mentioned
- Requires significant computational resources for 1.5B model
- No explicit free tier or paid pricing mentioned
- Limited to academic purposes per disclaimer
- No web interface or demo audio players visible
- Training requires massive dataset (1 million hours)
- Post-training techniques still developing
- Language coverage limited to 9 languages plus dialects
- No mobile or edge deployment mentioned
- Real-time streaming latency not quantified
Frequently asked questions about Cosyvoice
What is CosyVoice 3?
CosyVoice 3 is an advanced text-to-speech (TTS) system based on large language models designed for zero-shot multilingual speech synthesis in the wild. It improves upon CosyVoice 2 in content consistency, speaker similarity, and prosody naturalness, with training data scaled from 10,000 to 1 million hours and model parameters increased from 0.5B to 1.5B.
What languages does CosyVoice 3 support?
CosyVoice 3 supports 9 languages and 18 Chinese dialects. The languages include Chinese (ZH), English (EN), Japanese (JA), Korean (KO), German (DE), Spanish (ES), French (FR), Italian (IT), and Russian (RU). It also supports Chinese dialects including Cantonese, Dongbei, Tianjin, Sichuan, and Shanghai.
What is zero-shot in-context generation?
Zero-shot in-context generation allows CosyVoice 3 to synthesize speech in a target voice without additional training, using only a short audio prompt. The model analyzes the prompt's speaker characteristics and generates speech matching that voice while reading the input text in the same speaker identity.
Can CosyVoice 3 express emotions?
Yes, CosyVoice 3 supports emotionally expressive voice generation including happy, sad, fearful, angry, and surprised emotions. The model can generate speech with appropriate emotional tone based on the context or explicit instruction, making the synthesized speech more natural and expressive.
What is the hotfix capability?
Hotfix capability allows CosyVoice 3 to correct pronunciation errors by providing phonetic specifications for problematic words. Users can specify exact phonemes (like [j][ǐ] for Chinese or [IH1][N][V][AH0][L][IH0][D] for English) to fix mispronunciations of polyphonic words or proper nouns.
How does instructed voice generation work?
Instructed voice generation allows users to control voice characteristics through explicit instructions. Users can specify emotions (neutral, angry, sad, happy, fearful, surprised), styles (Cantonese, Chongqing dialect, Xi'an dialect), speed (fast, slow), voice types (Peppa Pig, robot), and fine-grained controls like [breath] markers or轻声 (light tone).
What is mixed-lingual generation?
Mixed-lingual in-context generation allows CosyVoice 3 to synthesize speech that naturally mixes multiple languages within a single utterance. For example, a Chinese sentence can naturally incorporate Japanese words like アイスクリーム (ice cream) or Korean words like 찌개 (stew) while maintaining consistent speaker identity.
What are the model size options?
CosyVoice 3 offers two model sizes: CosyVoice 3.0-0.5B with 0.5 billion parameters and CosyVoice 3.0-1.5B with 1.5 billion parameters. The larger 1.5B model provides enhanced performance on multilingual benchmarks due to greater model capacity.
What is the novel speech tokenizer?
The novel speech tokenizer in CosyVoice 3 improves prosody naturalness through supervised multi-task training that includes automatic speech recognition, speech emotion recognition, language identification, audio event detection, and speaker analysis. This comprehensive training enables more natural-sounding speech synthesis.
Can I fine-tune CosyVoice 3 for specific speakers?
Yes, CosyVoice 3 provides target speaker fine-tune models for both Chinese/English speakers (like longcheng, longhua, longshu, longbella) and minority language speakers. These fine-tuned models can generate speech in specific speaker voices across multiple languages while maintaining the target speaker's characteristics.