dots.tts – Xiaohongshu and Shanghai Jiao Tong University Open-Source Base Model for Text-to-Speech

1. What is dots.tts

dots.tts is a 20-billion-parameter fully continuous autoregressive text-to-speech base model jointly open-sourced by Xiaohongshu's dots team and the X-LANCE Lab at Shanghai Jiao Tong University. The model generates 48kHz audio block by block directly in a continuous latent space, achieving state-of-the-art voice similarity and content accuracy on benchmarks such as Seed-TTS-Eval. It supports zero-shot cloning, streaming output, and low-latency duplex conversation. The full project code and weights are open-sourced, with extended capabilities for precise audio editing through dots.tts.edit.

Technical Positioning and Domain: Belongs to the text-to-speech (TTS) domain, utilizing a fully continuous autoregressive architecture that directly models in a continuous latent space, abandoning traditional discrete acoustic Token approaches. It is positioned as a general-purpose TTS base model, balancing high-quality generation, zero-shot cloning, and real-time interactive capabilities.

Development Background: Jointly developed by Xiaohongshu's dots team and the X-LANCE Lab at Shanghai Jiao Tong University. The team has deep expertise in speech and multimodal domains, aiming to address issues such as information bottlenecks, loss of voice detail, and insufficient real-time performance in existing discrete speech synthesis solutions, and to promote the development of open-source TTS technology.

Core Value: Solves the problem of reduced voice fidelity caused by quantization errors in traditional discrete Token approaches, achieving high-fidelity speech generation through fully continuous autoregressive modeling. It also provides capabilities for zero-shot cloning, streaming low-latency interaction, and precise editing, offering a unified base model for scenarios such as voice assistants, content creation, and multilingual services.

Technical Features: Key innovations include semantic continuous representation (AudioVAE), a causal semantic encoder that suppresses error propagation, an autoregressive flow matching head (DiT) for generating acoustic patches, and post-training acceleration techniques such as SOAR self-correcting alignment, MeanFlow distillation, sCM, and DMD, enabling high-quality generation from multi-step to single-step processes.

2. Key Features

• Zero-shot Voice Cloning: Only 3–10 seconds of reference audio is needed to replicate the voice for reading any new text, with support for cross-language cloning. On the Seed-TTS-Eval benchmark, both content accuracy and speaker similarity reach leading levels, offering an efficient solution for personalized speech generation.

• Text-to-Speech Synthesis: Provides high-quality speech generation from random voice sampling to specific speakers. Based on a 2B parameter continuous autoregressive model, it directly outputs 48kHz audio without requiring an additional vocoder, delivering natural and refined audio quality.

• Streaming Generation and Low Latency Interaction: Naturally supports streaming output, generating audio in blocks after inputting the text prefix. In the 1T1A dual-stream mode, the downstream speech side responds immediately as the upstream LLM outputs each text token, achieving a first audio packet latency as low as 54.4 milliseconds. This is ideal for real-time speech agents and full-duplex conversations.

• Precise Speech Editing: Allows text replacement, emotion adjustment, pitch and speaking rate modification, and insertion or deletion of pauses in existing recordings. Multiple operations can be combined into a single instruction for one-time execution, while unedited sections remain unchanged, significantly improving the efficiency of audio post-production.

• Multilingual Support: Covers 24 languages, achieving an average speaker similarity of 83.9 in the MiniMax multilingual evaluation. Nineteen language variants achieved first place in individual categories, providing a solid foundation for global multilingual speech applications.

• Multiple Inference Speed Options: Offers several checkpoints, including Base, SOAR, MeanFlow (4-step), two-step sCM, and single-step DMD. Users can flexibly choose between audio quality and inference speed based on their application scenarios, meeting diverse needs from high-fidelity offline generation to real-time online services.

3. How to Use

1. Environment Requirements and Installation: Install using pip install dots.tts, or clone the source code from GitHub and install locally with pip install -e .. For high-concurrency scenarios, you can additionally deploy SGLang Omni to gain CUDA Graph acceleration and support for continuous batching. It is recommended to use an NVIDIA GPU with at least 24GB of VRAM.

2. Command-line Synthesis: Use the dots.tts entry point and specify --model-name-or-path, --text, and the target --prompt-audio and --prompt-text to achieve zero-shot voice cloning. Omitting --prompt-text will degrade to x-vector cloning, and omitting --prompt-audio will result in random voice sampling.

3. Command-line Editing: Use the dots.tts.edit entry point, providing --source-audio and an XML-style --instruction, to perform precise edits on existing recordings, such as text replacement, emotion or prosody modification.

4. Python API Streaming Output: By calling generate_stream(), you can retrieve audio tensors in blocks and push them in real-time to a player or WebSocket, enabling a low-latency experience where audio is played as it is generated. This is suitable for applications requiring real-time audio output.

5. Web Interface: Run python apps/gradio/app.py to launch the visual synthesis interface, or run apps/edit_playground/app.py to start the Edit Playground. Upload audio files in the browser, annotate editing segments, and preview results in real-time, reducing the barrier to entry.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For applications that prioritize the highest voice fidelity, broad multilingual support, and low-latency real-time interaction, dots.tts stands out with its fully continuous architecture and rich acceleration options, making it particularly suitable for voice agents, multilingual content generation, and precise editing scenarios. If the project primarily targets Chinese and is sensitive to hardware requirements, CosyVoice 3 is optimized for Chinese tasks and has a smaller parameter count, resulting in lower deployment costs. For lightweight applications or research exploration, ChatTTS and FishSpeech offer smaller models and faster deployment capabilities, though their multilingual and voice cloning capabilities are relatively limited. Developers should choose based on specific requirements for audio quality, real-time performance, and available hardware resources.

6. Editor's Summary

As a joint open-source TTS foundation model launched by Xiaohongshu and Shanghai Jiao Tong University, dots.tts has made significant innovations in its technical approach. The fully continuous latent space autoregressive architecture fundamentally avoids the information bottleneck caused by discrete quantization, achieving new heights in preserving vocal nuances and naturalness. Its semantic continuous representation, causal semantic encoder, and multi-stage post-training acceleration techniques (SOAR, MeanFlow, sCM, DMD) form a complete technical system that balances generation quality and inference efficiency. Performance in the Seed-TTS-Eval and MiniMax multilingual benchmarks has validated its effectiveness, particularly demonstrating a clear advantage in multilingual voice similarity.

In terms of practical value, dots.tts not only provides high-quality TTS and zero-shot voice cloning, but also expands its application boundaries through its native streaming duplex mode and precise editing capabilities, making it suitable for a variety of scenarios such as real-time voice assistants, audiobook content production, and podcast post-production. The full open-source release (Apache 2.0) along with training, fine-tuning, and distillation code significantly lowers the usage barriers for researchers and developers.

The target audience includes speech technology researchers, AI application developers, content creators, and global product teams requiring multilingual speech capabilities. In the future, with increasing community contributions and continuous model iteration, dots.tts is expected to further develop in real-time interaction, emotional expression, and more lightweight deployment. Although there is still room for improvement in terms of hardware requirements and long-text stability, its technical approach and open-source ecosystem have already provided an important reference for the field of speech synthesis.

7. Application Scenarios

• Real-time Voice Agent: The 1T1A dual-stream mode achieves a first packet latency as low as 54.4 milliseconds, suitable for LLM-driven real-time duplex conversations and voice assistants, enabling a natural interactive experience with simultaneous generation and playback.

• Zero-shot Voice Cloning: Only a few seconds of reference audio are needed to replicate any voice, supporting cross-language cloning. It is applicable to audiobook recording, voice-over, and personalized content creation, significantly reducing the cost of voice production.

• Multilingual Content Generation: Supports 24 languages while maintaining high speaker similarity, suitable for multilingual voice announcements in global products, localization services, and educational content, enhancing user experience.

• Precise Voice Editing: Replace text, adjust emotion, modify prosody, and adjust pauses in existing recordings using dots.tts.edit, applicable to podcast post-production and audio content refinement, improving editing efficiency.

• Live Streaming and Real-time Broadcasting: Combined with LLM streaming output, it enables simultaneous generation and playback, suitable for news broadcasting, live streaming sales, and real-time information delivery scenarios, enhancing interactivity and timeliness.

8. FAQ

Q: Is dots.tts completely open source?
A: Yes, the project uses the Apache 2.0 license. The code, model weights (6 checkpoints), and full training, fine-tuning, and distillation code are open-sourced on GitHub, supporting both commercial and academic use.

Q: What hardware is required to run dots.tts?
A: It is recommended to use an NVIDIA GPU with at least 24GB of memory. The 2B parameter model requires approximately 4GB of memory under FP16 inference, but actual usage also depends on input/output length and batch size. For the best experience, it is recommended to use an A100 or higher configuration.

Q: How can I fine-tune dots.tts for a specific voice or domain?
A: Run accelerate launch scripts/train_dots_tts.py and configure your own data path to fine-tune on top of the public checkpoints. The project provides complete fine-tuning scripts, and users can refer to the documentation to adjust parameters.

Q: Does dots.tts support Chinese speech synthesis?
A: Yes. dots.tts supports 24 languages, including Chinese. In multilingual evaluations, it performs exceptionally well in speaker similarity and content accuracy for Chinese tasks.

Q: How do I use the voice editing feature?
A: Use the dots.tts.edit command-line tool, providing --source-audio and an XML-formatted --instruction to specify text replacement, emotion adjustment, pitch and speaking rate modification, etc. Multiple instructions can be combined, and unedited regions will remain unchanged.

Q: How low is the latency for streaming generation?
A: In the 1T1A dual-stream mode, the first audio packet latency can be as low as 54.4 milliseconds. When deployed with SGLang Omni, a single H100 GPU can support 16 concurrent streams with a throughput of 4.76 req/s, meeting real-time interaction requirements.

9. Project Links

1. GitHub Repository: https://github.com/studio-dots-ai/dots.tts
2. arXiv Technical Paper: https://arxiv.org/pdf/2608.02673