Luna-TTS – A Text-to-Speech Large Model from VUI Labs

1. What is Luna-TTS

Luna-TTS is a text-to-speech large model independently developed by VUI Labs, a startup company affiliated with Shanghai Jiao Tong University. This model abandons the mainstream autoregressive generation approach and instead adopts a Masked Diffusion architecture. It is trained based on the Qwen3-0.6B language model and is paired with a self-developed Luna-Codec encoder-decoder, completing pre-training on 1 million hours of speech data across four languages: Chinese, English, Japanese, and Korean. In the Hugging Face TTS Arena V2 blind listening evaluation, Luna-TTS ranked first globally, and in the authoritative Artificial Analysis leaderboard, it placed third, surpassing competitors such as Google. Its Realtime version achieves a first-block latency of just 41.6 milliseconds, with an end-to-end real-time factor as low as 0.024, offering a viable technical solution for real-time speech interaction.

Image source: Official article
Image source: official article

Technical positioning and domain: Luna-TTS belongs to the text-to-speech large model category in the field of speech synthesis, focusing on high-quality, multilingual, and low-latency speech generation. Its non-autoregressive masked diffusion approach represents a cutting-edge exploration in both academia and industry. Compared to traditional autoregressive models, masked diffusion predicts masked positions iteratively, avoiding the issue of error accumulation along prefixes in autoregressive generation, while also supporting more flexible control (such as emotion and paralinguistic information). The model is positioned to meet the stringent demands of natural speech in scenarios such as audiobook production, podcasts, intelligent customer service, in-car navigation, and real-time interpretation. It has already demonstrated international competitiveness in technical metrics.

Development background: VUI Labs is a startup company affiliated with Shanghai Jiao Tong University, with its core team having accumulated years of experience in speech recognition, synthesis, and natural language processing. The motivation behind developing Luna-TTS was to overcome the limitations of autoregressive TTS models in terms of generation speed, control granularity, and naturalness. The team chose Qwen3-0.6B as the backbone model, leveraging its strong semantic understanding capabilities to improve the accuracy of speech content. Meanwhile, the self-developed Luna-Codec enables high compression rate encoding and decoding, laying the foundation for streaming transmission. The company also introduced GRPO reinforcement learning to optimize the denoising trajectory of the diffusion process, further enhancing content accuracy and speaker consistency.

Core value: Luna-TTS addresses the shortcomings of traditional TTS systems in naturalness, multilingual support, real-time performance, and control capabilities. It provides speech quality approaching that of humans, supports four languages: Chinese, English, Japanese, and Korean, and can stream output with extremely low latency. The fine-grained control over emotion and paralinguistic information makes the generated speech more expressive, suitable for a wide range of applications including audiobooks, podcasts, intelligent customer service, in-car navigation, and real-time interpretation. Ranking atop authoritative blind listening benchmarks validates the effectiveness of its technical approach and the superiority of its generation quality.

Technical features: The core technologies include the Masked Diffusion architecture, which treats the speech RVQ Token grid as a whole and predicts masked positions in parallel through multiple iterations, with the generation order dynamically determined by the model; the self-developed Luna-Codec encodes 24kHz waveforms into discrete Tokens with 25Hz sampling and 8 codebooks, achieving a bitrate of only 2.2kbps, and employs a causal encoder-decoder to enable streaming frame synchronization; bidirectional attention based on Qwen3 (standard version) and block-level causal attention (Realtime version), combined with KV Cache, achieve efficient inference; and the use of GRPO reinforcement learning to optimize the denoising trajectory, enhancing content accuracy and speaker consistency. These technologies collectively form a complete and efficient speech synthesis solution.

2. Key Features

• Multilingual Text-to-Speech: Supports high-quality voice synthesis for four languages: Chinese, English, Japanese, and Korean. Pre-trained on 1 million hours of multilingual data, the model captures pronunciation, prosody, and spoken language characteristics across all languages, generating natural-sounding voices with smooth prosody. In long-text scenarios (such as audiobooks and news broadcasting), it maintains consistent voice tone and stable speaking rate, with natural transitions when switching between languages.

• Emotion and Paralinguistic Control: Enables precise control over emotional states such as neutral, sad, and surprised, and supports paralinguistic details like breathing, pauses, laughter, and sighs. Users can specify emotions and paralinguistic elements through input tags or parameters, and the model dynamically adjusts prosody, pitch, and pause patterns during generation, giving the speech rich expressiveness and adaptability to different content and contexts.

• Real-time Streaming Output: The Luna-TTS Realtime version supports block-level streaming generation, with a first-block latency of just 41.6 milliseconds and an end-to-end real-time factor (RTF) as low as 0.024. This means the model can begin playing audio almost immediately after receiving text and continue generating subsequent content in real-time, making it highly suitable for latency-sensitive interactive scenarios such as real-time conversations, online meetings, and voice assistants.

• Voice Consistency and Cloning: The model supports voice cloning, allowing it to extract speaker characteristics from a small amount of reference audio (e.g., 5–10 seconds) and maintain voice stability throughout long-text generation. This is crucial for applications such as multi-speaker podcasts and personalized voice assistants, addressing the issue of voice variation in traditional TTS systems during long-text synthesis.

• High Compression Codec: The self-developed Luna-Codec encodes 24kHz waveforms into a discrete Token grid with a 25Hz frame rate and 8 codebooks, achieving a bit rate of only 2.2kbps. Compared to traditional codecs (such as Opus, which typically requires over 16kbps), this significantly reduces bandwidth and storage requirements, while supporting streaming frame-synchronous output, providing a foundation for real-time transmission and low-bandwidth scenarios.

3. How to Use

Currently, Luna-TTS is primarily made public in a research form, and the model weights have not been fully open-sourced. Users can follow the steps below to learn about and use the model (the following steps assume that the model weights are available or accessible via API):

1. Understand the model capabilities: Visit the project's official website (link to be updated after the official release).

2. Read the technical paper: Access the full technical paper on arXiv (https://arxiv.org/pdf/2608.11593) to gain a deeper understanding of the Masked Diffusion architecture, Luna-Codec design, training details, and experimental results. The paper includes model architecture diagrams, ablation experiments, and comparative data with competitors, which are helpful for understanding the technical principles.

3. Apply for API trial: If the official website provides an API trial application entry, fill out the form to obtain an API key. The API supports sending text and parameters (language, emotion, speaker ID, etc.) via HTTP requests and returns synthesized audio files or streaming audio data. Specific API documentation will be published on the official website.

4. Environment preparation (local deployment): It is recommended to use the Ubuntu 20.04 system with an NVIDIA A100 (40GB) or equivalent GPU computing power. Install CUDA 11.8 and Python 3.10, and use pip to install the dependencies: pip install torch==2.0.1 torchaudio==2.0.2 transformers soundfile librosa.

5. Model loading and inference: Assuming the model weights have been downloaded from an official source, use the Transformers library to load the model and processor:

   from transformers import AutoModelForTextToSpeech, AutoProcessor
   model = AutoModelForTextToSpeech.from_pretrained("path/to/luna-tts")
   processor = AutoProcessor.from_pretrained("path/to/luna-tts")
   inputs = processor(text="Hello, welcome to Luna-TTS.", return_tensors="pt")
   audio = model.generate(**inputs, emotion="neutral")
   soundfile.write("output.wav", audio[0].numpy(), samplerate=24000)
   

6. Streaming generation integration: For the Realtime version, streaming output can be achieved through the WebSocket interface. The client sends text in blocks, and the server returns audio segments in real time. The first-block latency can be controlled within 50ms, and the end-to-end RTF remains below 0.03. Specific protocols and example code can be referenced in the official documentation.

Note: The model weights are not yet publicly available, and the specific usage methods will be determined by the official release. It is recommended to follow the project's official website and subsequent announcements from Yusheng Yueban.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations:

For real-time interactive scenarios that demand top-tier audio quality and low latency, such as intelligent customer service, real-time interpretation, and in-car navigation, Luna-TTS stands out with its masked diffusion architecture and streaming output capabilities. It ranked first in the TTS Arena blind listening test with a rating of 1574, showcasing a clear quality advantage. Its Realtime version achieves a first chunk latency of just 41.6ms, significantly lower than Azure and Google's 100–200ms, offering a more natural conversational experience. However, if the requirement is to support dozens of languages with a limited budget, Microsoft Azure TTS or Google Cloud TTS provides broader language coverage and lower API costs (~$4/1M characters), although they may fall slightly short in naturalness compared to Luna-TTS.

For scenarios requiring rich vocal styles and a mature ecosystem, such as podcasting and audiobook creation, ElevenLabs offers support for 32+ languages and a diverse voice library, with capabilities for emotion and style adjustment. While its pricing is higher ($100/1M characters) and its latency is not as low as Luna-TTS Realtime, its platform maturity and ease of use remain highly appealing. Luna-TTS, on the other hand, provides more precise control over voice cloning and paralinguistic features, making it ideal for professional productions with high demands for expressiveness. Overall, users should weigh their language requirements, real-time needs, and budget: choose Azure or Google for multilingual, low-cost applications; Luna-TTS for high-quality, low-latency applications; and ElevenLabs for scenarios requiring diverse vocal styles and a mature platform.

6. Editor's Summary

Luna-TTS has opted for the non-mainstream Masked Diffusion architecture in its technical approach, a decision that has given it significant advantages in terms of speech naturalness and control flexibility. From an innovation perspective, applying masked diffusion to speech synthesis, combined with its self-developed Luna-Codec high-compression codec and GRPO reinforcement learning optimization, has formed a complete solution. Compared to mainstream autoregressive models, masked diffusion avoids error accumulation by using parallel iterative prediction, while also supporting finer-grained control (such as paralinguistic details). Its top performance in the Hugging Face TTS Arena V2 and its surpassing of Google on the Artificial Analysis leaderboard demonstrate that its generation quality has reached industry-leading levels. In terms of practical value, Luna-TTS's multilingual support, ultra-low-latency streaming output, and precise emotional control make it highly applicable in scenarios such as intelligent customer service, audiobooks, online education, in-car navigation, and real-time interpretation. Particularly, the Realtime version achieves a first-block latency of just 41.6ms, meeting the requirements for real-time conversations—a level many competitors have yet to achieve. The target users include speech application developers, content creators, enterprise users, and academic researchers, who can leverage Luna-TTS's high-quality synthesis capabilities to enhance product experiences. However, the model weights are not yet open-sourced, which limits its popularity and secondary development, and prevents the community from fully verifying and contributing to it. In the future, if Yusheng Yueban opens up the model weights or provides convenient API services, and expands language coverage and emotional diversity, Luna-TTS has the potential to become a major force in the TTS field, driving the industry toward more natural and controllable speech synthesis. Overall, this is a research achievement with highly competitive technical metrics, and its architectural innovation is worth the attention and follow-up of the industry.

7. Application Scenarios

• Smart Customer Service and Telemarketing: Replacing traditional robotic voice systems, Luna-TTS can generate natural, emotionally expressive speech and support real-time streaming responses. During customer conversations, it can switch between emotions such as neutral, sad, and surprised based on context, and incorporate paralinguistic details like breathing and pauses to enhance the realism of the interaction and reduce call drop rates. Combined with speech recognition and natural language understanding, it enables smoother human-machine conversations.

• Audiobooks and Podcasts: Generate high-quality long-text speech for audiobooks, news broadcasting, and educational podcasts. Luna-TTS maintains consistent voice quality throughout long texts and supports multi-emotion paragraph transitions (e.g., using neutral for narration and surprised for emotional peaks), making the content more engaging. The voice cloning feature allows creators to use custom voices, helping to build unique auditory brands.

• Online Education and Training: Provide clear and standard multilingual pronunciation for course explanations and language learning. Teachers can choose from encouraging, serious, or gentle emotional tones, combined with paralinguistic pauses, to enhance teaching effectiveness. Supporting Chinese, English, Japanese, and Korean, it is suitable for international education platforms and language learning applications, helping students master correct pronunciation.

• In-Vehicle and Smart Hardware: Leveraging the low-latency characteristics of Luna-TTS Realtime, it offers instant and natural voice interaction for in-vehicle navigation, smart home devices, and earphone assistants. Navigation instructions can include reminder tones, while smart home feedback can be delivered with a relaxed emotional tone, improving user experience. Its low-bitrate encoding and decoding also make it suitable for embedded devices.

• Real-Time Interpretation and Meeting Assistants: Combined with speech understanding capabilities, Luna-TTS enables low-latency, high-naturalness voice translation output in cross-language meeting scenarios. Its streaming capabilities allow for the simultaneous generation of target language speech, reducing waiting time and improving communication efficiency. It is applicable to international conferences, remote collaboration, and other similar scenarios.

8. FAQ

Q: Is Luna-TTS open source?
A: The technical paper and project website of Luna-TTS are now publicly available, but the model weights have not been open-sourced yet. Users can experience the results through online demonstrations on the official website, or keep an eye out for future official release plans. The open-source timeline will be announced by the official team.

Q: Which languages does Luna-TTS support?
A: Luna-TTS supports four languages: Chinese, English, Japanese, and Korean. The model has been pre-trained on 1 million hours of data across these four languages, achieving high cross-lingual synthesis quality, with natural pronunciation and prosody for each language.

Q: How can I apply for a Luna-TTS API trial?
A: Please visit the project's official website (link to be updated after official release) to apply for API access. Currently, the official website mainly showcases technical information and demonstrations.

Q: What is the latency of Luna-TTS Realtime?
A: The first-block latency of the Realtime version is only 41.6 milliseconds, with an end-to-end real-time factor (RTF) as low as 0.024. This means that from input text to hearing the first audio segment takes approximately 40ms, meeting the requirements for real-time dialogue and conference interpretation scenarios.

Q: What emotional and paralinguistic controls does Luna-TTS support?
A: Luna-TTS supports emotions such as neutral, sad, and surprised, as well as paralinguistic details like breathing, pauses, laughter, and sighing. Users can specify emotion tags and paralinguistic markers during generation, and the model dynamically adjusts prosody and pitch accordingly.

Q: How does Luna-TTS compare to ElevenLabs?
A: In the Hugging Face TTS Arena V2 blind listening leaderboard, Luna-TTS ranks first (Rating 1574), while ElevenLabs ranks tenth (Rating 1176). Luna-TTS excels in naturalness and fine-grained control, but ElevenLabs supports more languages (32+) and has a more mature ecosystem, offering a wide range of preset voices.

Q: What hardware is required for local deployment of Luna-TTS?
A: It is recommended to use an NVIDIA A100 (40GB) GPU or a GPU with equivalent computing power, requiring at least 16GB of VRAM. The model is based on Qwen3-0.6B, and consumer-grade GPUs (such as the RTX 3090 24GB) may be able to run it, but with slower inference speeds. Specific hardware requirements will be confirmed upon official release.

9. Project Links

1. Product Official Website: https://vuilabs-ai.github.io/luna-tts/
2. Paper/Technical Report: https://arxiv.org/pdf/2608.11593