AuK – Tencent HunYuan's Open-Source Foundation Model for Speech Generation and Editing

1. What is AuK

AuK is an open-source foundation model for speech generation and editing developed by the Tencent HunYuan team, featuring 1.5 billion parameters and utilizing a flow-matching diffusion architecture instead of traditional autoregressive approaches. Through a unified natural language instruction interface, the model can perform 16 different speech tasks, covering zero-shot/instructional TTS, speech content editing, acoustic modulation, paralinguistic editing, noise reduction, and voice separation across diverse scenarios. The core design philosophy of AuK lies in treating speech editing tasks as a denoising process starting from the original audio latent representation, thereby enabling precise modification of specified content while maximizing the preservation of the original timbre, melody, and integrity of unedited parts. Additionally, AuK provides a fast version called AuK-Flash with 4-step inference, achieving a speed improvement of approximately 4.5 times.

Image source: Official article
Image source: official article

Technical positioning and domain: AuK belongs to the field of foundation models for speech generation and editing, spanning five major task categories: text-to-speech (TTS), speech content editing, acoustic attribute adjustment, audio enhancement, and source separation. Its uniqueness lies in unifying the generation and editing directions under a single model, breaking through the previously distinct barriers between tasks.

Development background: This model was developed by the Tencent HunYuan large model team, leveraging its long-term accumulation in the areas of multimodal large models and speech technology. The team deeply integrated multimodal large models with audio diffusion architecture, establishing a complete pipeline from natural language instructions to speech outputs, demonstrating Tencent's systematic layout in foundation model research and development.

Core value: AuK addresses the core pain points of traditional speech models, which are task-specific and lack editing capabilities. Previously, modifying individual words or phrases in a recording required re-recording or using a combination of multiple tools, whereas AuK allows users to directly modify the content, acoustic attributes, and paralinguistic features of existing audio through simple text instructions, significantly reducing the technical barriers and labor costs of post-production in speech processing.

Technical features: AuK employs a hybrid Transformer architecture combining a 10-layer dual-stream MMDiT with a 20-layer single-stream DiT, with a model width of 1536. It uses an audio VAE to encode 24kHz waveforms into a 50Hz acoustic latent space. Training on approximately 3.03 billion instruction-audio pairs (accumulating to about 1.95 million hours) with a unified format alignment enables AuK to generalize across multiple tasks from a single interface. This data scale and architectural design place AuK at a leading level among open-source speech models.

2. Key Features

• Zero-shot TTS (Zero-shot Text-to-Speech): Users can provide a few seconds of reference audio, and the model can clone the voice and synthesize new speech content. This feature is suitable for personalized speech synthesis scenarios, such as customizing the voice of a virtual assistant or dubbing roles in audiobooks, achieving high similarity in voice cloning without additional training.

• Instruct TTS (Instruction-based Text-to-Speech): This is one of AuK's differentiating capabilities, allowing the generation of corresponding speech based purely on textual descriptions of voice characteristics—such as "a deep, magnetic male voice"—without requiring any reference audio. This feature significantly lowers the barrier to entry for speech synthesis, enabling content creators to obtain the desired voice without searching for specific voice samples.

• Speech Content Editing: Users can directly replace, insert, or delete text content in recordings without altering the original voice and tone. For example, correcting verbal mistakes in a podcast or updating place names in a voice navigation system. This solves the problem of voice inconsistency caused by traditional re-recording methods.

• Acoustic Attribute Editing: Supports fine-grained control over pitch (in semitones), speaking rate (in multiples), and volume (in decibels), enabling precise adjustments to the acoustic features of speech. This feature provides more natural and higher-quality processing results than traditional time-stretching and pitch-shifting algorithms, making it ideal for audio post-production.

• Emotion and Voice Editing: While keeping the text content unchanged, this feature allows modification of the emotional tone of the speech (e.g., from calm to excited), or converting the same content into different voices. It is highly practical for film dubbing and audiobook production, enabling quick adjustments to the performance emotion during post-production.

• Paralinguistic Event Editing: Supports adding or removing paralinguistic events such as breath sounds, laughter, and coughs, and can also switch between normal speech and whispering. These detailed adjustments significantly enhance the naturalness and expressiveness of the generated speech, making it closer to human vocal behavior.

• Accent Editing and Noise Reduction: Can remove accent features from recordings while also performing noise reduction and de-reverberation processing, improving speech clarity. This feature has direct applications in cross-border communication, language learning material creation, and restoration of old recordings.

• Speech and Music Separation: Supports splitting multi-speaker dialogue audio based on speaking order, or extracting the voice of a specific speaker based on content description. It can also separate vocals from instrumental tracks in songs. This feature provides convenient material processing tools for audio and video post-production and music remix creation.

3. How to Use

1. Verify the hardware environment: Prepare a machine equipped with an NVIDIA GPU having approximately 24GB of VRAM (such as RTX 3090 or RTX 4090), and ensure that the GPU driver and CUDA version meet the requirements for running PyTorch. If the VRAM is insufficient, inference may fail or require a significant reduction in batch size.

2. Clone the code repository: Clone the Tencent-Hunyuan/AuK repository from GitHub to your local machine. Execute the command git clone https://github.com/Tencent-Hunyuan/AuK and navigate to the project root directory to review the README document, which provides detailed guidance on environment configuration.

3. Install dependencies: Create a Python virtual environment according to the repository's requirements (it is recommended to use Python 3.10 or higher), and run pip install -r requirements.txt to install the required packages. Be sure to check that the torch version matches your local CUDA version to avoid computational incompatibility errors during runtime.

4. Download model weights: Download the AuK model weight files and the corresponding audio VAE files from Hugging Face or ModelScope, and place them in the designated directory within the repository. When downloading, confirm the model version (standard version or AuK-Flash distilled version), as the weight files from these two versions cannot be used interchangeably.

5. Download the semantic conditioning model: Download the multimodal large model Qwen2.5-Omni-3B separately. This model is responsible for parsing the user's natural language instructions and reference audio, generating semantic conditions for subsequent audio generation tasks.

6. Prepare input materials: Based on the task requirements, prepare the target text, reference audio (for Zero-shot TTS), or the original audio file to be edited. It is recommended to use WAV or FLAC formats for the audio, with a sampling rate that matches the model's requirements (24kHz).

7. Write natural language instructions: Compose clear and specific instructions according to the task type. For example, describe voice characteristics ("read with a deep, slow male voice"), specify sentences to be replaced ("change 'hello' to 'good morning'"), or indicate the required pitch/speech rate adjustments.

8. Run inference: Input instructions and specify the material path through the command-line interface (CLI), and execute the generation or editing task. Pay attention to the progress steps in the output logs, and adjust parameters such as instruction wording, sampling steps (e.g., increase to 32 steps), or CFG strength based on the quality of the resulting audio to achieve the desired outcome.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: If the core requirement is post-production speech editing—such as correcting recording errors, adjusting emotional tone, or removing accent noise—AuK is currently the most comprehensive open-source option, with its unified editing capabilities significantly outperforming CosyVoice 3, which only supports generation. If the application scenario emphasizes real-time interaction, such as voice assistants or online conversations, CosyVoice 3 offers a clear advantage with its 150ms first packet latency and streaming inference architecture. For lightweight conversational speech generation tasks, ChatTTS provides a lower deployment threshold and CPU inference capabilities, making it suitable for environments with limited resources.

6. Editor's Summary

AuK demonstrates a clear direction of technological innovation in the field of fundamental models for speech generation and editing. From an architectural perspective, it chooses a flow-matching diffusion approach over the autoregressive route, and unifies the editing task into a paradigm of "raw audio + instruction → output audio." This design inherently enables the model's capability for content modification while effectively preserving the acoustic characteristics of the unedited portions. The training scale of 3.03 billion instruction-audio pairs provides a data foundation for the model's cross-task generalization, and the introduction of human preference optimization and post-training with reinforcement learning further enhances the naturalness of the generated speech and the quality of instruction-following.

From a practical value standpoint, AuK compresses the previously complex post-production speech workflow—word modification, audio editing, emotional adjustment, noise reduction, and separation—into a single model's instruction-based operation, significantly lowering the technical barriers for audio production. For scenarios such as podcast creation, audiobook production, and film dialogue restoration, this "text-to-audio editing" method can effectively reduce the time and cost associated with repeated recording and manual post-production.

The model is positioned for professional users with a certain level of technical expertise: developers and audio engineers can use AuK to build automated post-production pipelines for speech, while content creators can quickly generate diverse character voices using Instruct TTS. It is worth noting that the 24GB memory requirement excludes some individual users, but this also encourages the community to explore lightweight solutions such as quantization and inference optimization. As the open-source community continues to develop and adapt AuK, its potential for application in more specialized scenarios is worth ongoing attention.

7. Application Scenarios

• Post-Production Dialogue Repair for Film/Shorts: When actors mispronounce lines or when the script is slightly adjusted, use AuK to directly replace individual words or phrases in the recording, eliminating the need for re-recording in the studio. This preserves the original emotional tone and intonation of the performance, significantly improving post-production efficiency.

• Podcast and Interview Production: Utilize the voice content editing features to remove verbal mistakes and filler words, adjust speaking pace and rhythm using acoustic attribute editing, and eliminate background noise and echo with the noise reduction function. This allows for the rapid organization of rough interview footage into polished, ready-to-publish content.

• Audiobook and AI Voice Production: Use Instruct TTS to generate distinct voice tones based on character traits. In post-production, the same narrator can adjust the emotional intensity of a specific chapter through emotion editing, reducing the production costs of multi-character audio content.

• Music Creation and Remixing: Use the lyric editing feature to rewrite lyrics of cover songs while preserving the original melody and vocal characteristics, or extract vocals from existing songs using music separation as Remix materials, providing musicians with flexible creative tools.

• Language Training and Accessibility Applications: Remove accents, adjust speaking rate and volume for users with accents or language barriers, generating clearer and more understandable audio content. This aids in the creation of language learning materials and provides communication support for people with special needs.

8. FAQ

Q: What is the fundamental difference between AuK and traditional TTS models?
A: Traditional TTS models can only generate entirely new speech from text and cannot modify existing audio content. In addition to its generation capabilities, AuK supports fine-grained editing of existing recordings at the content, acoustic properties, emotional, and paralinguistic levels, extending the capability from "only generation" to "generation + editing."

Q: What hardware configuration is required for AuK inference? Can it run on consumer-grade GPUs?
A: The standard version of AuK inference requires a GPU with approximately 24GB of VRAM, such as the RTX 3090 or RTX 4090. Consumer-grade GPUs (e.g., RTX 3060 with 12GB VRAM) typically cannot directly run the full model. Users can try using the distilled version, AuK-Flash, or explore model quantization solutions to reduce VRAM requirements.

Q: What is the difference between Zero-shot TTS and Instruct TTS? What are their respective use cases?
A: Zero-shot TTS requires providing a few seconds of reference audio to clone a specific voice, making it suitable for scenarios where a specific person's voice needs to be replicated, such as voice acting for a deceased actor. Instruct TTS, on the other hand, can generate speech based solely on textual descriptions, such as "a gentle female voice," making it ideal for quickly generating diverse character voices without the need to find reference audio samples.

Q: What languages does AuK support? How effective is it with Chinese?
A: The official list of fully supported languages has not been published, with examples and benchmark tests primarily focusing on Chinese and English. The model has been trained extensively on Chinese speech synthesis and editing data, delivering good results. However, the specific support for lesser-used languages and dialects remains to be confirmed by the official release.

Q: How can the voice consistency be maintained when editing speech with AuK?
A: AuK is based on a flow-matching diffusion architecture. During editing, the latent representation of the original audio is used as the starting point for denoising, and only the parts specified by the instruction are modified. As a result, the original voice characteristics, melody, and prosody of the unmodified content are well preserved. It is recommended to clearly specify the exact content to be modified in the instruction to avoid overly broad descriptions that may lead the model to unnecessarily reconstruct the entire audio.

Q: Can the model be used for commercial purposes?
A: AuK has been publicly released on GitHub and HuggingFace, with its code and weights open to the community. Specific commercial licensing terms should be confirmed by reviewing the LICENSE file in the model repository. Generally, open-source models can be used in commercial products as long as the corresponding license requirements (such as attribution and sharing agreements) are met.

9. Project Links

• Project Website: https://auk-project.github.io/
• GitHub Repository: https://github.com/Tencent-Hunyuan/AuK
• Hugging Face Model Library: https://huggingface.co/tencent/AuK
• arXiv Technical Paper: https://arxiv.org/pdf/2609.08936