AudioX-Turbo – A Unified and Efficient Audio Generation Framework Jointly Released by Noiz AI and Tsinghua University

1. What is AudioX-Turbo

AudioX-Turbo is a unified and efficient audio generation framework jointly developed by Noiz AI, the Hong Kong University of Science and Technology, and Tsinghua University. Based on a multimodal diffusion Transformer architecture with 2.7B parameters, the framework supports any combination of text, video, and audio as input, enabling the generation of high-quality sound effects and music segments. Through distribution matching distillation (DMD) and adversarial distillation techniques, the model reduces the number of inference steps from the traditional 50–200 steps to just 4 steps. It can generate 10 seconds of audio in only 0.24 seconds on a single RTX 4090 GPU, achieving near-real-time generation speed. The team also built a strong instruction dataset called IF-caps-Pro, containing approximately 9.2 million samples, and for the first time in audio generation, achieved precise timestamp control, allowing the model to understand and execute complex temporal instructions such as “have the cicada call for 3 seconds, then add a guitar at the 5th second.”

Technical Positioning and Domain: AudioX-Turbo belongs to the field of multimodal audio generation, positioned as a high-efficiency, controllable audio synthesis framework under a unified architecture. Unlike traditional single-task audio generation models (e.g., those that only support text-to-audio), this framework covers multiple tasks through a multimodal diffusion Transformer, including text-to-audio, text-to-music, video-to-audio, video-to-music, joint generation of text and video, audio completion and restoration, and image-to-audio (zero-shot), aiming to replace multiple specialized models with a single one.

Development Background: This framework was led by Noiz AI and co-developed with research teams from the Hong Kong University of Science and Technology and Tsinghua University. The team has deep expertise in diffusion models, flow matching, and audio signal processing. The motivation for development stems from the common pain points in current audio generation models, such as slow inference speed (typically requiring 50–200 sampling steps), lack of multi-task unification (requiring separate models for different modalities), and weak instruction-following capabilities (difficulty in controlling the temporal order of audio events).

Core Value: The core value of AudioX-Turbo lies in three aspects: first, ultra-fast inference, where 4 sampling steps can match the audio quality of the teacher model with 100 steps, reducing the number of function evaluations (NFE) by approximately 25 times, making real-time audio generation possible on consumer-grade GPUs; second, unified multimodal support, with a single model capable of handling any combination of text, video, and audio inputs, significantly reducing the complexity of deployment and inference; third, precise instruction-following, enabled by a structured dataset with timestamps, event counts, and instrument annotations containing around 9.2 million samples, allowing the model to accurately execute complex temporal instructions, a rare feature in previous audio generation models.

Technical Features: The core technologies include the multimodal diffusion Transformer architecture (based on the flow matching framework, where text, video, and audio modalities are uniformly projected and then input into a shared Transformer for cross-attention fusion), a two-stage distillation strategy (distribution matching distillation + adversarial distillation, compressing sampling steps to 4 and surpassing the teacher model in some metrics), and a three-stage progressive training strategy (pure text-audio pre-training → introduction of video multimodal joint generation → distillation acceleration), ensuring that the model maintains strong multimodal understanding capabilities while reducing inference costs.

2. Key Features

• Text-to-Audio (T2A): Generate ambient sounds and sound effects based on pure text prompts. Supports precise descriptions of sound types, styles, and scenarios, such as "the sound of raindrops hitting a metal roof with a slight echo," allowing the model to produce highly matching audio samples.

• Text-to-Music (T2M): Generate music clips according to textual instructions, specifying style, instrumentation, tempo, and emotion. Supports complex musical descriptions, such as "in C major, 120 BPM, with piano and strings, transitioning from melancholic to passionate," enabling the model to output music that meets the specified requirements.

• Video-to-Audio (V2A): Automatically add voiceover and sound effects (Foley) to silent videos. The model matches action sound effects and ambient sounds in real-time based on visual content, such as footsteps, door closing, or wind rustling through leaves, without requiring manual annotations.

• Video-to-Music (V2M): Analyze the emotion, tempo, and visual dynamics of a video to automatically generate synchronized background music. Supports various music genres (classical, electronic, jazz, etc.), ensuring the music aligns closely with the video's rhythm and mood, ideal for short video production.

• Text + Video Joint Generation (TV2A / TV2M): Combine video visuals with text instructions for precise control. Users can specify timestamp-level instructions such as "thunder at 3 seconds, guitar at 5 seconds," and the model will strictly follow the sequence to generate sound effects or music.

• Audio Completion and Restoration: Complete, restore, or transfer the style of audio based on context. For example, reduce noise in a recording or intelligently continue interrupted audio, maintaining consistency in tone and rhythm.

• Image-to-Audio (Zero-shot): No need for specialized training on images; the model can infer and generate corresponding ambient sounds or scene-specific sound effects based on static image content. For instance, inputting a beach photo can result in the generation of sounds like ocean waves and seagull calls.

3. How to Use

1. Environment Setup: Clone the GitHub repository (git clone <link to be updated after official release>), create a Python 3.8 virtual environment, and install FFmpeg, libsndfile, and the dependencies in requirements.txt. It is recommended to use CUDA 11.8 or later to enable GPU acceleration.

2. Weight Download: Download the pretrained weights from the Hugging Face model hub (HKUSTAudio/AudioX-Turbo) or the GitHub Release page, including the 4-step student model, VAE (Variational Autoencoder), and Synchformer video encoder. The weight files are large (approximately 5–6 GB), so ensure sufficient disk space.

3. Launch Inference: Run python run_gradio.py to start the local web interface, then open http://localhost:7860 (link to be updated after official release) in your browser to operate via the GUI. You can also directly call the Python API: from audiox import AudioXTurbo; model = AudioXTurbo.from_pretrained("path/to/weights").

4. Parameter Settings: Set input parameters based on the task type, such as video_path (video file path), text_prompt (text description), and audio_path (audio file path, for completion tasks). Supports arbitrary combinations of inputs, e.g., providing only text_prompt for T2A tasks, or both video_path and text_prompt for TV2A tasks.

5. Generate Audio: Run forward inference. By default, 4-step sampling generates 10 seconds of high-quality audio. It takes approximately 0.24 seconds on an RTX 4090 and about 0.5 seconds on an RTX 3090. The output is a 44.1 kHz WAV file, which can be saved using the save_audio() method.

Notes: The model has high VRAM requirements; a GPU with at least 24 GB of VRAM is recommended. If VRAM is insufficient, try reducing the sampling length or using FP16 inference. The first model load requires downloading weights; a stable network connection is recommended. For video input, ensure the video frame rate is at least 24 fps for optimal audio-visual synchronization.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For scenarios requiring ultra-fast inference and precise timestamp control (such as real-time game voice acting or film Foley production), AudioX-Turbo is currently the best choice, with its 4-step distillation and structured instruction-following capabilities being unique among similar products. If generation speed is not a priority and only text-to-audio/music is needed, AudioLDM 2 is a stable and reliable option, thanks to its mature community and MIT open-source license. For video-to-audio tasks, MMAudio has fewer parameters but more inference steps, resulting in slower performance compared to AudioX-Turbo. While Stable Audio offers high-quality output, it is not open source and only provides an API, making it suitable for users requiring commercial-grade stability, though it offers weaker controllability.

6. Editor's Take

AudioX-Turbo demonstrates outstanding innovation in technology. Its two-stage distillation strategy (DMD + adversarial distillation) reduces the sampling steps from 50–200 to just 4, while maintaining or even improving the quality of generated audio. This marks a milestone advancement in the field of audio generation. Combined with its self-built structured dataset IF-caps-Pro containing 9.2 million entries, it achieves precise timestamp control down to the second, enabling complex temporal instructions such as "first cicada chirping, then guitar," significantly enhancing the controllability of audio generation.

In terms of practical value, this framework unifies the generation tasks of text, video, and audio modalities. A single model can cover multiple scenarios, including T2A, T2M, V2A, and V2M, greatly reducing deployment and maintenance costs. Its ultra-fast inference capability makes it suitable for real-time interactive applications (such as game voice acting and live streaming audio effects), which previous audio generation models struggled to achieve.

Target audience: Primarily aimed at AI audio researchers, game developers, post-production professionals in film and television, and music creators. For researchers, the distillation method and structured dataset offer valuable references. For developers, the open-source Apache 2.0 license allows commercial use and further development. For creators, the ultra-fast inference and precise control can significantly boost work efficiency.

Future development potential: As model weights are further optimized (e.g., through quantization and pruning), it may be able to run on devices with lower memory capacity. Additionally, if the community contributes Chinese datasets and fine-tuning solutions, the current lack of Chinese support could be addressed. The scalability of the multi-modal unified architecture also suggests that it can seamlessly integrate more modalities (such as haptics and motion capture) in the future, enabling the construction of more complete immersive audio generation systems.

7. Application Scenarios

• Real-time Dubbing for Interactive Drama and Games: 4-step fast inference supports real-time sound effects and dynamic music generation within game engines. For example, in open-world games, the model can generate corresponding sound effects based on player actions (walking, jumping, fighting) in real time, without the need for pre-recorded audio libraries, enhancing immersion.

• Foley Sound Design for Post-production Video: Automatically generate precise environmental and action sound effects from silent video footage, reducing the cost of manual foley work. Input a chase scene video, and the model can automatically generate sounds such as footsteps, fabric friction, and background wind; only minor adjustments are needed in post-production.

• AI Live Streaming and Virtual Anchors: Generate background music or interactive sound effects in real time based on the visual content and live chat text. When a virtual anchor receives a gift, the model can generate corresponding sound effects based on the gift icon (e.g., a rocket); if the chat mentions "rain," the model automatically adds rain sounds.

• Assisted Music Composition: Quickly generate reference music segments based on text descriptions, supporting synchronized music with video emotions. If a composer inputs "a sad piano piece, adagio, with string accompaniment," the model can output multiple versions within seconds for reference, accelerating the creative process.

• Automated Audio Content Generation: Combine timestamps to precisely control the sequence of audio events, suitable for podcast and audio drama scenarios. For example, when producing a detective audio drama, you can specify "door opening sound at 2 seconds, footsteps approaching at 5 seconds, dialogue starting at 8 seconds," and the model strictly follows the timeline to generate a complete audio track.

8. FAQ

Q: Does AudioX-Turbo support Chinese text input?
A: The model is currently primarily trained on the English dataset IF-caps-Pro, and official Chinese test results have not been made public. Theoretically, the T5 encoder supports multiple languages, but the generation results for Chinese instructions may not be as stable as for English. It is recommended to prioritize using English prompts.

Q: Can the model be used for commercial purposes?
A: Yes. AudioX-Turbo uses the Apache 2.0 open-source license, allowing free use, modification, and commercial application, but the copyright notice must be retained. For specific terms, please refer to the LICENSE file in the GitHub repository.

Q: How much VRAM is required to generate a 10-second audio clip?
A: It can run smoothly on an RTX 4090 (24 GB VRAM). If using a GPU with 16 GB VRAM (such as the RTX 3080 Ti), it is recommended to reduce the sampling length to 5 seconds or enable FP16 inference. GPUs with 8 GB VRAM may not be able to load the full model, so model quantization should be considered.

Q: What are the main advantages of AudioX-Turbo compared to AudioLDM 2?
A: The main advantages are inference speed (4 steps vs. 50–200 steps) and precise timestamp control. AudioLDM 2 only supports global style descriptions and cannot accurately specify the exact time points when audio events occur, whereas AudioX-Turbo can achieve "thunder at 3 seconds, guitar at 5 seconds," which is crucial for narrative audio generation.

Q: How can I train my own dataset?
A: The official documentation currently does not provide detailed training scripts or tutorials. The community can refer to the construction method of the IF-caps-Pro dataset (structured timestamp annotations, event count, instruments, etc.), but this requires a high technical threshold. It is recommended to wait for the official release of a training guide.

9. Project Links

• GitHub Repository: https://github.com/NoizAI/AudioX-Turbo
• HuggingFace Model Hub: https://huggingface.co/HKUSTAudio/AudioX-Turbo
• arXiv Technical Paper: https://arxiv.org/pdf/2606.12555