SALMONN-2 – A General Audio Large Language Model Open-Sourced by Tsinghua University and Others

1. What is SALMONN-2

SALMONN-2 is a general audio large language model open-sourced by Tsinghua University, the Shanghai AI Laboratory, and the University of Cambridge. The model employs the SPEAR unified self-supervised audio encoder from ByteDance and multi-layer feature fusion adapters, using Qwen3-8B as its text base. With only approximately 18,000 hours of supervised data, it achieves state-of-the-art performance on three major comprehensive benchmarks: MMAU-Pro, MMAR, and MMSU, surpassing other open-source models of similar scale. SALMONN-2 is the first general audio large language model to systematically implement advanced capabilities such as sound event detection, audio forgery detection, speech quality assessment, and multi-modal context-aware speech recognition, offering an efficient and unified solution for audio understanding.

Technical Positioning and Domain: It belongs to the audio understanding direction within multi-modal large language models, focusing on general audio understanding that covers speech, environmental sounds, and music. Its uniqueness lies in using a unified self-supervised encoder instead of traditional dual encoders, achieving more balanced cross-domain capabilities. At the same time, it retains acoustic information from each layer of the encoder through multi-layer feature fusion adapters, enhancing performance on fine-grained tasks while simplifying the architecture.

Development Background: Jointly developed by Tsinghua University, the Shanghai AI Laboratory, and the University of Cambridge, SALMONN-2 is built upon the SPEAR self-supervised audio encoder open-sourced by ByteDance, inheriting and advancing the architectural ideas of the SALMONN series. The research team aimed to reduce reliance on large-scale supervised data while expanding the coverage of audio analysis tasks, achieving a breakthrough in data efficiency.

Core Value: It addresses the issues of high data requirements and limited task coverage in previous audio large language models. Using only 18,000 hours of supervised data, SALMONN-2 can reach or even surpass models trained on tens of thousands of hours of data. It is also the first to systematically support professional tasks such as sound event detection, forgery detection, and speech quality assessment, significantly lowering the application threshold and deployment costs.

Technical Features: It uses a single SPEAR self-supervised encoder combined with multi-layer feature fusion adapters to retain acoustic information from each layer. Timestamps are injected in natural language form to enable event localization. Multi-modal context learning improves the recognition of rare words. The architecture is simple yet highly extensible; performance continues to improve as the base model scales from 8B to 30B-A3B, demonstrating strong scale-up potential.

2. Key Features

• General Audio Understanding: Supports cross-domain tasks such as speech recognition, audio description, music understanding, emotion recognition, and speaker verification. The model processes all tasks within a single framework, eliminating the need for separate training for each task, greatly simplifying the deployment process while maintaining balanced performance across all tasks.

• Sound Event Detection (SED): Can locate and output the start and end time intervals of environmental sound events, such as dog barking, footsteps, and alarm sounds. The model accomplishes time localization through a natural language timestamp injection mechanism within a unified generation framework, without requiring a dedicated embedding layer. This makes it highly valuable for practical applications in security monitoring and content moderation.

• Audio Deepfake Detection: Determines whether speech is synthetically generated or edited, and identifies the start and end time intervals of the forged segments. This capability is especially critical in preventing telecom fraud and verifying audio authenticity. The model can simultaneously output authenticity judgments and time intervals, providing interpretable detection results.

• Speech Quality Assessment (SQA): Perceives low-level acoustic features such as noise, distortion, and clarity, and provides a quality score along with a natural language explanation. It can be used in automated evaluation scenarios such as customer service call recording quality checks and audio studio material screening, helping to quickly identify problematic segments and reduce the cost of manual review.

• Multimodal Context-aware Speech Recognition (MICL): During speech recognition, the model simultaneously receives a text bias vocabulary and reference pronunciation audio for these words as context, significantly improving the accuracy of recognizing obscure words and rare names. During training, a strategy of randomly discarding interfering words is introduced to prevent over-biasing, allowing the model to reasonably utilize contextual clues based on acoustic evidence.

• Timestamp Event Localization: The model inserts timestamps in natural language format (e.g., <2.0 seconds>) into the audio sequence and feeds them along with the audio embeddings into the language model. This enables event time localization within a unified generation framework. This mechanism allows SED and deepfake detection to output precise time intervals without requiring a dedicated timestamp embedding layer.

3. How to Use

1. Environment Requirements and Prerequisites: It is recommended to use the Linux operating system, Python 3.10, CUDA 11.8 or higher, and an NVIDIA GPU (with a recommended VRAM of 16GB or more). You need to install conda, git, and the HuggingFace CLI (used for downloading weights). Ensure that your network can access GitHub and HuggingFace.

2. Clone the Code Repository: Execute git clone https://github.com/bytedance/SALMONN/tree/salmonn2 to download the code locally, and navigate to the root directory of the repository. Note that this branch is salmonn2; the default branch may be different, so you need to switch to the corresponding branch.

3. Create a Virtual Environment and Install Dependencies: Run conda create -n salmonn2 python=3.10 to create a virtual environment. After activating it, execute pip install --upgrade pip setuptools wheel. Then, in the root directory of the repository, run pip install -r requirements.txt to install runtime dependencies, and execute pip install -e . --no-deps to install the SALMONN-2 package in development mode.

4. Download Pretrained Weights: Download the model checkpoints using the HuggingFace CLI: hf download marcoyang/SALMONN-2-8B --repo-type model --local-dir /path/to/salmonn-2-hf. Alternatively, you can use the from_pretrained method from the transformers library to automatically download the weights, but pre-downloading can help avoid network issues during runtime.

5. Load the Model and Perform Inference: Load the local weights using AutoProcessor and AutoModelForCausalLM, provide the audio file path and instruction text, and call model.generate() to obtain the audio understanding results. Refer to the example code in the repository's inference.py. Note that the recommended audio file format is 16kHz mono WAV. You can adjust parameters such as max_new_tokens during generation to control the output length.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For scenarios requiring professional audio analysis (such as sound event detection, forgery detection, and speech quality assessment), SALMONN-2 is currently the only open-source model that natively supports these features. Its data efficiency makes it more appealing for academic research and resource-constrained environments. If the task involves only general speech recognition and is less sensitive to data scale, MOSS-Audio may offer more stable performance on standard ASR tasks due to its million-hour training data, although it lacks advanced audio analysis capabilities. For scenarios with a strong need for Chinese speech understanding, Qwen-Audio, based on the Qwen foundation, has strong Chinese capabilities. However, it also does not support specialized tasks like SED, and specific performance data is not disclosed, making it suitable for scenarios with lower multi-task requirements.

6. Editor's Summary

SALMONN-2 demonstrates significant technological innovation in the field of large audio language models. Its core highlights lie in data efficiency and task scalability: by unifying a self-supervised encoder with multi-layer feature fusion adapters, it surpasses models trained on millions of hours of data using just 18,000 hours of supervised data. Moreover, it is the first to systematically integrate specialized capabilities such as sound event detection, audio forgery detection, and speech quality assessment within a general framework, marking a shift from single-function speech recognition to comprehensive audio understanding in large audio models. In terms of practical value, SALMONN-2 can be directly applied to scenarios such as meeting transcription, content moderation, quality monitoring, and archive retrieval. Its multimodal context-aware speech recognition capability effectively addresses the challenge of recognizing obscure words, reducing error rates during actual deployment. The target audience includes audio processing researchers, AI application developers, and enterprises requiring automated audio analysis. Although the current documentation and community ecosystem are still in their early stages, the model's architectural simplicity and proven scale-up potential (performance continues to improve as the base model is expanded) suggest strong scalability, positioning it as a promising foundational open-source tool in the field of audio understanding. For development teams seeking data efficiency and broad task coverage, SALMONN-2 offers a compelling starting point worth exploring in depth.

7. Application Scenarios

• Smart Meeting Assistant: Real-time transcription of meeting content, combined with participants' pronunciation examples to accurately recognize foreign names and technical terms. MICL capabilities allow the model to leverage reference pronunciation audio to improve recognition accuracy, making it suitable for cross-border corporate meetings and reducing manual correction costs.

• Audio Content Moderation: Automatically detect abnormal sound events in live streams or podcasts (such as gunshots, explosions), and identify deepfake voices to prevent fraud. The SED and deepfake detection features can output event timestamps in real-time, facilitating manual review and automated alerts, thereby enhancing the efficiency of content security moderation.

• Voice Quality Monitoring: Automatically score the quality of customer service calls and recording studio materials, identifying segments with noise and distortion. The SQA feature provides quality scores and natural language explanations, helping to quickly pinpoint problematic sections, and is applicable for large-scale audio quality inspection and audio material screening.

• Multimedia Archive Retrieval: Generate timestamped event descriptions for historical audio and radio programs, enabling precise content-based retrieval. The model can simultaneously output speech transcription and environmental event descriptions, enriching metadata and making audio archives searchable at the text level.

• Assistive Devices for Hearing-Impaired Users: Real-time text and timestamp notifications for environmental sound events (such as doorbells, alarms) to hearing-impaired users. The SED feature can recognize key sounds and convert them into visual information, helping hearing-impaired individuals perceive their surroundings and improving their quality of life.

8. FAQ

Q: What hardware configuration is required to run SALMONN-2?
A: It is recommended to use an NVIDIA A100 or a GPU of comparable performance, with at least 16GB of VRAM. It may be possible to run SALMONN-2 using consumer-grade GPUs (such as RTX 3090/4090), but the inference speed will be slower. We recommend at least 32GB of system memory, and the hard drive should reserve at least 20GB of space for storing code and model weights.

Q: Does SALMONN-2 support audio understanding in Chinese?
A: Yes. The base model Qwen3-8B has strong capabilities in Chinese, so SALMONN-2 can handle tasks such as Chinese speech recognition and emotion recognition. Capabilities like sound event detection are language-agnostic and also applicable in Chinese environments. The performance of Chinese speech recognition depends on the base model and training data, but overall it performs well.

Q: How can I fine-tune SALMONN-2 on my own data?
A: The official open-source code currently mainly provides the inference pipeline, and the training code has not been fully released. If you want to fine-tune, you can refer to the training scripts from previous SALMONN series models, but you will need to adapt them for the SPEAR encoder and MLF adapter. It is recommended to follow updates on the GitHub repository or contact the research team for access to the training code.

Q: What advantages does SALMONN-2 have over Whisper?
A: Whisper focuses on speech recognition, while SALMONN-2 is a general-purpose audio understanding model. It supports not only speech recognition (ASR) but also multiple tasks such as sound event detection, music understanding, forgery detection, and speech quality assessment. Although Whisper may be more mature for pure speech recognition, SALMONN-2 has a clear advantage in terms of task coverage and consistency.

Q: How does the audio encoder SPEAR in SALMONN-2 differ from the Whisper encoder?
A: SPEAR is a self-supervised encoder pre-trained on large-scale unlabeled data, capable of capturing semantic, phonetic, timbre, speaker, and acoustic environment information simultaneously. Whisper, on the other hand, is supervised-trained and primarily optimized for speech recognition. The unified representation of SPEAR enables SALMONN-2 to perform more balanced across various audio tasks, with lower data requirements.

9. Project Links

• Project Homepage (GitHub Repository): https://github.com/bytedance/SALMONN/tree/salmonn2
• HuggingFace Model Library: https://huggingface.co/marcoyang/SALMONN-2-8B
• Technical Paper (arXiv): https://arxiv.org/pdf/2607.17079