Xiaomi-CocktailASR-1: In-Depth Evaluation of a Target Speaker ASR Model Based on an End-to-End LLM Architecture

1. What is Xiaomi-CocktailASR-1

Xiaomi-CocktailASR-1 is Xiaomi's open-source Target Speaker ASR (TS-ASR) large model, designed using an end-to-end LLM architecture. It uses a reference speech as a speaker embedding prompt to accurately transcribe the content of the target speaker from mixed speech without requiring voice separation. The model achieves state-of-the-art (SOTA) performance on multi-speaker benchmarks, is compatible with single-speaker scenarios, and features negative sample rejection and chain-of-thought explainable reasoning capabilities. It is open-sourced under the Apache 2.0 license.

Technical Positioning and Domain: This model belongs to the intersection of speech recognition and speaker technology, focusing on transcribing the content of a target speaker in a multi-speaker mixed speech environment. Unlike traditional speech recognition systems, it directly injects speaker features as conditional prompts into the end-to-end recognition pipeline, bypassing the cascaded voice separation architecture. This represents a cutting-edge exploration in the TS-ASR direction.

Development Background: Developed and open-sourced by Xiaomi's research team, leveraging Xiaomi's technical expertise in smart hardware and voice interaction. The model is built upon improvements to the audio encoder of Data2Vec2 and the Qwen3-8B large language model backbone, reflecting the trend of migrating speech recognition toward large model architectures. Xiaomi has made its research results available under the Apache 2.0 license to promote the practical application and continuous iteration of speech recognition technology in multi-speaker scenarios.

Core Value: It addresses the pain points of traditional cascaded systems that combine voice separation and recognition, which suffer from error accumulation and insufficient robustness in multi-speaker environments. By implementing a reference speech prompt mechanism, the model enables direct transcription of the target speaker using a single model. It not only supports multi-speaker scenarios but is also compatible with single-speaker recognition, with substantive innovations in negative sample rejection and reasoning explainability, significantly enhancing usability in real-world applications.

Technical Features: An end-to-end architecture that uses reference speech as a conditional prompt, eliminating the need for a voice separation module; trained through multi-stage learning with approximately one million hours of data, integrating four capabilities: target extraction, single-scenario compatibility, negative sample rejection, and chain-of-thought reasoning; can be called with just a single line of code.

2. Key Features

• Target Speaker Speech Recognition: Given a reference speech and a mixed audio with multiple speakers, the model directly transcribes the content of the target speaker without requiring voice separation. On the AliMeeting Far field adaptation set, WER is as low as 20.63%, and on the LibriMix 2mix scenario, WER is only 4.11%, significantly outperforming previous SOTA levels (AliMeeting Far 27.5%).

• Single Speaker Scenario Compatibility: The same model can handle single-speaker recognition tasks without needing to switch models or add extra configurations based on the number of speakers. On the LibriSpeech clean test set, WER is 1.73%, and on the CommonVoice Chinese set, WER is 4.95%, with performance comparable to mainstream single-speaker ASR systems.

• Negative Sample Rejection: When the target speaker is not present in the audio, the model automatically outputs empty text instead of erroneously transcribing content. In LibriSpeech negative sample testing, the rejection rate reaches 79.59%, with an overall rejection rate range of 68%-80%. In contrast, models such as Qwen3-ASR and StepAudio have a rejection rate of 0%.

• Chain-of-Thought Explainable Reasoning: In CoT mode, the model outputs reasoning processes such as the number of speakers, gender, and voiceprint similarity before providing the final transcription result, balancing explainability and recognition accuracy. Experiments show that CoT mode not only improves reasoning transparency but can also slightly reduce WER.

• End-to-End Unified Architecture: Based on the improved Data2Vec2 audio encoder with 0.6B parameters, a lightweight linear Adapter, and the Qwen3-8B large language model backbone. Input is concatenated in the order of "reference speech + 1 second of silence + mixed speech," and the encoder outputs frame-level features that are aligned to the LLM's hidden space via the Adapter. The reference speech is treated as a conditional prompt, enabling the encoder to adaptively focus on the target speaker during feature extraction and preventing error accumulation from cascaded systems at the source.

• Multi-Stage Training for Balanced Capabilities: Approximately 400,000 hours of multi-speaker data are used for target extraction training, while around 600,000 hours of same-speaker reference-target pairs are used to prevent over-suppression in single scenarios. Around 10,000 hours of mismatched reference negative samples are used to internalize rejection capabilities. Additionally, CoT data trains the model to first output a reasoning chain before providing an answer. These four types of data work together to achieve a balanced set of capabilities.

• Out-of-the-Box Low-Barrier Usage: A single line of code is sufficient to perform inference, with support for automatic resampling, batch inference, and positive/negative sample demos. After installing dependencies via pip, weights can be downloaded from HuggingFace, and calling model("target.wav", "ref_speaker.wav") will return the transcription result.

3. How to Use

1. Environment Requirements: Requires Linux or Windows operating system, equipped with an NVIDIA GPU (recommended VRAM no less than 16GB), and Python version 3.8 or higher. The model is loaded with bfloat16 precision; CPU inference is slow and not recommended for production environments.

2. Install Dependencies: Run pip install torch torchaudio transformers soundfile to install the required environment. It is recommended that the transformers version be no lower than 4.40 to ensure compatibility with the model code.

3. Download the Model: Download all model weight files from the HuggingFace model library (Ease3/Xiaomi-CocktailASR-1), including config, tokenizer, and model weights. The total model size is approximately 8.6B parameters; ensure a stable network connection during download.

4. Load the Model: Load the model using the following Python code:

   from transformers import AutoModel
   model = AutoModel.from_pretrained(
       "Ease3/Xiaomi-CocktailASR-1",
       trust_remote_code=True,
       torch_dtype="bfloat16"
   ).cuda().eval()
   

   The trust_remote_code=True parameter is essential, as the model relies on remote code execution.

5. Prepare Audio: Prepare a 16k mono reference speech (recording the target speaker's voiceprint), as well as a target audio file containing one or more speakers. Audio with a sampling rate other than 16k will be automatically resampled; no manual processing is required.

6. Single Inference: Simply call model("target.wav", "ref_speaker.wav") to return the transcribed text of the target speaker. For negative sample inputs, the model automatically outputs empty text, without requiring additional parameters or post-processing.

7. Chain-of-Thought Inference: Append the cot=True parameter when calling model("target.wav", "ref_speaker.wav", cot=True). The model will output the reasoning process (such as the number of speakers, gender, and voiceprint similarity) within the thinking tag, and the final transcribed result within the <answer> tag.

8. Batch Inference: Organize the data into a 5-column TSV format (utt_id, wav, text, ref_wav, ref_id), run the tools/test_batch_scp.py script, and specify the model path, input TSV file path, and output file path to complete batch transcription, which is convenient for evaluating model performance on a test set.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For multi-person mixed speech scenarios such as intelligent meeting transcription and customer service quality inspection, Xiaomi-CocktailASR-1 is currently the most accurate open-source option for target speaker recognition, with its negative sample rejection capability offering a differentiating advantage in voice assistant products. If the primary requirement is single-speaker general speech recognition and multilingual support is needed, Qwen3-ASR has an advantage in language coverage, but its performance in multi-speaker scenarios is significantly lacking. StepAudio, as a general-purpose multimodal audio model, offers greater flexibility when handling diverse tasks such as speech and audio event processing simultaneously. However, it currently lacks publicly available data to support its TS-ASR capabilities, making it suitable as an alternative option for targeted testing.

6. Editor's Summary

The technical innovation of Xiaomi-CocktailASR-1 is reflected in three aspects: first, it integrates semantic and speaker information within a single network using the Data2Vec2 self-supervised masking prediction mechanism. By using speech as a conditional prompt to directly guide the encoder to focus on the target speaker, it avoids the structural flaws of traditional cascaded systems that suffer from error accumulation at the architectural level. Second, its multi-stage data proportion training strategy with approximately one million hours of data unifies four capabilities—target extraction, single-scenario compatibility, negative sample rejection, and chain-of-thought reasoning—into a single model. This approach to capability arrangement at the data level is worth emulating across the industry. Third, the introduction of negative sample rejection and CoT explainable reasoning ensures that the model not only has high-precision recognition capabilities but also provides reliability guarantees in its outputs, which is of significant importance for controlling false triggers in practical deployment scenarios.

In terms of practical value, the model's performance in reducing WER from 27.5% to 20.63% in the AliMeeting Far scenario and achieving a WER of 4.11% in the LibriMix 2mix scenario validates the technical feasibility of the end-to-end TS-ASR approach. Additionally, the Apache 2.0 open-source license significantly lowers the barrier to industry adoption. The model is applicable across three levels of users: speech technology researchers can use it as a strong baseline for TS-ASR research; product developers can leverage its target speaker identification and negative sample rejection capabilities to build differentiated voice interaction experiences; and industry users who need to process multi-speaker recordings can use the batch inference script to quickly complete transcription tasks.

The model's further development potential lies in expanding its multilingual capabilities, optimizing inference efficiency, and adapting for lightweight deployment on edge devices. The current parameter scale of 8.6B imposes certain requirements on the deployment environment, but future strategies such as distillation or quantization may enable it to penetrate more application scenarios. Overall, Xiaomi-CocktailASR-1 provides an open-source example worthy of in-depth study in the field of speech recognition, with both its technical approach and engineering implementation offering high reference value.

7. Application Scenarios

• Smart Meeting Targeted Transcription: In multi-person meeting scenarios, after the user provides their reference voice, the model transcribes only the content of the designated speaker, automatically filtering out interruptions and discussions from other participants. With a WER of 20.63% in the AliMeeting Far scenario, the model can output relatively reliable transcriptions when processing real-world far-field recordings in meeting rooms, meeting the needs of meeting minutes organization and speaker tracking.

• Voice Assistant Owner Command Recognition: Voice assistant devices such as smartphones, speakers, and car systems can use this model to identify the owner's voice, ensuring that commands from non-owner speakers are not triggered in environments with background voices or TV noise. The model's strong negative sample rejection capability ensures that the device does not respond erroneously when the owner is not present, enhancing the interaction experience and security boundaries.

• Customer Service and Quality Inspection Recording Analysis: Precisely extract the complete dialogue of a specified party from dual-party customer service call recordings for compliance checks, service evaluation, and script optimization. The model's WER of 1.73% in single-speaker scenarios approaches general ASR performance when the recording quality is clear, meeting the precision requirements for textual details in quality inspection scenarios.

• Smart Home Voice Control: In noisy home environments with multiple conversations, accurately recognize voice commands from the remote control holder or designated family member, preventing unintended operations caused by other speakers. Reference voices can be pre-recorded with family members' voiceprints, and during regular use, natural speech is sufficient to trigger the corresponding control commands.

• Accessibility Hearing Aid and Recording Devices: Designed for hearing-impaired individuals or users who need targeted voice recording, the model extracts speech from specific speakers in noisy social settings and converts it into text in real time. The model supports batch processing and CoT reasoning, and can be integrated with hearing aids or mobile applications, helping users focus on "hearing clearly" the person they want to listen to in multi-speaker conversation scenarios.

8. FAQ

Q: What is the fundamental difference between Xiaomi-CocktailASR-1 and traditional speech separation + recognition solutions?
A: Traditional solutions first use a speech separation model to extract the target speaker's voice from the mixed audio, then feed it into an ASR model for recognition. Errors in the separation stage can propagate and accumulate in the recognition results. This model directly inputs the reference speech as a conditional prompt into the encoder, allowing the feature extraction stage to adaptively focus on the target speaker and suppress interfering speech. It generates the transcription text in a single end-to-end step, thereby avoiding the error accumulation issues inherent in cascaded systems at the architectural level.

Q: What are the requirements for the reference speech? Will poor quality of the reference speech affect recognition performance?
A: The official recommendation is to use 16k mono-channel, clearly recorded reference speech with a duration of several seconds or more. The quality of the reference speech directly affects the accuracy of speaker feature extraction. If the reference speech contains significant noise, reverberation, or is truncated and incomplete, the encoder may fail to accurately model the target speaker's acoustic features, which in turn can affect recognition accuracy. Audio with a non-16k sampling rate will be automatically resampled, but this cannot compensate for the negative impact of insufficient signal-to-noise ratio.

Q: How is the negative sample rejection mechanism implemented? Is any additional threshold configuration required?
A: The model is trained on approximately 10,000 hours of negative samples with mismatched references, internalizing the behavior of "outputting empty text when the target is absent." During inference, no additional thresholds need to be set. When the input audio does not contain the target speaker, the model naturally outputs empty text, rather than erroneously transcribing content from other speakers. Experimental data shows a rejection rate of 68%-80%, and this capability can effectively reduce false trigger rates in scenarios such as voice assistants.

Q: What is the difference between CoT mode and standard mode? How should one choose between them?
A: Standard mode uses a unified prompt to cover three tasks: single-speaker recognition, multi-speaker target recognition, and negative sample rejection. It is suitable for scenarios where recognition speed is a priority. CoT mode uses an independent prompt to trigger the thinking inference tag, prompting the model to first output the reasoning process, such as the number of speakers, gender, and voiceprint similarity, and then provide the final transcription within the <answer> tag. While CoT mode improves interpretability and slightly reduces WER, it increases inference time slightly, making it suitable for scenarios requiring result interpretability or error analysis.

Q: Does the model support Chinese recognition? How does it perform with multiple languages?
A: The model achieves a single-speaker WER of 4.95% on the CommonVoice Chinese test set, indicating that its Chinese recognition capability is basically usable. Multi-speaker benchmark evaluations mainly use Chinese and English datasets (AliMeeting is a Chinese meeting dataset, and LibriMix is an English mixed speech dataset). Detailed evaluation data for other languages has not been officially published, and multilingual generalization performance still needs developers to conduct practical testing based on their specific language requirements.

Q: Can the model be used commercially? Are there any license restrictions?
A: Xiaomi-CocktailASR-1 uses the Apache 2.0 open-source license, which fully opens the code and model weights. It allows commercial use, free modification, and redistribution, provided that the original copyright declaration and license text are retained. This license is friendly to industry applications, and the model can be directly integrated into commercial products as long as the license terms are followed.

9. Project Links

• GitHub Repository: https://github.com/xiaomi-research/xiaomi-cocktailasr-1
• HuggingFace Model Library: https://huggingface.co/Ease3/Xiaomi-CocktailASR-1
• arXiv Technical Paper: https://arxiv.org/pdf/2609.11274