Qwen-Audio-3.0-Realtime – Alibaba's Real-Time Speech Interaction Model

1. What is Qwen-Audio-3.0-Realtime

Qwen-Audio-3.0-Realtime is a new generation of real-time speech interaction dialogue model introduced by Alibaba Cloud's Tongyi team, offering Plus and Flash versions. It maintains high inference depth while achieving millisecond-level response speed. Built on On-Policy Distillation and multi-teacher distillation technology, the model fully transfers the capabilities of large text models into the speech modality. It supports no-instruction Agent tool calling, dynamic emotional tone adjustment, and full-duplex conversation. The model is now available on Alibaba Cloud's BaiLian platform, suitable for scenarios such as intelligent customer service, education and training, and emotional companionship.

Image source: Official article
Image source: official article

Technical positioning and domain: This model belongs to the field of end-to-end real-time speech interaction large models, aiming to replace traditional cascaded speech systems (ASR + NLU + TTS) and achieve an integrated solution for speech understanding, reasoning, and generation. Its unique feature lies in simultaneously pursuing low latency and high intelligence, inheriting the reasoning capabilities of large text models through distillation techniques, rather than merely optimizing speech recognition or synthesis.

Development background: Developed primarily by the Alibaba Cloud Tongyi Qianwen team, which has extensive experience in the Qwen series of large models. Previously, the team launched the Qwen-Audio series of speech models. Qwen-Audio-3.0-Realtime represents a significant iteration in the direction of real-time interaction, aiming to address the pain points of existing speech assistants, such as slow response times, low intelligence, and the inability to autonomously call tools.

Core value: It solves the long-standing challenge in real-time speech conversations of achieving both speed and intelligence. Through On-Policy Distillation, the speech model is corrected in real-time by the text large model during answer generation, ensuring high-quality reasoning. At the same time, it supports no-instruction tool calling and full-duplex conversation, enabling the speech assistant to actively call tools such as API and MCP, and provide a natural interactive experience with simultaneous speaking and listening, and the ability to interrupt at any time.

2. Key Features

• Dual-Version Architecture: Offers two versions, Qwen-Audio-3.0-Realtime-Plus and Flash. The Plus version achieved a score of 90.5 in the VoiceBench spoken language evaluation, demonstrating strong reasoning capabilities, making it suitable for scenarios with high intelligence requirements. The Flash version has lower latency, with only 5.5% degradation in spoken language across multiple rounds of conversation, ideal for applications requiring extremely fast response times.

• Millisecond-Level Response: For latency-sensitive scenarios such as everyday conversations, the model can generate responses directly without waiting for the full inference chain. Its end-to-end architecture eliminates intermediate delays inherent in traditional cascaded solutions. Real-world testing shows that the first character latency can be as low as hundreds of milliseconds, while maintaining high response quality without significant degradation due to speed.

• No-Instruction Tool Calling: The model is equipped with agentic capabilities, allowing it to autonomously determine and invoke external tools (such as MCP, API, knowledge bases) based on context, without requiring users to explicitly state trigger phrases like "open XX." The results of these tool calls are automatically integrated into the multi-turn memory, enabling follow-up questions to be answered seamlessly and achieving a frictionless tool integration experience.

• Dynamic Emotional Expression: The model can dynamically adjust tone, rhythm, pitch, and emotion based on the conversation context, supporting paralinguistic signals such as laughter and sighs. In the Chinese emotional expression evaluation VStyle, the Plus version achieved a tied first place with a score of 4.22, delivering empathetic responses at a human-like level.

• Full-Duplex Conversation: Built-in multimodal perception full-duplex control sub-model enables simultaneous analysis of audio signals, semantic content, and speaker voiceprint features to determine the conversation pace. It supports speaking while listening, interruptions, and interjections, with minimal false interruptions in noisy environments and the ability to identify the main speaker in multi-person discussions.

• Voiceprint Locking: By uploading an audio sample of the target speaker via the audio_prompt field, the model can lock onto that specific voiceprint and focus the conversation accordingly. In multi-speaker scenarios, this feature ensures the model only responds to the designated individual, avoiding confusion. It is suitable for use cases such as meeting transcription and targeted interaction.

3. How to Use

1. Environment Requirements and Prerequisites: To use Qwen-Audio-3.0-Realtime, you need an Alibaba Cloud account and must enable the BaiLian platform model service. It is recommended to use a stable network environment to ensure low latency for WebSocket bidirectional communication. The development environment should support a WebSocket client (such as the websockets library in Python or the WebSocket API in JavaScript).

2. Obtaining an API Key: Log in to the Alibaba Cloud BaiLian console (bailian.console.aliyun.com), go to the "Model Marketplace" and search for Qwen-Audio-3.0-Realtime. Follow the on-page instructions to activate the model service and obtain an API-Key or Access Token. Be sure to distinguish between the Plus and Flash versions' endpoints, as they may differ in terms of endpoint and pricing.

3. Integrating into Applications: Connect to the model service using the WebSocket protocol. For example, in Python, use the websockets library to establish a connection, send an authentication request containing the API-Key, and then begin transmitting the audio stream. The official help documentation provides complete request formats and sample code, including audio encoding (PCM/OPUS) and message structure.

4. Configuring Tools and Knowledge Bases: Based on the FunctionCall protocol, declare callable tools (MCP service, custom API, knowledge base) in the request. The model will automatically trigger these calls during the conversation, without requiring explicit user instructions. You must configure the tool descriptions and parameter schema in advance on the BaiLian platform to ensure the model correctly understands the tool's purpose.

5. Customizing Voiceprints (Optional): If you need to lock in a specific speaker, upload an audio sample of the target speaker (in PCM or OPUS format) via the audio_prompt field when establishing the connection. The model will focus on this voiceprint throughout the session and ignore other speakers. This feature has certain requirements for the clarity and length of the audio sample; it is recommended to provide 3–5 seconds of clean speech.

6. Best Practices: It is recommended to implement audio buffering and streaming processing on the client side to fully leverage the model's low-latency characteristics. For bidirectional scenarios, handle audio input and output in parallel and use the model's interruption signals to manage the conversation state. Be aware that API calls may be subject to quota limits; in production environments, it is advisable to apply for a higher QPS and implement a reconnection mechanism to handle network fluctuations.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For scenarios requiring high intelligence and complex tool calling, Qwen-Audio-3.0-Realtime's automatic tool calling without explicit instructions and full-duplex capabilities are clearly advantageous, making it particularly suitable for applications such as intelligent customer service and office assistants that require autonomous decision-making. GPT-4o Realtime excels in maturity and global ecosystem integration, making it a better fit for cross-border operations or teams relying on the OpenAI ecosystem. If the application scenario is primarily focused on Chinese vertical domains (such as education or healthcare) and does not require ultra-low latency, Baidu Wenxin or iFLYTEK Spark may offer stronger advantages in industry-specific customization, though their overall interaction naturalness lags behind Qwen.

In emotional companionship and entertainment interaction scenarios, Qwen's dynamic emotional expression and paralinguistic signals are closer to human-like behavior, while iFLYTEK Spark's rich voice styles may be more suitable for specific role-playing applications. Developers should choose based on their balance of platform dependency, cost budget, and functional requirements. For scenarios that prioritize ultra-fast response times and low-cost simple conversations, the Flash version is competitive; however, for complex scenarios requiring deep reasoning and toolchain integration, the Plus version is a more appropriate choice.

6. Editor's Summary

From a technological innovation perspective, Qwen-Audio-3.0-Realtime's use of On-Policy Distillation and a multi-teacher distillation strategy represents a significant breakthrough in the voice domain. Traditional voice models typically acquire capabilities through knowledge distillation from text models, but this is mostly static. On-Policy Distillation, however, introduces a large text model in real-time during inference to correct the voice model, allowing it to maintain generation speed while achieving inference depth comparable to that of text models. The inclusion of four teachers—spoken language, general-purpose, Agentic, and audio understanding—ensures robust capabilities across different dimensions, avoiding the limitations that can arise from relying on a single teacher. The design of the duplex control sub-model also demonstrates a deep understanding of real-world interactive scenarios, enabling interruption handling and speaker identification with multi-modal perception, a feature that is relatively rare in similar products.

In terms of practical value, this model directly addresses two major pain points of voice assistants: insufficient intelligence and rigid interaction. The ability to perform tool calling without explicit instructions allows voice assistants to evolve from passive responders to proactive service providers, enabling users to perform complex tasks without memorizing specific commands. Dynamic emotional expression significantly enhances immersion in companion scenarios. For enterprise users, integration with the Alibaba Cloud BaiLian API simplifies deployment, but it also introduces challenges related to platform lock-in and cost control.

This model is primarily suitable for two groups: first, developers and enterprises looking to build intelligent customer service or voice assistants; and second, product teams aiming to provide natural voice interaction in education and training, as well as emotional companion domains. For individual developers, the free API quota can suffice for prototyping and validation, but cost evaluation is necessary for large-scale commercial use. Looking ahead, if Alibaba Cloud can offer more granular emotional control parameters or localized deployment options, the model's market competitiveness will be further enhanced. Overall, Qwen-Audio-3.0-Realtime has achieved a new level of technical integration in real-time voice interaction, making it one of the notable voice models in today's market.

7. Application Scenarios

• Smart Customer Service: In e-commerce, finance, and other industries, the model can respond to user inquiries in milliseconds, automatically invoking tools such as order lookup and return/exchange processing to complete closed-loop services. Its duplex capability allows users to interrupt and add information at any time, while the reduction in instruction calling minimizes user interaction steps, improving service efficiency.

• Education and Training: As a speaking practice tool, the model can correct pronunciation and grammar errors in real time and dynamically adjust its encouraging tone based on the learner's emotional state. It supports multi-turn dialogue practice, simulating real-life communication scenarios, and is suitable for language learning, interview training, and more.

• Emotional Companionship: By recognizing paralinguistic signals such as laughter and sighs, the model can provide empathetic responses and adjust its tone according to the user's emotional changes, offering natural emotional support. It is applicable in mental health assistance and elderly companionship scenarios, and its voiceprint locking feature ensures the conversation remains focused on a specific user.

• Office Meetings: In meeting scenarios, the model can lock onto the main speaker's voiceprint, record key points in real time, and automatically invoke calendar and email tools to execute follow-up tasks. Its duplex capability allows participants to interject and provide additional input at any time, without the model mistakenly interrupting, thereby enhancing meeting efficiency.

• Entertainment Interaction: Supports immersive interactions such as debates and role-playing, allowing users to interrupt or switch topics anytime. The model's dynamic emotional expression makes characters more lifelike, making it suitable for game NPCs and live-streaming interactions, thus enhancing user experience.

8. FAQ

Q: Does Qwen-Audio-3.0-Realtime support Chinese?
A: Yes, it supports Chinese and has been specifically optimized for spoken Chinese expressions. In the VStyle Chinese emotional expression evaluation, the Plus version achieved a tied first place. It also performs excellently in Chinese spoken language understanding on the VoiceBench benchmark.

Q: What are the main differences between the Plus and Flash versions?
A: The Plus version has stronger inference capabilities, scoring 90.5 on the VoiceBench spoken language evaluation, making it suitable for scenarios requiring high intelligence and complex reasoning. The Flash version has lower latency, with only 5.5% degradation in spoken language understanding during multi-turn conversations, making it ideal for real-time interaction scenarios with extremely high response speed requirements.

Q: How can I obtain an API key?
A: Log in to the Alibaba Cloud Bailing console (bailian.console.aliyun.com), search for Qwen-Audio-3.0-Realtime in the Model Plaza, and follow the on-screen instructions to activate the service and obtain an API-Key. For detailed steps, please refer to the official help documentation.

Q: Is local deployment supported?
A: Currently, the model is only available via the Alibaba Cloud Bailing API and does not support local deployment or model weight downloads. Users must connect to the cloud service through WebSocket to make calls.

Q: Is interruption during duplex conversation smooth?
A: The model includes a built-in multimodal perception duplex control sub-model, supporting natural interruptions and being less prone to false interruptions in noisy environments. During group discussions, it can identify the main speaker and maintain smooth interruption handling, which is ahead of similar products in the industry.

9. Project Links

• Aliyun Bailing Platform (Model Service Entry): https://bailian.console.aliyun.com/ (search for Qwen-Audio-3.0-Realtime)
• Official User Guide: https://help.aliyun.com/zh/model-studio/qwen-audio-realtime-user-guides
• Official Aliyun Qwen Organization (GitHub): https://github.com/QwenLM (model weights are not open-sourced; service is only available via API)