Fun-ASR-Realtime – Alibaba Qwen's Streaming Real-Time Speech Recognition Model

1. What is Fun-ASR-Realtime

Fun-ASR-Realtime is a streaming real-time speech recognition large model launched by Alibaba Qwen, designed for low-latency, high-precision speech-to-text scenarios. The model uses the WebSocket streaming protocol to achieve "output as you speak," compressing the first-word latency to the hundred-millisecond level while maintaining recognition accuracy close to that of offline models. In terms of dialect recognition, it covers 16 Chinese dialects and 30 languages, and outperforms similar products from Volcano Engine and Tencent in 12 dialect tests. The model also features advanced capabilities such as context understanding, hotword customization, emotion recognition, and timestamp output, maintaining industrial-grade robustness in complex backgrounds, distant noisy environments, and accented scenarios. Delivered through the Alibaba Cloud Bailian platform, Fun-ASR-Realtime provides an end-to-end speech recognition solution for real-time interaction scenarios such as live subtitles, meeting transcription, and customer service quality inspection.

Image source: Official article
Image source: official article

Technical Positioning and Domain: Fun-ASR-Realtime belongs to the speech recognition branch of the natural language processing field, focusing on streaming real-time speech recognition. Its unique positioning lies in the deep integration of large model architecture with streaming inference mechanisms, achieving high accuracy while maintaining low latency, creating a differentiated competition with traditional non-streaming or segmented speech recognition solutions. The model is designed for application scenarios with extremely high real-time requirements, such as live streaming, meetings, customer service, and voice input.

R&D Background: The model was developed by the Alibaba Qwen team based on their deep expertise in speech recognition and large models. The Qwen team has long-term research in speech recognition technology, including the release of models like Paraformer, which provided the technical foundation for Fun-ASR-Realtime. The R&D motivation stems from the rapidly growing market demand for real-time speech recognition and the shortcomings of existing solutions in dialect support, context error correction, and low latency.

Core Value: Fun-ASR-Realtime addresses the pain point of the trade-off between latency and accuracy in traditional real-time speech recognition. Its core value lies in providing hundred-millisecond-level first-word latency while achieving recognition accuracy close to offline models through context awareness and hotword customization mechanisms. Its breakthrough performance in dialect recognition enables it to serve a broad user base beyond Mandarin, filling the gap in the market for real-time dialect speech recognition.

Technical Features: The model adopts a large model backbone network based on Transformer or Conformer, achieving streaming inference through causal attention masking and chunk processing. Its pan-context enhanced training mechanism introduces historical dialogue context and real-time hotwords, giving the model contextual awareness. The multi-task joint modeling architecture synchronously outputs transcribed text, emotion labels, and sentence boundaries in a single forward pass, improving overall system efficiency.

2. Key Features

• Real-time Streaming Recognition: Achieves low-latency audio-to-text conversion through WebSocket duplex communication, supporting real-time character output. The model adopts a native streaming architecture, with first-character latency controlled at the hundred-millisecond level and extremely low tail-character latency, suitable for interactive scenarios requiring immediate feedback.

• Multi-dialect and Multilingual Support: Covers 16 Chinese dialects (including Cantonese, Wu, Min, Hakka, etc.) and 30 languages, suitable for scenarios such as overseas customer service and international conferences. In tests across 16 dialects, recognition accuracy for 12 dialect categories surpasses similar products from Volcano Engine and Tencent, with Wu dialect accuracy being industry-leading.

• Contextual Error Correction: Self-correction based on historical dialogue context and real-time hotwords, e.g., correcting "叶鹿" to "夜鹭" based on context. This feature is achieved through generalized context-enhanced training, allowing the model to dynamically leverage contextual information during inference to correct misrecognized content and improve recognition accuracy.

• Emotion Recognition: The Qwen-ASR model supports 7 fine-grained emotion categories, while Paraformer supports 3 polarity-based emotion categories. This feature is implemented through multi-task joint modeling, outputting emotion labels synchronously alongside the primary speech recognition task, suitable for scenarios such as customer service quality inspection and user sentiment analysis.

• Timestamp Output: Fun-ASR and Paraformer output sentence-level and character-level timestamps by default, facilitating subtitle alignment and keyword highlighting. The model predicts the start and end times of each character/word in the audio during decoding based on CTC or attention alignment paths, meeting the needs of video subtitle production and keyword localization.

• Hotword Customization: Supports improving recognition accuracy for brand names, person names, and specialized terms through hotwords. Users can dynamically inject a hotword list during recognition, and the model prioritizes matching hotword content during decoding, significantly enhancing recognition performance for specific vocabulary.

• VAD Intelligent Sentence Segmentation: The server automatically detects speech start and end points, supporting custom silence thresholds and sensitivity. Through a jointly trained voice activity detection task, the model accurately identifies when a speaker starts and stops speaking, achieving natural and smooth sentence segmentation.

• Non-human Sound Filtering: Automatically filters out non-human sound content to improve recognition purity. This feature identifies non-human sound signals such as background noise and music through acoustic feature analysis, excluding them from the output results to ensure the accuracy of the transcribed text.

3. How to Use

1. Obtain Credentials: Get your API Key from the Alibaba Cloud Bailian console (link to be updated after official release) and configure it in your environment variables. This credential is used for authentication and billing management and must be kept secure.

2. Install the SDK: Install the latest version of the DashScope SDK via pip (Python) or Maven (Java). Python users can run the pip install dashscope command, ensuring version 2.0.0 or later is installed for full streaming recognition support.

3. Configure Parameters: Create a Recognition instance, set the model name to fun-asr-realtime, and specify the audio format (e.g., pcm or wav) and sample rate (typically 16000Hz). You can configure hotword lists, language types, and emotion recognition switches based on scenario requirements.

4. Establish Connection: Establish a duplex connection with the Alibaba Cloud inference service via WebSocket. The SDK automatically handles connection establishment and keep-alive logic; developers only need to call the start() method to initiate a session. It is recommended to configure connection timeout and retry mechanisms to enhance robustness.

5. Send Audio: Read microphone or audio file data in fixed frame lengths and send it. The recommended frame length is 100-200ms to balance latency and network transmission efficiency. For real-time microphone input, use an audio library (e.g., PyAudio) to capture the data stream.

6. Receive Results: Handle the real-time on_event events in callbacks to obtain intermediate results and sentence-level final results. Intermediate results can be used for real-time captioning, while sentence-level final results include timestamps and emotion labels, suitable for subsequent processing.

7. End the Task: After sending all audio, call stop() or send a finish-task command to close the recognition session. It is recommended to clean up resources after the session ends and record the session ID for future queries and debugging.

4. Pros and Cons Analysis

5. Comparison of Similar Tools

Selection Recommendations: For Chinese real-time speech recognition scenarios requiring ultra-low latency and strong dialect support, Fun-ASR-Realtime is the optimal choice, especially in applications with high real-time demands such as live subtitles and meeting transcription. Its contextual error correction and emotion recognition features offer unique advantages in customer service quality inspection scenarios. For scenarios requiring global language coverage and strong language understanding, GPT-4o-Transcribe is more suitable, but higher latency and costs must be accepted. For enterprises already deeply embedded in the Volcengine or Tencent Cloud ecosystems, their respective speech recognition services offer advantages in integration convenience and private deployment.

6. Editor's Take

Fun-ASR-Realtime excels in technological innovation, deeply integrating large model architecture with streaming inference to achieve first-character latency at the hundred-millisecond level and accuracy close to that of offline models. This technical breakthrough marks a milestone in the field of real-time speech recognition. Its performance in dialect recognition is particularly noteworthy, leading competitors in 12 out of 16 dialects, with top-tier accuracy in the Wu dialect group. This is attributed to the Alibaba Qwen team's long-term investment in dialect data collection and model training. The model's context-aware error correction mechanism, achieved through generalized context-enhanced training, provides contextual awareness—a rare feature in existing speech recognition products, demonstrating technological foresight.

In terms of practical value, Fun-ASR-Realtime addresses the core needs for real-time speech recognition in scenarios such as live streaming, meetings, and customer service. Its low latency makes real-time captioning feasible, as validated during the 100-hour live stream "Return to the Desert Island" by FilmStorm, which generated 1.32 million characters of real-time captions. The emotion recognition feature offers a new analytical dimension for customer service quality inspection, while hotword customization improves recognition accuracy in specific scenarios. However, reliance on cloud services and opaque pricing strategies may limit its application in budget-sensitive or poor network environments.

This model is suitable for developers and enterprises requiring high-precision real-time speech recognition, particularly live streaming platforms, meeting system developers, customer service system providers, and voice input method developers. In the future, as the Alibaba Qwen team continues to optimize the model architecture and expand dialect support, Fun-ASR-Realtime is expected to find applications in more vertical domains. It is recommended that the team enhance SDK language support and documentation completeness to lower the integration barrier for developers.

7. Application Scenarios

• Live Streaming Subtitles: Provides millisecond-level subtitle output for long-duration live streams and multi-person interactive scenarios, supporting frequent speaker changes and complex competition environments. It has been validated in the 100-hour live stream "Return to the Desert Island" by Film Storm, generating a total of 1.32 million characters of real-time subtitles, proving its stability in extreme scenarios.

• Meeting Real-Time Transcription: In scenarios such as remote meetings and offline forums, it outputs text as it is spoken, providing structured text with sentence-level and word-level timestamps. This facilitates quick post-meeting review of key statements and time point localization, improving meeting efficiency and document organization speed.

• Customer Service Call Quality Inspection: Real-time recognition of two-way conversations between customer service representatives and users, with simultaneous emotion state detection. It supports custom hotwords to accurately capture brand names, product names, and complaint keywords. This enables real-time risk alerts during calls, helping businesses improve service quality.

• Voice Input Method: Provides low-latency dialect input capabilities for mobile devices, in-vehicle terminals, and IoT devices, supporting real-time transcription of 16 dialects including Cantonese, Wu, and Min. It enhances input efficiency and experience for non-Mandarin users, expanding the coverage of voice interaction.

• International Conference Simultaneous Interpretation: In cross-border conference scenarios, it supports multi-language real-time transcription across 30 languages, combined with context-based error correction to ensure accurate recognition of professional terminology. It provides instant text records for attendees, reducing language communication barriers.

• Educational Scenario Assistance: Provides real-time subtitles in online classrooms and lectures, supporting dialect and foreign language transcription to help students better understand teaching content. The emotion recognition function can also be used to analyze student emotional feedback, optimizing teaching effectiveness.

8. FAQ

Q: What audio formats does Fun-ASR-Realtime support?
A: The model supports PCM and WAV audio input, with a recommended sample rate of 16000Hz. For other formats (such as MP3, AAC), you need to transcode the audio on the client side before sending. The SDK provides auxiliary tools for audio format conversion that developers can call directly.

Q: What is the specific first-word latency? How can it be optimized?
A: Under standard network conditions, the first-word latency is approximately 100-200ms. Optimization methods include: using shorter audio frame lengths (recommended 100ms), ensuring sufficient network bandwidth, and selecting the nearest Alibaba Cloud node. For high-concurrency scenarios, it is recommended to configure according to the best practices guide on the Alibaba Cloud Bailian platform.

Q: How do custom hotwords take effect? How many hotwords are supported?
A: Hotwords are passed via the hotwords parameter of the Recognition instance, supporting up to 1000 hotword entries. Hotwords are assigned higher weights during decoding, significantly improving recognition accuracy for brand names, person names, and specialized terms. It is recommended to keep hotword lengths between 2-10 Chinese characters; overly long or short hotwords may have reduced effectiveness.

Q: What is the accuracy of emotion recognition? What emotion categories are supported?
A: The Qwen-ASR model supports 7 fine-grained emotion categories, including happiness, sadness, anger, surprise, fear, disgust, and neutrality, with an accuracy exceeding 85% on the test set. Paraformer supports 3 polarity-based emotion categories (positive, negative, neutral), suitable for quick emotion judgment scenarios.

Q: Does it support private deployment?
A: Currently, Fun-ASR-Realtime primarily provides cloud API services through the Alibaba Cloud Bailian platform, and a complete private deployment solution is not yet available. For enterprises with private deployment needs, it is recommended to contact the Alibaba Cloud sales team for customized solutions. Some basic models (such as Paraformer) are open-sourced on ModelScope and can be used for local inference.

Q: How are usage costs calculated? Is there a free tier?
A: For specific billing standards, please refer to the pricing page on the Alibaba Cloud Bailian platform, which typically charges based on audio duration. New Alibaba Cloud users usually receive a certain amount of free credits, subject to the latest platform policies. It is recommended to estimate costs before formal use to avoid overspending.

Q: What is the timestamp accuracy in recognition results?
A: Sentence-level timestamp accuracy is approximately 100ms, and word-level timestamp accuracy is approximately 50ms. Timestamps are predicted synchronously during decoding based on CTC or attention alignment paths, meeting the needs for subtitle alignment and keyword localization. For scenarios requiring extremely high accuracy, it is recommended to use word-level timestamps for post-processing.

9. Project Links

• Alibaba Cloud Bailian Platform Fun-ASR-Realtime Product Page: https://help.aliyun.com/zh/model-studio/real-time-speech-recognition-user-guide
• ModelScope Model Library (Paraformer and other base models): https://modelscope.cn/models?name=paraformer
• Alibaba Qwen Official Organization (GitHub): https://github.com/QwenLM