Qwen3.8-Omni-Flash – A Native Multimodal Model Launched by Alibaba Qwen

1. What is Qwen3.8-Omni-Flash

Qwen3.8-Omni-Flash is a native multimodal model launched by Alibaba Qwen. It jointly models four modalities—text, image, audio, and video—within a single architecture, supporting a context length of up to 1M tokens, enabling hours of audio and video content to be natively input without the need for segmentation. At its core is the design philosophy of "from understanding to delivery," not only providing cross-modal comprehension capabilities but also enabling the model to autonomously complete end-to-end workflows such as video editing, MV creation, short drama translation, movie commentary, and meeting minutes generation through its built-in Agent capabilities. Compared to the previous generation model, average performance has improved by over 25%, and the API audio input cost has dropped by more than 98%. Additionally, Alibaba Qwen has open-sourced two supporting frameworks: Qwen-MM-Plugins and Qwen-Live Harness, offering a complete toolchain for the development and deployment of audio and video Agent applications.

Image source: Official article
Image source: official article

Technical Positioning and Domain: Belongs to the intersection of multimodal large models and intelligent agents (Agent). This model is positioned as a foundational model integrating full-modal perception and delivery. It covers traditional visual-language understanding tasks and extends to creative tasks such as audio and video generation, editing, and translation, achieving a complete closed-loop from understanding to delivery ahead of similar products. Its technical approach balances academic frontiers (unified architecture for all modalities) with industrial implementation (Agent toolchain), offering a differentiated position within general-purpose multimodal large models.

Development Background: Developed by the Alibaba Tongyi Qwen team. This team has long focused on large language models and multimodal models, accumulating a complete technology stack from basic model training to cloud deployment. The launch of Qwen3.8-Omni-Flash represents a significant technical advancement in the native multimodal domain, aiming to address the core pain point of existing audio and video understanding models that can only "understand" but not "act." This model pushes the capabilities of the model from the perception layer to the execution layer.

Core Value: Solves the two major challenges of long audio and video understanding and multi-step task delivery. Previously, processing audio and video content longer than an hour typically required segmenting, analyzing each segment individually, and manually assembling the results, which was fragmented and inefficient. This model achieves global understanding and precise localization of long audio and video content through its native long context and Agentic on-demand perception mechanism. At the same time, its "from understanding to delivery" capability transforms tasks such as meeting minutes generation, video commentary creation, and short drama translation and dubbing from manual processes into automated pipelines, significantly reducing the human and time costs of content production.

Technical Features: The model employs a native unified multimodal architecture, jointly modeling four modalities within a single model. It achieves cross-modal understanding through modality-aligned encoding and a unified representation space, maintaining text capabilities comparable to those of pure text models of the same size. Its unique Agentic on-demand perception and multi-round evidence collection mechanism allow the model to autonomously decide "what to look at and what to listen to," focusing computational resources on key segments. This results in a reduction of approximately 45.7% in token consumption while improving understanding accuracy.

2. Key Features

• Agentic Long Audio-Video Understanding: Starting from the user's question, the model autonomously determines which segments of the video and audio to focus on. Through a multi-stage verification process, it progressively identifies key information. Compared to traditional full-scan approaches, this mechanism improves accuracy while reducing token consumption by approximately 45.7%, making it suitable for long video content Q&A and event retrieval scenarios.

• Controllable Audio-Video Captioning: Users can freely specify the description target (person, object, scene), time range, information granularity (summary or detailed), and output format (JSON, Markdown, etc.), enabling on-demand descriptive capabilities that align with "what the user wants to know" rather than "what the model sees." This feature provides precise structured input for downstream content analysis, retrieval, and re-creation.

• Long Meeting Understanding and Execution: Natively supports one-hour audio-video input, completing speaker segmentation, speech transcription, identity alignment, meeting note generation, and task extraction in an end-to-end manner. In complex meeting scenarios with overlapping speech and multiple speakers, the model leverages visual information to help resolve references and entity ambiguities, significantly improving recognition metrics.

• Music2MV: The model deeply understands the structure, rhythm, and emotional shifts of songs, outputting timestamped lyrics and executing the full MV creation workflow—from music comprehension, scene planning, image generation, to final quality inspection. Creators only need to input a song to obtain a complete MV output.

• Short Drama Translation: Completes the entire process of character dialogue recognition (ASR), colloquial translation, voice cloning for dubbing, audio track remixing, and quality inspection with a single sentence. This enables automated delivery of translated content. Compared to traditional multi-step manual translation and dubbing workflows, the end-to-end solution significantly shortens production cycles.

• Audio-Video Deep Research: Combining video content with user needs, the model automatically retrieves multimodal internet resources, cross-validates information, and generates rich,图文并茂 (image-rich) research reports centered around the video. This feature is ideal for in-depth research scenarios requiring a comprehensive analysis of video content and external materials.

• Video2Note: Automatically compresses hours of video into timestamped PDF notes with corresponding text, and includes automatic review and iterative correction capabilities. Students can convert course recordings into structured study notes, while researchers can turn lecture videos into searchable knowledge documents.

• Real-Time Speaking Practice Companion: Jointly models pronunciation and semantics, understanding user accent deviations and generating standard pronunciation examples in real time. As the user provides new speech input, the model continuously updates its assessment, distinguishing between critical errors that affect comprehension and natural accent variations, delivering a low-latency interactive speaking practice experience.

3. How to Use

1. Direct Web Usage: Access the Qwen AI platform (link to be updated once officially released).

2. API Calling: Apply for an API Key from Alibaba Cloud DashScope and call the API through an interface compatible with the OpenAI format. When making a request, you can input image, audio, or video links (or Base64 encoding) along with text questions, and the model will return the understanding results. The API method is suitable for developers to integrate the model's capabilities into their own applications.

3. Install Agent Plugin: Install the Qwen-MM-Plugins plugin in Agent development environments such as Claude Code and Qwen Code. After installation, you can issue instructions as if chatting, for example, “@video.mp4 help me create a commentary video.” The model will automatically complete the entire process of material understanding, task planning, tool calling, and result delivery.

4. Deploy Real-time Interactive Service: If you need to use real-time spoken language practice, low-latency audio-video interaction, and other features, you can integrate with the Qwen3.8-Omni-Flash-Realtime service. This service receives audio-video streams via WebSocket/WebRTC and completes perception and response simultaneously with input. The first token latency is approximately 600-980ms, and the audio generation RTF is about 0.153. Developers need to establish audio-video capture and streaming transmission channels on the frontend.

5. Notes: When using the API, pay attention to quotas and billing policies; when using the Agent plugin method, ensure that the runtime environment has the necessary dependencies (such as FFmpeg and other audio-video processing tools); although 1M context can natively accommodate long videos, it is recommended to control input length for faster response speeds.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: In scenarios involving long audio and video understanding and multi-speaker recognition, Qwen3.8-Omni-Flash demonstrates a clear advantage in localized contexts such as Chinese meetings and dialect support, and its API pricing is significantly lower than that of Gemini 3.8 Flash and GPT-4o, making it suitable for large-scale deployment and cost-sensitive commercial applications. If extreme benchmark performance is required and the primary content is in English, Gemini 3.8 Flash still maintains a certain level of leadership in LVOmniBench and FLEURS WER, and can be considered as an alternative. Developers should make their choice based on their language requirements, context length needs, and budget constraints. For teams requiring agent-based delivery capabilities, Qwen3.8-Omni-Flash is currently the only option that provides a complete open-source toolchain.

6. Editor's Summary

Qwen3.8-Omni-Flash demonstrates several noteworthy breakthroughs in terms of technological innovation. First, its native multimodal unified architecture discards traditional multi-model concatenation approaches, achieving joint modeling of four modalities within a single architecture. This design has substantial implications for improving the consistency of cross-modal reasoning and reducing system complexity. Second, the Agentic on-demand perception and multi-turn verification mechanism effectively addresses computational efficiency challenges in long audio and video processing. While reducing token consumption by approximately 45.7%, it also improves accuracy, providing a viable efficiency path for practical deployment of long-context models. Third, the multi-speaker audio-visual collaborative recognition strategy introduces visual information into the audio disambiguation process, achieving a DER score of 3.4 on the AliMeeting benchmark, significantly outperforming competing products and validating the effectiveness of cross-modal information fusion in specific tasks.

In terms of practical value, this model covers the complete workflow from content understanding to content delivery. Capabilities such as video narration, short drama translation, and automated meeting minutes can be directly applied to fields like media production, enterprise office operations, and online education. The release of the open-source toolchain lowers the integration threshold for developers, enabling the model's capabilities to be quickly embedded into existing Agent workflows and accelerating the transformation of technology into products. For audio-visual content production teams, enterprise knowledge management departments, and AI application developers, this model provides an integrated solution from perception to execution. Its differentiated advantages in the Chinese market scenario and pricing also give it strong market adaptability in the context of domestic model replacement and international competition.

7. Application Scenarios

• Batch Video Content Production: Self-media teams can input a video and a one-sentence requirement, and the model will automatically complete full video understanding, plot extraction, voiceover script writing, dubbing and music selection, editing and rendering, and quality inspection. This end-to-end automated delivery enables movie commentary, MV creation, and short drama translation and dubbing, significantly shortening the content production cycle and reducing labor costs.

• Automated Meeting Efficiency: Enterprise users upload an hour of meeting recordings, and the model natively performs speaker segmentation, speech transcription, identity alignment, and minutes generation. It automatically extracts action items, sends emails, or creates tasks, compressing the post-meeting organization work from hours to minutes. Additionally, visual information is used to enhance the accuracy of speaker identification in multi-speaker scenarios.

• Online Learning and Knowledge Management: Students or researchers input several hours of course recordings or lecture videos into the model, which automatically generates timestamped PDF notes with corresponding text and images. It supports automatic review and iterative refinement. For knowledge-intensive video content, this feature transforms passive watching into active knowledge accumulation.

• In-depth Video Research: Researchers input research requirements for specific video content, and the model automatically retrieves multimodal data from across the web, cross-verifies the information, and generates a rich,图文并茂 (image-rich) research report centered around the video. This is suitable for scenarios requiring a combination of video and external data for comprehensive analysis, such as product analysis, competitive research, and technical investigations.

• Real-time Interactive Services: Based on the Realtime architecture, the model supports low-latency audio and video interaction, applicable to real-time spoken language practice, intelligent customer service, and sound-based navigation. The first token latency is approximately 600-980ms, and the audio generation RTF is about 0.153, enabling a natural and smooth real-time conversation experience.

8. FAQ

Q: What is the fundamental difference between Qwen3.8-Omni-Flash and traditional multimodal models?
A: Traditional multimodal models typically treat text as the core, using images, audio, and video as supplementary inputs for single-round understanding tasks. Qwen3.8-Omni-Flash employs an end-to-end native multimodal unified architecture, jointly modeling four modalities at the model's foundation layer, and possesses equal cross-modal representation capabilities. Its Agent capabilities allow it not only to "understand" content, but also to autonomously plan and execute multi-step tasks, outputting complete deliverables (such as edited videos, reports, and notes), making it a next-generation multimodal Agent model.

Q: What does a 1M context window actually mean?
A: A 1M token context window can accommodate millions of Chinese characters or several hours of high-definition video frames and audio tracks. This means the model can process entire movies, meeting recordings, or long-form video courses in one go, without the need for segmentation and reassembly, fundamentally avoiding the issue of information fragmentation caused by segmented processing and ensuring the coherence of global understanding.

Q: Is the API calling cost really that low?
A: According to official data, the price for audio input via API has dropped by over 98%, and the price for audio and video input has decreased by over 93%, offering a significant price advantage compared to similar international products. The specific cost depends on the selected billing tier and usage volume. Developers are advised to check the latest detailed pricing table on the Alibaba Cloud DashScope console and estimate their budget accordingly.

Q: What roles do Qwen-MM-Plugins and Qwen-Live Harness play?
A: Qwen-MM-Plugins provides the model with on-demand perception, tool calling, and workflow execution capabilities, and can integrate with mainstream Agent frameworks such as Claude Code and OpenClaw, enabling the model to perform content understanding, task planning, tool execution, and result delivery within existing development environments. Qwen-Live Harness focuses on supporting real-time interactive scenarios, offering input and output management for audio and video streams and implementing streaming interaction protocols.

Q: What languages and dialects does the model support?
A: The model performs excellently in 60 language speech recognition tests, supporting 39 Chinese dialects, with coverage leading among domestic models. Its multilingual ASR capabilities achieve a WER of 9.3 on the FLEURS benchmark, which is still behind international competitors, but it demonstrates strong competitiveness in Chinese scenarios.

Q: What is the difference between the open-source version and the API version?
A: The open-source content currently includes two supporting frameworks: Qwen-MM-Plugins and Qwen-Live Harness. The model weights have not yet been released. The API version provides full access to the model's capabilities. Developers can call API capabilities and orchestrate workflows within their local Agent environments via plugins, enabling flexible application integration.

Q: What is the difference between the Qwen3.8-Omni-Flash-Realtime version and the standard version?
A: The Realtime version optimizes the streaming interaction architecture, receiving audio and video streams via WebSocket/WebRTC and performing perception and response simultaneously with input. It achieves a first token latency of approximately 600-980ms and an audio generation RTF of about 0.153. The standard version is designed for asynchronous understanding and generation tasks, suitable for handling long inputs and complex workflows. Developers should choose the appropriate version based on their business requirements for real-time performance.

9. Project Links

• Product Official Website: https://www.qianwenai.com/models/qwen3.8-omni-flash
• GitHub Repository - Qwen-MM-Plugins: https://github.com/QwenLM/Qwen-MM-Plugins
• GitHub Repository - Qwen-Live Harness: https://github.com/QwenLM/Qwen-Live-Harness