Wan-Streamer v0.2 – A Full-Modal Understanding and Generation Model from Alibaba Tongyi

1. What is Wan-Streamer v0.2

Wan-Streamer v0.2 is an end-to-end full-modal understanding and generation model introduced by Alibaba Tongyi Lab, designed for real-time full-duplex interaction. This model processes real-time understanding and synchronized generation of text, audio, and video using a single Transformer. It achieves an end-to-end response latency of just 550ms, with video output resolution of 640×368@25FPS. It supports full-duplex interaction where the AI can listen, speak, and respond simultaneously, enabling face-to-face video communication with AI that feels as natural as human interaction, significantly improving the naturalness and real-time performance of human-AI interaction.

Image source: Official article
Image source: official article

Technical Positioning and Domain: Wan-Streamer v0.2 belongs to the field of multimodal large models, focusing on real-time video conversation scenarios. It differs from traditional cascaded pipeline solutions by adopting a native end-to-end architecture, unifying perception, understanding, and generation into a single model. This enables synchronized understanding and output of audio and video with extremely low latency, filling the gap in current AI models for real-time video interaction.

Development Background: The model was developed by Alibaba Tongyi Lab, whose team has accumulated extensive experience in multimodal and inference optimization based on the Tongyi series of large models. In response to issues such as high latency, inability to listen, watch, and respond simultaneously, and reliance on external module assembly in current AI video interaction systems, the team proposed the Thinker-Performer dual-path architecture and a streaming unit closed-loop mechanism, aiming to address computational bottlenecks in real-time video generation and to advance AI from "polling-based dialogue" to "human-like communication."

Core Value: This model eliminates the fragmented feeling of "waiting for the user to finish speaking before responding" in AI video conversations, achieving real-time feedback akin to human interaction. At the same time, the end-to-end unified modeling avoids error accumulation and latency overhead caused by external module assembly, such as ASR, LLM, and TTS. It provides a reliable technical foundation for scenarios like AI video calling, virtual character role-playing, and accessible interaction, allowing AI to listen, watch, and respond in real time, just like humans.

Technical Features: The native streaming architecture maps user input and agent output onto the same causal timeline, completing a full closed-loop of perception, understanding, generation, and decoding every 160ms. The Thinker single-card is responsible for low-latency perception and language reasoning, while the Performer multi-card employs the Ulysses parallel strategy for high-definition video generation. By using temporal overlapping scheduling, the video generation cost is decoupled from the latency-sensitive path, achieving a balance between low latency and high video quality.

2. Key Features

• Real-time Audio-Video Conversation: Supports face-to-face interaction in the style of a video call, where AI real-time perceives the user's video feed and audio input, and simultaneously generates responses with expressions, gestures, and voice. This enables natural and smooth two-way communication, allowing users to interrupt or respond without waiting for the entire input to finish.

• Multimodal Understanding: Natively supports real-time understanding of three modalities: text, audio, and video, without requiring external ASR or visual models. The model internally performs semantic fusion and context modeling on multimodal inputs, enabling it to simultaneously understand the user's spoken content, tone and emotion, as well as facial expressions and body movements.

• Synchronized Audio-Video Generation: Outputs synchronized speech and high-definition video, ensuring precise alignment of lip movements, expressions, and voice through end-to-end unified modeling. This eliminates the common audio-video synchronization issues found in traditional pipeline approaches, enhancing the realism of interactions.

• Micro-expression and Gesture Generation: Capable of generating natural eye contact, head movements, body postures, and gesture actions. The output is not limited to the face but also includes upper body movements and scene details. This makes the AI character more lifelike, suitable for immersive interaction scenarios that require a strong sense of presence.

• Flexible Role-playing: Supports real-time generation of any character image through natural language descriptions for conversation. Users can customize the character's appearance, personality, and speaking style. This feature simplifies the implementation of scenarios such as virtual idols, historical figures, and game NPCs.

• Full-duplex Interaction: Users can interrupt or respond at any time without waiting for the AI to finish speaking. The model continuously perceives the input stream and immediately adjusts the generated content. This human-like speaking, listening, and responding pattern significantly improves the efficiency and naturalness of communication, avoiding the rigidity of traditional polling modes.

3. How to Use

Currently, Wan-Streamer v0.2 is released as a research project, and the model weights and code are not yet publicly available. The following is a reference guide based on the existing information. Please follow the official announcements from the Alibaba Tongyi Lab for specific open dates.

1. Environment Requirements: It is recommended to use a server equipped with multiple NVIDIA A100 or H100 GPUs. The Thinker component runs low-latency inference on a single GPU, while the Performer requires multiple GPUs (recommended: 4 or more) for Ulysses parallel video generation. The operating system is recommended to be Linux (Ubuntu 22.04+), and CUDA 12.x, PyTorch 2.0+, and corresponding deep learning dependency libraries must be installed. The total GPU memory is recommended to be no less than 80GB.

2. Obtaining the Model and Code: At present, users can learn about the architecture details through the project's official website (wan-streamer.com) and the technical paper (arXiv:2607.04443). The model weights and inference code are not yet available for download. It is recommended to follow the official organization repository on GitHub from the Alibaba Tongyi Lab and obtain the resources as soon as they are open-sourced.

3. Setting Up the Runtime Environment: Assuming the code is open-sourced in the future, users need to clone the code repository and install Python dependencies according to the README document (e.g., transformers, accelerate, decord). Configure the model weight path and download the pre-trained weights (estimated size is several tens of GB). Registration and access permissions may be required (e.g., Hugging Face or Alibaba Cloud ModelScope).

4. Starting the Service: Run the official startup script (e.g., python run_server.py), configure input and output devices (camera, microphone, display). The model will automatically begin real-time audio and video conversations. Integration with other applications can be done via WebRTC or custom APIs. It is recommended to first test within a local network to ensure network latency is below 50ms for the best experience.

5. Notes: This is currently a research version and may be optimized for specific scenarios only. Network latency significantly affects the end-to-end experience (approximately 350ms included). It is recommended to use a wired network or low-latency Wi-Fi. For multi-GPU parallel processing, NVIDIA NCCL and Ulysses communication libraries must be configured. If using consumer-grade GPUs (e.g., RTX 4090), achieving 25FPS output may not be possible; consider reducing resolution or frame rate.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: If the application scenario requires real-time video generation and interaction (such as AI in video calls, virtual characters, or immersive game NPCs), Wan-Streamer v0.2 is currently the only solution that supports end-to-end video output. Although it is not yet open-sourced, its technical approach is forward-looking. If only real-time voice conversation is needed and full control over the model is desired, Moshi, as an open-source solution, offers low latency and can be deployed locally, making it suitable for researchers and privacy-sensitive scenarios. If mature and stable multimodal understanding (text, image, audio) is required but video output is not necessary, GPT-4o Realtime stands out in overall capabilities due to OpenAI's optimizations, though it relies on API and has higher costs.

For enterprise-level applications, if low latency and full-duplex interaction are priorities and the budget is sufficient, keep an eye on Wan-Streamer's future open-source developments. If there is an urgent need to launch voice conversation features, Moshi or GPT-4o Realtime are more practical choices. In educational and accessibility scenarios, Wan-Streamer's video generation capabilities can provide a richer dimension of interaction, but hardware requirements should be evaluated in advance.

6. Editor's Summary

Wan-Streamer v0.2 demonstrates significant innovation in its technical architecture. Its native end-to-end Transformer design unifies text, audio, and video modeling, completely abandoning the traditional cascaded pipeline and fundamentally eliminating information loss due to error propagation and latency accumulation between modules. The Thinker-Performer dual-path architecture, combined with the Ulysses parallel and temporal overlapping scheduling mechanism, achieves real-time high-definition video generation while maintaining extremely low model-side latency (200ms). This design approach offers important reference value for the development of real-time multimodal models.

In terms of practical value, this model fills a product gap in the field of AI real-time video interaction. While existing mainstream models (such as GPT-4o, Gemini) support multimodal understanding, they are all unable to generate video frames synchronously. On the other hand, specialized video generation models (such as Sora) cannot achieve real-time interaction. Wan-Streamer unifies "understanding" and "generation" on the timeline, enabling AI to observe, listen, and respond simultaneously, just like a human. This opens up new possibilities for scenarios such as video call assistants, online education, game NPCs, and accessibility communication.

In terms of target users, this model is more suitable for research institutions and enterprise developers with high-performance computing resources. It is not yet open-sourced, but the core design of the technical paper has been made public, allowing academic researchers to draw inspiration from its architectural ideas for further improvements. Enterprise users can evaluate its compatibility with their own business applications in advance and quickly implement it once the model is released. For individual developers or small teams, the current hardware requirements and lack of open-sourcing limit direct usage. However, they can keep an eye on the future release of a lightweight version or cloud API services.

In terms of future development potential, if the Tongyi Lab at Alibaba can open-source the model weights and release a lightweight version that can run on consumer-grade GPUs, Wan-Streamer has the potential to become a benchmark model in the field of real-time video interaction. Its architectural scalability (the decoupling of Thinker and Performer) also leaves room for future improvements in resolution and the addition of more modalities (such as haptics and depth information). However, the fundamental demand for computing power in real-time video generation will not disappear. How to balance video quality, latency, and hardware costs will be the key to its large-scale deployment.

7. Application Scenarios

• Video Call AI Assistant: Simply turn on the camera to engage in face-to-face conversations with AI, ideal for scenarios requiring a sense of presence such as spoken language practice, interview simulations, and psychological counseling. The AI can observe the user's facial expressions and body language, and adjust its response strategy in real time to provide a more empathetic interaction experience.

• Contextualized Education: By observing students' expressions and level of attention through the camera, the AI teacher can assess their understanding of key concepts and adjust the teaching pace and explanation style accordingly. Additionally, the AI teacher exhibits natural micro-expressions and gestures, enhancing the emotional impact of the teaching process.

• Immersive Game NPCs: Non-player characters (NPCs) in games feature lifelike facial expressions, body language, and real-time responsiveness, enabling true "face-to-face" conversations with players. Players no longer encounter fixed animations but instead experience dynamic, personalized interactions, significantly enhancing the sense of immersion in the game.

• Accessible Interaction: Generate real-time video responses with precise lip-reading and gestures for hearing-impaired users, allowing them to participate more naturally in video calls. For visually impaired users, provide real-time environmental descriptions (via video perception) and use voice feedback to convey surrounding scene information, assisting with daily travel and social interactions.

• Virtual Character Role-Playing: Users can have real-time video conversations with historical figures, literary characters, or custom-designed avatars. By describing a character's traits in natural language, the model instantly generates the corresponding image and maintains the character's personality, suitable for cultural education, entertainment interaction, and creative expression.

8. FAQ

Q: When will Wan-Streamer v0.2 be open-sourced?
A: The model is currently released as a research project, and its weights and code have not been made public. Please follow the official announcements from the Tongyi Lab and the project website (wan-streamer.com) for updates on the open-source plan.

Q: What hardware configuration is required to run Wan-Streamer?
A: It is recommended to use multiple NVIDIA A100 or H100 GPUs (suggested: 4 or more cards). The Thinker card handles inference, while the Performer cards generate video in parallel. Total GPU memory should be no less than 80GB. Consumer-grade GPUs (e.g., RTX 4090) may not achieve a 25FPS output.

Q: Does Wan-Streamer support Chinese?
A: Models from the Tongyi Lab typically support both Chinese and English. However, the specific language coverage of Wan-Streamer v0.2 has not yet been disclosed. From a technical standpoint, the support for languages in an end-to-end multimodal model depends on the training data, and it is expected that both Chinese and English will be well-supported.

Q: What is the core difference between Wan-Streamer and GPT-4o Realtime?
A: The main differences lie in video output and architectural design. Wan-Streamer supports real-time video generation (640×368@25FPS) and uses a native end-to-end Transformer architecture, eliminating the need for external modules. In contrast, GPT-4o Realtime does not support video output and employs a cascaded pipeline architecture (ASR + LLM + TTS). Additionally, Wan-Streamer's full-duplex interaction is more comprehensive, allowing users to interrupt at any time.

Q: How is the 200ms latency on the model side achieved?
A: It is achieved through a native streaming architecture and the Thinker-Performer dual-path design. The Thinker card completes perception, understanding, and reasoning within 160ms, while the Performer cards use Ulysses parallel and overlapping scheduling to overlap video generation computation with the reasoning process, thereby decoupling video generation from the latency-critical path.

9. Project Links

• Project Website: https://wan-streamer.com/
• arXiv Technical Paper: https://arxiv.org/pdf/2607.04443
• Alibaba Tongyi Lab GitHub Organization (awaiting model open source): https://github.com/alibaba