Wan-Streamer – Alibaba's Open-Source Real-Time Full-Duplex Multimodal Foundation Model

1. What is Wan-Streamer

Wan-Streamer is an end-to-end real-time full-duplex multimodal foundation model open-sourced by Alibaba DAMO Academy. It unifies text, audio, and video input/output tokens into a single causal sequence through a unified causal Transformer architecture, enabling sub-second real-time bidirectional video interaction. The model achieves a response latency of only 200ms, with an end-to-end total latency of 550ms, and stably outputs synchronized audio and video at 25FPS. This model abandons traditional multi-module concatenation approaches (ASR+LLM+TTS+Rendering), completing the entire process of perception, understanding, and generation with a single model, providing a new technical paradigm for real-time digital human interaction.

Image source: Official article
Image source: official article

Technical Positioning and Domain: Wan-Streamer belongs to the intersection of multimodal foundation models and real-time interaction systems, specifically designed for digital human dialogue scenarios requiring low latency and synchronized audio-video output. Its unique positioning lies in being the industry's first publicly open-sourced single causal Transformer model with full-duplex video interaction capabilities, distinguishing it from voice-only solutions like GPT-4o Realtime or traditional approaches relying on multi-module cascading.

R&D Background: Developed by Alibaba DAMO Academy's Vision and Multimodal Team, leveraging Alibaba's deep expertise in cloud computing, large models, and multimodal understanding. The R&D motivation stems from common issues in existing digital human systems such as high latency, lip-sync mismatches, and inability to interrupt, aiming to fundamentally improve the naturalness and real-time performance of human-computer interaction through an end-to-end architecture.

Core Value: Solves the challenge of synchronizing "listening, seeing, speaking, and expression" in real-time video conversations, compressing end-to-end latency to 550ms, far below the industry standard of over 1 second. Additionally, the full-duplex mechanism allows users to interrupt at will and enables AI to proactively ask questions, creating an interaction experience close to real human conversation, significantly enhancing user satisfaction in scenarios such as virtual customer service and AI companionship.

Technical Features: Adopts a fully causal technology stack, strictly adhering to causal constraints from encoders and decoders to attention layers, ensuring predictions are based solely on historical information. Introduces a Thinker-Performer dual-GPU inference architecture that decouples context updates from audio-video generation for parallel execution, maintaining low latency while supporting long-sequence consistency. A three-stage training process (multi-task pre-training, full-duplex fine-tuning, streaming distillation) ensures the model's generalization capability and potential for lightweight deployment.

2. Key Features

• Real-time Audio-Video Dialogue: Supports two-way video calls between users and AI digital humans. The AI can simultaneously output speech and facial expressions, with a stable video frame rate of 25FPS. Lip movements and speech are naturally aligned without the need for post-correction. This feature is directly aimed at scenarios requiring visual feedback, such as virtual customer service and live streaming interactions.

• Full-Duplex Interaction: Users can interrupt the AI at any time while it is speaking, and the AI can also proactively ask questions, achieving a natural and smooth conversational rhythm. Technically, the model continuously monitors the input stream through 160ms streaming chunks, dynamically adjusting the generation direction under a causal attention mechanism, avoiding the mechanical feel of traditional half-duplex systems where "you speak, then I speak."

• Multimodal Input Understanding: Simultaneously receives and understands the user's video, speech, and text inputs, which enter the model as a unified token sequence. For example, the user's expressions, gestures, and tone can all be captured by the model and influence the response content, achieving true multimodal contextual awareness.

• Streaming Chunk Generation: Uses 160ms short-term streaming chunks, allowing the model to generate feedback while receiving user input, without waiting for a complete video frame or speech segment. This mechanism is the core guarantee for low latency, enabling a model-side response time of only 200ms, far superior to traditional frame-by-frame processing solutions.

• Long-Sequence Consistency: The global KV context caching mechanism ensures that the AI digital human's character image, tone, and expression style remain stable and consistent during long conversations. Even if the dialogue lasts for tens of minutes, the model will not "forget" previously set character traits or experience expression drift.

• Single Model End-to-End Processing: Integrates perception (video encoding, speech encoding), understanding (semantic reasoning, sentiment analysis), and generation (text prediction, audio synthesis, video rendering) into a single Transformer, eliminating information loss and synchronization errors caused by multi-module splicing. Training uses a three-stage progressive approach, while inference employs Thinker-Performer dual GPU collaboration, balancing model capacity and real-time performance.

3. How to Use

Currently, Wan-Streamer has only released the paper and project website demo. The complete code and model weights have not been open-sourced yet, so local deployment is not possible at this time. Below is an expected usage guide for future deployment based on the officially announced technical approach.

1. Environment Requirements: At least two NVIDIA GPUs (A100 or H100 recommended) with a combined VRAM of no less than 80GB. The operating system should be Ubuntu 20.0+ with CUDA version ≥ 11.8. The Thinker and Performer are deployed on different GPUs, sharing the KV cache via high-speed interconnects (NVLink/PCIe 4.0).

2. Installation Steps: Once the code is open-sourced, clone the repository using git clone <link to be updated after official release> and install dependencies with pip install -r requirements.txt. Model weights need to be downloaded from Hugging Face or the official mirror site and placed in the designated directory.

3. Starting the Service: Run python launch.py --thinker_gpu 0 --performer_gpu 1 to start the dual-process inference service. After startup, the service listens on port 8080 by default and can accept camera and microphone streams via WebRTC or RTMP protocols.

4. Configuration Notes: Key configuration parameters include streaming chunk length (default 160ms), video resolution (currently 192p in the prototype, adjustable in the future), KV cache size (affects long-dialogue stability), and whether to enable interruption detection thresholds. Adjustments should be made based on actual network bandwidth and GPU VRAM.

5. Important Notes: Since the model uses a fully causal architecture, input streams must be sent in chronological order and cannot be out of sequence. Context synchronization between the Thinker and Performer relies on shared memory, requiring GPU inter-communication latency to be below 1ms. The current resolution limit (192p) is a prototype design; higher-resolution versions will require larger VRAM support.

4. Pros and Cons Analysis

5. Comparison with Similar Tools

Selection Recommendations: For scenarios requiring real-time video interaction with extreme sensitivity to latency (e.g., virtual customer service, AI companionship), Wan-Streamer's end-to-end architecture and full-duplex capability offer clear advantages, though hardware costs must be evaluated after open-sourcing. If only voice conversation is needed and ~0.8s latency is acceptable, GPT-4o Realtime is a mature commercial choice. For live streaming or educational scenarios with pre-built video templates and no need for real-time interruption, HeyGen and Synthesia provide high-quality video output but with lower interaction naturalness. If Wan-Streamer can improve resolution and reduce hardware barriers after open-sourcing, it has the potential to become infrastructure in the real-time digital human field.

6. Editor's Take

Wan-Streamer has reached an industry-leading level in technological innovation. Its most outstanding contribution lies in proving that a single causal Transformer can simultaneously handle multimodal input and output while achieving sub-second full-duplex video interaction, breaking the long-standing industry consensus that "real-time digital humans must rely on multi-module splicing." The design philosophy of the fully causal tech stack—prohibiting future information leakage from the encoder to the attention layer—while increasing training difficulty, ensures streaming compatibility and temporal consistency during inference, providing a reusable architectural template for subsequent low-latency multimodal models. The Thinker-Performer dual-GPU inference scheme cleverly decouples compute-intensive tasks (audio/video denoising and generation) from lightweight context updates, achieving stable 25FPS output on limited hardware—a significant breakthrough in engineering deployment.

In terms of practical value, Wan-Streamer directly targets scenarios requiring high interaction naturalness, such as customer service, live streaming, and education. Traditional solutions where users must wait for the AI to finish speaking, or the "uncanny valley effect" caused by lip-sync mismatches, are effectively addressed by Wan-Streamer. The 550ms end-to-end latency means users barely perceive any waiting, which is enough to shift the psychological threshold of human-machine dialogue. However, the current 192p resolution is clearly insufficient for commercial applications, and the hardware requirements limit trials by small and medium-sized teams. If Alibaba can provide a lightweight version in the open-source release that runs on a single GPU at lower resolutions, it would greatly accelerate ecosystem development.

Target audience: Primarily AI digital human R&D teams, cloud service providers, and enterprises needing real-time video customer service. For individual developers or startups, it is recommended to wait for official lighter model variants or API services.

Future development potential: With continuous improvements in GPU computing power and the maturation of model distillation techniques, Wan-Streamer is expected to achieve 720p output and reduce VRAM requirements within six months. Its open-source strategy will attract significant community contributions, potentially spawning a multimodal real-time interaction ecosystem similar to Stable Diffusion. However, compared to GPT-4o Realtime, Wan-Streamer still lags in semantic understanding depth and general knowledge, and may need to integrate larger language models in the future.

7. Application Scenarios

• Virtual Customer Service: Deploy AI digital humans in banking, e-commerce, government services, and other scenarios. Users can ask questions directly via video, and the digital human synchronously outputs speech and expressions, supporting interruptions and follow-up questions, achieving an interaction experience similar to a real human agent. Wan-Streamer's low latency and full-duplex capabilities can significantly improve customer satisfaction and problem resolution efficiency.

• Live Streaming Interaction: AI hosts can view audience comments and voice questions in real time, instantly generating expressions and voice responses, and can interact with multiple viewers simultaneously. In traditional solutions, AI hosts can only read pre-written scripts. Wan-Streamer's real-time interruption mechanism makes interactions more natural, suitable for scenarios such as live commerce and virtual idols.

• AI Companionship: Emotional companion digital humans support long-duration video chats, maintaining consistent character appearance and tone, and can remember conversation history. Suitable for scenarios such as elderly people living alone, children's education, or psychological counseling, providing an immersive companionship experience.

• Game NPCs: In open-world games, NPCs can engage in real-time video conversations with players, dynamically adjusting responses based on the player's expressions and tone, enhancing game immersion. Wan-Streamer's streaming chunk generation mechanism is well-suited for the real-time interaction needs in games.

• Online Education: AI virtual teachers can conduct one-to-many or one-to-one real-time video Q&A sessions, capturing student expressions to gauge understanding and proactively adjusting the pace of explanation. Supports students asking questions at any time, with the teacher responding without delay, improving teaching efficiency.

8. FAQ

Q: Can Wan-Streamer be deployed locally now?
A: Currently, only the paper and project website demo have been released. The full code and model weights are not yet open-sourced, so local deployment is not possible. Alibaba DAMO Academy has stated that they will release it later, with the specific timeline to be announced.

Q: What is the main difference between Wan-Streamer and GPT-4o Realtime?
A: The core difference lies in video output. GPT-4o Realtime only supports voice conversations, while Wan-Streamer can synchronously output digital human videos with lip-synced expressions. Additionally, Wan-Streamer uses a single causal Transformer end-to-end architecture, eliminating the need for ASR+LLM+TTS concatenation, resulting in lower latency and better lip-sync accuracy.

Q: What are the hardware requirements for Wan-Streamer?
A: It is recommended to use at least two NVIDIA A100 or H100 GPUs with a combined VRAM of no less than 80GB. The Thinker and Performer are deployed on separate GPUs, sharing KV cache via NVLink. Consumer-grade GPUs (e.g., RTX 4090) cannot run it directly due to VRAM and bandwidth limitations.

Q: The current video resolution is only 192p. Will higher resolution be supported in the future?
A: The current 192p is a prototype validation version. The official plan is to increase the resolution to 720p or higher in subsequent iterations. Higher-resolution versions will require more VRAM and computing power, which may be mitigated through model distillation or quantization techniques to reduce resource consumption.

Q: Does Wan-Streamer support Chinese?
A: The paper is primarily trained on English data, but the model architecture itself supports multilingualism. The effectiveness of lip-sync and speech synthesis in Chinese scenarios will need to be tested after open-sourcing. It is expected to be adaptable through fine-tuning on Chinese multimodal data.

Q: How much data is needed to train Wan-Streamer?
A: The paper does not disclose the specific data volume, but the three-stage training requires a large amount of multimodal dialogue data (video + speech + text), as well as annotated data for full-duplex interaction. It is difficult for ordinary teams to replicate, so it is recommended to directly use the official pre-trained weights.

Q: Are there privacy and security risks?
A: Since the model needs to process user video in real time, data encryption and compliance requirements must be considered during deployment. It is recommended to deploy locally in a private environment to avoid uploading video data to the cloud. The official open-source version should include a data anonymization interface.

9. Project Address

• Project Website: https://wan-streamer.com/
• arXiv Technical Paper: https://arxiv.org/pdf/2606.25041
• Official Code Repository (To Be Open-Sourced): Planned for release on GitHub, specific address to be announced by the official team. Follow Alibaba DAMO Academy's GitHub organization at https://github.com/alibaba