EchoWM – Interactive Audio-Visual World Model Open-Sourced by JD.com

1. What is EchoWM

EchoWM is an interactive audio-visual world model open-sourced by the JD.com Exploration Research Institute. It continues the native audio-visual generation capabilities of JoyAI-Echo, allowing users to move and explore in real-time within an AI-generated world from a first- or third-person perspective. The visual elements, spatial relationships, ambient sounds, and speech remain consistently synchronized over long durations without drifting. The model is trained using a unified camera intent interface and a world data engine, achieving an average score of 81.7 on the WBench navigation benchmark, ranking first.

Image source: Official article
Image source: official article

Technical positioning and domain: Belongs to the category of Interactive Audio-Visual World Models, spanning multiple interdisciplinary areas including computer vision, natural language processing, speech generation, and real-time interaction. Compared to traditional video generation models that focus solely on visual realism, EchoWM jointly generates ambient sounds and speech alongside the video, while supporting real-time exploration within the generated world. This significantly enhances the immersive interactive experience.

Development background: Developed and open-sourced by the JD.com Exploration Research Institute, based on the team's self-developed JoyAI-Echo audio-visual generation framework. The institute has accumulated substantial technical expertise in the field of multimedia content generation, and the release of EchoWM fills the gap in domestic open-source world models that simultaneously support joint audio-visual generation and real-time interaction.

Core value: Addresses the common issue in most world models of "generating only visuals, not sounds" and the lack of interactive capabilities. Traditional video generation models typically end after producing a fixed-length video, preventing users from exploring within the scene. EchoWM, however, maps keyboard instructions to continuous 6-DoF trajectories through a unified camera intent interface, enabling users to navigate the generated scene in real-time from a first- or third-person perspective. It maintains natural synchronization between ambient sounds, speech, and visuals as the scene evolves, supporting multi-round, long-duration exploration.

Technical features: The model integrates a unified camera intent interface, a world data engine combining four complementary data sources, a progressive four-stage training strategy, and long-duration streaming interaction supported by a sink-plus-FIFO caching mechanism. In terms of inference performance, it employs distributed matching distillation to compress the process into four sampling steps, maintaining high response speed while controlling KV cost.

2. Key Features

• Real-time Explorable World Generation: Users can freely move, turn, and backtrack within AI-generated scenes using WASD directional keys. The world unfolds in real-time as the user interacts. The system converts discrete keyboard commands into continuous 6-DoF camera trajectories, keeping response latency within interactive limits to deliver an immersive "walking into the scene" experience.

• Native Audio-Video Joint Generation: Within the same framework, 720p video, ambient sound, music, and character speech are generated synchronously. The audio naturally evolves with the visual changes. Unlike post-processing voice-over solutions, EchoWM establishes audio-visual alignment during the generation phase, ensuring consistency in sound source location, volume, and events with the visual content.

• First-/Third-person Dual Perspective Control: In first-person mode, the camera moves with the observer, offering a handling experience similar to first-person games. In third-person mode, the camera automatically follows the movement of characters, vehicles, and other main subjects. The model learns the coupling relationship between subject motion and camera tracking from training data, eliminating the need to explicitly define trajectory parameters.

• Multi-turn Long-duration Continuation: The audio-video content from the previous segment is used as the context for the next round, allowing users to continue exploring based on previously generated content. After multiple interactions, the scene layout, subject appearance, and audio remain consistent, avoiding common issues of scene drift in long video generation.

• Action DSL Action Instruction System: Provides a structured action description language with the format <key>-<frame count>, connected by commas. For example, w-60,a-60 indicates moving forward for 60 frames and then left for 60 frames. Supports key combinations such as w/s/a/d/i/j/k for precise control of the camera movement path.

• Progressive Four-stage Training Pipeline: Stage one involves joint pre-training with AV-rich data; stage two freezes the backbone network and trains only the lightweight trajectory branch to enhance control responsiveness; stage three is joint fine-tuning; and stage four is autoregressive post-training, incorporating Teacher Forcing and Self-Gradient Forcing mechanisms to improve the stability of long-duration generation.

• Efficient Four-step Sampling Inference: By leveraging distribution-matching distillation, the generation process is compressed into four sampling steps, significantly reducing inference time compared to autoregressive frame-by-frame generation. Combined with the sink-plus-FIFO caching mechanism, it maintains low control KV costs while supporting long-duration streaming interactions, enabling real-time operation.

3. How to Use

1. Clone the repository: Execute git clone https://github.com/jd-opensource/JoyAI-Echo to download the project code and enter the echo_wm/ directory. Note that this directory is independent of the long video project echo_longvideo/ and should be used separately.

2. Create environment: Use conda create -n echo-wm python=3.11 to create an isolated Python 3.11 environment and activate it, avoiding dependency conflicts with the system Python environment.

3. Install dependencies: First install PyTorch 2.9.1 (CUDA 12.8 version), then run pip install -r requirements.txt to install the remaining dependencies. After installation, verify GPU availability using torch.cuda.is_available() to ensure that the CUDA environment is configured correctly.

4. Download model weights: Use hf download Echo-Team/Echo-WM to download the model checkpoint. Additionally, you need to download the Gemma 3 text encoder (from a gated repository), which requires first accepting the license agreement on the Hugging Face page and then running hf auth login to complete authentication.

5. Run official examples for verification: Execute python scripts/run_wm_case.py --case examples/wm_cases/0010 and other built-in complete cases to confirm that the environment is configured correctly, the model weights are loaded properly, and the generation pipeline operates as expected.

6. Custom generation and action instructions: Run inference_wm.py, passing in the first frame image, a six-field prompt, and an action sequence (Action DSL) to generate a video with sound. The action instruction format is <key>-<frame number>, connected by commas, and supports the combination of w/s/a/d/i/j/k to achieve precise camera motion control.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

**

Selection Recommendation: If the core requirement is a highly immersive audio-visual interaction experience—where users need to "enter" an AI-generated world and explore freely, while requiring synchronization between ambient sound, speech, and visuals—EchoWM is currently the leading choice in overall performance. Its unified control interface supports both first-person and third-person operations, making it highly suitable for scenarios such as game prototype validation and interactive narrative creation.

Alternative Considerations: If the application scenario emphasizes a director's perspective for video creation, with a focus on camera direction rather than free exploration, HappyOyster's director mode may better align with practical workflows. If the project requires low deployment barriers or lightweight integration, it is also possible to evaluate engineering compatibility factors beyond WBench scores, selecting the most appropriate solution based on your own computational resources.

6. Editor's Summary

EchoWM has made a valuable differentiated choice in its technical approach. While most world model projects focus on improving visual generation quality, EchoWM treats audio generation, real-time interaction, and long-term consistency as equally important core metrics. By unifying the camera intent interface, it maps discrete keyboard commands to continuous 6-DoF trajectories, simultaneously driving first-person and third-person perspective control, demonstrating a well-designed architecture with good scalability.

The world data engine integrates four complementary data sources: internally collected gameplay, human gameplay recordings, UE simulations, and general internet videos. It uses the 90th percentile of maximum translational amplitude of trajectories as a global "ruler" for normalization, preserving relative displacement differences between various trajectories. This provides a solid data foundation for the model to learn the coupling relationship between "subject movement—camera following." The progressive four-stage training strategy also reflects careful engineering considerations: from pre-training to control branch reinforcement, and then to joint fine-tuning and autoregressive post-training, each stage has clear objectives and effectively balances control responsiveness with generation quality.

In terms of practical value, EchoWM offers an accessible implementation path for scenarios such as game prototyping, interactive storytelling, and virtual tours. Its leading performance on the WBench navigation benchmark with an average score of 81.7 and a user preference rate of 63.13% demonstrates tangible advantages in interactivity and consistency. The open-source strategy further reduces the barrier to entry for the ecosystem, allowing developers to perform secondary development and customization on the existing architecture.

In terms of target users, EchoWM is aimed at world model researchers, game developers, interactive content creators, and technical professionals in the virtual production field. Currently, attention should still be paid to the usage limitations caused by hardware resource requirements and deployment complexity. However, the completeness of the model architecture and training strategy lays a solid foundation for future continuous optimization.

7. Application Scenarios

• Game Prototype Development: Creators can enter a generated scene for testing, validate spatial layout, route design, and camera experience before deciding on the production direction. Compared to traditional prototype development that requires building levels and placing objects, EchoWM allows planners and level designers to obtain an explorable preliminary scene within a few hours, reducing early validation costs.

• Interactive Storytelling: The plot progresses through user choices and continuously generates new content, replacing pre-recorded materials. This is used for interactive dramas and novels. Creators inject their scripts into the generation process as prompts, and each user action changes the story's direction. The audio and visuals update synchronously with the plot, maintaining narrative coherence across multiple interactions.

• Immersive Content: From "watching" to "entering," film and advertising content becomes interactive. Audiences can freely explore the world within the generated scenes. Production teams can use EchoWM to build exploratory virtual scenes for camera previsualization before formal filming, while advertising creative teams can quickly generate multiple versions of virtual scenes for proposal submissions.

• Virtual Tours: Low-cost digital roaming of scenarios such as museums, tourist attractions, and real estate, without the need for frame-by-frame modeling. Users move freely in the generated scene from a first-person perspective, with ambient sounds and visuals matching in real time, providing a cost-controlled digital display solution for cultural and tourism institutions and individual creators.

• Digital Human Performance: Audio, actions, and dynamic environments, camera angles, and sounds around the character are co-generated, suitable for virtual anchors and digital employees. EchoWM unifies character performance and scene changes within the same generation framework, simplifying the production process for digital human content, especially for application formats that require the scene to respond to user interactions.

8. FAQ

Q: What is the core difference between EchoWM and regular video generation models?
A: Regular video generation models generate a fixed video based on prompts and then end, with no interaction with the generated content. EchoWM allows users to explore the generated scene in real-time using WASD directional keys, with the visuals, spatial relationships, and ambient sounds updating synchronously with the user's actions. It is an "enterable" world model rather than a one-way video output.

Q: How can you switch between first-person and third-person perspectives?
A: Both perspectives use the same camera intent interface. In first-person mode, the camera moves with the observer. In third-person mode, the model automatically learns the coupling between "subject movement—camera following," with the camera tracking the movement of entities such as people or vehicles. No separate controller parameters are needed for different perspectives.

Q: How are sound and visuals kept synchronized?
A: EchoWM jointly generates video and audio within the same framework, with trajectory information injected only into the video branch. Audio is synchronized with visual events during the generation phase, rather than through post-processing voice synthesis. This ensures consistency in sound source position, volume, and visuals at a fundamental mechanism level.

Q: What hardware environment is required for deployment?
A: A NVIDIA GPU with CUDA 12.8 support and PyTorch 2.9.1 (cu128) installed is required. Generating 720p audio-visual content requires significant GPU memory and computational power; specific values can be found in the environment configuration of the official repository. It is recommended to use a high-performance GPU to ensure a smooth interactive experience.

Q: Will the scene drift after multiple rounds of interaction?
A: EchoWM supports long-duration multi-round interactions through autoregressive post-training and a sink-plus-FIFO context caching mechanism. The end of the previous audio-visual segment is used as the context for the next round, maintaining coherence in scene layout, subject appearance, and sound after multiple operations. The model demonstrates outstanding performance in long-duration consistency metrics in official evaluations.

Q: How do you write Action DSL action instructions?
A: The format is <key>-<frame count>, with multiple actions connected by commas. For example, w-60,a-60 means moving forward for 60 frames and then left for 60 frames. It supports key combinations such as w/s/a/d/i/j/k to precisely describe the camera's movement path.

9. Project Links

• Project Website: https://echo-team-joy-future-academy-jd.github.io/Echo-1.5-Page/wm/
• GitHub Repository: https://github.com/jd-opensource/JoyAI-Echo
• arXiv Technical Paper: https://arxiv.org/pdf/2608.23189