Xiaomi Auto World Model – Xiaomi's Assisted-Driving World Model

1. What Is the Xiaomi Auto World Model

Xiaomi Auto's assisted-driving world model (Xiaomi Auto World Model) is the first to deeply couple 3D reconstruction (WorldRec) with video generation (WorldGen) into one driving-scene understanding and simulation stack. Sparse 3D anchors reconstruct 10 seconds of video in 10 seconds; two-stage training plus ODE distillation cuts denoising to 4 steps (~0.19 s/frame) with up to ~1 minute continuous generation. Waymo reconstruction PSNR 28.48; nuScenes generation FVD 64.97—both leading benchmarks. Deployed for synthetic data, closed-loop simulation, and assisted-driving training academy—moving world models from papers to production.

Image source: Official article

Technical positioning and domain: Autonomous-driving visual world model for 3D reconstruction + video generation—not pure generative fantasy, but geometry-accurate, consistency-aware scene modeling.

Development background: In-house Xiaomi Auto team addressing long-tail data scarcity and slow simulation—vision + AD stack experience from architecture through deployment.

Core value: Breaks the data-scarce → fragile model → insufficient testing loop: fast accurate geometry, controllable generation into unobserved space/time, feeding perception training, validation, and driver education with less real-road dependency.

Technical characteristics: Sparse 3D anchors vs dense Gaussians for multi-view consistency; two-stage training + ODE distillation (~12× faster inference); closed loop where reconstruction constrains generation and generation extends reconstruction boundaries.

2. Key Features

• WorldRec 3D reconstruction: Sparse query anchors aggregate multi-camera multi-time features with visibility weighting—10 s video → 10 s rebuild; Waymo PSNR 28.48 (~+1 vs DGGT).

• WorldGen video generation: Bidirectional temporal pretrain + causal finetune; ODE distillation 50→4 steps; 0.19 s/frame single-view; up to 81 frames (~1 min) including future views and occluded regions; nuScenes FVD 64.97.

• Rec–gen coupling: 3D priors stabilize generation; generation fills unseen spacetime; mutual loss structure reduces long-horizon drift.

• Extreme scenarios: Heavy rain/snow/fog, animal intrusion, jaywalking—controllable synthesis with consistent geometry.

• Synthetic data at scale: 100k+ driving clips with 3D labels for perception training.

• Closed-loop simulation: Replay and stress-test accidents with tunable parameters for perception/planning validation.

3. How to Use

1. Requirements: Xiaomi vehicle (e.g., SU7) with assisted-driving academy software; network for updates; stationary safe parking to launch simulations.

2. Open academy: From AD/ intelligent driving menu → Assisted Driving Academy (exact UI naming may vary by firmware).

3. Pick scenarios: Urban, highway, rural templates plus weather/night/animal edge cases with previews.

4. Configure parameters: Weather intensity, traffic density, time of day—system renders first-person instructional video.

5. Learn: Play/pause/rewind with annotated hazards and recommended maneuvers.

6. Safety: Academy only when parked; rules tuned for China road standards; start basic before extreme cases.

4. Pros and Cons

5. Comparison with Similar Tools

Selection guidance: Not directly usable outside Xiaomi—reference architecture for rec+gen coupling. Waymo/Tesla similarly closed. NVIDIA Drive Sim for open toolchain teams with budget for integration. Xiaomi's coupling excels when strict scene structure matters.

6. Editor's Take

Architecturally meaningful: rec and gen aren't bolted modules—they co-constrain through losses, reducing geometric drift while filling occlusions. SOTA on Waymo/nuScenes supports the thesis.

Industrial closure is the story: 100k clips, accident replay sims, and consumer-facing academy tie R&D to product and users. Third parties can't run it today; platform lock-in caps research reuse.

Worth watching if APIs or cloud services open; transferable to robotics/digital twin rec+gen patterns.

— strong tech and shipping proof; −1 for exclusivity and opacity.

7. Use Cases

• Synthetic long-tail data: Weather, night, rare hazards for perception training.
• Closed-loop sim: Directed tests on perception/planning under parameterized danger scenes.
• Driving academy: First-person lessons for complex maneuvers in-car.
• Cockpit assistant visuals: On-demand scenario playback from voice queries ("highway lane change in rain").

8. FAQ

Q: Need internet?
A: First load often online; some base scenes cache offline; complex weather gen may still need network updates.

Q: Third-party vehicles?
A: No—tied to Xiaomi hardware/sensor stack; no public third-party API announced.

Q: Commercial use of synthetic data?
A: Internal R&D asset; academy video for viewing only—not exportable for external commercial datasets.

Q: vs Waymo world model?
A: Xiaomi emphasizes rec+gen coupling for long-horizon stability and faster on-device-friendly inference (0.19 s/frame).

Q: Custom scenarios?
A: Limited sliders (weather, traffic, time); core geometry/rules preset for safety—no full user-authored worlds.

9. Project Links

• Project site: https://JointWM.github.io/
• arXiv paper: https://arxiv.org/pdf/2605.18137