Xiaomi OneVL – Open-Source Autonomous Driving Model from Xiaomi Embodied Intelligence

1. What Is Xiaomi OneVL

Xiaomi OneVL is an open-source autonomous driving model from Xiaomi's embodied intelligence team—the first framework to unify VLA (vision-language-action), world modeling, and latent-space reasoning in one stack. Dual auxiliary decoders supervise compact latents during training, delivering “think as fast as not thinking, perform better than thinking” in benchmarks. At 0.24s onboard latency (4.16 Hz), it hits state-of-the-art on NAVSIM, ROADWork, Impromptu, and Alpamayo-R1. Weights and code ship under Apache 2.0 for academic and commercial reuse.

Image source: Official article

Technical positioning and domain: End-to-end motion planning with a unified VLA inference frame—scene understanding, language reasoning, and driving actions in one network instead of modular perception-prediction-planning stacks. Latent reasoning fused with a visual world model enables causal reasoning at realtime speeds.

Development background: Built on Qwen3-VL-4B-Instruct with Emu3.5-VisionTokenizer—lightweight, high-performance choices from Xiaomi's AV and robotics work. Motivation: explicit chain-of-thought (CoT) is accurate but too slow for vehicles; direct prediction is fast but opaque. OneVL targets both speed and interpretability via latent CoT.

Core value: Breaks the speed-vs-accuracy tradeoff in driving LLMs. Latent reasoning matches direct-prediction latency (~0.24s, 32% faster than explicit autoregressive CoT) while beating explicit CoT on NAVSIM (PDM-score 88.84 vs larger 8B rivals like AdaThinkDrive and LaST-VLA). Dual language + visual explanations reduce black-box risk.

Technical characteristics: Dual auxiliary decoders (language rebuilds text CoT; vision decoder predicts future frames); three-stage progressive training (trajectory → language → world model alignment); one-step parallel latent prefilling at inference with decoders dropped.

2. Key Features

• Unified VLA reasoning: Single framework from perception to action—less information loss and pipeline delay vs modular AV stacks.

• World model forecasting: Predicts +0.5s/+1.0s frames, forcing internalized road geometry and dynamics for safer lane changes and avoidance.

• Latent CoT: Deep reasoning in compact latents; parallel prefilling at inference—0.24s @ 4.16 Hz, first latent approach reported to beat explicit CoT accuracy.

• Dual-decoder training supervision: Language and vision decoders shape latents to encode semantics and physics; decoders removed at deploy time.

• High-precision trajectories: Qwen3-VL-4B backbone + MLP head; NAVSIM PDM 88.84 beating 8B competitors at 4B scale.

• Dual interpretability: Natural-language rationales plus predicted future visuals for audit and debug—e.g., “slow because lead vehicle braked.”

3. How to Use

1. Environment: Linux (Ubuntu 20.04+), 32GB VRAM NVIDIA GPU (A100/V100) recommended, Python 3.8+, PyTorch 2.0+, Transformers 4.30+, CUDA 11.7+. Use a dedicated conda env.

2. Clone and install:

   git clone https://github.com/xiaomi-research/onevl.git
   cd onevl && pip install -r requirements.txt
   

   For vision tokenizer issues: pip install git+https://github.com/baaivision/Emu3.git

3. Download weights: ~8GB checkpoint from Hugging Face or project site into ./checkpoints/ (e.g., onevl_qwen3_4b_vla.pth). Use git lfs or wget.

4. Inference:

   python scripts/inference.py --checkpoint ./checkpoints/onevl_qwen3_4b_vla.pth --input ./data/sample_scenario.json
   

   Outputs trajectories, language explanations, and future frames. Batch: scripts/batch_inference.py.

5. Config (configs/default.yaml): Tune backbone, training stages, latent token count (default 64—more tokens, higher accuracy, slower). For vehicle deploy, inference.fp16=True for mixed precision.

6. Notes: First run downloads Qwen3-VL-4B—set HF_ENDPOINT if needed. All three training stages are mandatory (skipping one can cost 15–21+ PDM points). Batch size 1 for minimum latency; TensorRT/ONNX can target <0.15s.

4. Pros and Cons

5. Comparison with Similar Tools

Selection advice: Production teams wanting realtime + interpretability → OneVL. Need fastest pure planning on nuScenes with LiDAR → VAD. Research on latent reasoning/world models → OneVL's open code. EMMA is reference-only (closed, slow CoT).

6. Editor's Review

OneVL is among the most innovative open AV LLM releases lately—unifying VLA, world modeling, and latent CoT with evidence that latent beats explicit CoT without sacrificing latency. That matters for an industry stuck choosing fast opaque planners vs slow explainable ones.

Apache 2.0 and 4B scale lower the bar vs closed giants like EMMA. Dual explanations help move AV toward auditable decisions.

Fit: AV engineers, researchers, integrators—not beginners. Training is finicky; camera-only sensing is a real limit in adverse weather.

Future: LiDAR fusion, sub-100ms optimizers, better long-tail data—likely aligned with Xiaomi's broader intelligent driving roadmap.

— Innovation 5/5, utility 4.5/5, performance 4.5/5, ecosystem 4/5.

7. Use Cases

• Production L2+/L3 stacks: Replace modular pipelines with unified camera-in → trajectory-out at realtime latency.

• Complex urban scenes: World-model futures for merge/yield decisions with text rationales for validation.

• Development audit: Visualize language + predicted frames to catch bad reasoning during sim/log replay.

• Research forks: Swap backbones, adjust stages, fine-tune for highway fleets under Apache 2.0.

8. FAQ

Q: Consumer GPUs?
A: RTX 3090 (24GB) runs with higher latency (~0.5s+); 32GB A100/V100 ideal. 12GB needs compression tricks.

Q: Skip a training stage?
A: Don't—each stage is required; skipping language or vision alignment costs ~15–21+ PDM points.

Q: vs EMMA?
A: OneVL is faster, fully open, smaller; EMMA may generalize broader tasks but is closed and CoT-slow.

Q: LiDAR/radar?
A: Not in current release—camera-only via Emu3.5 tokenizer.

Q: Long-tail robustness?
A: Augment/simulate rare scenes; fine-tune on target geography; test heavily before deploy.

Q: 0.24s measured how?
A: A100, FP16, 640×480 input, 64 latent tokens—your hardware/resolution/token count will shift results.

9. Project Links

• Website: https://xiaomi-embodied-intelligence.github.io/OneVL/
• GitHub: https://github.com/xiaomi-research/onevl
• arXiv: https://arxiv.org/pdf/2604.18486