UniWorld-View – RabbitZoom Intelligence Collaborates with Peking University and Others to Open-Source a World Model

1. What is UniWorld-View

UniWorld-View is an open-source world model jointly developed by RabbitZoom Intelligence, Peking University, and the鹏城实验室 (Pengcheng Laboratory). It has topped the WorldScore world model evaluation leaderboard from the team of Fei-Fei Li. Based on a single image or a video, the model can generate new perspective views according to a specified camera trajectory, achieving a unified architecture for single-image 3D generation and video 4D generation. The model has completed compatibility with the domestic Ascend computing power, with both code and weights open-sourced, providing an efficient and controllable solution for world model research and application.

Technical Positioning and Domain: UniWorld-View belongs to the intersection of computer vision and generative models, focusing on novel view synthesis and 3D/4D scene generation. On the WorldScore leaderboard, the model achieved the highest scores in dimensions such as static scenes (85.53), camera control (97.72), and 3D consistency (91.63). It is positioned as a controllable, unified world model framework, differing from traditional novel view synthesis methods that only support static scenes or video generation models that lack precise control.

Development Background: This model was jointly developed by RabbitZoom Intelligence, Peking University, and the Pengcheng Laboratory. The team has deep expertise in areas such as point cloud rendering, diffusion models, and 3D vision. The motivation for development stemmed from the shortcomings of existing world models in terms of camera control precision and task unification — most models cannot simultaneously handle static images and dynamic videos, and lack the ability for precise camera trajectory control.

Core Value: UniWorld-View addresses the unification problem of generating multi-perspective views from single images or videos. It effectively improves generation quality through occlusion-aware point cloud rendering technology. Additionally, by open-sourcing the model and adapting it to domestic Ascend computing power, it significantly lowers the usage threshold for world models. Its unified architecture design reduces the need for developers to switch between different models for various tasks, thereby enhancing development efficiency.

Technical Features: The model employs an architecture that integrates occlusion-aware point cloud rendering with video diffusion models. It introduces a dual projection and normal filtering mechanism, resolving issues of occlusion and backside light leakage in point cloud rendering. It supports precise six-degree-of-freedom camera trajectory control, enabling large baseline perspective switching while maintaining temporal consistency in the generated visuals.

2. Key Features

• Single Image Novel View Synthesis: Input a single image, and users can specify any camera path (push, pull, pan, dolly, 360° orbit). The model automatically generates high-quality images for the corresponding view through point cloud rendering and diffusion generation processes. This feature enables visual expansion from a single 2D image to a 3D scene, suitable for rapid visualization of concept art.
• Dynamic Video Novel View Synthesis: Input a monocular video, and the model can infer unobserved spatial regions to generate novel views with temporal consistency. The video diffusion model ensures frame continuity, making it ideal for 4D scene reconstruction and dynamic content creation.
• Unified Generation Framework: The same model and code support novel view generation for both static images and dynamic videos, without the need to switch architectures or retrain. This unified design reduces development and deployment complexity, offering convenience for multi-task applications.
• Camera Trajectory Control: Supports precise six-degree-of-freedom camera pose control. Users can define any trajectory using a JSON file, enabling large-baseline view switching and cinematic camera movements. The control accuracy scores 97.72 on the WorldScore camera control dimension, verifying its effectiveness.
• Occlusion-Aware Point Cloud Rendering: Features a dual projection and normal filtering mechanism. Dual projection involves projecting the source image to the target view and then back-projecting it, cumulatively accumulating occlusion masks frame by frame; normal filtering removes back-facing point clouds, effectively resolving issues such as foreground-background tearing and back-facing light leakage, thereby improving image quality.
• Adaptation to Domestic Ascend Compute Power: The model has been adapted and optimized for Huawei's Ascend NPU, enabling efficient inference on domestic hardware and meeting requirements for information innovation and autonomous controllability. The adaptation process has been optimized, and inference performance is close to that of mainstream international GPUs.
• Open-Source Weights and Code: The model weights and full code are open-sourced under the Apache 2.0 license, supporting both commercial and academic use. Developers can freely download, modify, and redevelop the model, and the community can contribute improvements.

3. How to Use

1. Environment Setup: Clone the GitHub repository (git clone https://github.com/PKU-YuanGroup/UniWorld-View), create a Python virtual environment (recommended Python 3.8–3.10), and install dependencies (pip install -r requirements.txt). Dependencies include PyTorch (≥1.13), torchvision, diffusers, opencv-python, etc. It is recommended to use CUDA 11.7 or higher with at least 16GB of video memory. For Ascend deployment, install CANN and torch_npu.
2. Model Download: Download the pre-trained weights from Hugging Face (https://huggingface.co/Drexubery/UniView) or the GitHub Release page, and place them in the checkpoints/ directory. You can also run the download script to automatically fetch the model. The model file is approximately 2–5GB in size; ensure a stable network connection and consider using a resume-download tool.
3. Data Input: Prepare the source material: a single image (supporting JPEG/PNG, recommended resolution no less than 512×512) or a short video (MP4 format, recommended duration no more than 30 seconds, frame rate 24–30fps). At the same time, define the target camera trajectory by specifying the camera pose for each frame in a JSON file, including position (xyz), rotation (quaternion or Euler angles), focal length, and other parameters. The official provides an example trajectory generation script (generate_trajectory.py) that can generate common trajectories such as orbiting or zooming.
4. Run Inference: Execute the inference script, for example: python run_inference.py --input image.jpg --trajectory trajectory.json --output results/ --resolution 1024 --num_steps 50. The model will automatically generate new perspective views, outputting them as a sequence of images or a video file. Adjustable parameters include resolution (512/1024), sampling steps (default 50), batch size, etc., to balance quality and speed.
5. Result Output and Post-processing: The generated results are saved by default as a sequence of PNG images. You can also directly output an MP4 video by using the --output_video parameter. Users can use tools like FFmpeg for post-editing and compositing.
6. Ascend Deployment (Optional): If you want to run the model on the domestic Ascend NPU, install the Ascend toolkits (CANN, torch_npu), and set the --device npu parameter. The official provides compatibility guidelines and allows the use of Ascend acceleration units for inference. Note the compatibility between the Ascend version and the PyTorch version, and it is recommended to use the officially recommended mirror environment.

Notes: The first run requires downloading the model; it is recommended to use an SSD for storage. When generating high-resolution long videos, the video memory usage is significant; consider reducing the resolution or using chunked inference. The camera trajectory definition must follow the right-hand coordinate system; it is recommended to test with example trajectories first. If video memory is insufficient, try enabling gradient checkpointing with the --use_checkpointing parameter.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations:
For professional users requiring precise camera control and the ability to handle both static and dynamic scenes, UniWorld-View is currently the top choice among open-source world models. Its unified architecture reduces the complexity of multi-task deployment, and its open-source strategy and compatibility with domestic computing power meet the requirements of self-reliance and controllability. If the user primarily focuses on the quality of dynamic scene generation and has low requirements for camera control, WorldScape-0.2 has a slight advantage in dynamic scores, but its open-source status and domestic compatibility are unclear and require further evaluation.

For high-quality novel view synthesis of static scenes, NeRF-based methods (e.g., Instant NGP) still lead in detail preservation, but they require scene-specific training and do not support dynamic inputs. Stable Video Diffusion excels in video generation speed but cannot precisely control camera trajectories, making it suitable for quick previews rather than professional production. Overall, UniWorld-View has set a new benchmark in controllability and uniformity, but there is still room for improvement in complex scene generalization and inference speed, making it ideal for high-quality offline production scenarios.

6. Editor's Summary

UniWorld-View demonstrates notable breakthroughs in technological innovation. Its proposed occlusion-aware point cloud rendering mechanism effectively addresses the long-standing issues of occlusion and back-side light leakage in point cloud rendering through dual projection and normal filtering. This solution has been quantitatively validated in the WorldScore benchmark, with top performance across multiple metrics confirming the effectiveness of its technical approach. The unified static and dynamic generation architecture reduces model fragmentation, offering new insights for the practical application of world models. Compared to existing methods, UniWorld-View shows a particularly significant improvement in camera control precision, with a score of 97.72 clearly outperforming competitors.

In terms of practical value, the model's open-source strategy and compatibility with domestic computing power lower the barrier to entry, enabling professionals in fields such as film production, virtual reality, and robot simulation to access high-level new perspective generation capabilities at a relatively low cost. The Apache 2.0 license further encourages commercial applications and community-driven secondary development. Although the current documentation and community ecosystem are still in the early stages of development, the core technology has already reached a usable level of maturity.

The target audience primarily includes computer vision researchers, post-production professionals in film, game developers, robot simulation engineers, and cultural heritage digitalization workers. For scenarios requiring high-quality controllable generation, UniWorld-View provides a rare solution among currently available open-source world models. The current version still has room for optimization in terms of inference speed and long video support, which are key areas for future improvements. The model still has limitations in terms of hardware requirements and generalization across complex scenarios, and the team has indicated that they will iterate on these aspects.

7. Application Scenarios

• Film and Advertising Production: Directors or post-production personnel can quickly generate multi-camera preview images based on a single concept image or a segment of real footage, simulating camera movements such as push-ins, pull-outs, panning, and tracking shots. Compared to traditional 3D reconstruction methods, this approach eliminates the need for modeling and texture mapping, significantly reducing pre-production costs and accelerating creative iteration.
• Virtual Reality and Gaming: Developers can generate interactive 360° immersive environments from a single scene concept image, used for VR experiences or game level design. The multi-perspective images generated by the model can be directly used as environment maps or scene backgrounds, enhancing immersion and development efficiency.
• Robotics and Autonomous Driving Simulation: By generating new perspective data from real-world environments, the model provides diverse training samples for visual perception models. For example, generating street scenes from monocular video enhances the model's generalization ability under unknown perspectives and reduces the cost of on-road data collection.
• Digitalization of Cultural Heritage: After a single shot of cultural relics or ancient architecture, the model generates a 360° digital display. Museums can use this technology to quickly create online 3D exhibits without the need for expensive 3D scanning equipment, preserving valuable artifacts while expanding their reach.
• E-commerce and Real Estate: Merchants can generate multi-angle display content from a single product image or room video. Consumers can view product details or room layouts from any angle, improving browsing experience and purchase intent, especially suitable for scenarios requiring multi-dimensional presentation such as furniture and electronics.

8. FAQ

Q: What distinguishes UniWorld-View from traditional novel view synthesis methods like NeRF?
A: UniWorld-View employs an architecture that integrates point cloud rendering with diffusion models, supporting dynamic video input and precise camera control. In contrast, NeRF is primarily designed for static scenes and requires scene-specific optimization. UniWorld-View outperforms NeRF in multiple metrics on the WorldScore benchmark and can generate novel views without training, making it more user-friendly.

Q: What hardware requirements does the model have?
A: It is recommended to use high-end GPUs such as NVIDIA A100 (40GB) or Ascend 910, with a memory of at least 32GB. Consumer-grade GPUs like the RTX 3090 (24GB) can run low-resolution generation (e.g., 512×512), but may lack sufficient memory for high-resolution video (e.g., 1024×1024), requiring chunked inference or reduced resolution. CPU inference is not supported.

Q: How is the camera trajectory defined?
A: Users must provide a JSON file containing the camera pose parameters for each frame: position (xyz coordinates), rotation (quaternion or Euler angles), and focal length (or field of view). The official repository provides an example script (generate_trajectory.py) to generate common trajectories such as orbiting or zooming. It is recommended to first test with example trajectories before customizing complex ones.

Q: Does the model support commercial use?
A: Yes, the model is licensed under Apache 2.0, with both code and weights open-sourced. It allows commercial use, modification, and redistribution. However, users must ensure that content generated using the model complies with relevant laws and regulations and cannot be used for infringing or illegal purposes.

Q: How long does it take to generate a video?
A: Generating a single frame takes approximately 5-10 seconds (at 1024 resolution, using NVIDIA A100). A 10-second video (30fps) would take about 25-50 minutes. Inference speed is affected by resolution, sampling steps, and hardware performance. The current version is suitable for offline production; real-time applications require further optimization. Users can speed up generation by reducing resolution or sampling steps.

Q: Does the model support multi-GPU parallel inference?
A: The official version does not currently provide native support for multi-GPU inference. However, users can modify the inference script themselves using the PyTorch distributed framework to achieve data parallelism. The official team has not yet announced plans for multi-GPU support, so users are advised to keep an eye on repository updates.

Q: What input formats does the model support?
A: The model supports JPEG and PNG formats for images, and MP4 and AVI formats for videos. It is recommended that input resolution be no less than 512×512, video duration not exceed 30 seconds, and frame rate be between 24-30fps. For non-standard formats, users can first convert them using FFmpeg.

9. Project Links

• GitHub repository: https://github.com/PKU-YuanGroup/UniWorld-View
• HuggingFace model library: https://huggingface.co/Drexubery/UniView