H3-World – A World Model with Action Controllability Developed by Tencent in Collaboration with Universities

1. What is H3-World

H3-World is an action-controllable world model developed collaboratively by Tencent, National University of Singapore, and The Hong Kong Polytechnic University. It is based on the MiniMax-H3 video generator with 33B parameters and undergoes lightweight fine-tuning. This model innovatively translates keyboard operations into English instructions, reusing the pre-trained text pathway within the model and leveraging temporal attention routing to precisely bind each instruction to the corresponding video frame, achieving temporal alignment control between actions and visual progression. By using only 8,000 game videos and fine-tuning 0.199% of the parameters with rank-32 LoRA, H3-World enables the model to generate controllable visuals in real-time based on keyboard input and generalize to unseen scenarios, marking an important technical exploration toward interactive generative worlds.

Image source: Official article
Image source: official article

Technical positioning and domain: H3-World belongs to the intersection of video generation and world modeling. Its core technical approach leverages the pre-trained capabilities of an existing video generator, adapting it through lightweight modifications to become an action-controllable world model that supports interactive control, differing from the traditional approach of training self-regressive world models from scratch.

Development background: This model was jointly developed by Tencent, National University of Singapore, and The Hong Kong Polytechnic University. The motivation behind its development was to address the core pain points of existing video generation models, which can only be controlled via global text prompts and struggle with precise control over action timing. The project explores upgrading high-quality video generators into interactive world models at extremely low cost.

Core value: H3-World solves three major challenges in video generation: the difficulty of precisely controlling action timing, high training costs, and weak generalization capabilities. Compared to traditional conditional injection methods, this language-grounded approach to action control generates coordinated character and camera movements without disrupting the scene content, while also enabling the model to generalize combinations to unseen scenarios.

Technical features: Key technical highlights include action language representation, reusing pre-trained text pathways, temporal attention routing, and lightweight LoRA fine-tuning. These technologies together achieve an elegant "zero new modules" design, unlocking interactive control capabilities of the 33B model by fine-tuning just 0.199% of the parameters, reducing training costs by several orders of magnitude compared to training a world model from scratch.

2. Key Features

• Keyboard Actions to Video Control: Real-time conversion of keyboard inputs for character movement and camera operations into corresponding video frame changes, enabling an interactive world simulation where "pressing a key equals control." Users can issue control instructions to the model via predefined actions or keyboard keys, and the model generates the corresponding visuals in real-time, suitable for gaming, virtual environment interaction, and other scenarios.

• Temporal Precise Control: Supports executing different action instructions at different time intervals within the same video, precisely controlling when each action occurs. The model binds each instruction to its corresponding video latent variable frame via temporal attention routing, preventing control leakage between adjacent actions and addressing the pain point of global prompts being unable to precisely control action timing.

• Multi-Environment Generalization: Trained on game visuals, the model generalizes to diverse visual environments such as neon cities, desert oases, and fantasy churches while maintaining scene content stability. This capability stems from the reuse of pre-trained text pathways, allowing the model to naturally understand action semantics without needing to relearn action encodings.

• Compositional Action Generalization: The model can combine character/camera action primitives seen during training into new, previously unseen action pairs, enabling compositional generalization control. A single model handles various control types, including first- and third-person perspectives, character movement, and camera motion, without requiring separate training for different tasks.

• Long-Term Generation: Supports continuous controllable video generation over longer time sequences, where action instructions can sustainably drive the evolution of the visual scene. The generation results can be autoregressively extended, allowing users to input action instructions sequentially, with the model generating subsequent video frames continuously to create an interactive generation experience.

• Multi-Modal Control Support: A single model can handle multiple control types, including first- and third-person perspectives, character movement, and camera motion, offering strong versatility. Users can control either the character via keyboard inputs or manipulate the virtual camera's movement, providing flexible support for various application scenarios.

3. How to Use

H3-World provides two usage options: online experience and local deployment, allowing users with different technical backgrounds to choose based on their needs.

1. Environment Requirements: For local deployment, create a conda environment with Python 3.10 + CUDA 12.8, install PyTorch 2.10 and dependencies listed in requirements.txt. The base model weights are approximately 135GB, so sufficient disk space and a GPU that meets the memory requirements must be prepared.

2. Online Experience Process: Open the interactive demo Space on Hugging Face (hugging-apps/h3-world-action-demo) to experience keyboard-driven control without downloading the 135GB weights. Upload or select an initial frame image, input a scene description prompt, and then use predefined actions or keyboard keys to enter control instructions. The model will generate a controllable video clip in real-time with a duration of approximately 5.2 seconds and a resolution of 832×480.

3. Local Deployment Steps: Clone the official H3-World repository and also clone DiffSynth-Studio into a specified directory. Switch to a specific commit and apply the official attention patch (this step cannot be skipped, otherwise LoRA will be ineffective). Download the MiniMax-H3 base model weights and H3-World LoRA (step-10000.safetensors) from Hugging Face and place them in the corresponding directory within the repository.

4. Configuration and Inference: Run env.sh to point the Hugging Face, Torch, and Triton caches to the repository's internal directories. During inference, provide the first frame image, static semantic conditions, and an action sequence in a timeline format (each segment corresponds to a character or camera instruction). The model will generate video latent variables according to the action timeline and decode them for output, supporting autoregressive extension for longer durations.

5. Notes: During local deployment, the attention patch installation cannot be skipped, otherwise the LoRA parameters will not take effect. When downloading weights, be mindful of the network environment; the ModelScope mirror may offer a more stable download option for users in China.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: If the research goal focuses on exploring low-cost transformation paths from video generators to world models, or requires precise temporal action control, H3-World is the most suitable choice. Its lightweight fine-tuning characteristics and open-source nature provide convenient conditions for academic reproduction, and the generation quality from the 33B foundation significantly outperforms small-scale world models trained from scratch. For users seeking an "out-of-the-box" real-time gaming experience, GameNGen performs better in single-game scenarios, but its lack of generalization capability limits its application scope.

If a world model with the broadest scene coverage is needed, Genie 3's large-scale pre-training approach offers stronger generalization, but its closed-source nature and extremely high training costs restrict secondary development potential. Genie 1, as an early exploratory work, still holds academic reference value, but its actual control capabilities and generation quality are lagging behind subsequent solutions.

6. Editor's Summary

H3-World demonstrates a unique innovation in its technical approach. Current world model research is primarily divided into two paths: training autoregressive models from scratch or modifying existing video generators. H3-World chooses the latter and fully implements the concept of "reusing pre-trained text pathways." By translating actions into linguistic instructions and reusing the original text encoding pathway from MiniMax-H3, the model naturally inherits the rich semantic representations learned during the video pre-training phase, eliminating the need to learn action semantics from scratch. This design not only significantly reduces training costs (only fine-tuning 0.199% of parameters) but also achieves an elegant architecture with "zero new modules."

The temporal attention routing mechanism is another key technical contribution. Precise temporal action control in video generation has long been a challenge. H3-World addresses this by binding instructions to video frames through directional attention masks, effectively preventing control leakage between adjacent actions. This mechanism provides a general and controllable modification strategy for video generation models, offering methodological insights for future research.

In terms of practical value, H3-World skillfully balances the threshold for generating high-quality videos with interactive control capabilities. With a training dataset of 8,000 game videos and 10,000 optimization steps, teams with limited computational resources can still reproduce and iterate upon the model. Its compositional generalization ability further demonstrates that lightweight fine-tuning does not significantly compromise the model's adaptability to out-of-distribution scenarios.

The primary target users of this model include: researchers studying world models and controllable video generation, who can conduct secondary experiments using H3-World's code and weights; game prototype developers and interactive content creators, who can quickly validate interactive generation concepts using the online Space; and technical professionals interested in the latest developments in multimodal large models.

H3-World represents a pragmatic direction in world model research: rather than building complex world simulators from scratch, it unlocks interactive capabilities with minimal modifications on top of existing powerful generative models. As the underlying video generation models continue to improve, the potential of this lightweight modification approach will also grow, possibly becoming one of the mainstream methodologies in this field.

7. Application Scenarios

• Game Prototype Development: Designers can input initial visuals and instantly "play" through virtual scenarios, quickly validating level concepts and cinematography, without needing to build a complete game engine first. By using keyboard controls to manipulate character movement and camera perspectives, an explorable virtual environment can be generated within minutes, significantly shortening the early-stage design verification cycle.

• AI Agent Training Sandbox: Provides an interactive virtual environment for embodied agents. Agents explore the world and receive visual feedback through keyboard actions, reducing the cost of training in real-world settings. Combined with reinforcement learning frameworks, agents can be trained on navigation and interaction decision-making capabilities in diverse visual environments, bridging the domain gap between simulation and real-world scenarios.

• Film and Animation Previsualization: Directors can use keyboard controls to manipulate virtual camera movements (pan, dolly, crane), generating real-time storyboard previews and drastically reducing the pre-visualization production cycle. Through an action timeline, precise sequencing of camera movements can be arranged, offering an intuitive previsualization tool for complex scene cinematography and minimizing rework during actual filming.

• Robot Operation Strategy Validation: Map robotic arm or drone movement instructions into action language and test the temporal validity of action sequences in the generated simulation environment. Researchers can rapidly generate diverse environmental scenarios to test strategy robustness, reducing reliance on physical simulators and accelerating algorithm iteration efficiency.

• Immersive Content Creation: Combined with VR/game interactions, this allows users to drive the evolution of personalized virtual worlds using natural keyboard inputs through UGC tools. Anyone can generate "playable videos" without requiring professional skills. Creators can build interactive narrative content, providing technical support for new content formats such as interactive films and virtual tourism.

8. FAQ

Q: How much GPU VRAM is required to deploy H3-World locally?
A: H3-World has a base model with 33B parameters and a weight size of approximately 135GB, requiring a GPU with at least 40GB of VRAM for inference (such as A100 80GB or equivalent). If VRAM is insufficient, you can try a quantization scheme, but the official does not currently provide a quantized version, which may affect the user experience.

Q: Is there a difference in performance between the online experience and local deployment?
A: The online Space and local inference use the same model weights and inference logic, resulting in consistent generation outcomes. The online experience may have limitations on single-generation duration and concurrency, making it suitable for quick validation. Local deployment allows for a complete reproduction of the paper's experiments and supports longer autoregressive extensions.

Q: With only 8000 video training samples, will the generated content be too limited?
A: No. H3-World inherits the broad video semantics from the pre-trained MiniMax-H3 model. The 8000 game videos are used only to learn action control mappings. The model can generalize to unseen scenarios such as neon cities or desert oases, and action primitives can be combined to produce action sequences not present in the training data.

Q: Does H3-World support Chinese instruction input?
A: The official demo and documentation use English action instructions by default. Since the model reuses the pre-trained text pathway, in theory, Chinese action descriptions can be translated into English and used as input. However, the official has not conducted specific robustness testing for Chinese input, so it is recommended to prioritize English instructions for stable results.

Q: What are the roles of the LoRA weights and the DiffSynth patch?
A: The LoRA weights (step-10000.safetensors) are fine-tuned on the H3 self-attention projection and serve as the core carrier of action control capabilities. The DiffSynth patch modifies the attention calculation path to enable temporal attention routing. Without the patch, the LoRA parameters cannot take effect. Both components are essential and cannot be omitted.

9. Project Links

1. Project Website: https://danzer1xxxxchan.github.io/H3-World/
2. GitHub Repository: https://github.com/Danzer1xxxxChan/H3-World
3. HuggingFace Demo Space: https://huggingface.co/spaces/hugging-apps/h3-world-action-demo
4. Model Weights (ModelScope): https://modelscope.cn/models/DANNY621/H3-World
5. Technical Paper: https://modelscope.cn/papers/2609.01560