HOMIE – An Open-Source Digital Human Video Generation Framework from The Hong Kong University of Science and Technology

1. What is HOMIE

HOMIE is an open-source digital human video generation framework developed by the Department of Computer Science and Engineering at The Hong Kong University of Science and Technology. It is built upon the Wan2.1-T2V-14B backbone network from Alibaba and integrates the Qwen3-VL multimodal large model. This framework can simultaneously process four types of visual elements—digital humans, products, logos, and text—within a unified pipeline, addressing four major technical challenges: human-object interaction, brand element integration, OCR text fidelity, and multi-view consistency. The model training requires only approximately 5.5K steps / 10,000 A100 hours, achieving an OCR accuracy improvement of 38.7% over the strongest open-source model. It supports inference from 480P on a single GPU to 720P across multiple GPUs, significantly reducing the video production threshold in scenarios such as e-commerce live streaming and virtual anchors.

Image source: Official article
Image source: official article

Technical Positioning and Domain: HOMIE belongs to the field of multimodal video generation, focusing on multi-agent collaboration and semantic understanding in controllable digital human video generation. Unlike traditional digital human frameworks that only focus on facial animation, HOMIE incorporates non-human elements such as products, logos, and text as first-class citizens in the generation process, achieving a leap from "digital humans speaking" to "digital humans promoting products."

Development Background: The project is led by the YIYANGCAI team at The Hong Kong University of Science and Technology, leveraging the university's long-term expertise in computer vision and multimodal learning. The research motivation stems from the urgent need in e-commerce and brand marketing scenarios for integrated videos combining "digital humans + products + brand elements." Existing methods either fail to handle multi-agent interactions or require switching between independent models for different tasks, resulting in low efficiency.

Core Value: HOMIE addresses three major pain points in digital human video generation: "human-object disconnection, loss of brand elements, and text blurriness." By unifying the generation of four key elements within a single framework, e-commerce professionals no longer need to separately produce digital human videos, product display animations, and logo overlays. Instead, they can directly drive complete product promotion videos using text prompts, reducing the production cycle from days to minutes.

Technical Features: It employs an architecture combining "Wan2.1 backbone + Qwen3-VL inference + three-stage progressive training," injecting multimodal semantics without disrupting the original text-to-video alignment. Two innovative mechanisms—Global Multimodal Self-Attention Guidance (GMG) and Modality-Reference Embedding (MRE)—ensure consistency in the appearance of multiple agents and correct semantic attachment. The training steps required are only 1/5 to 1/10 of those needed by similar methods.

2. Key Features

• Human-Object Interactive Video Personalization: Supports multiple people and various objects appearing simultaneously in the same video. By leveraging reference images, the architecture constrains the appearance of each subject and generates natural interactive actions (e.g., holding, showcasing, passing). Users only need to provide photos of people and product images to generate targeted promotional video clips, eliminating the need for frame-by-frame hand-drawing or post-production compositing.

• Precise Logo/Abstract Concept Attachment: Utilizes the semantic reasoning capabilities of the Qwen3-VL multimodal large model to automatically attach abstract visual elements such as brand logos to the most semantically relevant objects in the video. Users do not need to explicitly describe the logo's position in the prompt; the model automatically completes the attachment based on common knowledge such as "soda bottle → bottle body" or "phone → back panel."

• High-Fidelity OCR Text Generation: When paired with OCR reference images, ensures that text on product packaging and labels remains clear, readable, and free from blurring or character misalignment in the generated video. In open-source evaluations, the OCR accuracy reaches 0.452, a 38.7% improvement over the previous state-of-the-art model, effectively addressing the industry challenge of text appearing as a blurry mass in AI-generated videos.

• Multi-View Consistency Generation: Given multi-view reference images of the same object (e.g., front, side, back), the model maintains consistent appearance and details when the object rotates or moves within the video. This feature is particularly important for display scenarios involving figurines, 3D models, and toys. The multi-view consistency metric DINOrec reaches 0.685, an 8.2% improvement over similar methods.

• Multimodal Semantic Understanding and Fusion: Introduces Global Multimodal Guidance (GMG) into the DiT self-attention layer, aligning the semantic features extracted by Qwen3-VL with VAE visual tokens on a global scale. Each video generation token can "see" the global semantic information from all reference images, enabling accurate understanding of the potential relationships between people, products, and logos.

• Three-Stage Progressive Training Strategy: The training process is divided into three stages: single-subject 480P → multi-subject 480P → high-definition 720P, completed in just 5.5K steps. Compared to similar methods that typically require 30–40K steps, this strategy improves training efficiency by 5–10 times, significantly reducing development costs. This approach also enables the model to rapidly converge during the low-resolution phase and make fine adjustments during the high-resolution phase.

• Flexible Inference and Multi-Resolution Output: Supports generating 832×480 resolution videos using a single GPU card, and can be scaled up to 1280×720 high-definition vertical videos using multi-GPU FSDP. Users can choose between 480P or 720P based on their available computational resources, and the output format is directly compatible with short video platforms such as Douyin and Kuaishou, eliminating the need for secondary encoding.

3. How to Use

1. Environment Setup: Clone the HOMIE code repository from GitHub (git clone https://github.com/YIYANGCAI/HOMIE.git), create a Python 3.10 virtual environment, and install the dependencies listed in requirements.txt. It is recommended to use an NVIDIA A100 (40GB) or higher GPU configuration, with a minimum VRAM of 24GB. For 480P inference on a single GPU, at least 24GB VRAM is required. For 720P multi-GPU inference, it is recommended to use 4×A100 GPUs.

2. Weight Download: Download the official pre-trained weights from the HuggingFace model library (yychai/homie-r2v-wan2.1) and place them in the checkpoints/ directory. The weight files are approximately 14GB in size and include fine-tuned parameters for the Wan2.1 backbone and the Qwen3-VL adaptation layer.

3. Material Preparation: Prepare reference images: frontal portrait of a person (recommended resolution: 512×512 or higher), multi-angle images of the product (at least one main image), and reference images for the Logo/text (prefer PNG with transparent background). All reference images must be named according to the official format and placed in the reference/ subdirectory. Multi-view images of the same subject should include a view label (e.g., product_front.jpg, product_back.jpg).

4. Prompt Writing: Compose text prompts in the format "Subject Action + Scene Description + Interaction Details". For example: "A young woman holding a bottle of red-packaged beverage in her right hand, smiling at the camera, with the brand text on the beverage bottle clearly visible." There is no need to describe the Logo's position in the prompt; the model will automatically attach it.

5. Run Inference: Execute python inference.py --prompt "your prompt" --person_image person.jpg --product_image product.jpg --logo_image logo.png. The default output resolution for a single GPU is 832×480 video. For 720P resolution, add --resolution 1280 720 and use --fsdp to enable multi-GPU parallelism.

6. Result Output: The generated video is saved in the output/ directory in MP4 format, with a default frame rate of 24fps. Users can control the video length using --num_frames (default: 81 frames, approximately 3.4 seconds), and use --seed to fix the random seed for reproducibility.

Notes: The first inference requires loading approximately 14GB of weights, which takes 1–3 minutes. Subsequent inferences take about 30–60 seconds per video. If VRAM is insufficient, reduce --num_frames or use --half to enable FP16 inference. It is recommended to first test the pipeline using the official example materials before replacing them with your own.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For e-commerce product promotion and brand marketing scenarios, HOMIE is currently the best open-source solution, with its unified four-factor framework and leading OCR/multi-view capabilities directly addressing industry pain points. SkyReels-V3 has similar functionality but higher training costs and slightly inferior OCR details, making it suitable for teams with existing custom video model foundations for secondary development. MuseTalk and Sadtalker focus on audio-driven facial animation and cannot handle commodities or logos, but they offer fast inference and low hardware requirements, making them ideal for live streaming or conversation scenarios requiring only digital human facial expressions. If the project requires both digital human identity consistency and product display, HOMIE is the only framework that covers the entire pipeline. If only quick deployment of digital human speech is needed, lightweight solutions like MuseTalk can be considered.

In academic research, HOMIE's three-stage progressive training strategy and its method of lossless MLLM integration offer valuable insights and are suitable for improvement as a foundational framework for multi-modal video generation. For small and medium-sized teams, the Apache 2.0 license and low training cost make HOMIE easy to implement, but A100-level computing power is required.

6. Editor's Summary

HOMIE achieves key innovations in its technical architecture: by introducing Global Multi-modal Guided Attention (GMG) and Modal-Reference Embedding (MRE), it injects the semantic reasoning capabilities of multi-modal large models into the DiT backbone without compromising the alignment between text and video. This design avoids the costly realignment training, enabling the model to simultaneously understand the semantic relationships between people, products, logos, and text, and generate physically plausible interactive actions. In practice, the 38.7% increase in OCR accuracy and 8.2% improvement in multi-view consistency represent not just fine-tuning enhancements, but architectural breakthroughs—GMG allows each video token to perceive the global semantics of the reference image, while MRE prevents multi-agent information confusion through independent embeddings. Training requires only 5.5K steps, demonstrating the efficiency of this architecture and providing a reproducible baseline for teams with limited resources.

In terms of practical value, HOMIE directly addresses a critical need in e-commerce and brand marketing: replacing the multi-tool pipeline of "digital human synthesis + product animation + logo overlay + text rendering" with a single framework, reducing the production cycle from days to minutes. The 1280×720 vertical output format is compatible with short video platforms, further lowering distribution costs. For scenarios such as virtual anchors and product showcases, the multi-view consistency feature solves the issue of visual "breakdown" that occurs when objects rotate in traditional generation methods, offering tangible commercial value.

The primary users include: e-commerce operators and brand marketers (who directly use the pre-trained model to generate sales videos); AI video generation researchers and developers (who use the open-source code for secondary development or academic research); and medium-sized MCN agencies (who leverage the low training cost to customize dedicated digital humans). In terms of future development potential, HOMIE can evolve in three directions: first, supporting longer video durations and real-time inference to meet the needs of live streaming scenarios; second, enhancing support for Chinese and multiple languages to expand into international markets; and third, integrating voice-driven capabilities to achieve full-modal control combining "text + voice + visual." Currently, the framework is in the early stages of open-source, and the pace of community contributions and documentation improvements will determine the speed of its ecosystem growth.

7. Application Scenarios

• E-commerce Live Streaming Sales: Merchants provide product images, digital human avatars, and brand logos. HOMIE automatically generates sales videos featuring the digital human holding the product and performing specified actions. It supports 1280×720 vertical video output, which can be directly uploaded to platforms like Douyin and Kuaishou without requiring post-production editing. The OCR function ensures that text on product packaging, such as ingredients and prices, is clear and readable, reducing the risk of content review issues.

• Brand Advertising Production: Marketing professionals only need to input the advertising copy and brand logo. HOMIE automatically attaches the logo to the most relevant object in the video (e.g., the side of a beverage bottle or the casing of an electronic product). There is no need to specify the placement in the prompt; the model uses multi-modal semantic understanding to achieve intelligent placement, resulting in more natural and consistent brand element presentation in the generated ad videos.

• Virtual Host/Avatar Content Creation: Provides a unified identity image for virtual hosts while supporting natural interaction with various products. For example, a virtual host can sequentially showcase a smartphone, headphones, and a power bank in the same video, maintaining consistent appearances for each product and smooth, coherent host actions. This is suitable for scenarios such as live stream previews and product reviews.

• Multi-View Product Display: Given multi-view reference images (front, side, back) of figures, 3D models, or toys, HOMIE generates a video showing the object rotating, maintaining consistent appearances across all angles. This is applicable for e-commerce product detail pages and crowdfunding promotional videos, providing multi-angle visual materials without the need to shoot physical items.

• Packaging Text Fidelity Output: When combined with OCR reference images, ensures that text on product packaging—such as food, cosmetics, and health supplements—is clear and legible in the generated videos. This solves issues of blurred, distorted, or garbled text in traditional AI video generation, making it ideal for scenarios requiring the display of ingredient lists, usage instructions, and production dates.

8. FAQ

Q: What hardware configuration is required to run HOMIE?
A: During inference, a single card with 480P (832×480) output requires at least 24GB of VRAM, with the recommended options being NVIDIA A100 (40GB) or RTX 4090 (24GB). For 720P multi-card inference, it is suggested to use 4×A100. During training, the official team used approximately 10,000 A100 hours to complete 5.5K steps. If users wish to perform fine-tuning, at least 8×A100 (40GB) is recommended. Consumer-grade GPUs (e.g., RTX 3090 24GB) can attempt inference with reduced resolution or frame rate, but performance will be slower.

Q: Does HOMIE support Chinese prompts and Chinese text generation?
A: The framework is based on Wan2.1 and Qwen3-VL, which have been thoroughly optimized and evaluated for English scenarios. Chinese prompts can be input theoretically, but the model's performance in understanding Chinese semantics has not been publicly tested. The OCR functionality is currently primarily designed for English text, and the fidelity of Chinese text processing remains to be validated. It is recommended that users first try English prompts, and wait for future versions to improve Chinese support.

Q: Is the training cost of HOMIE really that low? Can it be reproduced on consumer-grade GPUs?
A: According to the official report, HOMIE completed training in 5.5K steps using approximately 10,000 A100 hours, which is significantly lower than the tens of thousands of steps required by similar methods. However, this cost is based on an A100 cluster. Consumer-grade GPUs (e.g., RTX 4090) are limited by VRAM and computational power, and cannot directly reproduce the full training process. Users can attempt experimental fine-tuning by using a smaller resolution (e.g., 320P) or reducing the number of training steps, but the results may be suboptimal.

Q: What is the open-source license for HOMIE? Can it be used commercially?
A: HOMIE is licensed under Apache 2.0. The code and model weights are fully open-sourced, allowing commercial use, modification, and redistribution without additional authorization. However, note that the backbone networks Wan2.1-T2V-14B and Qwen3-VL also use open-source licenses, and users must comply with their respective terms (both allow commercial use). It is recommended that users read the LICENSE files of each component before use.

Q: What is the fundamental difference between HOMIE and digital human frameworks like MuseTalk and Sadtalker?
A: MuseTalk and Sadtalker primarily focus on generating facial animations driven by audio, producing only the face and upper body of a person. They cannot handle non-human elements such as products, logos, or text. HOMIE is a full-scene text-to-video generation framework that can simultaneously control digital humans, products, and brand elements, outputting complete scene videos. In e-commerce live-streaming scenarios, HOMIE can directly generate videos of "digital humans holding and showcasing products," whereas frameworks like MuseTalk require post-processing to integrate product visuals.

9. Project Links

• Product Website: https://yiyangcai.github.io/homie-page.github.io
• GitHub Repository: https://github.com/YIYANGCAI/HOMIE
• HuggingFace Model Library: https://huggingface.co/yychai/homie-r2v-wan2.1
• arXiv Technical Paper: https://arxiv.org/pdf/2607.18217