MAGI-2-preview – Sand.ai's Open-Source Multimodal Video Generation Model

1. What is MAGI-2-preview

MAGI-2-preview is a multimodal video generation model developed and open-sourced by Sand.ai, utilizing a Mixture of Experts (MoE) architecture with a total parameter count of 114B, activating only 6B parameters during inference. Based on a single-stream Transformer architecture, this model unifies the joint modeling of text, images, videos, and audio from the input layer, enabling native multimodal generation. It ranks sixth on the AA video generation leaderboard, with its code and pre-trained weights open-sourced under the Apache 2.0 license, exploring new pathways for efficient scalability in the video generation domain.

Image source: Official article
Image source: official article

Technical positioning and domain: Belongs to the field of multimodal video generation, focusing on achieving low-cost, high-efficiency video generation through a sparsely activated MoE architecture. It represents the first open-source implementation of a trillion-parameter MoE model in video generation, filling a gap in the open-source community for such large-scale video generation models.

Development background: Developed by the Sand.ai team, which has deep expertise in video generation and MoE architecture. The team previously launched the MAGI-1 series of models and validated the streaming generation capabilities. MAGI-2-preview is their latest achievement, aimed at verifying a new, efficient scaling path in the video generation domain and addressing the contradiction between traditional dense model parameter scale and inference cost.

Core value: Resolves the contradiction between parameter scale and inference cost in video generation models, achieving a significant reduction in inference resource requirements by activating only 6B parameters out of a total of 114B through MoE sparsity. At the same time, the single-stream architecture enables native fusion of audio and video, improving the quality of multimodal generation and providing a reproducible baseline for trillion-parameter video generation in both academia and industry.

Technical features: Employs a single-stream Transformer for unified modeling, directly exchanging cross-modal information via self-attention mechanisms, abandoning the traditional cross-attention concatenation approach; optimizes expert parallel communication for video ultra-long sequences, enabling efficient distributed training and inference; dynamic routing and expert load balancing ensure training stability.

2. Key Features

• Multimodal Video Generation: Supports generation from text, images, and videos, with native integration of audio understanding to achieve synchronized audio-visual output. Users can input text descriptions or reference images, and the model will generate corresponding video content, while also allowing audio prompts to control sound effects, eliminating the need for post-production alignment.

• MoE Sparse Activation: Utilizes a Mixture-of-Experts (MoE) architecture, activating only 6B parameters out of the total 114B during each inference. This significantly reduces computational costs and memory usage, enabling the trillion-parameter model to run under reasonable hardware conditions, with inference costs far lower than those of similarly sized dense models.

• Single-Stream Unified Architecture: Abandons traditional cross-attention concatenation methods, jointly modeling text, image, video, and audio tokens starting from the first layer of the Transformer. The self-attention mechanism directly exchanges cross-modal information, improving audio-visual synchronization accuracy and avoiding delays and quality degradation caused by post-production alignment.

• Efficient Scaling: Reconstructs MoE communication and training stability for video generation scenarios, addressing the cross-card communication bottleneck under ultra-long sequences, enabling effective model scaling, and laying the foundation for training even larger models.

• Distributed Communication Optimization: Tailored for the characteristics of ultra-long video sequences, the token routing and communication strategies in expert parallelism are restructured to reduce cross-GPU data transfer overhead, enhance multi-card training efficiency, and make training trillion-parameter models feasible with limited hardware resources.

• Training Stability Assurance: Integrates three stability assurance mechanisms—dynamic routing, expert load balancing, and joint multimodal data training—to ensure stable convergence of the MoE model in video generation scenarios, preventing expert collapse and routing oscillation.

3. How to Use

1. Environment Setup: Clone the GitHub repository SandAI-org/MAGI-2-preview and configure dependencies according to the README. It is recommended to use a Linux operating system (Ubuntu 20.04 or higher), and install Python 3.8+, PyTorch 2.0 or higher, as well as dependency libraries such as transformers and diffusers. Specific dependencies can be found in the requirements.txt file in the project documentation.

2. Download Model Weights: Obtain the pre-trained weights from GitHub Releases or the HuggingFace model repository (sand-ai/MAGI-2-preview) and place them in the designated directory of the project. The weight files are large (approximately tens of gigabytes), so it is recommended to use git lfs or directly download them using the HuggingFace download tool, ensuring sufficient disk space is available.

3. Configure Inference Script: Modify the configuration file and script parameters to support three modes: t2v (text-to-video), i2v (image-to-video), and v2v (video-to-video). Refer to the run.sh format from MAGI-1 for execution, setting parameters such as input prompts, reference image paths, and output directories. You can adjust video resolution, frame count, and other generation parameters as needed.

4. Hardware Requirements: Specific configurations will be updated in the official documentation. It is recommended to refer to the 24B multi-GPU H100 or 4.5B single-card 24GB VRAM setup from MAGI-1. Since MAGI-2-preview has a total parameter count of 114B and an activation size of 6B, it is advised to use GPUs with at least 24GB of VRAM (such as NVIDIA A10G, A100, H100) for inference. Using multiple GPUs can significantly improve generation speed.

5. Run Inference: Execute the launch command, and the model will generate a video based on the input. The first run will require loading the weights, which may take a long time. Pay attention to adjusting parameters such as resolution and frame count to balance quality and speed. It is recommended to first test with short prompts to confirm that the environment is correctly configured before proceeding to more complex generations.

6. Notes: Ensure a stable network connection and avoid interruptions during weight downloads; be mindful of VRAM usage during inference to prevent exceeding hardware limits; when using multiple GPUs, correctly set the distributed environment variables; if using a Windows system, it is recommended to run the model via WSL2 or Docker.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For scenarios requiring native audio-video fusion generation and low inference costs, MAGI-2-preview's MoE sparse activation offers a clear advantage, making it suitable for academic research and enterprise private deployment. If video editing and camera movement control capabilities are a priority, Wan2.7-Video excels in functional richness, making it ideal for film production and content creation. Open-Sora 1.2 stands out in lightweight deployment and quick onboarding, suitable for individual developers and prototype validation. CogVideoX provides better support for Chinese, making it well-suited for Chinese content generation scenarios.

6. Editor's Summary

MAGI-2-preview represents a milestone in technological innovation: it is the first to successfully apply a trillion-parameter MoE architecture to the field of video generation, and it achieves native fusion of audio and video through a single-stream Transformer, a first in the open-source community. From a technical implementation perspective, the model includes targeted designs for MoE communication optimization and ensuring training stability, addressing core challenges in large-scale distributed training under long sequences. This provides reproducible experience for the development of future, even larger video generation models. Its sparse activation strategy, with a total parameter count of 114B but only 6B activated, effectively reduces inference costs, making a trillion-parameter model operable under reasonable hardware conditions, which holds significant value in practical deployment.

In terms of practical value, MAGI-2-preview's open-source strategy (Apache 2.0) greatly lowers the usage threshold, enabling both academic research and commercial applications to perform secondary development and customization based on this model. Its multimodal generation capabilities cover text, images, video, and audio, making it particularly suitable for scenarios requiring audio-visual synchronization, such as advertising production and film previsualization. However, the model has not yet clearly defined key metrics such as video duration and resolution, and its generation quality still lags behind leading commercial models, which limits its application in high-quality film and video content production.

In terms of target users, this model is primarily aimed at AI video researchers, multimodal model developers, and enterprise teams requiring private deployment. Individual creators with a certain technical foundation can also attempt to use it through the open-source code, but this would require significant hardware investment. In terms of future development potential, as Sand.ai continues to optimize the model and the community contributes further, the MAGI series is expected to improve further in video generation quality and functional richness, becoming a major force in the open-source video generation field.

7. Application Scenarios

• Long-form Video Ads and Short Films: The MoE architecture supports complex narrative modeling, enabling the generation of coherent, minutes-long video content. Advertising agencies and content creators can leverage its multimodal capabilities to input scripts and reference images, quickly producing video ads or short films with complete storylines and reducing production costs.

• Multimodal Film Previsualization: Native audio understanding capabilities allow for the synchronized generation and editing of voiceovers, sound effects, and visuals. Film production teams can use this model during the previsualization phase to rapidly generate concept videos with sound effects, accelerating creative iteration and reducing post-production workload.

• AI Video Academic Research: The open-source trillion-parameter video MoE weights provide a reproducible Scaling baseline for the academic community. Researchers can use this model for theoretical studies on video generation, architectural improvements, and performance evaluation, advancing the field.

• Enterprise Private Deployment: The model supports the construction of private video generation capabilities in sensitive industries such as finance and healthcare, based on the Apache 2.0 protocol. Enterprises can deploy the model on internal servers to generate training videos, product demonstrations, and other content, ensuring data security.

• Real-time Streaming Video Generation: Sparse activation reduces inference costs, enabling low-latency real-time video generation and live streaming scenarios. Live streaming platforms and social applications can integrate this model to generate personalized video content in real-time based on user input, enhancing interactive experiences.

8. FAQ

Q: What are the hardware requirements for MAGI-2-preview?
A: Specific configurations will be updated in the official documentation. It is recommended to refer to the 24B multi-card H100 or 4.5B single-card 24GB VRAM setup from MAGI-1. As the model has a total parameter count of 114B and an activation size of 6B, it is recommended to use GPUs with at least 24GB of VRAM (such as NVIDIA A10G, A100, H100) for inference. Using multiple GPUs can improve speed.

Q: Does the model support Chinese input?
A: The model is primarily trained for English scenarios, but theoretically supports multi-language text input. Actual performance depends on the coverage of Chinese data in the training set. It is recommended to use English prompts for more stable results.

Q: How can I obtain the pre-trained weights?
A: The pre-trained weights can be downloaded from GitHub Releases or the HuggingFace model repository (sand-ai/MAGI-2-preview). It is recommended to use git lfs or the HuggingFace download tool, as the weight files are large. Ensure a stable network connection and sufficient disk space.

Q: What are the maximum duration and resolution for video generation?
A: The official documentation has not yet clearly specified the maximum video duration or resolution. The model inherits MAGI-1's streaming generation capability, theoretically allowing for the generation of longer videos. However, practical results are limited by VRAM and inference time. It is recommended to adjust parameters based on your hardware conditions and start with short clips.

Q: Does the model support audio generation?
A: The model natively integrates audio understanding and supports synchronized audio-visual generation. Users can input audio prompts or reference audio, and the model will generate the corresponding audio alongside the video, eliminating the need for post-processing alignment.

Q: Can the model be used for commercial purposes?
A: Yes, the model is released under the Apache 2.0 license, with full open access to both code and weights. It supports commercial and academic use, but the terms of the license must be followed, and copyright notices must be retained.

Q: How can I contribute to the community?
A: You can follow the GitHub repository and submit Issues or Pull Requests. The Sand.ai team welcomes community contributions, including model optimization, documentation improvements, and application development. You can also follow the official blog for the latest updates.

9. Project Links

• Project Website: https://sand.ai/blog/magi-2-preview
• GitHub Repository: https://github.com/SandAI-org/MAGI-2-preview
• HuggingFace Model Library: https://huggingface.co/sand-ai/MAGI-2-preview