Wan-Dancer – The Open-Source Human Figure Dance Video Generation Model from Alibaba Tongyi Wanxiang

1. What is Wan-Dancer

Wan-Dancer is an open-source music-driven human figure dance video generation model developed by Alibaba Tongyi Wanxiang. Users only need to provide a single正面 (front-facing) portrait photo and a piece of music to automatically generate rhythmically accurate, fluid, and stylistically distinct dance videos. This model has, for the first time, broken through the bottleneck of minute-level temporal generation, supporting the output of ultra-long, coherent 720p/30fps videos ranging from 15 seconds to 3 minutes. It covers five dance styles: Chinese classical dance, K-pop dance, street dance, tap dance, and Latin dance. Additionally, it allows for personalized dance style customization through LoRA technology. Wan-Dancer excels in long-sequence consistency, beat alignment, and identity preservation, providing an efficient technical solution for dance content creation, virtual idol performances, and choreography assistance.

Image source: Official article
Image source: official article

Technical positioning and domain: Wan-Dancer belongs to the video generation branch of generative artificial intelligence, focusing on music-driven human figure dance synthesis. Its uniqueness lies in the use of a hierarchical decoupled dual DiT (Diffusion Transformer) architecture, separating global motion planning from local detail refinement. This addresses common issues in traditional models, such as temporal drift and repetitive motions in long video generation, positioning it as an open-source dance generation tool aimed at professional creation and personalized customization.

Development background: This model was developed by the Alibaba Tongyi Wanxiang team, leveraging Alibaba's deep expertise in the fields of computer vision and multimodal understanding. The motivation for its development stems from the current limitations of dance video generation models, which are typically constrained to short durations (usually no more than 20 seconds), lack diversity in movements, and suffer from imprecise beat alignment. The goal is to achieve stable, high-quality dance video generation at the minute level through innovative architectural design.

Core value: The core value of Wan-Dancer lies in advancing dance video generation from the "short clip experimentation" phase to the "minute-level usability" stage, enabling non-professional users to quickly produce diverse dance content. At the same time, its LoRA-based personalization capabilities reduce the threshold for replicating specific dance styles, providing a practical technical foundation for dance education, virtual IP operations, and creative previsualization.

2. Key Features

• Music-Driven Dance Generation: Input a person's photo and an audio file, and the model automatically analyzes the music's beat, rhythm, and style to generate a dance video that is highly synchronized with the music. The system includes a dedicated Music Encoder to extract musical features, which are injected into the generation process via cross-attention mechanisms, achieving precise alignment at the beat level. Users do not need to manually annotate rhythm points.

• Minute-Level Long Video Generation: Breaking through the traditional dance generation model's limitation of around 20 seconds, this system supports stable output of coherent videos ranging from 15 seconds to 3 minutes. Through a hierarchical decoupled architecture, the global DiT plans long-term motion skeletons, while the local DiT fills in details, effectively avoiding pose drift and content repetition in long sequences. Visual consistency is maintained at 720p/30fps resolution.

• Multi-Style Dance Coverage: Natively supports five distinct dance styles, including Chinese classical dance, K-Pop, street dance, tap dance, and Latin dance. Users can switch between styles using prompts or predefined style tags. The model was trained to separately model the motion characteristics of each style, resulting in clearly differentiated outputs in terms of limb amplitude, rhythm patterns, and motion texture.

• LoRA Personalization: Only a small amount of specific dance video clips (approximately 10–20 seconds) are needed to train a custom LoRA weight, enabling the replication of the same dance moves. This feature is suitable for brand IP customization, dance teaching demonstrations, and the creation of exclusive motion libraries for virtual characters. The training process can be completed on consumer-grade GPUs.

• Keyframe Secondary Editing: Keyframes generated in the global phase support manual editing. Users can adjust specific poses, change clothing, or modify motion trajectories. The local refinement phase will regenerate transition frames based on the modified keyframes, enabling precise motion-level control and offering greater flexibility for professional creators.

• Dynamic Frame Rate Adaptation: By introducing rotational position encoding (RoPE), the model maps absolute time information to each step of the generation process, allowing it to automatically adjust the frame rate distribution based on the actual duration of the input music. Whether the music is 15 seconds or 3 minutes long, the rhythm density and temporal relationships of the dance movements remain naturally coherent, eliminating motion distortion caused by time scaling in traditional methods.

3. How to Use

1. Environment Requirements and Prerequisites: It is recommended to use a Linux system equipped with an NVIDIA A100 (80GB) or higher memory GPU, PyTorch version ≥2.0, and CUDA ≥11.8. For local deployment, ensure that Git, Python 3.10 or higher, and at least 100GB of available disk space for storing model weights (14B parameters) are installed. Windows users can run the system via WSL2 or Docker.

2. Online Experience (Quick Start): Visit the ModelScope Studio page and upload a 9:16 vertical portrait photo of a single person (recommended resolution of 720×1280 or higher) and a 10–30 second MP3/WAV music file. After selecting the target dance style, click "Generate." The system will output the dance video within 5–15 minutes. This method is ideal for quickly verifying results without any local setup.

3. Local Deployment and Inference: Clone the GitHub repository (git clone （link to be updated after official release） install -r requirements.txt). Download the 14B weights (approximately 28GB) via ModelScope. Run the global keyframe inference script (python run_global.py --config configs/global.yaml) and the local temporal refinement script (python run_local.py --config configs/local.yaml) sequentially. Note that the global phase generates approximately 64 keyframes, and the local phase refines them into a full video. The total inference time depends on the video length and GPU performance.

4. DiffSynth-Studio Integrated Inference: After installing the DiffSynth-Studio library, call the WanVideoPipeline through the Python API. Load the global and local model configurations, set the reference image path, music file path, keyframe mask parameters, and output resolution. This method supports batch processing and more flexible parameter adjustments, making it suitable for developers who need to integrate it into existing workflows. Example code can be found in the examples/ directory of the project's GitHub repository.

5. Key Configurations and Notes: When generating a 3-minute video, it is recommended to set the number of global keyframes to 128 frames and the local refinement steps to 50 steps for optimal quality. For LoRA training, prepare a 10–20 second video of the same dance style. Use the train_lora.py script with a learning rate of 1e-4 and train for 2000 steps. Note that the background of the person's photo should be simple, avoiding multiple people or obstructions, as this may affect the identity preservation effect.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For creators requiring high-quality dance videos at the minutes level with precise alignment to music beats, Wan-Dancer is the optimal choice. Its hierarchical decoupled architecture and dynamic frame rate adaptation capabilities offer clear advantages over similar tools. If only short clips (4–5 seconds) are needed and music synchronization is not a priority, MuseDance's end-to-end U-Net architecture provides faster inference, although it is limited in output resolution and duration. For scenarios primarily requiring character animation (e.g., generating actions for virtual characters) without music-driven requirements, Animate Anyone offers mature identity preservation and pose control capabilities, along with a more complete community ecosystem. However, it requires additional input in the form of a pose sequence, making it less automated than Wan-Dancer. MagicAnimate performs well in short clip animation generation but lacks music understanding capabilities, making it more suitable as a video in-betweening or motion transfer tool. Overall, Wan-Dancer leads in functional completeness and technical advancement within the vertical domain of dance generation, though users must consider its higher hardware requirements and relatively steeper learning curve.

6. Editor's Summary

Wan-Dancer has made a meaningful innovation in the niche area of music-driven dance video generation. Its hierarchical decoupled dual DiT architecture is not a simple stacking of modules, but rather a systematic solution to the core contradiction in long video generation: "global consistency" and "local detail richness." The global DiT is responsible for planning the motion skeleton at the minute-level timescale, while the local DiT focuses on the physical plausibility of transitions between frames. This division of labor allows the model to maintain coherent action logic and stable visual quality even for videos up to three minutes in length, avoiding the common issues of posture degradation and content repetition seen in traditional diffusion models for long sequences.

In terms of technical implementation, the introduction of RoPE dynamic frame rate adaptation and optical flow loss functions enhances the usability of the generated results from two dimensions: temporal synchronization and motion smoothness. These design ideas offer valuable references for research in the same field.

In terms of practical value, Wan-Dancer advances dance video generation from the "experimental fragment" stage to the "deployable content" stage. The generation range of 15 seconds to 3 minutes covers the mainstream content lengths on short video platforms, and the output specification of 720p/30fps meets the basic commercial threshold. The LoRA personalization customization feature further reduces the cost of replicating dance styles, enabling scenarios such as brand IP, virtual idol operations, and dance education to quickly obtain customized materials. However, the model's high hardware resource requirements (recommended A100-level GPU) and relatively complex inference process limit its popularity among individual creators to some extent. In the future, if a lightweight version is released or more efficient inference optimizations are provided, its application scope will be further expanded.

In terms of target users, Wan-Dancer is most suitable for content creators with a certain technical background, virtual IP operation teams, and dance education institutions. For ordinary short video users, the online experience platform provides an accessible entry point for trial use, but local deployment and LoRA training still require a certain level of engineering capability. In terms of technical development trends, the hierarchical planning and refinement paradigm demonstrated by Wan-Dancer has the potential to be extended to a broader range of video generation tasks (such as action-based game character animations and sports instructional videos). As the open-source community continues to contribute and optimize the model weights, its inference efficiency and dance genre coverage are expected to improve, thereby establishing a more solid technical barrier in the field of AI-driven dance generation.

7. Application Scenarios

• Short Video Content Creation: Creators can upload a portrait photo and a piece of trending music to quickly generate a dance video synchronized with the music's rhythm, suitable for content publishing on platforms like Douyin, Kuaishou, and Xiaohongshu. No need for real people to appear on camera or professional choreography and filming equipment, enabling mass production of diverse dance content. This is especially suitable for virtual character operations and anonymous content creation.

• Virtual Idols and Digital Human Performances: Virtual idol operation teams can use Wan-Dancer to generate coherent performance materials tailored to specific dance styles for their characters, which can be used for live stream warm-ups, virtual concert clips, or daily short video content. Compared to traditional motion capture or hand-drawn animation, the generation cost is significantly reduced, and the output duration is sufficient to support full song performances.

• Dance Teaching and Choreography Assistance: Dance instructors can utilize the model's beat alignment and keyframe preview features to automatically generate corresponding dance action sequences from a piece of music, serving as reference videos for teaching demonstrations. Additionally, by adjusting keyframes, instructors can modify the amplitude and rhythm of specific movements, aiding in the design of choreography plans and improving teaching efficiency.

• Advertising and Film Previsualization: Directors and creative teams can use Wan-Dancer to quickly generate previsualizations of dance shots at the storyboard level before filming, verifying the coordination between dance arrangements and musical rhythms, as well as the visual presentation of different dance styles in the scene. This helps in rapidly iterating creative concepts before formal filming, reducing trial-and-error costs.

8. FAQ

Q: What are the requirements for input photos in Wan-Dancer?
A: It is recommended to upload a 9:16 portrait-oriented single-person photo with a resolution of no less than 720×1280. The background should be simple, and the subject should be fully visible without obstructions. The facial and body contours of the person in the photo should be clearly visible. Avoid side views, downward glances, or group photos, as these may affect the accuracy of identity preservation and motion generation.

Q: How long does it take to generate a 3-minute video?
A: On an NVIDIA A100 (80GB) GPU, global keyframe inference takes approximately 3–5 minutes, and local temporal refinement takes about 10–20 minutes, totaling around 15–25 minutes. If using a consumer-grade GPU (such as RTX 4090), the time may extend to 40–60 minutes. Due to resource limitations, online experience platforms typically take 5–10 minutes to generate a 30-second video.

Q: Does Wan-Dancer support audio formats other than custom music?
A: Currently, it supports MP3, WAV, and FLAC formats, with a recommended sampling rate of 44100Hz or 48000Hz. The model does not impose strict restrictions on music style, but clearer rhythm and beat (such as pop, electronic, or dance music) will yield better results. Pure vocals or audio with unclear rhythm may lead to mismatched movements and music.

Q: How much data is required for LoRA training? What hardware is needed?
A: To train custom LoRA weights, prepare 10–20 seconds of video clips from the same dance style. The person's movements in the video should be complete and not frequently obstructed. For hardware, it is recommended to use a GPU with at least 16GB of VRAM (such as RTX 4080). Training for 2000 steps takes approximately 30–60 minutes. The training data must be preprocessed into 720p/30fps format, and the corresponding audio files must be extracted.

Q: Will the character's identity features change with the length of the video?
A: Wan-Dancer employs global keyframe anchoring constraints and VAE encoding, which provide strong identity preservation in long sequences. However, slight changes in facial details may occur under extreme movements (such as rapid spinning or bending) or complex lighting conditions. It is recommended that the input photo features a neutral expression, even lighting, and avoids the use of photos with exaggerated makeup or accessories.

Q: How can I modify the motion details in the generated video?
A: Users can export the skeleton sequence during the global keyframe stage and use image editing tools to adjust the pose of specific frames (such as changing arm angles or clothing colors), then rerun the local refinement script. The project provides tools for visualizing and exporting keyframes, but some programming knowledge is required. A simpler alternative is to adjust the overall motion style through LoRA training.

9. Project Links

• Project Website: https://humanaigc.github.io/wan-dancer-project/
• GitHub Repository: https://github.com/Wan-Video/Wan-Dancer
• HuggingFace Model Library: https://huggingface.co/Wan-AI/Wan-Dancer-14B
• arXiv Technical Paper: https://arxiv.org/pdf/2607.09581
• ModelScope Online Experience: https://www.modelscope.cn/studios/Wan-AI/Wan-Dancer