LongCat-Video-Avatar 1.5 – Meituan's Open-Source Digital Human Video Generation Model

1. What Is LongCat-Video-Avatar 1.5

LongCat-Video-Avatar 1.5 is Meituan LongCat's open audio-driven digital human video framework, built on the 13.6B-parameter LongCat-Video base. Upgrading the audio encoder from Wav2Vec2 to Whisper-Large-v3 dramatically improves lip-sync; Step Distillation cuts inference to 8 steps; 480P/720P and INT8 quantization are supported. It delivers production-grade physical plausibility and temporal stability for solo/multi-person dialogue, singing, and stylized animation—fully open under MIT for deploy and extension.

Image source: Official article

Technical positioning and domain: CV and multimodal generation—audio-driven talking head generation. It combines large video models with fine-grained audio features, distillation, and quantization for high-precision, stable, commercially usable open digital humans.

Development background: Meituan LongCat team with prior LongCat-Video releases. Demand from e-commerce livestream, edtech, and virtual support; open solutions lacked lip-sync, long-video identity drift, and speed—addressed here via open high-precision, efficient tooling.

Core value: Balances lip-sync accuracy vs. stability and speed vs. quality. Whisper-Large-v3 improves lip match; Step Distillation ~5× faster at similar quality. Native dual-audio interaction, continuation, and stylization out of the box.

Technical characteristics: Coarse-to-fine spatiotemporal generation with Block Sparse Attention; multi-reward GRPO for text alignment, visual quality, and motion coherence; INT8 and multi-GPU context parallelism on consumer GPUs.

2. Key Features

• Single-audio driven video: One audio track plus prompt or reference image yields dynamic, lip-accurate talking video. Whisper-Large-v3 phoneme-level features maintain sync under speed and emotion changes.

• Multi-audio interaction: Dual audio streams for two-person dialogue, turn-taking, etc. Time alignment handles switching and overlap—no manual editing for multi-role visuals.

• Video continuation and long video: Native continuation extends minutes without color drift or identity break via segmented generation and latent state handoff—courses, live replay, long talks.

• Stylization and animation: Generalizes to anime, animals, complex real-world conditions, 3D animation styles—audio-driven capability beyond photoreal faces.

• Singing and performance: Music-driven expressions and stable full/half-body performance for virtual idols and music video use.

• Multi-task base model: Underlying LongCat-Video supports text-to-video, image-to-video, and continuation in one framework.

3. How to Use

1. Environment: Clone repo, Conda env, PyTorch 2.6.0, FlashAttention-2, dependencies. Linux (Ubuntu 20.04+), NVIDIA GPU 16GB+, CUDA 12.1+.

2. Download weights: huggingface-cli download meituan-longcat/LongCat-Video-Avatar-1.5 --local-dir ./weights (~26GB).

3. Single-audio generation: JSON with audio path, prompt, reference image:

   {
     "audio_path": "./input/audio.wav",
     "prompt": "A person speaking naturally",
     "ref_image_path": "./input/ref.jpg"
   }
   

   Run run_demo_avatar_single_audio_to_video.py with --model_type avatar-v1.5 --use_distill --use_int8. Output in ./output.

4. Multi-audio: JSON with two audios and character info; run run_demo_avatar_multi_audio_to_video.py. Merge (overlap) or Concatenation (turn-taking).

5. Continuation: Add --num_segments 4 (etc.) for longer continuous video; ~15 s per segment by default.

6. WebUI: streamlit run ./run_streamlit.py for interactive upload and parameters.

Notes: INT8 slightly lowers quality—use when VRAM tight; --use_distill recommended; set CUDA_VISIBLE_DEVICES for multi-GPU.

4. Pros and Cons

5. Comparison with Similar Tools

Selection advice: Open, customizable digital human video for dev teams/SMEs—best pick with MIT, INT8, multi-audio, and continuation. Need 4K or turnkey cloud—HeyGen et al. Researchers gain a strong lip-sync + distillation baseline.

6. Editor's Take

LongCat-Video-Avatar 1.5 innovates by pairing Whisper-Large-v3 with large video models—a first in open lip-sync at this level. Eight-step distillation without quality collapse matters for deployment. Multi-reward GRPO balances alignment, visuals, and motion thoughtfully.

It solves drift, multi-person generation, and stylization for live commerce, edtech, and virtual support—clear commercial paths. MIT lowers the bar for quality previously commercial-only.

Audience: e-commerce ops, schools cutting course cost, creators, multimodal researchers. Not ideal for 4K purists; 720P suffices for most apps.

Architecture scales—expect Chinese fine-tunes, higher res, richer style templates with community growth.

— Resolution and Chinese gaps; otherwise leading open package.

7. Use Cases

• AI spokesperson and e-commerce: Product audio + reference image → batch lip-sync promo video; continuation for multi-minute listings on short-video platforms.

• Online education: Course audio → stable virtual instructor with gestures; multi-audio for Q&A scenes.

• Virtual customer service: Branded digital agents with multi-turn dialogue visualization on web or outbound calls.

• Short video and social: Sing/perform audio → stylized (anime, realistic) clips for TikTok, Bilibili, etc.

8. FAQ

Q: Minimum VRAM?
A: 16GB with INT8 for smooth 480P; 24GB+ for 720P or FP16.

Q: Chinese speech?
A: Supported but weaker than English; fine-tune on Chinese audio for production.

Q: Identity drift on long video?
A: Use continuation with sensible --num_segments (15–30 s each); keep reference lighting/angle consistent.

Q: INT8 quality impact?
A: Slightly less detail/color; lip-sync and motion largely unchanged. FP16 if VRAM allows.

Q: Commercial use?
A: MIT allows commercial use/modification; generated content must comply with law.

Q: Control two speakers' order?
A: Timestamp two audios in JSON; Concatenation for turns, Merge for simultaneous speech.

9. Project Links

• GitHub: https://github.com/meituan-longcat/LongCat-Video
• Project site: https://meigen-ai.github.io/LongCat-Video-Avatar-1.5-Page/
• Hugging Face: https://huggingface.co/meituan-longcat/LongCat-Video-Avatar-1.5