JoyAI-Echo – JD's Open-Source Long-Form Audio-Video Generation Framework

1. What Is JoyAI-Echo

JoyAI-Echo is an open-source long-form audio-video generation framework from JD Future Academy, designed for minute-scale multi-shot story generation. Through four technical innovations—a paired cross-modal memory bank, memory-driven post-training, Director Agent conversational editing, and lightweight real-time super-resolution—it addresses core pain points in long video generation: character face drift, voice inconsistency, and slow generation. It is the first system to deliver up to 5 minutes of highly consistent, interactive, HD long-form video. Built end-to-end on LTX-2.3 and Gemma-3-12B, JoyAI-Echo marks JD's entry into the global top tier of long video generation and provides a deployable open-source solution for virtual content creation, digital humans, and film pre-visualization.

Image source: Official article

Technical positioning and domain: JoyAI-Echo sits in multimodal generation, focused on joint long video and audio synthesis. Unlike short-clip models (seconds to tens of seconds), it targets cross-shot consistency, interactivity, and real-time performance at minute scale—positioned as a professional long-form content production tool rather than a simple text-to-video demo.

Development background: Led by JD Future Academy (Joy Future Academy), the team has deep experience in video generation, multimodal retrieval, and diffusion acceleration. The project builds on LTX-2.3 (Lightning Transformer) and Google's Gemma-3-12B text encoder, aiming to break bottlenecks in duration, consistency, and interactivity. JD serves its own e-commerce marketing and digital human business while advancing open-source progress in long video generation.

Core value: JoyAI-Echo tackles three industry challenges: (1) character identity drift—inconsistent facial features and voice across shots; (2) slow generation—minute-scale video requiring hours of inference; (3) high creation barrier—users struggle to control multi-shot narrative in natural language. Paired cross-modal memory preserves consistency; DMD distillation delivers ~7.5× speedup; Director Agent enables conversational local edits so non-experts can produce up to 5 minutes of high-quality story video.

Technical characteristics: A "paired cross-modal memory bank" binds visual and audio features for reuse across shots; Distribution Matching Distillation (DMD) compresses multi-step diffusion to few steps for practical streaming generation; Director Agent expands natural language into structured scripts, shots, characters, and scenes with closed-loop editing; a lightweight single-step super-resolution module upscales 736×1280 to 1472×2560 under streaming latency for professional output.

2. Key Features

• Minute-scale multi-shot coherent stories: Generate multi-shot video sequences up to 5 minutes from a single prompt or JSON file. The system splits text into coherent shots while preserving character, scene, and narrative continuity—breaking the short-clip limit of existing models.

• Cross-modal joint audio-video generation: A single pipeline outputs synchronized video and audio—character speech, ambient SFX, and background music—without manual alignment, reducing production complexity.

• Paired cross-modal memory bank: Core component using slot-paired binding of visual memory (face, appearance) and audio memory (speaker timbre). Each new shot is conditioned on prior identity features for story-level consistency up to 5 minutes; user preference rate 59.4% vs. 27.7% for competitors.

• DMD few-step inference acceleration: Distribution Matching Distillation reduces dozens of diffusion steps to ~7, achieving ~7.5× speedup while preserving quality—making minute-scale streaming practical; per-shot time drops from minutes to seconds.

• Director Agent conversational editing: Interact via natural language; the agent expands intent into structured script, characters, scenes, and shots. Supports plan-generate-review-local-revise workflow—edit a problematic segment without rerunning the full video, turning static generation into dynamic collaboration.

• Lightweight real-time audio-video super-resolution: Single-step module sharpens 736×1280 base output to 1152×1920 or 1472×2560 under streaming latency without breaking real-time generation.

• Fully open source and reproducible: Code and weights under Apache 2.0 (academic/non-commercial); built on LTX-2.3 and Gemma-3-12B with inference scripts, sample prompts, and docs for reproduction.

3. How to Use

1. Environment and prerequisites: Python 3.11, PyTorch 2.8, CUDA 12.8, and a compatible NVIDIA GPU (24GB+ VRAM recommended, e.g., RTX 4090 or A100). Install ffmpeg for AV processing. Use conda or uv for dependencies.

2. Clone and install: Run git clone https://github.com/jd-opensource/JoyAI-Echo.git, create env with conda create -n joyai python=3.11, then pip install -r requirements.txt. Install PyTorch manually for your CUDA version.

3. Download weights: From Hugging Face, download ~46GB echo-longvideo-release.safetensors and ~24GB gemma-3-12b encoder into checkpoints/. Use resumable download for stability.

4. Write story prompt JSON: Create JSON describing each shot—characters/subjects, actions/dialogue, style, camera motion, background, SFX/BGM. See examples/. Multiple shots auto-maintain cross-shot consistency.

5. Run inference: Execute python inference.py; model loads once and processes all prompt files. Output defaults to inference_result/outputs/. First load ~2–3 minutes; per-shot generation ~5–15 seconds depending on resolution and complexity.

6. Director Agent editing (optional): Run python director_agent.py --input your_story.json for conversational edits—e.g., "change the third shot background to dusk"—for localized regeneration without full reruns.

4. Pros and Cons

5. Comparison with Similar Tools

Selection advice: For multi-minute, multi-shot stories with high consistency (animators, digital human ops), JoyAI-Echo is the only complete open solution. For budget short social ads, VideoCrafter2 or AnimateDiff may suffice. HappyOyster is close commercially but closed. For synced audio-video, JoyAI-Echo is the standout open option.

6. Editor's Take

JoyAI-Echo shows JD's depth in long video generation. The paired cross-modal memory bank is not simple feature caching—it binds visual and audio via slot-pairing with latent conditioning in diffusion—a novel industrial and academic approach. DMD uses distribution matching to map teacher multi-step distributions to student few-step paths for 7.5× speedup—a distinct route from LCM with stronger theoretical grounding.

Practically, it addresses "face drift," "voice jumps," and slow generation. Director Agent upgrades generative AI from one-shot output to interactive collaboration—ideal for marketing teams iterating brand video. Hardware (24GB+) and Chinese support remain gaps for individual creators.

Audience: professional animators, digital human producers, pre-viz teams, researchers. Enterprises need license review for commercial use.

Future potential is strong—distillation maturity and hardware cost drops could make JoyAI-Echo a long-video open benchmark. Watch for more languages, lower VRAM, stronger multilingual encoders, and richer editing (background swap, outfit change).

— Strong innovation (memory + DMD), high utility (5 min + conversational edit), open-friendly.

7. Use Cases

• Virtual storytelling and animation: Produce multi-minute coherent animation; Director Agent iterates plot and shots with consistent look, voice, and character—80%+ time savings vs. frame-by-frame work.

• Digital human content and live streaming: Long-form virtual host/customer content with stable face and voice; real-time SR for HD live push.

• Brand marketing rapid iteration: Conversational edits to scripts and shots—indoor/outdoor swaps, outfit changes—in hours not weeks.

• Film pre-viz and storyboards: Natural-language long-form pre-viz for shot language, blocking, and pacing; 5-minute coverage for key scenes reduces shoot risk.

• Education and training: Multi-shot teaching video—historical reenactments, lab demos—with consistent presenter identity and local revision support.

8. FAQ

Q: Which operating systems are supported?
A: Primarily Linux (Ubuntu 20.04+) and Windows 10/11 (WSL2 or native CUDA). macOS lacks NVIDIA GPU for full inference—prompt authoring only.

Q: Minimum hardware?
A: Minimum 16GB VRAM (e.g., RTX 3080) with speed/resolution limits. Recommended 24GB+ (RTX 4090) for smooth 1080p. CPU: 8+ cores suggested.

Q: Poor Chinese prompt results?
A: Translate to English or mix EN for key names/scenes. Future versions may add multilingual encoders; Gemma fine-tuning can help.

Q: Output formats?
A: Default MP4, H.264 video, AAC audio. Adjust ffmpeg in inference script for GIF/WebM. SR supports 1152×1920 or 1472×2560.

Q: Single GPU for full pipeline?
A: Yes with ≥24GB VRAM. Use --low-memory for sharded loading if tight. Full model ~18–20GB VRAM per shot complexity.

Q: Commercial use?
A: Academic/non-commercial license; contact JD Future Academy for commercial authorization. Personal non-commercial use is allowed.

Q: What Director Agent commands work?
A: Reorder shots, change appearance/background, camera motion, add/remove SFX—e.g., "change outfit to red in shot 2," "insert close-up after shot 5." Emotional nuance may need multiple iterations.

9. Project Links

• Project site: https://echo-team-joy-future-academy-jd.github.io/Echo-LongVideo-Page/
• GitHub: https://github.com/jd-opensource/JoyAI-Echo
• Hugging Face weights: https://huggingface.co/jd-echo