JoyAI-Video-Edit – JD.com's Open-Source Real-Time Streaming Video Editing Model

1. What is JoyAI-Video-Edit

JoyAI-Video-Edit is a real-time streaming video editing model developed and open-sourced by JD.com. It is based on a self-regressive diffusion architecture with 16B parameters, achieving end-to-end inference at 30FPS in 720P resolution. It supports stable streaming editing for videos of arbitrary duration. Users can modify characters, scenes, styles, and objects in real-time during video playback through natural language instructions, without having to wait for the entire video to be generated. This model outperforms all streaming editing methods in the OpenVE-Bench evaluation, leading comprehensively in all metrics, and can also provide a large-scale data synthesis pathway for embodied intelligence.

Image source: Official article
Image source: official article

Technical positioning and domain: JoyAI-Video-Edit belongs to the video generation and editing domain, focusing specifically on real-time streaming video editing, which differs from traditional offline video editing methods. Traditional methods require the complete generation or editing of a video before output, whereas JoyAI-Video-Edit enables an "edit while playing" interactive mode, processing video frames instantly as they arrive, without needing to know the full sequence in advance. This technical approach advances video editing from offline batch processing to real-time interaction, significantly improving creation efficiency and user experience.

Development background: This model was primarily developed by the JD Explore Research Institute, leveraging JD.com's technical expertise in computer vision and multimodal large models. The team proposed a series of innovative components based on the latest advancements in self-regressive diffusion models, addressing core challenges in video editing such as real-time performance, duration limitations, and temporal drift. These components include an MLLM condition encoder, causal video VAE, 16B parameter MMDiT backbone, self-regressive distribution matching distillation, and bounded KV caching. The project has been open-sourced, aiming to promote the popularization and application of video editing technology.

Core value: JoyAI-Video-Edit addresses three major pain points in video editing: first, long waiting times, as traditional models require several minutes or even longer to generate editing results; second, duration limitations, as most models only support editing of second- or minute-level clips; third, temporal instability, where long video editing is prone to drift and flickering. By leveraging a streaming architecture, bounded KV caching, and distribution matching distillation, this model achieves stable output for arbitrary durations, improving video editing efficiency by an order of magnitude. It provides real-time feedback capabilities for scenarios such as short video creation, live streaming special effects, and film previsualization.

2. Key Features

• Real-time Streaming Editing: Edit processing occurs instantly as video frames arrive, without waiting for the complete video sequence. Users can issue instructions at any time during playback, and the model responds immediately, modifying the current and subsequent frames to deliver a true "edit-as-you-play" interactive experience. This feature completely transforms the traditional offline workflow of video editing.

• Open Instruction Control: Based on an MLLM conditional encoder, the model can understand natural language instructions and supports various editing operations, such as global style transfer (e.g., turning daytime into nighttime), local object addition/removal/modification (e.g., adding or removing a person), background replacement, and action adjustment. Users don't need to write code or use complex tools; they can drive the editing process using everyday language.

• Arbitrary Duration Processing: Breaking away from the limitations of traditional video editing models that only support second- or minute-level clips, JoyAI-Video-Edit can perform stable editing continuously as the video plays, without experiencing temporal drift or quality degradation. Thanks to bounded KV caching and long temporal domain optimization, the model maintains consistent editing performance even on videos lasting tens of minutes.

• 720P High Throughput Deployment: The end-to-end pipeline achieves an inference speed of 30.19 FPS at 720×1280 resolution, meeting the requirements for real-time video processing. This performance enables the model to be directly applied to live streaming, real-time monitoring, and other scenarios without the need to reduce resolution or frame rate.

• Multi-dimensional Editing Capabilities: Covers tasks such as character outfit changes, scene transitions, style conversions, object replacements, and subtitle editing. For example, users can change a character's clothing from red to blue or transform an indoor scene from a modern style to a retro style, and the model maintains visual consistency across all these tasks.

• Reference Image-Guided Editing: In addition to natural language instructions, the model also supports using a reference image as an editing condition. When a user provides an image of the target style, the model can transfer the entire video or specific parts to that style, achieving more precise visual control. This feature is particularly useful in post-production for films and design previews.

3. How to Use

1. Environment Requirements and Preparation: It is recommended to use a Linux operating system (Ubuntu 22.04 or higher), Python 3.10, CUDA 12.1, and cuDNN 8.9. In terms of hardware, due to the model's parameter size of 16B, it is advised to use a GPU with at least 48GB of VRAM (such as NVIDIA A100 80GB or H100). Consumer-grade GPUs (such as RTX 4090 24GB) may not be able to fully load the model and would require model parallelism or quantization solutions. Create a Conda environment and install dependencies:

   conda create -n joyai python=3.10
   conda activate joyai
   pip install -r requirements.txt
   

2. Download Model Weights: Download the model files from the official Hugging Face repository (jdopensource/JoyAI-Video-Edit). The weight files are approximately 30GB in size, so ensure that there is sufficient disk space available. Place the checkpoints according to the following directory structure:

   JoyAI-Video-Edit/
   ├── ckpts/
   │   ├── mmdit.safetensors
   │   ├── vae.safetensors
   │   └── mllm_encoder.safetensors
   └── ...
   

3. Start Local Service: Navigate to the deploy folder in the project root directory and execute the startup script:

   cd deploy
   bash run_server.sh
   

   The script will automatically load the model and launch an inference service based on FastAPI, which listens by default on 0.0.0.0:8080. The startup logs will display the model loading progress and readiness status.

4. Access the Interactive Interface: Open http://localhost:8080 in your browser to access the Web UI of JoyAI-Video-Edit. The interface includes a video upload area, a webcam input option, an instruction input box, and an editing preview window. Users can upload local video files (supporting common formats such as MP4, AVI, and MOV) or directly call the webcam for real-time streaming.

5. Real-time Editing Operations: While playing a video or previewing the webcam feed, describe the desired editing effect in natural language within the instruction input box (e.g., "Replace the background with a beach"), then click the "Apply" button. The model will immediately process the current frame and continuously output the edited video stream. The editing results can be previewed in real-time or saved to a local file with one click.

Notes and Best Practices:

• The first time the model is loaded, it may take a long time (approximately 5–10 minutes), so it is recommended to keep the service running continuously.
• For long video editing, ensure a stable network bandwidth and GPU temperature to avoid inference interruptions.
• If using a webcam stream, confirm that the webcam supports 720P resolution and that the necessary drivers (such as v4l2) are installed on the system.
• Editing instructions should be as clear as possible, avoiding vague expressions; complex edits can be broken down into multiple instructions and executed step by step.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations:

For users seeking real-time interaction, high-quality editing, and with access to powerful GPU resources (such as professional video studios or live streaming platforms), JoyAI-Video-Edit is the optimal choice. The editing quality and temporal stability offered by its 16B parameters are difficult to match with other streaming methods, making it particularly suitable for scenarios requiring long-term stable output (such as film pre-visualization or embodied data synthesis).

If hardware resources are limited (such as a single RTX 4090 GPU), lightweight streaming models like SANA-Streaming can be considered. Although the editing quality and resolution are lower, they can operate at higher frame rates, making them suitable for scenarios with extremely high real-time requirements but less stringent quality demands, such as simple special effects.

For offline HD editing tasks (such as movie-level post-production), traditional models like Stable Video Diffusion still have advantages in specific style transfer and frame-level quality. However, they require full generation time and are not suitable for real-time interaction.

6. Editor's Summary

JoyAI-Video-Edit has achieved a significant technological breakthrough in the field of video editing, advancing real-time streaming editing from concept to a deployable product-level stage. Its core innovation lies in: unifying open semantic control with high-fidelity editing for the first time within a streaming framework, through the use of a self-recursive diffusion Transformer with 16B parameters combined with an MLLM conditional encoder. The causal video VAE and bounded KV caching mechanism effectively address the issue of temporal drift in long-duration editing, enabling stable output for any duration. Compared to similar products, the model's comprehensive superiority on the OpenVE-Bench validates its advantages in editing quality and temporal consistency, while its inference speed of 30FPS at 720P demonstrates its level of engineering maturity.

In terms of practical value, JoyAI-Video-Edit directly addresses the core pain point in video creation—“waiting for generation”—reducing editing feedback time from the minute level to the frame level. Short video creators can adjust styles and content while playing back the video, live streamers can instantly change backgrounds and effects, and film directors can rapidly experiment with different visual concepts during the editing phase. The efficiency gains in these scenarios are revolutionary. Additionally, the model provides a data synthesis pathway for embodied intelligence—converting manual video operations into robotic arm operation materials—showcasing its potential beyond the entertainment industry.

In terms of target users, this tool is most suitable for professional users and technical teams with access to high-end GPU resources, including video content creators, technical departments of live streaming platforms, post-production film companies, and researchers in embodied intelligence. For individual creators or users with hardware limitations, it is recommended to wait for the community to release a quantized version or lightweight variant.

Looking ahead, with advancements in model distillation techniques and hardware efficiency, the 16B parameter model is expected to run on a broader range of consumer-grade GPUs. JD.com has open-sourced this project under the Apache 2.0 license, which will attract community contributions of more application plugins and optimization strategies, further lowering the barrier to entry. Meanwhile, the technical approach of this model (self-recursive streaming editing + bounded KV caching) may become a new paradigm in the video editing domain, encouraging more teams to follow and conduct further research.

7. Application Scenarios

• Short Video Creation: Creators can replace characters' clothing, adjust scene styles, or add virtual elements in real-time during the video editing process. For example, when producing a fashion video, users simply need to input "Change the top to a white shirt," and the model will instantly modify the video during playback, without the need for re-shooting or post-production rendering, significantly improving content production efficiency.

• Live Streaming Real-Time Effects: During live streaming, hosts can change the background, add virtual items, or alter their own appearance on the fly. For instance, a gaming streamer can replace the live stream background with a game scene in real-time, while a product promotion streamer can switch the product display environment with one click, achieving interactive visual enhancement and keeping viewers engaged.

• Home Decor and Interior Design: Users can replace furniture, wall materials, and lighting effects in real-time while browsing room videos. For example, inputting "Replace the sofa with a blue fabric one" or "Change the walls to light gray," the model will immediately update the visuals, helping users intuitively compare different renovation plans and support decision-making.

• Film Post-Production Preview: Directors and editors can quickly experiment with different visual styles and scene replacements during the editing phase. For instance, converting a daytime video to a nighttime style in real-time or replacing buildings in the background, reducing trial-and-error costs and accelerating creative iteration.

• Embodied Intelligence Data Synthesis: Convert human hand operation videos (such as grasping or assembling) into robotic arm operation materials. By editing the appearance of hands and the background in the video, diverse robotic training data can be generated, enabling low-cost expansion of the dataset scale and improving the generalization ability of robots.

8. FAQ

Q: What hardware configuration is required to run JoyAI-Video-Edit?
A: It is recommended to use a GPU with at least 48GB of VRAM (such as NVIDIA A100 80GB, H100, or domestic Ascend 910B). Due to the model's 16B parameters, consumer-grade GPUs (such as RTX 4090 24GB) may not be able to fully load the model and would require model parallelism or a 4-bit quantization approach. However, this will significantly reduce inference speed. CPU memory should be 64GB or more, and at least 100GB of free disk space is required (including weights and temporary files).

Q: Does the model support Chinese instructions?
A: The MLLM condition encoder is based on a multimodal large language model and has strong understanding capabilities for Chinese instructions. Users can describe editing requirements in Chinese (e.g., "Replace the background with a beach"), and the model can accurately parse and execute them. However, for complex or obscure descriptions, it is recommended to use concise and clear Chinese expressions to achieve the best results.

Q: Can the edited results be saved as video files?
A: Yes. In the Web UI interface, the edited video stream can be previewed in real time, and supports one-click export as an MP4 file. The original video's resolution and frame rate are preserved during export, and users can also choose to reduce the bitrate to minimize file size. For camera streams, edited results can also be recorded and saved.

Q: Does JoyAI-Video-Edit support batch processing of multiple videos?
A: The current version is primarily designed for real-time interactive scenarios. The Web UI supports real-time editing of a single video or camera stream. For batch offline processing, users can call the REST API provided by the deployment service and write scripts to submit tasks in a loop. However, when processing a large number of videos consecutively, GPU VRAM may be insufficient. It is recommended to introduce appropriate intervals or use a VRAM cleanup mechanism.

Q: What advantages does JoyAI-Video-Edit have compared to other video editing tools (such as Runway, Pika)?
A: The core advantage lies in its real-time performance. Tools like Runway and Pika require the full video to be submitted first, and it can take several minutes or even longer to obtain the editing results, typically only supporting short clips (<10 seconds). JoyAI-Video-Edit responds instantly during video playback and supports videos of any length. In addition, its editing quality leads in the OpenVE-Bench evaluation. However, these tools have unique advantages in certain stylized generation tasks and require lower hardware specifications.

9. Project Links

• GitHub Repository: https://github.com/jd-opensource/JoyAI-Video-Edit
• Hugging Face Model Library: https://huggingface.co/jdopensource/JoyAI-Video-Edit
• arXiv Technical Paper: https://arxiv.org/abs/2608.03974