InstructAV2AV – An Open-Source Audio-Visual Joint Editing Model Developed by BAAI and Peking University

1. What is InstructAV2AV

InstructAV2AV is an open-source audio-visual joint editing model jointly developed by the Beijing Academy of Artificial Intelligence (BAAI) and Peking University. With just a single natural language instruction, users can simultaneously edit the video frames and corresponding audio during the end-to-end generation process, without the need for manual masking or step-by-step processing. The model supports operations such as modifying dialogue, replacing characters, inserting or deleting objects, while precisely preserving background and ambient sounds and maintaining temporal synchronization between the edited audio and video. This provides a new, efficient paradigm for post-production video editing and creative content generation.

Image source: Official article
Image source: official article

Technical Positioning and Domain: InstructAV2AV belongs to the field of multi-modal generation and editing, specifically focusing on audio-visual joint editing. Unlike traditional video editing models (which handle only visuals) or audio editing models (which handle only sound), this model enables synchronized editing of video frames and corresponding audio, while maintaining background consistency. It is the first open-source model that supports instruction-driven audio-visual joint editing, filling a gap in open-source solutions in this area and advancing multi-modal editing technology from single-modal approaches toward joint editing.

Development Background: This model was jointly developed by the Beijing Academy of Artificial Intelligence (BAAI) and the Visual and Auditory Information Processing Laboratory at Peking University. BAAI has deep expertise in foundational models and multi-modal research, having previously launched large models such as the WuDao series. Peking University has strong academic foundations in computer vision, audio processing, and cross-modal learning. The collaboration aims to address core pain points in current audio-visual editing, such as the separation of audio and video processing, the tediousness of manual masking, and the lack of synchronization between edited audio and video. By integrating joint modeling of video and audio, the model reduces the barrier to entry for content creation and improves the naturalness and efficiency of editing.

Core Value: The core value of InstructAV2AV lies in liberating audio-visual editing from cumbersome manual workflows. Users no longer need to separately handle video and audio tracks, manually create masks, or adjust frame by frame. Instead, they can modify both the visuals and audio with just one natural language instruction. The model automatically maintains the stability of background, ambient sounds, and irrelevant objects, while ensuring that the edited audio and video remain naturally synchronized. This significantly reduces the technical barriers of audio-visual editing, enhances creation efficiency and experience, and is especially suitable for scenarios requiring frequent modifications, such as film post-production and short video creation.

Technical Features: The model is based on a modified Ovi symmetric dual-stream diffusion Transformer architecture, incorporating three core technologies. First, Source Latent Concatenationconcatenates the latent variables of the source video and source audio with noise across channels, allowing the source content to act as a structural condition that constrains the model to preserve background and ambient sounds, avoiding the issue of "changing one target and redrawing the entire world." Second, SIGA Gated Attention (Source-Instruction Gated Attention) uses a learnable soft gate to determine the modification degree at each position, achieving a dynamic balance between content preservation and instruction execution. Third, the Dual-Stream Interaction Architectureallows the video and audio branches to exchange features through bidirectional cross-modal attention after self-attention within each branch, ensuring that visual motion and audio events constrain each other and remain synchronized. In addition, a two-stage training strategy (first training single-modal editing capabilities by disabling cross-modal attention, then joint fine-tuning) and an automated data pipeline (constructing the InsAVE-80K dataset) further ensure the model's performance.

2. Key Features

• Identity-preserving Voice Modification: Modify the speech content and dialogue of a character while preserving their identity, appearance, and voice. This feature utilizes the SIGA gated attention mechanism to precisely identify and modify voice-related feature regions, ensuring that the character's visual appearance remains unaffected. It is suitable for post-production voice dubbing corrections and dialogue adjustments in film and television, significantly reducing the need for reshoots.

• Audio-Visual Instance Replacement: Fully replace a specified character in the video, including their voice and dialogue. The model retains background and ambient sounds through source latent variable concatenation, while the dual-stream interaction architecture ensures precise temporal synchronization between the new character and their new voice. This is ideal for character replacement or virtual streamer image changes.

• Instance Insertion: Add new objects (such as characters or items) into the video and automatically generate synchronized audio. The model generates visual content based on the described attributes of the object and simultaneously synthesizes corresponding audio (such as speech or sound effects), achieving a seamless audio-visual insertion effect and enhancing the flexibility of content creation.

• Instance Removal: Remove a specified object and its corresponding audio from the video while maintaining the continuity of the background and ambient sounds. The model identifies and removes the target region's visual and auditory information using attention mechanisms, then fills in the background and smooths the audio. This is useful for eliminating unwanted characters or objects from the video.

• Attribute Combination Editing: Allows for independent modification and flexible combination of a character's appearance, speech content, and voice. Users can change dialogue without altering the voice or switch voice without changing the appearance, offering fine-grained control and meeting complex creative demands.

• Background and Ambient Sound Preservation: During the editing process, the model automatically preserves unmodified background visuals and ambient sounds using source latent variable concatenation, preventing the entire scene from being redrawn due to localized edits. This feature ensures the naturalness and coherence of the edited content and is a core advantage that differentiates this model from other editing solutions.

• Audio-Visual Temporal Synchronization: Based on the bidirectional cross-modal attention mechanism in the dual-stream interaction architecture, the model enforces mutual constraints between video motion and audio events during editing, ensuring precise temporal alignment between lip movements and speech, as well as actions and sound effects. This eliminates common synchronization issues.

3. How to Use

1. Environment Requirements: Ensure the system has Python 3.8 or higher, a CUDA 11.7 or higher environment, and it is recommended to use an NVIDIA GPU (with at least 16GB VRAM, such as RTX 3090/4090 or A100). The operating system is recommended to be Linux (Ubuntu 20.04+); for Windows, confirm CUDA support. Reserve at least 20GB of disk space for model weights and datasets.

2. Clone the Code: Execute git clone https://github.com/suimuc/InstructAV2AV.git to clone the repository to your local machine. Use git lfs to track large files; it is recommended to install Git LFS first.

3. Set Up the Environment: Enter the project directory and create and activate a conda environment: conda create -n instructav2av python=3.8 -y && conda activate instructav2av. Then install the dependencies: pip install -r requirements.txt. If a specific PyTorch version is required, install it from the PyTorch website according to your CUDA version.

4. Download Weights: Download the weights from the Hugging Face model library: https://huggingface.co/suimu/InstructAV2AV. You can use git lfs clone https://huggingface.co/suimu/InstructAV2AV or download directly through your browser and place the files in the checkpoints directory. Ensure the weight files are complete, approximately 5GB in size.

5. Prepare Materials: Prepare a video file with audio as input. It is recommended that the video duration does not exceed 30 seconds and the resolution does not exceed 1024×1024, with formats such as MP4 or AVI. The audio must be synchronized with the video, and the model will automatically extract the audio stream. The content of the material should be clear, avoiding excessive shaking or complex scenes.

6. Run Inference: Run the inference script via the command line. Example command: python inference.py --video input.mp4 --instruction "Change the dialogue of the person in the video to 'Hello, world'". You can add the --audio parameter to specify an independent audio file, or use --cfg_scale to adjust the instruction strength. Inference time depends on the video length and GPU performance; a 30-second video typically takes about 2-3 minutes on an A100.

7. Save Results: After inference is complete, the output directory (default is outputs/) will generate the edited video file, including a synchronized audio track. Check if the results meet your expectations. If not, adjust the instruction or parameters and rerun. It is recommended to keep a backup of the original materials.

Notes: The model processes longer videos more slowly and consumes more VRAM, so it is recommended to edit in segments; instructions should be concise and clear, avoiding vague descriptions; the first run requires downloading dependent models (such as CLIP, VQVAE, etc.), so ensure a stable internet connection.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For scenarios requiring simultaneous editing of video frames and corresponding audio, such as film post-production and short video creation, InstructAV2AV is currently the only open-source end-to-end solution. Its capabilities in background preservation and audio-visual synchronization are significantly superior to step-by-step processing approaches. If only video frame editing is needed (without audio involvement), InstructVideo is more lightweight and has a mature community, making it suitable for pure visual editing tasks. For audio generation or editing only (e.g., music production, sound effect design), AudioLDM 2 offers greater advantages in audio quality and flexibility. For generating audio-visual content from scratch (rather than editing), Ovi provides a complete generation pipeline, suitable for creative content prototyping. Overall, InstructAV2AV fills the open-source gap in audio-visual joint editing, but still requires improvements in handling complex scenarios and Chinese language support.

6. Editor's Summary

InstructAV2AV demonstrates a clear technical vision in its innovation: it addresses the challenge of background preservation through source latent variable concatenation, achieves a dynamic balance between instructions and source content using the SIGA gated attention mechanism, and ensures audio-visual synchronization based on a dual-stream interactive architecture. These designs are not merely a stack of modules, but rather systematic responses to the three core questions in joint audio-visual editing: "what to modify, what to retain, and how to synchronize." Compared to existing single-modal editing models, InstructAV2AV represents a paradigm shift from "separate editing" to "joint editing," and its end-to-end nature significantly reduces the complexity of the editing process.

In terms of practical value, this model directly lowers the technical barriers to audio-visual editing. Post-production professionals no longer need to master complex masking and track operations, short video creators can quickly iterate through content, and virtual human operation teams can flexibly adjust the appearance and voice of digital humans. The model's open-source Apache 2.0 license also makes it suitable for commercial integration, positioning it as a potential foundational component of multi-modal editing infrastructure.

The primary target users include: post-production professionals (for dialogue correction and character replacement), short video and social media creators (for rapidly generating multiple versions of content), virtual human operation teams (for adjusting digital human attributes), and multi-modal AI researchers (as a baseline model or platform for improvement). For professional cinematic-grade applications, the current version still requires manual post-processing in complex scenarios, but it has already significantly reduced repetitive labor.

In terms of future development potential, the InstructAV2AV architecture offers good scalability: it can incorporate strategies for processing longer videos, support more language instructions, and improve editing resolution and detail quality. As community contributions grow, the model is expected to make breakthroughs in real-time editing and multi-object interactive editing. Overall, InstructAV2AV marks an important open-source milestone in the field of joint audio-visual editing, and its technical approach is worth close attention and follow-up.

7. Application Scenarios

• Post-production in Film and Television: Film and television post-production professionals use InstructAV2AV to replace actors' dialogue and synchronize lip movements without the need for reshoots. For example, modifying the lines in a conversation, the model automatically maintains the actor's appearance and background while generating a new voice that matches the lip movements, significantly reducing the cost of reshoots and improving post-production efficiency.

• Short Video Production: Short video creators can modify the characters and voices in the material with a single instruction to quickly generate different versions of the video. For example, replacing pedestrians in the video with specific characters and automatically generating corresponding dialogue voices, which is suitable for template-based content production and multi-platform distribution.

• Virtual Human Operations: Virtual human operation teams use the model to adjust the appearance, voice, and speaking content of digital humans, enabling personalized interactions. For example, changing the hairstyle, clothing, and voice of a virtual host according to user needs while maintaining the continuity of actions and background, enhancing the diversity and appeal of virtual human content.

• Advertising Creativity: Advertising professionals can generate multiple versions of promotional videos with different characters and dialogue for A/B testing. For example, replacing the spokesperson in an advertisement with different personas and synchronously modifying the ad copy, while the model keeps the background and product presentation unchanged, quickly producing multiple creative versions and shortening the production cycle.

• Interactive Content Production: Game and education professionals can dynamically generate multimedia materials with consistent audio and visual elements. For example, inserting a character who explains new knowledge into an educational video, the model automatically generates the corresponding voice and ensures synchronization with the visuals, suitable for the rapid production of personalized educational content.

8. FAQ

Q: What hardware is required to run InstructAV2AV?
A: It is recommended to use an NVIDIA GPU with at least 16GB of VRAM (such as RTX 3090/4090 or A100) and CUDA version 11.7 or higher. If using a consumer-grade GPU (such as RTX 3060 12GB), you may need to reduce the video resolution or shorten the duration, which will significantly slow down the inference speed. CPU inference is currently not supported.

Q: Does the model support Chinese instructions and content editing?
A: The model is primarily trained on English data, and its semantic understanding of Chinese instructions is limited. For Chinese content editing (such as modifying Chinese dialogue), the results may not be as ideal as with English, and there may be semantic deviations or lip-sync issues. The community is exploring Chinese adaptation solutions, but for now, it is recommended to prioritize using English instructions.

Q: What is the quality of the edited video? Can it maintain the original resolution?
A: The model maintains the input video's resolution during editing, but slight blurriness or artifacts may appear in the edited regions, especially in areas with fast motion or complex textures. It is recommended that the input video resolution does not exceed 1024×1024. The overall quality after editing is generally acceptable, but professional applications may require post-processing such as noise reduction or sharpening.

Q: How to handle long videos longer than 30 seconds?
A: The model's VRAM usage and inference time increase significantly for long videos. It is recommended to split long videos into multiple short segments (each no longer than 30 seconds), edit them separately, and then concatenate them. Note that audio-visual discontinuities may occur at the splice points, requiring manual adjustment of the transitions. Future versions may introduce optimizations for long video processing.

Q: Does the model support real-time editing?
A: Real-time editing is not currently supported. Inference speed depends on video length and GPU performance. A 30-second video takes approximately 2–3 minutes on an A100 GPU and is even slower on consumer-grade GPUs. The model is primarily intended for offline editing scenarios, and real-time applications would require further lightweight optimization.

Q: How can I obtain the model weights and code?
A: The code is hosted on GitHub (github.com/suimuc/InstructAV2AV), and the model weights are published on Hugging Face (huggingface.co/suimu/InstructAV2AV), both of which can be downloaded directly. The weights are approximately 5GB in size, and it is recommended to use Git LFS or a tool for resuming interrupted downloads.

Q: Can the model be used for commercial projects?
A: Yes. The model is licensed under Apache 2.0, allowing for commercial use, modification, and redistribution, provided that the copyright notice is retained. Before use, please confirm the license compatibility of the dependent components (such as CLIP, VQVAE, etc.).

9. Project Links

• Project Website: https://hjzheng.net/projects/InstructAV2AV/
• GitHub Repository: https://github.com/suimuc/InstructAV2AV
• HuggingFace Model Library: https://huggingface.co/suimu/InstructAV2AV
• arXiv Technical Paper: https://arxiv.org/pdf/2605.18467