YuE2: In-Depth Evaluation of the Symbolic Planning-Based Music Generation Model Open-Sourced by Hong Kong University of Science and Technology and the M-A-P Team

1. What is YuE2

YuE2 is an open-source music generation model jointly developed by the Hong Kong University of Science and Technology and the M-A-P team. Its core design concept introduces an explicit symbolic musical score planning layer between text/audio conditions and the final waveform, representing melodies and chords using the ABC notation format. The model employs a hybrid Transformer architecture with approximately 3.6 billion parameters, capable of generating editable musical scores based on lyrics and style prompts, which are then rendered into full 48kHz songs. This transforms music creation from an uncontrollable black-box process into a white-box, controllable composition workflow. In the WildSongBench benchmark test, YuE2 achieved an average SongBench score of 6.9632, surpassing mainstream commercial products such as Suno v5, and setting a new performance benchmark for open-source music generation.

Image source: Official article
Image source: official article

Technical Positioning and Domain: YuE2 represents a cutting-edge exploration in the field of AI-generated content, specifically in audio and music generation. It differs from traditional end-to-end audio synthesis approaches by introducing an editable symbolic musical score as an intermediate representation, thereby enabling a paradigm shift from blind, lottery-style creation to an explicit, intervenable creative process. The model lies at the intersection of generative AI, music information retrieval, and computational auditory perception, with a particular focus on structured music generation and controllable creation.

Development Background: This model was developed jointly by the Hong Kong University of Science and Technology and the M-A-P team. The M-A-P team has previously accumulated extensive expertise in pre-training and generation within the music audio domain, having launched several well-known music representation learning models. The team has long been committed to addressing pain points in music generation, such as insufficient controllability of the creative process, complex cover song workflows, and the difficulty for users to make fine-grained modifications to generated results. YuE2 is a product of systematic framework reconstruction based on the team's prior technical accumulations.

Core Value: YuE2 addresses the core issues of lack of interpretability and controllability in the music generation process. By explicitly presenting melodies and chords in ABC notation format, users—whether human or Agent entities—can read, modify, and reuse the work before rendering the audio. Additionally, the model supports three modes: original composition, zero-shot cover singing, and Agent-based conversational editing, all sharing the same set of weights, significantly simplifying the technical workflow for music creation and re-creation.

Technical Features: The model uses a single hybrid Transformer backbone, unifying autoregressive symbolic planning with flow-matching acoustic rendering within the same checkpoint. Its multi-stage inference API breaks down the generation process into four stages: score planning, semantic generation, acoustic synthesis, and decoding. All intermediate outputs are saved to disk, providing open technical interfaces for intervention and reproduction in the creative process.

2. Key Features

• Symbolic Composition Planning: After users input lyrics and style prompts, YuE2 first generates an editable ABC score containing melody and chords, serving as an explicit intermediate planning layer. It then renders this score into a complete 48kHz song with vocals and accompaniment. This design breaks down the creative process into two stages: planning and execution. Users can review or even modify the score before proceeding to acoustic rendering, achieving full control across the entire pipeline from inspiration input to final composition output.

• Zero-shot Cover Version Generation: After users transcribe a source recording into a score using the配套的 SheetSage2 tool, they can input new lyrics or specify a target style, and the model will re-perform the song in a completely new interpretation. No fine-tuning is required for specific songs during the cover process. The system defaults to melody-only mode, automatically adapting the accompaniment to the new style, preserving the source song's recognizability while offering ample room for re-creation.

• Agent-based Conversational Editing: YuE2 treats the score as a readable and writable interface for intelligent agents. Through the Agent framework that supports SKILL.md, users can modify the score using natural language conversations, covering dimensions such as harmonic progression adjustment, melody optimization, tempo changes, and form restructuring. After making changes, users can trigger re-rendering with a single click, achieving a closed-loop conversational music creation process.

• Audio-to-Score Transcription (SheetSage2): This tool transcribes any audio recording into a viewable and editable ABC score, serving as the entry point for cover versions and editing workflows. It establishes a conversion pathway between the audio world and the symbolic score world, providing an engineered route for reusing existing music assets.

• Music Understanding (MERT2): Offers full-track level music representation capabilities, providing high-quality semantic support for downstream tasks such as transcription. This representation model achieved the best results in 14 out of 15 evaluations on the MARBLE benchmark, demonstrating its technical depth in music understanding and description tasks.

• Stage-wise Intervenable Inference API: The inference process is explicitly divided into four stages: plan(), generate_semantic(), synthesize(), and decode(). The score, semantic tokens, acoustic latent representations, and generation parameters are all saved to disk. This design allows users to intervene at any stage of the process, such as exporting the score for editing and re-rendering, or reusing existing planning results with a different decoder, balancing creative flexibility, engineering debugging convenience, and result reproducibility.

3. How to Use

1. Environment Requirements: To use YuE2, you need a Linux operating system, Python 3.12 runtime, and an NVIDIA GPU that supports BF16 precision calculations. Model weights will be automatically downloaded from the Hugging Face repository during the first run. It is recommended to confirm in advance that your network has stable access to the Hugging Face domain. According to the official documentation, a single GPU with 24GB of VRAM is sufficient for offline operation.

2. Installation and Deployment: After cloning the GitHub repository, create a Python virtual environment and run pip install . to complete the installation. Dependencies include deep learning frameworks, audio processing libraries, and tokenization components. It is recommended to use a package manager with caching during installation to accelerate dependency resolution and downloads.

3. Quickly Generate Your First Song: Run the command python examples/generate.py --output outputs/first-song to generate a complete song from built-in example lyrics and style parameters. The output directory will contain audio files in FLAC format, corresponding ABC music score files, and all intermediate outputs (semantic tokens, acoustic latents), making it easy for users to inspect the data flow during the generation process.

4. Python API Usage: Load the pre-trained pipeline using YuE2Pipeline.from_pretrained("m-a-p/YuE2-3B"), then input the style description and lyrics text to trigger inference. The cot parameter controls the planning mode: "full" generates an editable full music score (default behavior), "melody" plans only the melodic direction, and "off" directly performs acoustic generation. These three modes correspond to different creative or editing scenarios.

5. Cover a Song: First, use SheetSage2 to transcribe the original recording into melody ABC format. Then, when calling the generation function, specify cot="melody" and input new lyrics or target style prompts. The model will re-perform the song in melody-only mode, while the accompaniment is automatically generated by the model based on the new style.

6. Editing Existing Works: Use pipe.plan() to export the music score of an existing song. Modify the ABC file manually or with the help of an Agent, then re-run the rendering process with the parameters --abc-file edited.abc --cot full to generate the updated version of the song. This workflow is suitable for refined editing scenarios such as structural adjustments, harmonic replacements, and tempo changes.

7. Agent Skill Usage: Import the skills/yue2-music/ directory from the repository into an Agent application that supports the SKILL.md specification. This allows you to naturally issue commands for songwriting, transcribing, editing, and version comparison within a conversation, and the Agent will autonomously orchestrate tool calls to complete the creative task.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For development teams and researchers with certain engineering capabilities who prioritize controllability of generation results and data privacy, YuE2 stands out due to its white-box sheet music mechanism, fully open-source ecosystem, and local deployment capabilities. It is particularly advantageous for projects requiring deep customization of the music generation process, especially in the development of secondary music creation tools, AI music education products, or studios needing to generate editable music assets in bulk.

For individual users or content creators who prioritize quick output and lack experience in configuring deep learning infrastructure, commercial services like Suno offer a more convenient solution with zero deployment cost, mature lyric clarity, and cloud computing resources for immediate creation. Enterprises prioritizing commercial copyright protection should carefully evaluate the differences between YuE2's Apache 2.0 license, MusicGen's non-commercial restrictions, and AIVA's commercial licensing system, and make choices based on actual business needs.

6. Editor's Summary

YuE2 introduces a symbolic planning intermediate layer in the architectural design of music generation models, inserting explicit ABC sheet music between text-based conditioning and waveform output. This design choice is not a localized optimization but rather a structural challenge to the black-box end-to-end generation paradigm—sheet music serves both as an interface that can be read, modified, and understood by humans and agents, and as a structural scaffold that constrains acoustic generation. This transforms the creative process from a style-based generation into a guided, verifiable, and replayable engineering workflow. According to the evaluation results from WildSongBench, this architectural adjustment has not compromised generation quality, and in fact, it has surpassed several closed-source commercial products in average SongBench scores, demonstrating that symbolic representation constraints and high-quality acoustic rendering are not in zero-sum conflict.

From a practical value perspective, YuE2 unifies three production pathways—creation, cover singing, and editing—under the same weight. Combined with the intermediate output storage mechanism of its staged inference API, it significantly reduces the cost of reusing and iterating on music assets. The open-source Apache 2.0 license, along with the full-stack tools (SheetSage2, MERT2, WildSongBench) provided, enables the model to support a complete transition from academic research to commercial product development. It is important to note, however, that there are still subtle differences between the model's lyrical pronunciation clarity, text-music semantic alignment accuracy, and commercial products. Additionally, the Agent editing ecosystem's reliance on third-party frameworks introduces a variable in its engineering implementation.

The core target audience for this model includes three groups: academic institutions and algorithm engineers engaged in music generation research, startup teams developing music creation tools or content products based on the open-source license, and enterprise technology departments aiming to build customizable music generation pipelines within local data environments. As the integration of symbolic representation with neural generation continues to be validated, the planning-rendering separation paradigm established by YuE2 is likely to become one of the important directions in the evolution of music generation technology. Continued attention is warranted for its future iterations in dimensions such as multi-track interaction, fine-grained editing, and control over long structural elements.

7. Application Scenarios

• Personal Music Creation: Ordinary users without professional knowledge of composition or arrangement can input lyrics and style descriptions to generate a complete song featuring vocals and accompaniment. Since the sheet music is explicitly presented in ABC format, users can also adjust the melodic direction, chord progressions, and structural segments after generation, gradually shaping vague musical ideas into complete works.

• Cover Versions and Remixing: Creators use SheetSage2 to transcribe existing song recordings into sheet music, and then adapt the song into new styles such as jazz, rock, or electronic using the cot="melody" mode, or replace the lyrics to create multilingual versions. This process does not require fine-tuning the model for cover tasks, significantly shortening the remix production cycle and meeting the fast-paced production needs of short video soundtracks, fan covers, and ad adaptations.

• Demo Music for Film, Games, and Advertising: Production teams can quickly generate multiple candidate versions of background music, including vocal versions and pure instrumental versions, to complete style previews at a low cost in the early stages of a project. Once the candidate options are finalized, they can decide whether to hire a human music producer for refinement, effectively reducing the communication and trial-and-error costs in the early stages and improving the iteration efficiency of the music component in the production process.

• Music Education and Harmony Analysis: Teachers use YuE2 to generate example compositions with specified chord progressions, musical forms, and melodic characteristics. Students can compare the ABC sheet music to intuitively learn composition techniques and harmonic rules. Because the sheet music and audio are aligned one-to-one, students can also modify the sheet music and re-render the audio for listening, experiencing in real time how harmonic changes affect the auditory experience, thus forming a complete teaching loop that is audible, visible, and editable.

8. FAQ

Q: How long does it take for YuE2 to generate a complete song?
A: The actual inference time depends on the GPU model, memory bandwidth, and the duration of the generated audio. The official documentation does not provide precise real-time rate data, but generating a 30- to 60-second complete song (including planning and acoustic rendering) typically takes several seconds to a few minutes on an NVIDIA RTX 4090-level GPU. It is recommended that users first run the full process using the example script and output timing information to establish a quantitative understanding of their hardware's throughput capacity.

Q: Can YuE2 run on a Windows system?
A: The official requirements specify a Linux system environment, but most users can successfully run it through the Windows Subsystem for Linux 2 (WSL2). Note that GPU drivers must be installed on the Windows host, and WSL2 must use a version of the Linux CUDA toolkit that matches the driver. Additionally, Python 3.12 and PyTorch have minimal compatibility issues under WSL2, but some audio processing dependencies (e.g., soundfile) may require additional compilation and installation.

Q: What does a PER of 9.79% mean?
A: The pronunciation error rate (PER) measures the degree to which lyrics in the generated song can be accurately recognized. Lower values indicate clearer articulation of the lyrics. A PER of 9.79% means that approximately one out of every ten syllables has a pronunciation deviation. In scenarios where lyric clarity is extremely important (e.g., lyric subtitle synchronization), this may cause minor issues. If users require high lyric clarity, they can try switching planning modes (via the cot parameter) or adjusting the length of the lyric text to observe differences in the output.

Q: Does the cover song feature involve copyright issues?
A: Copyright issues related to cover songs are determined by the usage context and local laws. YuE2 itself provides a technical capability and does not guarantee the legality of the cover content. In most jurisdictions, non-commercial cover song adaptations enjoy some degree of fair use. However, commercial release of cover songs still requires obtaining mechanical licenses or synchronization rights for the original composition. It is recommended that users consult with legal professionals and obtain the necessary permissions before commercially publishing cover songs using this feature.

Q: What additional environment setup is required for the Agent-based conversational editing?
A: You need to import the skills/yue2-music/skill package from the repository into an Agent framework that supports the SKILL.md specification. Currently supported platforms include some open-source Agent development frameworks and commercial Agent platforms. The machine where the Agent runs must be able to access the runtime environment where YuE2 is hosted (either local or remote server). If the Agent and YuE2 are not on the same machine, you will also need to configure API forwarding or a remote calling channel.

Q: Can the specific length of the audio be controlled during the generation process?
A: The generation duration in YuE2 is primarily determined by the number of lyrics and the musical structure of the score. Users can directly influence the final audio length by editing the number of sections, repeat signs, and measures in the ABC notation. After modifying the score, re-rendering will produce output versions of different lengths. This is also a structural advantage of the model compared to pure black-box generation tools in terms of length control.

9. Project Links

• Product Homepage: https://map-yue2.github.io/
• GitHub Repository: https://github.com/multimodal-art-projection/YuE