MuScriptor – An Open-Source Multi-Instrument Music Transcription Model Jointly Released by Kyutai and Mirelo

1. What is MuScriptor

MuScriptor is an open-source multi-instrument music transcription model jointly developed by Kyutai and Mirelo. It can automatically transcribe audio of music from various genres in the real world into standard MIDI format. As the first model of its kind trained on a large-scale real-world dataset covering 170,000 songs, it employs a pure decoder Transformer architecture, offering four parameter scales ranging from 60M to 1.4B. It also supports instrument-conditioned control, with the code released under the MIT license.

Technical Positioning and Domain: MuScriptor belongs to the field of audio signal processing and music information retrieval (MIR), focusing on multi-instrument automatic music transcription (AMT). Unlike previous models that relied on synthetic data or limited real recordings, MuScriptor directly addresses real-world music in complex mixed environments, filling a critical technical gap in high-accuracy transcription for real multi-instrument scenarios.

Development Background: This model was jointly developed by Kyutai, a non-profit AI research laboratory based in France, and Mirelo, a music technology company. The motivation for its development stemmed from the insufficient generalization capabilities of existing AMT models on real recordings—most models are trained only on synthetic MIDI data, leading to a sharp increase in error rates when encountering real-world mixes. Kyutai's expertise in generative audio modeling and Mirelo's experience in music signal processing provided the technical foundation for this breakthrough.

Core Value: The core value of MuScriptor lies in its significant improvement in the accuracy of transcribing real-world multi-instrument music. Through fine-tuning on 170,000 real-world music tracks and post-training with reinforcement learning to align with high-quality outputs, it achieves a Multi F1 score of approximately 41.6 in complex mixing scenarios, far surpassing models trained on synthetic data. Additionally, the instrument-conditioned control feature allows users to specify target instruments, greatly enhancing the controllability and consistency of the output.

Technical Features: The model uses a pure decoder Transformer, taking 5-second audio slices as input and predicting MIDI-like token sequences in an autoregressive manner to achieve end-to-end transcription. A three-stage training strategy (synthetic pre-training → real-world fine-tuning → reinforcement learning post-training) ensures that the model has foundational knowledge from large-scale data while adapting to the diversity of real-world audio. The four parameter scales cater to the needs of both edge devices and high-precision applications.

2. Key Features

• Multi-instrument Audio Transcription: Capable of automatically converting real music audio containing 36 types of instruments, such as piano, guitar, drums, and bass, into standard MIDI format. The model is specifically optimized for multi-instrument mixing scenarios, enabling it to simultaneously recognize and separate notes from different instruments, and output structured symbolic representations.

• Support for Multiple Music Genres: Covers various music genres including pop, classical, rock, and heavy metal. The training data includes a rich collection of real recordings, allowing the model to adapt to timbre, rhythm, and mixing characteristics across different styles, maintaining stable transcription quality even in complex acoustic environments.

• Instrument Condition Control: Users can specify the target set of instruments to be transcribed by using prefix embeddings, enabling customized output. This mechanism not only simplifies the MIDI output (e.g., extracting only the piano track) but also leverages classifier-free guidance to ensure consistent instrument allocation across audio segments, avoiding fluctuations in instrument recognition between segments.

• Multi-scale Model Selection: Offers four parameter scales: 60M, 103M, 307M, and 1.4B. Users can flexibly choose based on their computational resources and accuracy requirements. Smaller models are suitable for edge devices or fast inference, while larger models perform better on complex audio, balancing deployment flexibility and performance.

• End-to-end Automatic Processing: The model takes 5-second audio slices as input and automatically completes the entire process of framing, mel-spectrogram feature extraction, note detection, and MIDI generation without human intervention. Users only need to input an audio file to directly obtain standard MIDI output, significantly lowering the usage barrier.

• Reinforcement Learning Post-training Optimization: After pre-training and fine-tuning, the model undergoes post-training using GRPO-style reinforcement learning on 300 high-quality annotated datasets, aligning with high-quality outputs. This step significantly improves the F1 scores at the note onset, frame, and offset levels, reducing missed and false detections, making the transcription results closer to human-annotated standards.

3. How to Use

1. Environment Setup: Clone the GitHub repository (git clone (link to be updated after official release)), and ensure you have Python 3.8 or higher, PyTorch 2.0 or above, and CUDA 11.7 or higher (for GPU inference). CPU inference is also possible but will be significantly slower.

2. Download Model Weights: Obtain model weights from the HuggingFace model library (link to be updated after official release). Three versions are available: 103M, 307M, and 1.4B. The 60M version has not been released yet. The weights are licensed under CC BY-NC 4.0 and can be used free of charge for non-commercial purposes.

3. Load the Model and Configure: Initialize the model using the inference script provided in the repository. If you enable instrument condition control, specify the target instrument list in the configuration file (e.g., "piano,guitar,bass"). Example command:

   python inference.py --model_path ./weights/307M.pt --audio input.wav --instruments piano,guitar
   

4. Input Audio and Generate MIDI: Feed the audio file to be transcribed (supporting common formats such as WAV, MP3, and FLAC) into the inference script. The model automatically segments the audio into 5-second clips for processing and then merges them to output a complete standard MIDI file. The output file can be directly imported into DAWs (e.g., Ableton Live, FL Studio) or notation software (e.g., MuseScore) for further editing.

5. Notes and Best Practices: For audio longer than 5 minutes, it is recommended to process it in segments to avoid memory overflow; the instrument condition control feature can significantly improve consistency across segments and is recommended for use when stable instrument allocation is required; the 1.4B model requires at least 8GB of GPU memory, while the 307M model can run on 4GB memory; if using CPU inference, ensure that the system has sufficient memory (recommended: 16GB or more).

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For users prioritizing transcription accuracy in real-world scenarios, MuScriptor is currently the preferred open-source solution, particularly excelling in multi-instrumental mix transcription and cross-segment consistency. If computational resources are limited or only piano transcription is required, consider the lightweight version of Google MT3, though its performance on real audio is weaker. YourMT3+ offers some architectural insights but has been surpassed by MuScriptor in overall performance and is less open-sourced.

If users require dynamic information (e.g., velocity) or more precise note control, MuScriptor currently does not support these features. They may need to wait for future updates or combine it with other post-processing tools. For academic research purposes, MuScriptor's open-source weights and complete training recipe provide an excellent baseline for further improvements and comparative experiments.

6. Editor's Summary

MuScriptor has achieved a significant breakthrough in the field of multi-instrument automatic transcription. Its technological innovations are primarily reflected in three aspects: first, it is fine-tuned using a large-scale dataset of real music (170,000 tracks), fundamentally addressing the domain shift issue between synthetic data and real-world scenarios; second, it introduces a reinforcement learning phase during training, employing GRPO-style alignment strategies to bring the model's output close to the quality of human-annotated data at the note level; third, it features an instrument condition control mechanism, which not only enhances user controllability but also stabilizes consistency across segments through classifier-free guidance, solving the previous pain point of unstable instrument allocation in long audio.

In terms of practical value, MuScriptor significantly lowers the barrier to entry for music transcription. Music producers, educational institutions, and researchers can now quickly convert existing recordings into editable MIDI data without the need for expensive professional equipment or manual transcription. The design of four parameter scales allows it to adapt to various deployment scenarios, ranging from mobile devices to servers. The open-source code under the MIT license further promotes secondary development and integration.

The primary target users include: music producers (to quickly obtain MIDI tracks for composition), music educators (to automatically generate sheet music for teaching), music information retrieval researchers (to obtain high-quality symbolic inputs), and developers of generative music models (to leverage MuScriptor for producing large-scale training data). For consumer-level users, the 307M model is recommended to balance speed and accuracy.

In terms of future development potential, MuScriptor's architectural design offers strong scalability. If it can incorporate support for dynamics information, improve inference speed, and further expand the diversity of musical genres in its training data, it has the potential to become a standard tool in the field of multi-instrument transcription. Additionally, its pure decoder architecture opens up new possibilities for joint modeling of end-to-end music generation and transcription.

7. Application Scenarios

• Music Production Assistance: Producers can quickly convert existing recordings without MIDI (such as old records or live performance recordings) into editable MIDI tracks for re-arranging, mixing, or sampling. For example, extract the piano and bass parts from a pop song, import them into a DAW, and replace the timbre or adjust the rhythm.

• Automatic Score Generation: Music education institutions and publishers can automatically convert teaching recordings or performance audio into sheet music, significantly reducing the cost of manual transcription. Teachers can quickly generate sheet music versions of practice pieces, and students can convert their improvisations into visual scores for analysis.

• Music Information Retrieval: Provide high-quality symbolic input for downstream tasks such as chord recognition, key analysis, and style classification. Researchers can use MuScriptor to transcribe large audio libraries into MIDI, enabling the training of more accurate music analysis models and enhancing the overall performance of MIR systems.

• Generative Music Modeling: Supply large-scale, high-quality MIDI training data for music generation models. MuScriptor's accurate transcription capabilities make it a powerful data annotation tool, useful for building multi-instrument, multi-style symbolic music datasets and advancing research in symbolic music generation.

• Music Archaeology and Archiving: Convert unstructured audio such as historical recordings and live performances into structured MIDI data, facilitating digital preservation and analysis. Archives and musicologists can use this to batch process valuable recordings, extract musical structural information, and provide data support for music history research.

8. FAQ

Q: What types of instruments does MuScriptor support?
A: The model supports 36 common instrument types, including piano, guitar, bass, drums, violin, cello, saxophone, trumpet, trombone, clarinet, flute, and more. The full list of instruments can be found in the project documentation. Users can specify the set of instruments to be transcribed using the instrument conditioning control feature.

Q: What hardware configuration is required to run MuScriptor?
A: It is recommended to use an NVIDIA GPU with at least 4GB VRAM (for the 307M model) or 8GB VRAM (for the 1.4B model). CPU inference is also possible, but it is slower; we recommend a system with at least 16GB of RAM. The 60M model (to be released) can run on mobile devices or embedded systems.

Q: How to enable instrument conditioning control?
A: Specify the target instruments during inference using the --instruments parameter, for example: --instruments piano,guitar,bass. The model will only output MIDI tracks for these instruments and use classifier-free guidance to ensure consistent instrument allocation across segments.

Q: How is the quality of MuScriptor's output MIDI?
A: In real audio scenarios, MuScriptor achieves a Multi F1 score of approximately 41.6, significantly outperforming previous open-source models. Note onset and offset detection accuracy is relatively high, but the current version does not support dynamic velocity information, and all output velocities are fixed values. For most composition and educational needs, the quality is already sufficiently practical.

Q: What advantages does MuScriptor have over Google MT3?
A: The main advantage lies in its generalization capability in real-world scenarios. MT3 was trained on synthetic data and has a relatively high error rate when encountering real-world mixes. MuScriptor, on the other hand, was fine-tuned on 170,000 real-world music tracks and further trained using reinforcement learning, achieving an F1 score that is approximately 10–15 percentage points higher on real audio. Additionally, MuScriptor supports instrument conditioning control, while MT3 does not.

Q: Does the model support real-time transcription?
A: Real-time processing is not currently supported. The model performs inference on 5-second audio slices, with the 1.4B model taking approximately 1–2 seconds to process 5 seconds of audio on an A100 GPU, and longer on consumer-grade GPUs. The 307M model can achieve near-real-time performance (approximately 0.5 seconds for 5 seconds of audio), making it suitable for offline batch processing.

Q: What music genres are included in the training data?
A: The training set covers multiple genres, including pop, classical, rock, heavy metal, jazz, electronic, and folk. The 170,000 real-world music tracks come from public datasets and licensed recordings, ensuring diversity. However, the proportion of some niche genres (such as world music and experimental music) may be lower, which could affect transcription performance.

9. Project Links

• GitHub Repository: https://github.com/muscriptor/muscriptor
• HuggingFace Model Library: https://huggingface.co/MuScriptor
• arXiv Technical Paper: https://arxiv.org/pdf/2607.08168v1