Muse Voice Transcribe – Meta's First Real-Time Audio Perception Model

1. What is Muse Voice Transcribe

Muse Voice Transcribe is Meta's first real-time audio perception model, integrating streaming automatic speech recognition (ASR), speaker segmentation, and voice endpoint detection. The model supports over 70 languages and native language code switching, capable of handling long audio recordings exceeding one hour and scenarios with more than 20 simultaneous speakers. It ranks first in both streaming transcription and voice segmentation benchmark tests. Utilizing adaptive latency technology based on reinforcement learning, it dynamically balances response speed and recognition accuracy. Currently, Muse Voice Transcribe is integrated into Meta AI and Muse Code, positioned as the real-time perception layer for AI systems.

Image source: Official article
Image source: official article

Technical positioning and domain: Muse Voice Transcribe belongs to the speech recognition branch within natural language processing. However, its product positioning is not a traditional offline transcription tool, but rather an audio perception infrastructure for real-time interaction scenarios. It achieves end-to-end streaming generation from audio to text by unifying the modeling of soft tokens and text tokens, differing from the traditional cascaded (VAD first, then ASR) pipeline architecture.

Development background: The model was developed by Meta based on the Muse Spark series of autoregressive models, continuing Meta's technical accumulation in the fields of speech and multimodal AI. Its development motivation directly addresses the perception bottlenecks of current AI assistants and agents in real-time conversations, namely the inability to accurately determine when a user starts or stops speaking, and the lack of effective voice attribution in multi-speaker environments.

Core value: This model solves three key issues in real-time speech interaction: first, streaming transcription with extremely low latency, outputting text in 80ms units; second, automatic speaker segmentation in multi-person scenarios, allowing voice attribution without post-processing; third, precise voice endpoint detection, enabling AI systems to clearly perceive the start and end boundaries of speech. These capabilities allow Muse Voice Transcribe to serve as the real-time perception layer for AI systems, significantly enhancing the performance of AI assistants in complex auditory environments.

Technical features: It employs a unified autoregressive streaming training framework, integrating three tasks—ASR, speaker segmentation, and endpoint detection—into a single model, with task expansion achieved through special tokens. Its core differentiation lies in the adaptive latency strategy based on reinforcement learning, allowing the model to dynamically decide whether to wait for more context or output immediately based on the recognition difficulty of each word, achieving Pareto optimality between speed and accuracy.

2. Key Features

• Real-time Streaming ASR: The model processes audio and outputs text in real-time in 80ms intervals, without waiting for the entire audio segment to finish. This mechanism breaks through the limitations of traditional ASR systems, which must wait until the user finishes speaking before starting recognition, providing a response experience for real-time voice assistants that closely matches the rhythm of human conversation.

• Multi-speaker Diarization: Automatically identifies and labels more than 20 different speakers, distinguishing who spoke and when during the streaming output process. This feature achieves the current state-of-the-art DER score of 17.5 on public speech segmentation benchmarks, making it suitable for multi-speaker scenarios such as meeting transcripts and interview records, without requiring any post-processing steps.

| - Voice Endpoint Detection: Accurately determines the start and end of speech by introducing special tokens <|speech_onset|> and `<|speech_endpoint|

• Multilingual and Code-switching Support: Covers over 70 languages, natively supporting arbitrary code-switching within or between sentences, and accurately recognizing mixed expressions such as code-switching between Chinese and English. This capability is particularly important for daily communication in multinational corporations and multilingual communities, without the need to pre-specify language types.

• Long Audio and Multi-speaker Meeting Handling: Can process continuous audio input lasting over one hour without segmentation. The model continuously receives audio data through a streaming token mechanism and directly outputs complete transcription results and speaker labels, making it ideal for long meetings, lectures, and interviews.

• Reinforcement Learning Adaptive Latency: Combines word error rate rewards and latency rewards multiplicatively through RL, allowing the model to dynamically decide whether to wait for more context or output immediately based on the difficulty of recognizing each word. This mechanism achieves a Pareto optimal balance between speed and accuracy, showcasing notable technical advancement among peer streaming ASR models.

3. How to Use

1. Select the Integration Entry Point: Muse Voice Transcribe offers three integration methods, including Meta Model API (suitable for developer integration), Mac version of Meta AI (suitable for individual users), and Muse Code (suitable for developers). API integration requires accessing the Meta Developer Platform to apply for credentials.

2. Start Voice Transcription: Press the ⌘ + v shortcut key in Muse Code, or click the voice button in the Mac version of Meta AI to begin transcription. The system will automatically activate the Muse Voice Transcribe model for real-time audio perception and text output.

3. Cross-Application Global Invocation: Hold down the Fn key in any Mac application to invoke Meta AI for voice transcription. This enables voice input capabilities across the entire system without switching application windows, allowing direct voice-to-text input in document editors, chat windows, or code editors.

4. API Integration Invocation: Through the Meta Model API, developers can integrate Muse Voice Transcribe into their own applications or workflows on a pay-per-use basis (3 USD per 1000 minutes). The API supports streaming return of transcription results and speaker labels, making it easy to build custom real-time voice processing systems.

5. Configuration and Best Practices: For long audio scenarios, it is recommended to enable streaming mode and set a reasonable listening timeout threshold during API calls. In multi-speaker meeting scenarios, speaker labels can be used for subsequent text structuring. The system requires macOS platform for desktop integration, while the API is compatible with mainstream operating systems.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

If building a low-latency voice interaction layer for a real-time AI assistant, Muse Voice Transcribe stands out as the most targeted solution due to its comprehensive capabilities in adaptive latency, multi-speaker segmentation, and endpoint detection. However, if the requirement involves handling voice input, voice output, and tool calling simultaneously, Gemini 3.1 Flash Live's multimodal conversation capabilities offer a more holistic advantage. For developers prioritizing cost control and local deployment, building a customized pipeline based on the Whisper open-source model is a viable option, though they will need to address streaming processing and speaker segmentation independently.

6. Editor's Summary

Muse Voice Transcribe demonstrates a clear path of technological innovation in the niche area of real-time audio perception. By unifying streaming ASR, speaker segmentation, and endpoint detection into a single autoregressive model, and achieving word-by-word adaptive latency through reinforcement learning, this architectural design exhibits high technical uniqueness within the field of speech recognition. It achieved a score of 3.1% on the AA-WER Streaming Index in the Artificial Analysis streaming speech-to-text benchmark, validating the practical effectiveness of this technical approach.

In terms of practical value, Muse Voice Transcribe addresses key perception challenges in real-world dialogue scenarios for AI systems. AI assistants not only need to understand the content of speech, but also need to know when a user starts speaking, when they stop, and who is speaking—this is precisely the core value of the model. As a real-time perception layer for AI systems, it provides foundational audio understanding capabilities for a variety of applications, including voice assistants, meeting transcription, and real-time subtitles.

This model is suitable for the following users: AI application developers who need to build real-time voice interaction capabilities, enterprise users with needs for transcribing cross-border meetings and long audio recordings, and teams that rely on multilingual communication scenarios. Through the Meta Model API, developers can quickly integrate its capabilities, reducing the complexity of building their own speech processing pipelines.

Muse Voice Transcribe advances speech recognition from simply "understanding content" to "perceiving dialogue," providing AI systems with a more complete auditory comprehension capability. As AI agents and voice interaction applications continue to grow, the application potential and technical evolution of this type of real-time audio perception model are worth close attention.

7. Application Scenarios

• Real-time Meeting Minutes for International Conferences: Supports scenarios with more than 20 participants speaking simultaneously and meetings lasting over one hour. It automatically distinguishes each speaker and accurately transcribes discussions with mixed Chinese and English content. Meeting organizers can obtain structured, speaker-tagged minutes, eliminating the need for tedious manual organization and labeling.

• AI Voice Assistants and Agents: Acts as a real-time perception layer, offering low-latency streaming ASR and precise voice endpoint detection capabilities. AI assistants can accurately determine when a user starts and ends their speech turn, avoiding unintended responses during pauses in user thinking, significantly enhancing the naturalness and smoothness of interactions.

• Podcast and Interview Subtitle Generation: Processes hours-long audio content, natively supporting segmentation of multi-person conversations. It automatically generates independently labeled subtitle tracks for each interview guest. Podcast producers and video creators can obtain fully labeled subtitle files without the need for manual speaker annotation in post-production.

• Multilingual Customer Service Quality Assurance: Real-time transcription of customer service agent and customer conversations in mixed languages. It improves the accuracy of professional terminology recognition through context preference and automatically archives call records. QA teams can quickly search for specific scripts and customer feedback, enabling intelligent analysis of large volumes of call content.

• Developer Voice Input Tools: Through the Meta AI for Mac or Muse Code's Fn key for global invocation, enables voice transcription across any application, including coding, documentation writing, and messaging. Developers can break free from keyboard input constraints and complete daily development and communication tasks using voice.

8. FAQ

Q: What is the fundamental difference between Muse Voice Transcribe's streaming ASR and traditional speech recognition?
A: Traditional speech recognition typically requires waiting for the user to finish speaking an entire sentence or paragraph before processing, resulting in higher latency. Muse Voice Transcribe processes audio streams in real-time in 80ms intervals, allowing the model to output text as it receives audio. This achieves true streaming inference, with latency controlled within the natural pauses of human conversation, making it more suitable for real-time interactive scenarios.

Q: How does the model handle mixed-language input with both Chinese and English?
A: Muse Voice Transcribe natively supports code-switching and does not require pre-specified language. During training, the model learned arbitrary language switching patterns both within and between sentences, enabling it to accurately recognize expressions that mix Chinese and English, such as using both Chinese and English words within a single sentence. The model identifies and outputs the corresponding language text based on context.

Q: What is the billing method and cost for API calls?
A: When using Muse Voice Transcribe via the Meta Model API, billing is based on usage, with a price of $3 per 1000 minutes, or approximately $0.18 per hour. Compared to some general-purpose multimodal models that require paying for both audio input and text output, Muse Voice Transcribe offers a simpler cost structure for pure transcription. Specific usage and billing details can be viewed on the Meta Developer Platform.

Q: Which languages does the model support, and does it cover less commonly spoken languages?
A: Muse Voice Transcribe supports over 70 languages, including major international languages and numerous regional languages. This multilingual capability is based on the multilingual training data from the Muse Spark series of models. Users do not need to specify the language each time they make a call; the model automatically identifies the language based on the audio content.

Q: How can I use Muse Voice Transcribe for global voice input on Mac?
A: After enabling Muse Voice Transcribe in the Mac version of Meta AI or Muse Code, you can invoke the global voice transcription feature by holding down the Fn key in any Mac application. The system will capture microphone audio and output the transcribed text in real-time at the current cursor position, providing a voice input experience that works across applications. This feature is currently limited to the Mac platform only.

9. Project Links

• Official Product Introduction: https://research.meta.ai/blog/introducing-muse-voice-transcribe
• Meta AI Product Portal: https://www.meta.ai
• Muse Code: https://muse.ai/muse-code
• Meta Model API (Developer Access): https://developers.facebook.com