Nemotron-Labs-Diffusion – NVIDIA's Open-Source Three-Mode Language Model

1. What is Nemotron-Labs-Diffusion

Nemotron-Labs-Diffusion is a series of three-mode language models launched by NVIDIA, unifying autoregressive (AR), diffusion, and self-speculative decoding within a single architecture. Through joint AR-diffusion objective training, the model can dynamically switch decoding modes across different concurrency scenarios while maintaining high throughput. The series includes base, instruction, and vision-language model variants with 3B, 8B, and 14B parameters, surpassing existing open-source AR and diffusion language models in both accuracy and speed, offering unprecedented flexibility and efficiency for inference deployment.

Technical Positioning and Domain: This belongs to the cutting-edge intersection of natural language processing and generative AI, specifically positioned as "unified multi-mode decoding language models." It breaks through the limitations of traditional single-decoding-paradigm language models by integrating the sequential generation capability of autoregression, the parallel planning capability of diffusion models, and the acceleration capability of self-speculation. This design enables coverage of high-concurrency cloud inference, low-latency edge interaction, and complex tasks requiring long-range logical planning, achieving scenario-adaptive capability at the model architecture level.

R&D Background: Developed by NVIDIA's research team, leveraging its deep expertise in GPU computing, deep learning frameworks (such as TensorRT, SGLang), and large-scale language model training. The R&D motivation stems from the "mode selection dilemma" faced by existing language models during deployment: autoregressive models suffer from throughput limitations due to sequential decoding, diffusion models can parallelize but lack language priors, and multi-token prediction methods require additional model heads. NVIDIA aims to create a universal foundation model that "adapts one model to multiple scenarios," reducing the cost of maintaining multiple model pipelines.

Core Value: Solves the difficulty of balancing concurrency scenarios and latency requirements in traditional language models. A single model can seamlessly switch between AR mode (high-concurrency batch inference), diffusion mode (maximum parallelism, suitable for long-text planning), and self-speculation mode (low-latency interaction) without additional training or architectural modifications. Its 8B model decodes 6 times more tokens per forward pass than Qwen3-8B, achieving 4x throughput improvement on SPEED-Bench with GB200 GPUs, significantly reducing deployment costs in both cloud and edge environments.

Technical Features: Employs block-level diffusion formulation for intra-sequence parallel decoding, generating multiple tokens in a single forward pass; self-speculation mode uses the same model as both draft and verifier, sharing KV cache without auxiliary prediction heads; supports FP8 low-precision inference, combined with the SGLang framework to achieve high-throughput serving on GB200 GPUs. Joint AR-diffusion objective training effectively balances gradient competition between the two objectives through a two-stage strategy and global loss averaging.

2. Key Features

• Three-Mode Decoding Switching: The model supports three decoding modes: Autoregressive (AR), Diffusion, and Self-Speculative. Users can seamlessly switch between modes within the same model by simply changing the attention mask pattern, without needing to load different weights or modify the network architecture. AR mode is suitable for high-concurrency batch requests, Diffusion mode is ideal for long-text tasks requiring parallel generation, and Self-Speculative mode focuses on low-latency interactive scenarios.

• Block-Level Diffusion Parallel Generation: Utilizing a block-level diffusion formulation, the output sequence is divided into multiple blocks, with tokens within each block generated in parallel via bidirectional attention denoising. A single forward pass can output multiple tokens, significantly improving decoding speed. This mechanism differs from the token-by-token generation of traditional diffusion language models, refining sequence-level diffusion into block-level operations to balance parallelism and linguistic coherence.

• Self-Speculative Draft Verification: The Diffusion mode can quickly generate multiple tokens as a draft, which is then verified by the same model in AR mode. Draft generation and verification share the KV cache, eliminating the need for additional prediction heads or auxiliary models. Compared to Multi-Token Prediction (MTP) methods, this mode achieves better acceptance rates and practical device efficiency without increasing the parameter count.

• Vision-Language Understanding: A Vision-Language Model (VLM) variant is provided, supporting image understanding and text-image reasoning. By combining a vision encoder with the three-mode language backbone, the model can handle tasks such as image captioning and visual question answering, expanding the boundaries of multimodal applications. The vision variant also supports three-mode switching, offering flexible deployment capabilities for multimodal reasoning.

• Efficient Inference Deployment: Natively supports the FP8 low-precision format, leveraging NVIDIA hardware (e.g., GB200) FP8 compute units to significantly reduce memory usage and inference latency while maintaining model accuracy. Combined with the SGLang inference framework, the Linear Self-Speculation strategy can be configured to achieve production-grade high-throughput services.

• Joint AR-Diffusion Training Mechanism: The model simultaneously optimizes the autoregressive next-token loss and the block-level diffusion denoising loss, with a weighted combination (α=0.3) balancing the two. The AR objective provides a left-to-right language prior for diffusion, preventing diffusion training from wasting capacity on arbitrary token permutations; the diffusion objective enhances the model's forward-planning capability. A two-stage training strategy (first pure AR pre-training, then joint training) ensures the harmonious integration of both modes.

3. How to Use

1. Environment Requirements: Requires Python 3.8+, with PyTorch 2.0+ and CUDA 11.8+ installed. NVIDIA GPUs (e.g., A100, H100, GB200) are recommended, with at least 16GB VRAM (for the 8B model). Run pip install "transformers>=5.0" torch peft to install basic dependencies. For FP8 support, additionally install torchao and sglang related libraries.

2. Obtain Model Weights: Pull the corresponding model from the HuggingFace model hub. For example, for the 8B base model, use the command: git lfs clone (link to be updated after official release) and tokenizer_config.json. Visual variants require additional download of vision encoder weights.

3. Select Decoding Mode and Run Inference: Choose the corresponding script based on the concurrency scenario:

   • High concurrency batch: Use chat_ar.py, set --mode ar.
   • Maximum parallelism: Use chat_dlm.py, set --mode diffusion, and adjust block size (e.g., --block-size 64).
   • Low-latency interaction: Use chat_linear_spec.py, enable self-speculative mode, optionally attach a LoRA draft adapter (--lora_weights path/to/lora).
     Run single-turn or multi-turn dialogue inference with support for streaming output.

4. Production Deployment: Start the inference service based on the SGLang framework. Configuration example: python -m sglang.launch_server --model-path nvidia/Nemotron-Labs-Diffusion-8B --speculative-mode linear --enable-fp8. Integrate into applications via REST API, with support for dynamically switching decoding modes (specified via request parameters). Note: FP8 quantization requires calibration on first load; it is recommended to warm up with a representative dataset.

5. Notes and Best Practices: Diffusion mode is suitable for scenarios with small batch sizes to fully leverage parallelism; AR mode offers optimal throughput with large batch sizes. In self-speculative mode, the LoRA draft adapter can improve acceptance rate but increases VRAM overhead; use it when VRAM is sufficient. For visual variant inference, preprocess images to the specified size (e.g., 336×336) and pass the pixel_values tensor.

4. Pros and Cons Analysis

5. Comparison with Similar Tools

Selection Recommendations:

For cloud service scenarios pursuing maximum throughput and deployment flexibility, Nemotron-Labs-Diffusion is the optimal choice. Its diffusion mode can multiply throughput in batch inference, and the three-mode switching capability allows switching to AR mode during off-peak hours to save energy. If the team primarily uses NVIDIA hardware (e.g., H100, GB200) and has strong FP8 inference requirements, the model's native support can significantly reduce optimization costs.

For general tasks requiring a mature ecosystem and broad community support, Qwen3-8B or LLaMA 3.1-8B remain safer choices. These models have rich fine-tuning tools, quantization solutions, and third-party inference frameworks (e.g., vLLM, TGI), with more comprehensive benchmarks in multilingual and long-context scenarios. If the project is latency-sensitive and does not require parallel generation, pure AR models combined with speculative decoding (e.g., Medusa) can also achieve good results.

6. Editor's Take

Nemotron-Labs-Diffusion marks a significant step forward in architectural innovation. Traditional language models have long been constrained by the sequential bottleneck of autoregressive decoding, while diffusion models, though capable of parallel generation, lack language priors. NVIDIA, through joint training and structured attention mechanisms, integrates the strengths of both paradigms into a single model, and additionally introduces a self-speculative mode, forming a versatile "multi-mode decoder" foundation. This design is academically forward-looking—it demonstrates that AR and diffusion objectives can coexist harmoniously, and the two-stage training strategy effectively avoids gradient conflicts. From a practical standpoint, the 8B model's 4x throughput improvement on SPEED-Bench is not just theoretical; it directly translates into reduced cloud inference costs. Particularly noteworthy is the self-speculative mode: using diffusion to generate drafts, AR for verification, and sharing KV cache, this "self-consistent" acceleration scheme is more elegant and parameter-efficient than MTP.

Of course, the model is not without its shortcomings. Its training cost is relatively high, and its strong dependence on NVIDIA hardware limits deployment in heterogeneous environments. The performance of its vision variant has yet to undergo large-scale third-party evaluation, and its ecosystem maturity lags far behind LLaMA and Qwen. However, given NVIDIA's dominance in AI infrastructure and the ongoing optimization of the SGLang inference framework, Nemotron-Labs-Diffusion has the potential to become the standard foundation for next-generation inference deployment.

Target Audience: Teams requiring high-throughput cloud inference services (e.g., AI dialogue platforms, content generation services), academic institutions researching multi-mode decoding paradigms, and developers looking to explore diffusion language models for edge applications.

Future Development Potential: As NVIDIA's next-generation GPUs (e.g., the Rubin architecture) further optimize FP8 and block-level parallelism, the model's performance advantages will become even more pronounced. If the open-source community builds a rich toolchain around it (e.g., quantization, fine-tuning, RAG integration), it could very well challenge LLaMA's position in the open-source domain.

7. Application Scenarios

• Real-time Conversational Assistant: In low-concurrency scenarios (such as personal AI assistants or customer service dialogues), use the self-speculative mode to achieve low-latency interaction. The model can quickly generate a draft and verify it after user input, compressing response time to under 200ms while maintaining reply quality. Combined with the LoRA draft adapter, the first-token latency can be further reduced.

• Cloud Inference Services: In high-concurrency batch request scenarios (such as API gateways or content moderation), switch to AR mode. AR mode fully leverages GPU compute density and, combined with FP8 quantization, can support thousands of concurrent requests on a single GB200 GPU, significantly reducing cloud deployment costs. The PagedAttention mechanism in the SGLang framework further optimizes memory utilization.

• Code and Mathematical Reasoning: In tasks requiring multi-step logical deduction (such as code generation or mathematical proofs), use the diffusion mode. Block-level diffusion generates multiple candidate tokens in parallel, enhancing the model's forward-planning capability and avoiding the "local optimum" trap of autoregressive decoding. Experiments show that the diffusion mode achieves better pass@k metrics on HumanEval and GSM8K compared to AR models with the same parameters.

• On-device Local Inference: The 8B model can run on consumer-grade GPUs with 16GB of VRAM (such as the RTX 4090), with three modes to adapt to different workloads. For daily use, set it to AR mode to save power; switch to diffusion mode when processing long documents for acceleration. The model weights are approximately 16GB (FP16), which can be further compressed to 6GB via 4-bit quantization, making it suitable for mobile devices.

• Vision-Language Applications: The vision-language model variant supports image understanding and visual question answering, suitable for tasks such as intelligent document analysis (e.g., invoice recognition, contract summarization) and multimodal interaction (e.g., image caption generation). Users can upload images and ask questions, with the model generating answers in parallel using diffusion mode, balancing speed and accuracy.

8. FAQ

Q: How to choose among the three decoding modes of Nemotron-Labs-Diffusion?
A: Choose based on concurrency and latency requirements. Use AR mode for high-concurrency batch inference (e.g., API services); use diffusion mode for scenarios requiring maximum parallelism with small batch sizes (e.g., long text generation); use self-speculative mode for low-latency interactions (e.g., real-time dialogue). The model can dynamically switch at runtime by modifying the attention mask without reloading.

Q: How to set the block size for diffusion mode?
A: The block size controls the granularity of parallel decoding. Start with 64 as a baseline; larger blocks increase parallelism but may affect local coherence. For the 8B model, a block size of 64-128 is recommended. For vision variants, reduce it appropriately (e.g., 32) to balance local dependencies in image sequences.

Q: Does self-speculative mode require additional training?
A: No. Self-speculative mode leverages the model's own diffusion capability to generate drafts, with AR mode for verification, sharing the same parameters. However, an optional LoRA draft adapter (requires fine-tuning) can be attached to improve draft quality, thereby increasing acceptance rate and actual speedup. The LoRA adapter can be trained with minimal data (e.g., 1000 dialogue samples).

Q: How to enable FP8 inference? How much does it affect accuracy?
A: Add the --enable-fp8 parameter when starting SGLang. FP8 quantization uses dynamic per-layer calibration and typically does not cause significant accuracy degradation (loss <0.5% on benchmarks like MMLU). Calibration is required on first load (about 5 minutes), with no additional overhead for subsequent inference. Memory usage can be reduced by approximately 50%.

Q: Does the model support Chinese?
A: Nemotron-Labs-Diffusion is primarily optimized for English scenarios, with its tokenizer and training data predominantly in English. Performance on Chinese tasks may be inferior to Chinese-specific models of similar parameter size (e.g., Qwen3). For Chinese use cases, it is recommended to use AR mode with minimal Chinese instruction fine-tuning, or wait for NVIDIA to release a multilingual version.

Q: How to obtain the vision-language variant?
A: Search for nvidia/Nemotron-Labs-Diffusion-8B-VL in the HuggingFace model hub. When loading, specify --trust-remote-code and install transformers>=5.0 and torchvision. Image preprocessing uses the model's built-in processor (automatic resizing and normalization).

9. Project Links

• HuggingFace Model Hub (Official Weights & Tokenizer): https://huggingface.co/collections/nvidia/nemotron-labs-diffusion
• arXiv Technical Paper: https://arxiv.org/pdf/2607.05722
• NVIDIA Official Organization Page: https://huggingface.co/nvidia
• SGLang Inference Framework (Official Deployment Guide): https://github.com/sgl-project/sglang (Note: SGLang is an independent open-source project, not directly maintained by NVIDIA, but officially recommended for deployment)