Qwen3.8-27B-DFlash2 – Inco AI's Open-Source Speculative Decoding Draft Model

1. What is Qwen3.8-27B-DFlash2

Qwen3.8-27B-DFlash2 is a speculative decoding draft model developed and open-sourced by Inco AI, specifically designed for the Qwen3.8-27B target model. This draft model contains only 1.92B parameters and leverages innovative block-level parallel prediction, a lightweight path selector, and dual-head dynamic convolution technology. It can generate entire blocks of token sequences in parallel during a single forward pass, significantly enhancing inference speed. Under H200 single-card and SGLang single-concurrency configurations, its throughput reaches 2.7 to 3.4 times that of traditional autoregressive decoding. The average acceptance length is 4.80, significantly outperforming native multi-token prediction (MTP) methods and community-based DSpark solutions, while maintaining strictly lossless decoding output.

Image source: Official article
Image source: official article

Technical positioning and domain: This model belongs to the field of natural language processing, specifically focusing on inference acceleration through speculative decoding. Unlike traditional draft models that still rely on autoregressive, token-by-token generation, DFlash2 employs a non-autoregressive, block-level parallel generation strategy. As an auxiliary model for Qwen3.8-27B, it aims to significantly reduce the inference latency of large language models and enhance service throughput, making it suitable for production environments with high real-time requirements.

Development background: After in-depth research into existing speculative decoding methods, the Inco AI team found that traditional MTP and community-based solutions (such as DSpark) still suffer from serial bottlenecks during the draft generation phase, limiting acceptance length and acceleration ratios. Starting from block-level parallel prediction, the team combined lightweight path selection with local dependency modeling to propose the DFlash2 architecture, which was validated on the Qwen3.8-27B model.

Core value: This solution addresses the core issues of slow draft model generation and limited acceptance length in speculative decoding. With an extremely compact size of just 1.92B parameters (approximately 3.85GB), it achieves a 2.7–3.4 times inference speedup. At the same time, it ensures that the decoding output is strictly consistent with autoregressive results through rigorous validation. This approach significantly reduces the deployment cost of large models, making it particularly suitable for high-concurrency, low-latency scenarios.

Technical features: Three core technologies—block-level parallel draft generation (predicting entire blocks of tokens in a single forward pass), a lightweight path selector (with only 2M parameters, tracking coherent paths from top-16 candidates), and dual-head dynamic convolution (+3% parameters, modeling local dependencies within blocks)—work together to greatly improve the quality of the draft and the acceptance length with minimal additional overhead. The model is natively integrated with mainstream inference engines such as SGLang, vLLM, llama.cpp, and oMLX, and supports deployment on multiple platforms including NVIDIA GPU and Apple Silicon.

2. Key Features

• Parallel Draft Generation: As the speculative decoding draft model for Qwen3.8-27B, DFlash2 predicts entire blocks of tokens in a single forward pass, completely replacing the traditional autoregressive generation method that produces tokens sequentially. This significantly reduces latency during the draft phase and serves as the core foundation for improving acceleration ratios.

• Intelligent Path Selection: By using a lightweight path selector to track a coherent path among the top-16 candidates at each position, DFlash2 addresses the issue of unnatural token transitions that may arise from block-level parallel prediction. This selector adds only 2M parameters and 0.6% latency overhead, increasing the average accepted length by approximately 0.4 tokens.

• Local Dependency Modeling: A dual-head dynamic convolution is employed to specifically model local dependencies between adjacent positions within a block, effectively mitigating the "suffix decay" phenomenon at the end of blocks. This module achieves near-equivalent results to adding ten Transformer layers, with an extremely low cost of just +3% parameters and +0.7% latency.

• Lossless Inference Acceleration: DFlash2 achieves 2.7–3.4 times higher throughput than autoregressive decoding on mainstream engines such as SGLang, vLLM, and llama.cpp. It ensures complete consistency between the output of the decoding process and the autoregressive results through a rigorous validation mechanism, without any loss in quality.

• Minimal Model Size: The draft model contains only 1.92B parameters (approximately 3.85GB), making it significantly smaller in size compared to the target model Qwen3.8-27B (approximately 54GB). This results in extremely low additional storage and GPU memory overhead during deployment, making it suitable for co-deployment with the target model on the same GPU.

• Multi-Engine Ecosystem Compatibility: DFlash2 is natively integrated with mainstream inference engines such as SGLang, vLLM, llama.cpp, and oMLX. It supports deployment on multiple platforms, including NVIDIA GPUs and Apple Silicon. Users can choose the appropriate inference backend based on their environment, reducing integration costs.

3. How to Use

1. Model Preparation: Download the Qwen3.8-27B target model and the Qwen3.8-27B-DFlash2 draft model separately to your local machine via the ModelScope platform. The target model belongs to the Qwen series, while the draft model is a specialized version provided by Inco AI. Ensure the download path is correct and that the model files are complete.

2. SGLang Deployment: After installing the SGLang inference framework, execute the sglang.launch_server command. Specify the speculative decoding algorithm using the --speculative-algorithm DFLASH parameter, and set the draft model path using the --draft-model parameter. Once launched, the service will automatically load the target model and the draft model, providing an inference interface compatible with the OpenAI API.

3. vLLM Deployment: Install the vLLM version that supports DFlash, and start the service using the vllm serve command. In the --speculative-config configuration, set method to dflash and enter the name or path of the draft model. vLLM will automatically handle collaborative inference between the target model and the draft model, achieving acceleration.

4. llama.cpp Deployment: Pull and compile the PR branch that supports DFlash functionality. Prepare the GGUF files for both the target model and the draft model separately. When launching the server, use the --spec-type draft-dflash parameter to specify the draft type and load both model files. The lightweight nature of llama.cpp makes it particularly suitable for resource-constrained environments.

5. Apple Silicon Deployment: Use the precompiled oMLX package, and in the Model Manager, enable the DFlash option for the target model. Specify the draft model and set the Verify mode to dflash. Optimized for the Apple Metal API, oMLX enables efficient local inference on devices such as MacBook.

6. Command Line Tool: After installing the dflash package, you can directly run inference and evaluation using the dflash generate command in conjunction with transformers, MLX, or OpenAI-compatible backends. This tool supports custom parameters, making it convenient for quick testing and benchmark evaluation.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For users seeking maximum inference acceleration with the Qwen3.8-27B model, DFlash2 is the optimal choice, as it significantly outperforms DSpark and native MTP in terms of accept length and acceleration ratio. If the target model is not part of the Qwen series, DSpark or native MTP may offer better general applicability, although their acceleration performance is typically lower than that of DFlash2. In resource-constrained environments (e.g., single-card consumer-grade GPUs), DFlash2's minimal parameter overhead still provides substantial acceleration, though practical testing is required for confirmation. For teams looking to quickly integrate speculative decoding, DFlash2's multi-engine compatibility reduces migration costs, while DSpark and native MTP may require more custom development.

6. Editor's Summary

Qwen3.8-27B-DFlash2 demonstrates clear technological innovation in the speculative decoding domain. Its block-level parallel draft generation completely transforms the traditional paradigm where draft models must generate tokens autoregressively one by one, combining a lightweight path selector and dual-head dynamic convolution to achieve a significant increase in draft acceptance length at a minimal parameter cost. From a technical metric perspective, with an average draft acceptance length of 4.80 and a speedup ratio of 2.7–3.4 per concurrent request, this solution is ahead of its peers, and its lossless output characteristics ensure reliability in practical deployment.

From a practical value standpoint, this model directly addresses the pain points of high inference latency and throughput limitations in large-scale language models. For scenarios such as AI Agent services, real-time dialogue systems, and code generation, the acceleration effect can be translated into lower response costs and a better user experience. Its extremely compact size of 1.92B allows it to be easily deployed on the same hardware as the target model, eliminating the need for additional hardware investment and reducing the barrier to entry.

In terms of target users, this model is primarily aimed at AI service developers requiring high-throughput inference, researchers working on speculative decoding algorithms, and end users who want a smooth experience on local devices. For enterprise-level deployment, DFlash2's engine compatibility (SGLang/vLLM/llama.cpp) enables it to seamlessly replace existing inference stacks.

Looking ahead, the architectural design of DFlash2 is transferable—block-level parallel prediction and path selection mechanisms can be extended to other target models. If the Inco AI team continues to expand the range of supported models, DFlash2 has the potential to become an infrastructure-level tool in the speculative decoding field. However, its current limitation to only being compatible with Qwen3.8-27B also reminds users to pay attention to future official updates. Overall, this is a technically solid and highly effective open-source project that is worth in-depth evaluation by relevant professionals.

7. Application Scenarios

• AI Agent Services: In long-running scenarios requiring frequent tool calling and task planning with large models, DFlash2's 2.7–3.4 times acceleration significantly reduces token-by-token decoding costs, enhancing the Agent's response speed and concurrent processing capability. It is well-suited for applications such as intelligent customer service and automated workflows.

• Real-time Dialogue Systems: Chatbots and customer service assistants are highly sensitive to response latency. DFlash2's low-latency draft generation enables faster inference for each interaction, significantly improving the perceived smoothness of the conversation while maintaining output quality, making it ideal for interactive scenarios such as online customer service and virtual assistants.

• Code Generation Assistants: On programming benchmarks such as HumanEval and MBPP, DFlash2's high acceptance length means more correct tokens can be generated per validation step, accelerating code completion and generation. When integrated into IDE plugins or programming Copilots, it can significantly improve developers' coding efficiency.

• Long-form Text Creation: For long document generation tasks such as articles, reports, and marketing copy, the high acceptance length allows the model to accept more draft tokens in each validation step, reducing the number of inference rounds and thereby accelerating the overall generation process, while ensuring content coherence and quality.

• On-device Local Deployment: With its compact size of 1.92B and support from oMLX and llama.cpp, DFlash2 can run on local devices such as Apple Silicon notebooks, enabling efficient on-device inference. This makes it suitable for privacy-sensitive or offline scenarios.

8. FAQ

Q: What is speculative decoding? How does DFlash2 achieve acceleration?
A: Speculative decoding is an inference acceleration technique that uses a lightweight draft model to quickly generate candidate tokens, which are then verified in parallel by the target model, thereby reducing the number of serial decoding steps. DFlash2 further improves the draft model to perform block-level parallel prediction, generating an entire block of tokens in a single forward pass. Combined with a path selector and local convolution to enhance draft quality, this achieves a higher acceleration ratio.

Q: Does DFlash2 support target models other than Qwen3.8-27B?
A: Currently, DFlash2 is trained and optimized specifically for Qwen3.8-27B. To support other target models, you would need to retrain the draft model or wait for official extended support. The community may provide migration solutions, but this has not been officially confirmed.

Q: What hardware configuration is required to deploy DFlash2?
A: Official testing achieved the best acceleration ratio on a single H200 GPU. In theory, any GPU that supports inference for the target model can run DFlash2, but the acceleration effect depends on memory and computational power. It is recommended to use a GPU with at least 24GB of memory (such as the RTX 4090) to achieve noticeable acceleration. Apple Silicon devices can deploy DFlash2 via oMLX, with performance depending on the chip's specifications.

Q: Does DFlash2 support Chinese input and output?
A: The target model Qwen3.8-27B itself supports both Chinese and English. DFlash2, as a draft model, does not alter the language capabilities of the target model, so it fully supports Chinese tasks. The actual performance depends on the target model's own capabilities, while the draft model only affects generation speed.

Q: What open-source license does DFlash2 use? Can it be used commercially?
A: DFlash2 uses the Apache 2.0 license, and the code is fully open source, allowing both commercial and academic use. However, note that the target model Qwen3.8-27B follows the Tongyi Qianwen license. Please confirm its terms before commercial use.

9. Project Links

• Project Website: https://inco.ai/blog/dflash2/
• GitHub Repository: https://github.com/z-lab/dflash