AngelSpec – An End-to-End Speculative Decoding Training Framework Open-Sourced by Tencent HunYuan Team

1. What is AngelSpec

AngelSpec is an end-to-end speculative decoding training framework open-sourced by the Tencent HunYuan team, built upon TorchSpec. It supports six types of draft architectures, including MTP autoregressive and DFly block-parallel. This framework decouples inference from training, utilizing Mooncake/RDMA streaming transmission for hidden states. On the Hy3-A21B model, the DFly architecture achieved an end-to-end acceleration of 1.98–2.40 times and improved throughput by 10.5–11.8% compared to DFlash. AngelSpec aims to systematically address the trade-off between draft acceptance rate and verification overhead in speculative decoding, offering a unified training solution for accelerating inference in large-scale language models.

Image source: Official article
Image source: official article

Technical positioning and domain: Belongs to the large model inference acceleration direction within the field of natural language processing, focusing on the training phase of speculative decoding technology. AngelSpec is positioned as a unified end-to-end training framework that covers multiple draft architectures, differing from solutions that support only a single architecture, thus filling the gap in a unified training framework for multiple architectures.

Development background: Developed by the Tencent HunYuan team, built on TorchSpec. The team has accumulated extensive experience in inference optimization on large-scale models such as Hy3 and Qwen3. In response to the heterogeneity of real-world workloads (e.g., different scenarios such as dialogue, code, and mathematics), they proposed a workload specialization design philosophy to avoid a single draft model performing poorly across all domains.

Core value: Addresses issues with existing speculative decoding training frameworks, such as support for only a single architecture, lack of decoupling between training and inference, and inability to adapt to heterogeneous workloads. Through innovations like a unified training pipeline, decoupled scalability, and dynamic validation (D-Cut), it significantly improves the end-to-end acceleration ratio and throughput of speculative decoding, while also reducing deployment and maintenance costs.

Technical features: Supports six draft architectures (DFly, DFlash, DFlare, Eagle3, DSpark, MTP) sharing the same training pipeline, with switching achievable by simply modifying the configuration; inference and training are fully decoupled, allowing independent scaling; employs an acceptance rate alignment objective function (TV/LK/D-PACE, etc.) to directly optimize the draft acceptance rate; provides online real-time evaluation capabilities, reporting speculative decoding metrics in real time during training.

2. Key Features

• Unified Training of Six Draft Architectures: AngelSpec integrates six distinct draft generation architectures—DFly, DFlash, DFlare, Eagle3, DSpark, and MTP—into a single training pipeline. Users can simply modify the configuration file to switch between architectures, without needing to rebuild the training process. This unified design significantly reduces the experimental cost of trying different architectures, making it easier for researchers to compare and select the most suitable draft strategy for specific scenarios.

• MTP Training with TTT Expansion: Supports Multi-Token Prediction (MTP) training with a Training-Time Test (TTT) mechanism that exposes the draft model to self-generated trajectories, eliminating exposure bias caused by teacher forcing. Memory optimization approaches single-causal forward pass efficiency, and combined with Ulysses sequence parallelism, it supports up to 128k context length. This makes it ideal for long-sequence training and inference.

• Acceptance Rate Alignment Objective Function: Provides multiple acceptance rate optimization objectives, including cross-entropy (CE), top-k KL divergence, LK Loss, D-PACE weighted loss, and end-to-end Total Variation (TV) loss. Users can freely combine these objectives based on their task requirements, directly optimizing the draft model's acceptance rate during rejection sampling, rather than focusing solely on prediction accuracy. This enhances practical acceleration effects.

• Document-Aware Sequence Packing: Uses a fixed-length packing strategy inspired by Megatron, with strict cross-document isolation to prevent attention contamination between different documents. It also covers both DFlash and MTP training paths, ensuring efficient data utilization while maintaining training quality. This is suitable for mixed multi-document training scenarios.

• Online Real-Time Evaluation: During training, real inference is performed using engines like vLLM to execute speculative decoding, with key metrics such as average acceptance length and per-token acceptance rate reported in real time. This allows training performance to be directly reflected in inference capabilities, avoiding discrepancies between offline metrics and online performance, and accelerating iterative optimization.

• Multi-Inference Backend Support: Compatible with mainstream inference engines such as vLLM, SGLang, and HuggingFace. Users can deploy the draft model in different inference environments without modifying the model itself. This flexibility enables AngelSpec to be easily integrated into existing inference service stacks, reducing migration costs.

3. How to Use

1. Environment Requirements and Installation: The system requires a Linux environment that supports CUDA, and it is recommended to use an NVIDIA A100 or higher configuration GPU. First, run pip install -e ".[vllm]" to install the framework and vLLM backend, then execute pip install mooncake-transfer-engine to install the Mooncake transfer engine. Alternatively, you can use ./tools/build_conda.sh 1 vllm to one-click build a Conda isolated environment that includes Mooncake. Once activated, you can start using it.

2. Single-node Quick Start: On a node with 8 GPUs, allocate 4 GPUs for inference and 4 GPUs for training. Run ./examples/qwen3-8b-dfly/run.sh to launch the complete training and inference pipeline. The script will automatically configure the inference and training components and use Mooncake to transfer hidden states. Users only need to ensure that the dependencies are installed correctly.

3. Configuration Override and Customization: The framework uses YAML configuration files to manage training parameters. Users can directly override configuration items via the CLI, for example: training.learning_rate=5e-5 training.num_train_steps=500. This design facilitates rapid experimentation and parameter tuning without the need to manually edit configuration files, making it suitable for automated hyperparameter search.

4. Multi-node Distributed Deployment: By using the Inference Controller and Training Controller with Ray scheduling, combined with Mooncake's hidden state storage, cross-node decoupled training can be achieved. The inference and training GPU pools can be independently scaled to meet varying workload demands. For specific configuration details, refer to the multi-node deployment guide in the official documentation. You will need to set up the Ray cluster and Mooncake storage nodes in advance.

5. Draft Architecture Switching: Modify the draft_model_type parameter in the configuration file to switch between six different architectures. For example, set it to DFly, MTP, Eagle3, etc. The framework will automatically load the corresponding training logic and network structure without requiring any changes to other code.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: If the team needs to deploy multiple speculative architectures for experimental comparison, or is dealing with heterogeneous workloads such as dialogue, code, and mathematics, AngelSpec's unified training pipeline and workload-specialized design can significantly reduce switching costs, making it suitable for research teams and production environments that demand extreme acceleration performance. If the project is already based on the SGLang inference stack and primarily uses the EAGLE-3 speculative architecture, SpecForge may offer tighter integration and more optimized training efficiency, with its memory optimization techniques being particularly beneficial for large-scale model training.

For teams with limited resources, the single-architecture training code of Medusa or EAGLE-3 is more lightweight and easier to deploy, although the acceleration effect may not be as strong as that of specialized frameworks, and it lacks the flexibility to switch between architectures. If rapid validation of speculative decoding effects is needed, the pre-trained weights of Medusa and EAGLE-3 can be used directly, while AngelSpec is more suitable for scenarios requiring long-term iteration and deep optimization.

6. Editor's Summary

AngelSpec demonstrates significant technical innovation in the field of speculative decoding training frameworks. Its design that unifies six draft architectures breaks the previous paradigm of training each architecture independently, greatly reducing experimental and deployment costs. The concept of workload specialization (MTP for dialogue, DFly for code and math) directly addresses the challenge of heterogeneous requests in real-world scenarios, avoiding the issue of a single draft model performing poorly across all contexts. The hybrid target-conditioned backbone and predecessor autoregressive head in the DFly architecture, along with the D-Cut dynamic validation mechanism, provide new insights into block-parallel generation and resource-adaptive allocation. The complete decoupling of inference and training through Mooncake/RDMA allows both to scale independently, offering high practical value in production environments and enabling flexible resource allocation based on actual workloads.

In terms of practical value, AngelSpec achieves a 1.98–2.40× end-to-end acceleration on the Hy3-A21B model, and improves throughput by 10.5–11.8% compared to DFlash, with solid data backing these claims. Its online real-world evaluation feature enables training effectiveness to be directly mapped to inference performance, reducing the risk of inconsistency between offline metrics and online results. The primary users include researchers in large model inference optimization, AI service deployment engineers, and enterprise technical teams with high requirements for inference efficiency. For research institutions, AngelSpec offers a rich combination of architectures and training objectives, facilitating exploration into the forefront of speculative decoding. For enterprises, its decoupled architecture and dynamic validation mechanism help maintain service quality while reducing computational costs.

In terms of future development potential, as community contributions increase, documentation and tutorials are expected to be further improved, and compatibility with multiple backends will also expand. If AngelSpec can support more domestic hardware and inference engines, its application scope will be further broadened. Overall, AngelSpec provides a systematic, flexible, and efficient solution for speculative decoding training, and has the potential to become a key infrastructure in this field.

7. Application Scenarios

• Large Model Online Inference Acceleration: Deploy speculative decoding drafts for MoE/dense models such as Hy3 and Qwen3 to reduce autoregressive decoding latency. Using DFly or MTP drafters, achieve near 2x end-to-end acceleration while maintaining generation quality, suitable for online services requiring low-latency responses, such as smart assistants and real-time translation.

• High-Concurrency Dialogue Services: Utilize the D-Cut dynamic verification mechanism to maintain the highest average throughput within a concurrency range of 4–64. For high-concurrency scenarios such as customer service assistants and intelligent Q&A, AngelSpec can adaptively adjust verification depth, converting additional computing resources into throughput improvements, effectively handling traffic peaks.

• Code Generation and Mathematical Reasoning: DFly block-parallel drafting excels at capturing long predictable spans in code completion and mathematical problem-solving. In structured output tasks such as IDE plugins and problem-solving engines, it can significantly reduce the number of autoregressive steps, improve generation speed, while maintaining the accuracy of code or mathematical expressions.

• Long Context Training and Inference: With Ulysses sequence parallelism supporting up to 128k context, it is suitable for long-sequence scenarios such as document analysis, codebase retrieval, and long document summarization. During the training phase, long-sequence data can be processed directly, and during inference, the draft model can also leverage long context information to enhance performance on complex tasks.

8. FAQ

Q: What draft architectures does AngelSpec support?
A: AngelSpec supports six architectures: DFly, DFlash, DFlare, Eagle3, DSpark, and MTP. These share a unified training pipeline, and switching between them only requires changing the draft_model_type parameter in the configuration file.

Q: How are training and inference decoupled in AngelSpec?
A: Through the Mooncake/RDMA transfer engine, inference workers stream hidden states to training workers, allowing both to operate on different GPU clusters and scale independently. This design avoids suboptimal issues caused by a unified parallel strategy.

Q: How many GPU resources are required to use AngelSpec?
A: For a single-node quick start, it is recommended to use 8 GPUs (4 for inference and 4 for training). For multi-node deployment, resources can be flexibly configured based on the workload. Consumer-grade GPUs (e.g., RTX 4090) may not be able to run the full pipeline, and it is advised to use A100 or higher-grade GPUs.

Q: What advantages does AngelSpec's acceptance rate alignment objective function offer?
A: AngelSpec directly optimizes the acceptance rate of the draft model during rejection sampling, rather than focusing solely on prediction accuracy. It provides multiple options, including CE, top-k KL, LK Loss, D-PACE, and TV loss, which can be freely combined to make training results more closely aligned with actual inference acceleration.

Q: How can the practical performance of the draft model be evaluated during training?
A: The framework supports online real-time evaluation, where actual speculative decoding is executed using inference engines like vLLM. It reports the average acceptance length and per-token acceptance rate in real time, ensuring that training results directly reflect inference performance and avoiding offline metric biases.

Q: What are the distinguishing features of AngelSpec compared to other speculative decoding frameworks (e.g., SpecForge)?
A: AngelSpec supports more draft architectures (6 total), fully decouples inference and training, and offers workload-specialized design (MTP for dialogue, DFly for code and math), along with unique features like D-Cut dynamic validation. These make it well-suited for heterogeneous workloads and multi-architecture research.

9. Project Links

• Project Website: https://angelspec.readthedocs.io/
• GitHub Repository: https://github.com/Tencent/AngelSpec
• HuggingFace Model Library: https://huggingface.co/collections/AngelSlim/angelspec
• arXiv Technical Paper: https://arxiv.org/pdf/2607.25852