Instella-MoE – AMD's Open-Source Mixture-of-Experts Language Model Series

1. What is Instella-MoE

Instella-MoE is AMD's open-source Mixture-of-Experts (MoE) language model series, with a total parameter count of 16B and an activated parameter count of 2.8B. It employs a 27-layer decoder architecture. The model is trained from scratch using AMD Instinct MI300X / MI325X GPUs and the ROCm software stack, offering six fully open-source versions: pre-training, mid-training, long-context base, SFT, DPO, and RL optimization. These versions cover the full pipeline from foundational models to conversational models, achieving efficient synergy between training and inference on AMD hardware.

Image source: Official article
Image source: official article

Technical positioning and domain: Instella-MoE belongs to the large language model category in the field of natural language processing, utilizing a Mixture-of-Experts (MoE) architecture. It is positioned as an efficient and scalable open-source foundational model. Its unique feature lies in the sparse activation mechanism, which activates only 2.8B parameters while maintaining a total parameter capacity of 16B, significantly reducing inference computational costs. It is suitable for various scenarios such as dialogue, long document understanding, and code generation. The model runs natively within the AMD ecosystem and is deeply integrated with the ROCm software stack, forming a closed-loop solution from training to inference.

Development background: This model was developed by AMD, leveraging its technical expertise in GPU hardware and the ROCm open-source software stack. AMD launched Instella-MoE to build a fully open-source model series that operates entirely within the AMD ecosystem, with deep integration with ROCm throughout the training and inference stages. This demonstrates the competitiveness of AMD hardware in large model training and inference, while also promoting the development of the open-source community on AMD platforms. The model utilizes the Primus training framework and the Miles RL framework, reflecting AMD's systematic investment in AI infrastructure.

Core value: Instella-MoE addresses the high computational costs and hardware lock-in issues in large model deployment. By providing full-stack open-source access (including weights from six stages and the code), developers can freely obtain and use the complete model series—from pre-training to reinforcement learning—on AMD GPUs, significantly lowering the barriers for research and commercial applications. Its efficient MoE architecture enables performance comparable to dense large models under limited resources, making it particularly suitable for enterprise private deployment and resource-constrained edge scenarios. Furthermore, the full-stack open-source nature provides a reproducible baseline for academic research, promoting further exploration of the MoE architecture.

Technical features: The model employs a gated multi-head latent attention mechanism to enhance its capability in long-sequence modeling. It also introduces the FarSkip-Collective technology to achieve communication and computation overlap under expert parallelism, resulting in a 12.7% improvement in pre-training speed. In the post-training phase, it adopts a pipeline of SFT → DPO → two-stage RL, with the RL stage utilizing the MOPD multi-teacher distillation strategy. This strategy applies token-level reverse KL divergence constraints to maintain general capabilities while enhancing instruction-following and reasoning quality. These technologies collectively establish Instella-MoE's dual advantages in training efficiency and model quality.

2. Key Features

• Multi-stage Model Series: Offers six versions—pre-training, mid-training, long-context Base, SFT, DPO, and RL optimization—covering the full pipeline from foundation to dialogue. Developers can select the appropriate weights based on the task stage, eliminating the need for training from scratch. The pre-training version is suitable for continued pre-training or feature extraction, while the SFT/DPO/Think versions can be directly used for dialogue and instruction tasks. The long-context Base version is specifically designed for handling long documents. Together, these versions form a complete technical stack.

• Efficient MoE Inference: With a total parameter count of 16B, but only activating 2.8B parameters per forward pass, the model dynamically routes tokens to different expert networks via a gating routing mechanism. It achieves performance close to that of larger dense models while maintaining low computational overhead, significantly improving inference speed. On AMD Instinct hardware, combined with the SGLang inference framework, TTFT (Time to First Token) can be reduced by as much as 39.2%, and memory usage is also greatly reduced.

• Long Context Support: The Base version supports a context length of 64K, using a gated multi-head latent attention mechanism to reduce KV cache overhead and improve inference efficiency for long sequences. This mechanism introduces gating control within the latent attention space, enhancing the model's ability to capture long-range dependencies. It is suitable for understanding and summarizing long documents such as legal contracts and research papers.

• Instruction-following Capability: The SFT version achieves instruction tracking through supervised fine-tuning, enabling accurate understanding and execution of user instructions. The model performs well on benchmarks such as Multi-IF, making it suitable for dialogue systems, task-oriented interactions, and API calling scenarios. It produces responses in standardized formats and rarely deviates from the given instructions.

• Chain-of-Thought Reasoning: The Think version further enhances response quality through reinforcement learning, allowing the model to generate step-by-step reasoning processes. Compared to the SFT version, the Think version shows significant improvements in tasks involving mathematics, logic, and complex problem solving. Its reasoning chains are clear and logically consistent, making it ideal for applications requiring interpretability.

• Native AMD Optimization: Deeply optimized for the ROCm ecosystem and the SGLang inference framework, achieving the best inference efficiency on AMD Instinct series hardware. SGLang has been specifically tuned for the architectural characteristics of AMD GPUs, supporting expert parallelism and continuous batching. It fully leverages the hardware interconnect bandwidth to deliver low-latency, high-throughput inference services.

• Deep Overlap of Communication and Computation: During the pre-training phase, the FarSkip-Collective technology is used to overlap communication and computation under expert parallelism, increasing pre-training speed by 12.7%. This technology hides communication latency, allowing GPUs to continue computing while waiting for data, significantly reducing training time. The SGLang inference framework also supports expert parallelism, achieving efficiency optimization for both training and inference.

3. How to Use

1. Obtain model weights: Visit the Hugging Face model repository (amd/instella-moe) or the GitHub repository (AMD-AGI/Instella-MoE) to download the required version. Currently, Base, SFT, DPO, and Think versions are available, with each version organized in a separate folder. The weight files are in safetensors format and can be directly loaded using the Transformers library. It is recommended to select the appropriate version based on your task requirements to avoid loading unnecessary large files.

2. Environment setup: Install the AMD ROCm driver (recommended ROCm version 6.0 or higher) and PyTorch for ROCm. AMD provides pre-configured Docker images (such as rocm/pytorch) to quickly set up the runtime environment. The hardware requirements include at least one AMD Instinct MI300X or MI325X GPU, with a recommended VRAM of 64GB or more. If using SGLang for inference, ensure that the system has ROCm-compatible GPU drivers and library files installed.

3. Load the model: Use the Hugging Face Transformers library to load the model weights. Example code is as follows: from transformers import AutoModelForCausalLM, AutoTokenizer; model = AutoModelForCausalLM.from_pretrained("amd/instella-moe-base", torch_dtype=torch.bfloat16, device_map="auto"). Alternatively, you can use the SGLang inference framework, which provides optimized loading interfaces for AMD GPUs, supporting automatic parallelism and memory management. It is recommended for use in production environments.

4. Perform inference: Choose the appropriate version based on the task type. The Base model is suitable for tasks such as text continuation and embeddings, and can be directly called using model.generate(). The SFT/DPO/Think versions are suitable for dialogue and instruction-following tasks and require constructing appropriate dialogue templates. When performing inference, it is recommended to set parameters such as max_new_tokens and temperature to achieve better generation quality. SGLang provides an OpenAI-compatible API interface, allowing inference via HTTP requests.

5. Fine-tuning and deployment: Continue training the Base model with your own data to support parameter-efficient fine-tuning methods such as LoRA and QLoRA. When fine-tuning, prepare an instruction dataset in the correct format and use the Primus training framework or Hugging Face Trainer. The fine-tuned model can be deployed as an API service using vLLM or SGLang, providing a standard OpenAI-compatible interface for easy integration into existing applications.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: If your hardware environment is AMD Instinct GPU and you are looking for a fully open-source model across the entire pipeline, Instella-MoE is the best choice. Its MoE architecture provides significant advantages in inference efficiency, making it suitable for enterprise private deployment and long document processing. The full open-source nature allows researchers and developers to deeply explore model training details and perform customized fine-tuning. If you are using NVIDIA GPUs and require a longer context length (e.g., 128K), Qwen3.5-4B-Base, as a dense model, is more mature in terms of usability and ecosystem compatibility, but it has fewer parameters and may not perform as well as MoE models on complex tasks. DeepSeek-V2-Lite also uses the MoE architecture, but with a lower activated parameter count (2.4B), and performs well on NVIDIA platforms, making it suitable for resource-sensitive applications. Its 128K context length provides a stronger advantage for long document processing. Mixtral 8x7B has a larger parameter count and stronger generation capabilities, but with higher inference costs, making it suitable for scenarios requiring high-quality outputs. However, note its 32K context length limitation.

For cross-platform deployment needs, it is recommended to prioritize Qwen3.5-4B-Base or DeepSeek-V2-Lite, as they have more complete toolchains and community support within the NVIDIA ecosystem. Instella-MoE is more suitable for teams with significant investments in AMD hardware, allowing them to fully leverage hardware performance while contributing to the development of the AMD open-source ecosystem.

6. Editor's Summary

Instella-MoE represents a significant move by AMD in the field of large language models, with technological innovations primarily manifesting in three areas: first, the use of a Mixture-of-Experts (MoE) architecture to achieve parameter-efficient activation, where only 2.8B parameters are activated out of a total of 16B, reaching a leading level in inference efficiency within its class. This is made possible by the carefully designed gating routing mechanism and expert parallelism strategy; second, a full-stack open-source strategy, where weights from all six stages—from pre-training to reinforcement learning—are made available, providing academic and industrial communities with a complete baseline for model research. This level of transparency is relatively rare among open-source large models; third, deep integration with the AMD ROCm ecosystem, leveraging optimizations from FarSkip-Collective and SGLang to maximize performance for both training and inference on AMD hardware. Specific manifestations include a 39.2% reduction in time-to-first-token (TTFT) and a 12.7% increase in pre-training speed. These innovations not only demonstrate AMD's capabilities in AI infrastructure but also provide the open-source community with an alternative to the NVIDIA ecosystem, promoting competition at the hardware level.

In terms of practical value, Instella-MoE is well-suited for enterprises and research institutions that have access to AMD GPU resources, especially in scenarios involving private deployment with high data compliance requirements, where its full-stack open-source nature and controllability offer clear advantages. The model's MoE architecture allows for deploying larger model capacities under the same computational power, delivering strong performance on tasks such as long document analysis, code assistance, and dialogue systems. The multi-stage optimization in the post-training phase ensures the model's competitiveness in instruction-following and inference quality, with the Think version showing particular promise in complex tasks. However, its ecosystem dependency is a double-edged sword—while it performs exceptionally well on AMD hardware, the cost of cross-platform migration is relatively high, which may limit its user base in the short term.

The target audience mainly includes: AI infrastructure teams using AMD Instinct GPUs, scholars researching MoE architectures, small and medium enterprises requiring low-cost deployment of large models, and developers interested in the full-stack training process of open-source large models. For deep users of the NVIDIA ecosystem, the appeal of Instella-MoE is limited, though it can serve as a reference for research. In terms of future development potential, as the ROCm ecosystem matures and AMD GPUs become more widespread, Instella-MoE is likely to become a benchmark model on the AMD platform, encouraging more hardware vendors to participate in the competition for open-source large models. Additionally, the full-stack open-source strategy may attract community contributions for fine-tuned versions and application plugins, fostering a positive ecological cycle. However, in the short term, its community size and third-party support will require time to build, and the completeness of documentation and toolchains will be critical.

7. Application Scenarios

• Enterprise Private Deployment: Deploy dialogue and inference services on local GPU clusters based on the AMD ROCm ecosystem to meet data compliance requirements. Instella-MoE's fully open-source pipeline allows enterprises to perform customized fine-tuning and build dedicated AI assistants. Using SGLang to deploy API services enables low-latency, high-throughput inference, supporting concurrent access from multiple users, suitable for customer service, internal knowledge base Q&A, and similar scenarios.

• Long Document Analysis: Leverage the 64K long context capability of the Base version to handle understanding, summarization, and Q&A tasks for long texts such as legal contracts, research papers, and financial reports. The gated multi-head latent attention mechanism effectively reduces memory usage during long-sequence inference, allowing hundreds of pages of documents to be analyzed on a single AMD GPU. Combined with Retrieval-Augmented Generation (RAG) technology, the accuracy and efficiency of long document processing can be further enhanced.

• Code-Aided Generation: The SFT/Think versions support instruction-following for programming tasks and can be used in IDE plugins or code review tools. The model is capable of generating code snippets, explaining code logic, and assisting with debugging, and supports multiple programming languages. For code completion and bug fixing tasks, the Think version's chain-of-thought capability can provide more reasonable solutions, making it suitable for integration into development workflows.

• Research and Education: The fully open-source pipeline is well-suited for academic research on MoE architectures, reproduction of training methods, and instruction in large model courses. Researchers can continue pre-training or perform fine-tuning experiments based on pre-trained weights to explore the effects of different training strategies. Educators can use the multi-stage weights to explain the model training process, demonstrating step-by-step from pre-training to reinforcement learning, helping students understand the entire process of large model development.

• Edge and Hybrid Cloud Inference: The low activation parameter count (2.8B) makes it suitable for efficient inference on resource-constrained edge devices or as a base model for hybrid cloud AI services. Through model quantization (e.g., INT8) and knowledge distillation, the model size can be further reduced, enabling real-time responses on the edge. In a hybrid cloud architecture, Instella-MoE can serve as the cloud base model, while a lightweight version is deployed on the edge, enabling flexible service scheduling.

8. FAQ

Q: What is the total parameter count and activated parameter count of Instella-MoE?
A: The total parameter count is 16B (16 billion), and the activated parameter count is 2.8B (2.8 billion). It uses a Mixture-of-Experts (MoE) architecture, activating only a portion of the parameters during each inference, thereby reducing computational costs while maintaining large capacity.

Q: Which hardware platforms does the model support?
A: The model natively supports AMD Instinct MI300X / MI325X GPUs based on the ROCm software stack. It cannot run directly on NVIDIA GPUs and requires additional adaptation (e.g., modifying code to be compatible with CUDA), but lacks official support and performance optimization.

Q: How to choose the model version (Base/SFT/DPO/Think)?
A: The Base version is suitable for basic tasks such as text generation and embeddings; the SFT version is instruction-tuned and ideal for dialogue and instruction-following; the DPO version further optimizes preference alignment to reduce harmful outputs; the Think version enhances reasoning capabilities through reinforcement learning, making it suitable for complex tasks such as mathematics and logic. It is recommended to start with the Base version for evaluation and select a higher version based on task requirements.

Q: What is the context length supported by Instella-MoE?
A: The Base version supports a context length of 64K. The context length for other versions (SFT/DPO/Think) is not explicitly stated and may follow the Base configuration or be shorter. It is advised to check the official documentation or model configuration files before specific use.

Q: Does the model support fine-tuning? How to fine-tune it?
A: Yes, it supports fine-tuning. You can use the Base model as a starting point and train it on your own data using parameter-efficient fine-tuning methods such as LoRA or QLoRA. It is recommended to use PyTorch or Primus frameworks compatible with AMD ROCm for training. When fine-tuning, prepare datasets in instruction format and adjust the learning rate and batch size to accommodate the memory characteristics of AMD GPUs.

Q: How does the model's performance compare to Qwen3.5-4B?
A: On average, Instella-MoE achieves 76.7% performance, while Qwen3.5-4B-Base achieves 79.5%. However, Instella-MoE activates fewer parameters (2.8B vs 4B), offering higher inference efficiency and making it more suitable for resource-constrained scenarios. Instella-MoE has unique advantages in long-context handling and MoE architecture research.

Q: Does the model support quantization?
A: The official release does not provide quantized versions, but users can apply quantization tools such as GPTQ or AWQ themselves. Since the model is based on ROCm, it is important to ensure that the quantization tools support AMD GPUs. The SGLang inference framework has preliminary support for INT8 quantization, and more quantization options may be available in the future.

Q: Does the model support multi-turn dialogue?
A: The SFT/DPO/Think versions support multi-turn dialogue, but it is necessary to correctly construct the dialogue templates (e.g., using ChatML format). The model maintains contextual coherence in long conversations, but due to context length limitations, it is recommended to control the number of dialogue rounds.

9. Project Links

• Project Website: https://rocm.blogs.amd.com/artificial-intelligence/instella-moe/README.html
• GitHub Repository: https://github.com/AMD-AGI/Instella-MoE
• HuggingFace Model Library: https://huggingface.co/collections/amd/instella-moe