Mellum2 – JetBrains' Open Mixture-of-Experts Model

1. What Is Mellum2

Mellum2 is JetBrains' open MoE LLM for software engineering. Total size 12B with sparse activation—only 2.5B parameters per token—delivering ~2.5B dense inference cost with leading code generation, intelligent routing, and agent collaboration. Optimized for AI coding assistants and automation, it offers Thinking and Non-thinking modes and supports fully local private deployment—a performance, cost, and security option for enterprise AI.

Technical positioning and domain: NLP and code generation for software engineering workflows—not a general LLM but an "expert" for IDE completion, debug assistants, and agent systems integrated with JetBrains products (IntelliJ IDEA, PyCharm, etc.).

Development background: JetBrains AI research with deep IDE ecosystem and user base—motivation: in-house, commercializable, product-integrated high-performance model vs. third-party APIs.

Core value: Resolves enterprise AI assistant tension between performance and cost. Dense 7–8B models are expensive/slow; 2.5B dense models are weak. Mellum2 MoE matches larger dense code ability at ~2.5B active compute with full on-prem deployment—no code leakage or API compliance risk.

Technical characteristics: 64 experts, 8 active per token; Grouped-Query Attention, sliding window attention, multi-token prediction heads; three-stage curriculum pretrain and Muon optimizer for code/math depth.

2. Key Features

• Intelligent code generation and completion: Real-time completion plus full functions, classes, and refactor suggestions from context—strict style and logic for mainstream conventions.

• Dual-mode inference: "Thinking" for hard logic, refactors, debug; "Non-thinking" for millisecond completion—depth vs. speed on demand.

• Workflow routing and summarization: Routes complex requests to sub-agents/tools; summarizes long docs, repos, or intermediate reasoning for quick orientation.

• Tool calling and agent collaboration: Native function calling for APIs, DBs, dev tools, shell—fits CI/CD automation and multi-agent pipelines.

• Local private deployment: Open weights on intranet or workstation—100% source privacy for finance, defense, and regulated sectors.

• Long context: Layer-Selective YaRN extends window to 128K tokens for large files, docs, and multi-turn analysis.

3. How to Use

1. Environment: NVIDIA A100 40/80GB recommended; RTX 3090/4090 usable but slow on 128K or Thinking. Linux Ubuntu 20.04+, CUDA 12.1+, PyTorch 2.0+.

2. Download weights: Hugging Face or GitHub Release in standard Transformers format.

3. Start inference: vLLM example:

   python -m vllm.entrypoints.openai.api_server \
       --model /path/to/mellum2 \
       --tensor-parallel-size 4 \
       --max-model-len 32768 \
       --gpu-memory-utilization 0.9
   

   Set tensor-parallel-size to GPU count.

4. Mode selection: API extra_body for Thinking:

   {
     "model": "mellum2",
     "messages": [{"role": "user", "content": "Write a Python function for the first N Fibonacci numbers."}],
     "extra_body": {"mode": "thinking"}
   }
   

   Default is Non-thinking.

5. Best practices: Production throughput—vLLM with --enable-prefix-caching. Fine-tune with LoRA for org code style. Pure NL tasks may trail general models.

4. Pros and Cons

5. Comparison with Similar Tools

Selection advice:

• Enterprise secure coding assistant: Mellum2—Apache 2.0, local, top code benchmarks vs. Qwen3.5-9B generalism.
• Multimodal general assistant: Qwen3.5-9B.
• Lightweight prototype: DeepSeek-Coder-V2-Lite weaker than Mellum2; CodeLlama aging on license and context.

6. Editor's Take

Mellum2 marks another vertical open-model milestone—MoE, curriculum pretrain, and engineering tricks put large-model capability in a 2.5B-active envelope. Dual-mode inference elegantly splits depth vs. latency.

It hits enterprise pain: cost, performance, security—for JetBrains, it's the engine of next-gen intelligent IDEs. Community MoE maturity and fine-tune tooling will widen adoption.

Audience: enterprise dev teams, AI tooling vendors, compliance-heavy orgs; individual learners exploring MoE coding assistants.

— Top open code/agent choice; −1 for young ecosystem and 12B total load on hardware.

7. Use Cases

• Enterprise IDE plugins: Local backend for IntelliJ/VS Code—completion, refactor, multi-line generation without cloud exfiltration.

• Automated code review agents: Thinking mode on PRs—bugs, logic, perf fixes with suggested patches.

• DevOps orchestration: Parse build logs, call Jira/Slack, open tickets, attempt fix patches in CI/CD.

• Internal knowledge assistants: Query private repos/docs—answers with code examples as 24/7 tech advisor.

8. FAQ

Q: Hardware requirements?
A: 24GB+ VRAM (A10G, A100 40GB). Consumer 3090/4090 with 4-bit GPTQ/AWQ; CPU possible but impractical.

Q: Apache 2.0 meaning?
A: Use, modify, sell derivatives commercially without paying or open-sourcing your changes—very permissive.

Q: Fine-tune on our codebase?
A: LoRA/QLoRA on instruction datasets (Alpaca-style) via SFTTrainer/Unsloth; often freeze experts, adapt attention/output LoRA.

Q: vs. GPT-4 or Claude 3.5?
A: Thinking mode nears or beats GPT-4 on some code benchmarks; wins on zero API cost, privacy, customization—loses on general knowledge, creative writing, vision.

Q: Native 128K context?
A: Extended via Layer-Selective YaRN from 4K after ~10.6T token pretrain—slight degradation possible on extreme length, usually acceptable.

9. Project Links

• GitHub: https://github.com/JetBrains/mellum2
• Hugging Face: https://huggingface.co/jetbrains/mellum2
• Technical blog: https://blog.jetbrains.com/blog/2024/10/15/mellum2-a-new-open-source-model-for-software-engineering/