BitCPM-CANN – Edge LLM Open-Sourced by ModelBest and Tsinghua on Ascend

1. What Is BitCPM-CANN

BitCPM-CANN is China's first 1.58-bit ternary LLM trained end-to-end entirely on Huawei Ascend domestic compute, jointly released by ModelBest, Tsinghua University's NLP Lab, and OpenBMB. Using quantization-aware training (QAT), it ships in 0.5B, 1B, 3B, and 8B sizes—releasing ~6× VRAM at inference with 90.1%–97.2% capability retention. This milestone gives domestic NPUs a complete low-bit training stack, breaking the industry limitation that Ascend chips could only infer, not natively train, such models.

Technical positioning and domain: Extreme low-bit quantized LLMs on domestic Ascend—1.58-bit ternary weights trained and inferred natively on Ascend, filling the supply-side gap for low-bit models on domestic chips.

Development background: Co-developed by ModelBest, Tsinghua NLP Lab, and OpenBMB with prior GPU BitCPM work exceeding Microsoft BitNet training efficiency. Ascend adaptation driven by domestic compute autonomy and edge deployment memory/power constraints.

Core value: Enables native low-bit training on domestic chips—not just inference adaptation. ~6× VRAM release lets 8B models run on flagship phones, lowering hardware barriers. Scalable low-bit paradigm hedges HBM price/supply risk and foundations larger domestic training.

Technical characteristics: QAT learns ternary weights (-1, 0, +1) from training start—not post-training compression. MindSpeed × Megatron-LM on Ascend with env adaptation, 32K sequences, parallelism, and fused ops for train-infer integration. 1:1 benchmarks vs MiniCPM4 full precision on 11 tasks across sizes validate reproducibility and scale.

2. Key Features

• 1.58-bit ternary inference: 0.5B–8B sizes for efficient edge runs—8B on flagship phones with practical speed and quality for NLU/NLG.

• Full Ascend training loop: Data load through QAT to inference natively on Ascend—no foreign GPU pretrain required—true compute autonomy.

• Extreme VRAM savings: ~6× vs BF16—e.g., 8B on 16GB devices, lowering edge deployment barriers.

• Full-precision benchmarks: 1:1 vs same-size MiniCPM4 on 11 tasks—3B 97.2%, 8B 95.7%, 0.5B 90.1% retention.

• Open weights and training stack: Full series weights plus MindSpeed × Megatron-LM low-bit base for fine-tuning and research on Ascend.

• Scalable low-bit paradigm: 0.5B–8B engineering proof for future 50B–100B low-bit training reference.

3. How to Use

1. Environment: Ascend NPU (910B+ recommended), Linux (Ubuntu 20.04+), CANN 5.0+. Edge: flagship phones with 8GB+ RAM.

2. Download weights: Hugging Face 0.5B–8B series; use git lfs for full files, e.g., ./models/bitcpm-cann-8b/.

3. Training stack: Clone MindSpeed × Megatron-LM base; pip install -r requirements.txt; bash setup.sh for CANN and ops. Set export ASCEND_DEVICE_ID=0; tune config.yaml.

4. Edge inference example:

   from transformers import AutoModelForCausalLM, AutoTokenizer
   model = AutoModelForCausalLM.from_pretrained("./models/bitcpm-cann-8b", trust_remote_code=True)
   tokenizer = AutoTokenizer.from_pretrained("./models/bitcpm-cann-8b")
   inputs = tokenizer("Hello, please introduce yourself.", return_tensors="pt")
   outputs = model.generate(**inputs, max_length=100)
   print(tokenizer.decode(outputs[0]))
   

   Enable quantized inference on edge for lower memory.

5. Fine-tuning: train.py with model path, data, output dir, e.g., python train.py --model_path ./models/bitcpm-cann-8b --data_path ./data/finetune.json --output_dir ./output. Test small batches first.

6. Notes: Match NPU driver and CANN (5.1.0 recommended). Multi-card via --nproc_per_node. Reduce sequence length or streaming if OOM on edge.

4. Pros and Cons

5. Comparison with Similar Tools

Selection advice: Ascend or domestic closed-loop users—BitCPM-CANN is essentially the only end-to-end option with 6× VRAM gains. NVIDIA researchers use BitNet on GPU but lower training efficiency vs BitCPM. Snapdragon edge teams may prefer Qualcomm AI Hub for direct chip tooling but weaker quant/VRAM vs BitCPM-CANN.

6. Editor's Review

BitCPM-CANN marks a key step in domestic LLM supply chains—first complete 1.58-bit ternary training on Ascend, QAT from day one vs post-hoc compression, 3B at 97.2% retention proving low-bit need not sacrifice quality. 6× VRAM makes 8B on phones realistic; with MoE, 50B–100B on device is conceivable—revolutionary for phones, IoT, automotive.

Audience: chip vendors, domestic compute devs, edge app builders, quantization researchers. NVIDIA-only shops face migration friction.

Future: as Ascend matures, BitCPM-CANN may become a domestic training standard. Next priorities: long-context and better docs.

— breakthrough innovation and VRAM value; ecosystem and docs still catching up.

7. Use Cases

• Smartphone edge AI: Larger models in limited RAM—local assistants, translation, image captioning offline.

• Domestic chip ecosystem: Supply high-quality 2-bit-ready models for Snapdragon 8 Gen 4-class chips—hardware/software co-design.

• Enterprise edge: PCs, cars, IoT with low memory cost—hedging HBM supply risk; in-car voice assistants, smart home.

• Low-bit research: Reusable QAT infra on Ascend for finer quant (0.5-bit/binary) or vertical fine-tunes (medical, legal).

• Edge privacy: Local inference without cloud upload—hospital chart analysis on-prem, etc.

8. FAQ

Q: NVIDIA GPU support?
A: No—full stack is Ascend-native. Use GPU BitCPM series on GitHub for NVIDIA.

Q: Phone RAM for 8B?
A: ~2GB model memory at 1.58-bit plus system overhead—8GB+ recommended; current flagships qualify.

Q: How is retention calculated?
A: BitCPM-CANN scores divided by same-size MiniCPM4 full precision averaged over 11 tasks (MMLU, RACE, GSM8K, etc.).

Q: Train on Ascend 910A?
A: Yes, 910B preferred. 910A may need aggressive parallelism for 8B—start 0.5B/1B to validate env.

Q: Custom datasets on open base?
A: Yes—JSONL with instruction/response; respect 32K sequence limit in config.

Q: Inference frameworks?
A: Official Transformers Python API; community adapting vLLM/TGI; CANN APIs for edge acceleration.

9. Project Links

• Hugging Face collection: https://huggingface.co/collections/openbmb/bitcpm-cann
• GitHub (training base): https://github.com/OpenBMB/BitCPM
• Technical blog: https://openbmb.medium.com/