HPC-Ops – Tencent Hunyuan’s Open-Source Industrial LLM Inference Operator Library

1. What Is HPC-Ops?

HPC-Ops is an industrial-grade high-performance large-model inference operator library open sourced by Tencent’s Hunyuan AI Infra team. It targets production-scale, high-concurrency inference—not isolated kernel tuning, but the full chain from Attention and MoE through GEMM, fused communication, and sampling. Deep software–hardware co-design on NVIDIA H20 and other SM90 GPUs delivers end-to-end QPM (queries per minute) gains of 30%+ via dynamic scheduling and operator fusion, outperforming vLLM, FlashInfer, SGLang, and other mainstream stacks on multiple core metrics. It already powers large-scale Hunyuan production inference—a battle-tested “heavy weapon.”

Technical positioning and domain: HPC at the intersection with deep learning inference—extreme LLM kernel optimization. It is not a general inference framework but a deeply integrable operator library meant to be the “nuclear engine” inside vLLM, SGLang, and similar upper layers. It addresses static optimization’s inability to adapt to dynamic online load.

Research background: Built by Tencent Hunyuan AI Infra from large-scale internal AI operations experience. Motivation: open solutions like FlashInfer hit limits under production dynamic load. HPC-Ops codifies industrial best practices into reusable, extensible operators.

Core value: Systemic and battle-tested. It optimizes the full chain—not single-kernel peaks—to fix tail latency “barrel effects” in online inference. Dynamic Attention scheduling, high-precision Router GEMM, and fused communication/compute offer practical answers to mixed long/short requests, MoE routing precision, and tensor-parallel bottlenecks.

Technical characteristics: Runtime dynamic optimization (tile-level scheduling, greedy bin packing, CTA load balance); full-chain fusion (MoE, comms, sampling as bubble-free pipelines); precision vs throughput balance (dual-BF16 emulating FP32 on Tensor Cores).

2. Key Features

• Dynamic Attention scheduling: Core innovation for mixed long/short online requests and tail latency. Replaces static split-kv with runtime tile-uniform splitting and greedy bin assignment across CTAs. Up to 2.95× speedup vs FlashInfer-class baselines on long contexts.

• Router GEMM (dual-BF16 emulating FP32): For MoE routing and other precision-sensitive paths. Offline split of FP32 weights into high/low BF16 matrices; two BF16 Tensor Core GEMMs plus FMA correction—near-BF16 speed at FP32 accuracy. Error ~1/36 of TF32; up to 3.22× vs cuBLAS FP32.

• FusedMoE (bubble-free pipeline): Five MoE stages—routing index, Gate-Up GEMM, activation quant, Down GEMM, Top-K aggregation—into one pipeline. Cuts memory traffic and launch overhead; 1.2×–1.6× vs vLLM/SGLang.

• Fused AllReduce+Norm: Tensor-parallel comm fused with residual add and RMSNorm as NVLink-native ops. Prefill uses NVSwitch multicast; decode uses Lamport P2P. Up to 1.68× vs NCCL/FlashInfer separate paths.

• Sampler mega-fusion: Repetition penalty, temperature, Top-K/Top-P, Softmax, random sampling—10+ kernels into 2 CUDA kernels, reducing launches and memory traffic at the inference tail.

• GroupGEMM FP8: Expert parallel and grouped expert matmul; per-tensor and block-wise FP8—efficient for DeepSeek-V3-class FP8-trained MoE models.

3. How to Use

1. Environment:

   • Hardware: NVIDIA SM90 (H20, H100, etc.); NVLink recommended for comm fusion.
   • Software: Linux (Ubuntu 22.04+); Python 3.8+; CUDA 12.8+ with nvcc on PATH; C++17 compiler (GCC 9+); PyTorch 2.0+ matched to CUDA.

2. Build and install:

   • git clone https://github.com/Tencent/hpc-ops.git
   • cd hpc-ops → python setup.py bdist_wheel
   • pip install dist/**.whl

3. Configuration and calls:

   • import hpc
   • Prepare torch.Tensor inputs per operator API (dtype/shape as documented).
   • e.g. hpc.ops.fused_attention(query, key, value, ...) or hpc.ops.fused_moe(gate, input, ...)

4. Best practices:

   • Integrate as engine: Replace Attention/MoE in vLLM/SGLang model code for end-to-end gains.
   • Benchmark: Run scripts under benchmark/ on target hardware vs FlashInfer.
   • Correctness: Run tests/ before integration to avoid numeric drift.

4. Pros and Cons

5. Comparison with Similar Tools

Selection advice:

• AI Infra teams chasing peak performance: Deploying huge MoE (DeepSeek-V3), mixed long/short requests, TP bottlenecks—HPC-Ops first. Higher integration cost; massive performance return.

• SMB / fast validation: vLLM for turnkey serving; swap operators later for HPC-Ops gains. FlashInfer if you only need Attention-level optimization as a component.

6. Editor’s Take

HPC-Ops marks inference optimization moving from “single-kernel peaks” to “system optimum.” FlashInfer-class work excels on one op; production bottlenecks are coordination and scheduling. HPC-Ops’s dynamic scheduling, pipeline refactor, and comm fusion deliver 1+1>2 gains.

Innovations are grounded: dynamic Attention is a scheduling paradigm shift; dual-BF16 Router GEMM shows deep Tensor Core and numerics craft. For large LLM serving, 30% QPM and tail-latency fixes directly affect SLA and unit economics. Validated at Hunyuan scale—not a lab demo.

Audience: AI Infra, systems, and senior algorithm engineers building inference clusters. App developers benefit indirectly via frameworks.

— Unmatched on dynamic load, full-chain performance, and high-precision routing for its target users. One star withheld only for early ecosystem/docs maturity.

7. Use Cases

• High-throughput online APIs: Dynamic Attention + fused Sampler for consistent UX under variable request lengths; higher QPM.

• MoE deployment: FusedMoE + GroupGEMM FP8 for DeepSeek-V3, Qwen3-235B-class models—less memory traffic and kernel launch overhead.

• Tensor-parallel multi-GPU: Fused AllReduce+Norm eases cross-GPU comm on 8×H20 nodes—scale model size with less perf cliff.

• Precision-sensitive inference: Finance, legal, scientific LLM apps—Router GEMM at FP32-grade routing without full FP32 GEMM cost.

8. FAQ

Q: Relationship to vLLM—use both?
A: HPC-Ops is operators; vLLM is a framework. Complementary—replace vLLM Attention/MoE implementations with HPC-Ops for end-to-end speedup.

Q: H20-only?
A: Optimized for SM90 (H100, H20). Older GPUs (A100) may not build or won’t hit advertised gains. Other vendors depend on community/Tencent roadmap.

Q: Non-MoE models—still worth it?
A: Yes—dynamic Attention, fused Sampler, Fused AllReduce+Norm help all Transformer inference, especially variable-length online traffic.

Q: Dual-BF16 memory overhead?
A: ~2× router weight storage (high/low BF16). Router weights are a small fraction of total params—acceptable vs precision/perf gains.

Q: Verify correctness when integrating?
A: Run tests/ against PyTorch/reference implementations before production rollout.

9. Project Links

• GitHub: https://github.com/Tencent/hpc-ops (source, install guide, tests, benchmarks)