Colibrì – Open-Source Lightweight Local Inference Engine with Hierarchical Storage

1. What is Colibrì

Colibrì is an open-source lightweight local inference engine designed to run ultra-large-scale MoE (Mixture of Experts) models on consumer-grade hardware. It can operate the flagship GLM-5.2 model with a total of 744B parameters on a regular PC with just 25GB of system memory, without requiring expensive server clusters. Its core innovation lies in the hierarchical storage architecture: keeping resident parameters in memory while storing massive routing expert parameters on NVMe SSDs and dynamically loading them on demand, thereby breaking the traditional constraint that large models must be fully loaded into high-speed VRAM or memory. The engine is written entirely in C, has zero external dependencies, produces a minimal compiled output, starts quickly, and includes intelligent caching and prefetching mechanisms, significantly reducing disk I/O latency.

Technical Positioning and Domain: Colibrì belongs to the natural language processing domain as an inference engine, focusing on the localized deployment and efficient inference of large models with MoE architecture. It fills the gap in running tera-parameter-scale MoE models on consumer-grade hardware, offering a differentiated competitive edge over engines like llama.cpp and vLLM, with its core advantage being support for hierarchical storage of extremely large-scale MoE models.

Development Background: This project was created by open-source community developer JustVugg, aiming to address the stringent memory and VRAM requirements of inference for ultra-large MoE models. With the emergence of tera-parameter MoE models such as GLM-5.2, traditional inference solutions require hundreds of GBs of VRAM or memory, making them inaccessible to ordinary users. Colibrì significantly lowers the hardware barrier through software-level storage hierarchy and streaming loading.

Core Value: The core value of Colibrì lies in enabling individual developers, small teams, and researchers to run large models that were previously only deployable in data centers, directly on their local hardware. It eliminates reliance on expensive cloud resources, supports offline inference, processing of privacy-sensitive data, and rapid validation of model performance, promoting the democratization of ultra-large models.

Technical Features: Hierarchical storage inference (three-level caching: VRAM/memory/disk), pure C zero-dependency engine, global int4 quantization and hand-written AVX2 kernels, Router-Lookahead asynchronous prefetching, MLA KV-Cache extreme compression (only 576 values per Token), and an integrated visual routing heatmap panel.

2. Key Features

• Hierarchical Inference Storage: Divides model parameters into resident parameters (approximately 9.9GB) and routing expert parameters (approximately 370GB), storing them separately in system memory and NVMe SSDs. During inference, expert parameters are dynamically streamed from the disk based on routing results, forming a three-tier storage pool consisting of GPU memory (optional), system memory, and NVMe SSD, thereby overcoming the capacity limitations of a single storage medium.

• Ultra-large Model Local Deployment: Enables the GLM-5.2 MoE flagship model with 744B parameters to run on consumer-grade PCs with only 25GB of system memory, without requiring a GPU or large server memory. This allows individual developers to locally experience the generative capabilities of a trillion-parameter model, without needing to apply for expensive cloud resources.

• int4 Quantization Compression: Utilizes global int4 quantization to compress the 744B model to approximately 380GB of disk usage. Resident parameters, after int4 quantization, occupy only about 9.9GB of memory, while routing expert parameters are stored in the NVMe SSD, approximately 370GB. Combined with a hand-written AVX2 integer dot product kernel, efficient inference is achieved without relying on BLAS.

• Pure C Streaming CPU Inference: The entire engine is written in pure C, without relying on external frameworks such as Python, PyTorch, or CUDA. The compiled output is a single executable file with minimal size, capable of running directly on any x86-64 Linux system that supports AVX2. During inference, expert parameters are dynamically scheduled through streaming loading, eliminating the need to fully load the model into memory.

• Web-based Visualization Control Panel: Features an integrated Chat interface, offering a conversational experience similar to ChatGPT. It also displays real-time inference performance metrics such as TTFT (Time to First Token), throughput, and queue status, enabling developers to monitor and optimize performance efficiently.

• Expert Routing Brain View: The Brain page presents the routing heatmap and storage hierarchy distribution in a matrix format of 76 layers × 256 experts. Hovering the mouse over each expert reveals its topic preferences, helping researchers understand the internal routing behavior of the MoE model and supporting educational and analytical purposes.

3. How to Use

1. Environment Preparation: You need an x86-64 computer running Linux or WSL2, with a CPU that supports the AVX2 instruction set. At least 16GB of system memory (recommended 25GB or more) and a local NVMe solid-state drive with approximately 400GB of available space. You must install the GCC compiler (used for source code compilation).

2. Obtain and Compile the Engine: Execute git clone (link to be updated after official release) to clone the repository. Navigate to the colibri/c directory and run ./setup.sh to complete one-click compilation. This script will automatically compile a pure C executable with zero dependencies named coli. Windows 11 users can install GCC via MinGW-w64 or MSYS2, and then run make glm.exe in the c/ directory to compile the native executable.

3. Download the Pre-converted Model: Download the int4 quantized model container from the Hugging Face repository jlnsrk/GLM-5.2-colibri-int4. The model file is approximately 380GB and should be stored on a local NVMe solid-state drive, for example, /nvme/glm52_i4. You can use git lfs or the Hugging Face CLI for downloading.

4. Start Local Chat or API Service: Set the model path and start the chat: COLI_MODEL=/nvme/glm52_i4 ./coli chat. The first load will take about 30 seconds, and the model will occupy approximately 9.9GB of resident memory, allowing you to interact with GLM-5.2 directly in the terminal. If you need an API service, run COLI_MODEL=/nvme/glm52_i4 ./coli serve --host 0.0.0.0 --port 8000. This service is compatible with the OpenAI API protocol, and any client that supports OpenAI can connect directly.

Notes: Ensure that the NVMe drive has sufficient contiguous space, as frequent random read/write operations can affect inference speed. It is recommended to place the model files in a dedicated partition or directory. The first run will establish a cache, and subsequent startups will be faster. Currently, only the GLM-5.2 model is supported; other models require community adaptation.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For users who wish to run a 100B-parameter MoE model on a personal computer without a GPU, Colibrì is the only viable option. Its hierarchical storage design allows the 744B model to run with just 25GB of memory, but it requires a high-speed NVMe drive and some patience (CPU inference speed is limited). If users have a large memory workstation and a consumer-grade GPU, KTransformers offers more flexible heterogeneous computing and fine-tuning capabilities, making it suitable for research and development scenarios. For high-concurrency inference services in production environments, vLLM excels in throughput and latency thanks to PagedAttention and GPU acceleration, though the hardware cost is higher. As a general-purpose CPU inference engine, llama.cpp supports the widest range of models and has a mature community ecosystem, but lacks hierarchical storage optimization for ultra-large MoE models and has higher memory requirements. Overall, Colibrì fills the gap for running 100B MoE models without a GPU, making it ideal for individual developers, educational research, and privacy-sensitive scenarios; if performance and ecosystem are priorities, KTransformers or vLLM would be more suitable.

6. Editor's Summary

Colibrì has made an innovative choice in its technical approach: implementing a zero-dependency engine purely in C, and leveraging hierarchical storage and streaming loading to make it possible for consumer-grade computers to run trillion-parameter MoE models. This approach directly challenges the conventional belief that "large models must rely on GPU clusters," offering a practical inference solution for hardware-constrained scenarios. Its Router-Lookahead prefetch strategy and learning caching mechanism effectively alleviate disk I/O bottlenecks. Real-world testing shows that the first token latency is within acceptable limits (approximately 30 seconds of loading, with smooth subsequent generation), proving that software optimization can partially compensate for hardware limitations. In terms of practical value, Colibrì is particularly suitable for individual developers conducting preliminary research on model performance, intelligent decision-making in offline environments, and local deployment with strict data privacy requirements. It lowers the barrier to access cutting-edge MoE models, enabling low-cost academic research and teaching demonstrations. However, the current version has notable limitations: it only supports the single model GLM-5.2, lacks GPU acceleration and fine-tuning capabilities, and has basic visualization panel features. Its community is still in the early stages, and its ecosystem is far less mature than that of llama.cpp or vLLM. Looking ahead, if Colibrì can expand its model support (e.g., compatibility with DeepSeek, Mixtral, and other mainstream MoE models) and introduce optional GPU acceleration modules, its application prospects will be even broader. For users seeking extreme lightweight and low hardware requirements, Colibrì is a noteworthy open-source project, but production-level deployment should still carefully evaluate its performance and stability.

7. Application Scenarios

• Personal Local Deployment of Flagship Large Models: Developers with no server budget can experience the full capabilities of the 744B parameter GLM-5.2 on a home computer, for tasks such as content generation, code assistance, and knowledge-based Q&A. Through Colibrì's Web control panel or API service, they can interact with the local model just as they would with a cloud-based model.

• Offline Inference on Edge Devices: Run ultra-large MoE models on memory-constrained industrial PCs or edge boxes to meet the needs of intelligent decision-making in offline environments, such as industrial text analysis and offline document processing. The low dependency of the pure C engine makes it easy to integrate into embedded systems.

• Rapid Model Effect Pre-research: Researchers can validate the actual performance of ultra-large models on specific business tasks locally, without needing to apply for expensive cloud resources. By adjusting prompts or conducting a few tests, they can quickly evaluate model suitability and reduce trial-and-error costs.

• Processing of Privacy-sensitive Data: In scenarios such as healthcare, finance, and government affairs, sensitive data must not leave the local network. Colibrì enables complete local inference of the model, ensuring data never leaves the device, meeting strict data privacy compliance requirements while still benefiting from the intelligent analysis capabilities of large models.

• MoE Mechanism Teaching and Research: Universities and research institutions can use Colibrì's Brain view to visually display expert routing heatmaps and storage hierarchies, enabling low-cost research into MoE's sparse activation, caching strategies, and quantized inference techniques. Students can deepen their understanding of large-scale Transformer architectures by running the 744B model in practice.

8. FAQ

Q: Why does Colibrì require an NVMe SSD? Can't a regular SSD be used instead?
A: During inference, Colibrì frequently streams routing expert parameters from the disk (each inference requires reading tens of MB to hundreds of MB). The low latency and high throughput of NVMe SSDs are critical to ensuring generation speed. Regular SATA SSDs have insufficient random read performance, which can significantly increase inference latency and even cause stuttering. It is recommended to use an NVMe SSD with PCIe 3.0 or higher.

Q: Which models does Colibrì currently support? Can it run Llama or DeepSeek?
A: Colibrì currently only officially supports the GLM-5.2 model (an int4 quantized version is already available). Its architecture is specifically designed for the GLM series MoE models and is not yet compatible with other models. The community may expand support in the future, but it is not recommended to attempt running non-official models at this time, as they may cause errors due to parameter structure mismatches.

Q: How fast is Colibrì's inference speed? Can it achieve real-time conversation?
A: With the recommended configuration (e.g., i7-13700 + NVMe SSD + 32GB RAM), the first token generation takes about 30 seconds (model loading), and subsequent generation speed is approximately 2–5 tokens per second, depending on the CPU single-core performance and disk speed. This is sufficient for non-real-time text generation and Q&A, but it cannot match the streaming conversation experience of cloud GPU services. For long document generation, the speed is still acceptable.

Q: Can Colibrì run on Windows?
A: Yes. Windows 11 users can run the Linux version through WSL2, or compile a native Windows executable using MinGW-w64 / MSYS2 after installing GCC. Specific steps: execute make glm.exe in the c/ directory to generate glm.exe. However, note that NVMe drivers and file system performance on Windows may be slightly lower than on Linux.

Q: Does Colibrì support fine-tuning? How can I customize GLM-5.2?
A: Colibrì currently only provides inference capabilities and does not support fine-tuning. If you want to fine-tune GLM-5.2, you must use other frameworks (such as LLaMA-Factory or KTransformers) for LoRA or full-parameter fine-tuning, and then convert the fine-tuned model into a format supported by Colibrì (currently, the conversion tool is not publicly available; please follow community updates).

Q: Does Colibrì require a GPU?
A: No. Colibrì is entirely based on CPU inference and does not rely on a GPU. It leverages the AVX2 instruction set to accelerate int4 dot product operations, so the CPU must support AVX2 (most x86-64 CPUs released after 2013 support AVX2). Future versions may include GPU support, but the current version runs purely on the CPU.

9. Project Links

• GitHub Repository: https://github.com/JustVugg/colibri (includes source code, compilation scripts, and usage documentation)
• Hugging Face Model Repository: https://huggingface.co/jlnsrk/GLM-5.2-colibri-int4 (model container with int4 quantization)