FreeToken: Open-Source Edge-Side MoE Large Model Inference System Enabling Full-Throttle Operation on Consumer Hardware

1. What is FreeToken

FreeToken is an open-source edge-side MoE large model inference system jointly developed by UC Berkeley and MIT institutions, unifying a personal computer's CPU, memory, PCIe bus, and GPU into an elastic computing platform. Through core technologies including full-layer double buffering prefetch, bandwidth adaptive hybrid scheduling, Agent state reuse, and elastic memory hot expansion/contraction, the system enables full-throttle operation of ultra-large MoE models like Qwen3.6-35B and DeepSeek-V4-Flash 284B on single-card consumer hardware, significantly lowering the threshold for locally deploying cutting-edge large models.

Image source: Official article
Image source: official article

Technical positioning and domain: FreeToken belongs to the domain of large model inference optimization and edge-side deployment, focusing on efficient local execution of MoE (Mixture of Experts) architecture models. It achieves data-center-level model inference capabilities on consumer hardware, establishing a differentiated positioning compared to general-purpose inference frameworks like Ollama and llama.cpp.

Development background: The project was jointly developed by UC Berkeley and MIT institutions, addressing core challenges in edge-side MoE model deployment such as PCIe bandwidth bottlenecks, limited VRAM capacity, and redundant computation in multi-round Agent interactions. The system-level collaborative optimization solution stems from academic research in the intersection of systems and AI.

Core value: This tool resolves the practical issue of consumer hardware being unable to run ultra-large MoE models, pushing hardware throughput to theoretical limits through full-stack heterogeneous collaboration. By leveraging Agent state reuse, it reduces multi-round interaction latency, making local AI Coding Agents and large-scale model inference viable application solutions.

Technical features: Full-layer double buffering prefetch eliminates I/O wait bubbles, bandwidth adaptive scheduling dynamically distributes computational load, elastic memory hot expansion/contraction ensures service stability, and Agent state reuse minimizes redundant computation overhead. These four core technologies form a complete edge-side inference optimization system.

2. Key Features

• End-to-end Full-Power Inference on Edge Devices: Supports full-power operation of ultra-large MoE models like Qwen3.6-35B and DeepSeek-V4-Flash 284B on consumer-grade single GPUs such as RTX 4060/5090. The system unifies CPU, memory, and GPU scheduling, enabling local inference for large-parameter models without requiring data-center-level hardware, breaking through computational barriers between consumer devices and cutting-edge models.

• Full-Layer Dual Buffer Prefetching: While the GPU computes the current layer, the background prefetches the next layer's Expert weights via PCIe streaming, achieving complete overlap between computation and data transfer. This eliminates I/O waiting bubbles entirely, ensuring the Prefill process doesn't experience severe blocking from layer-by-layer Expert retrieval from system memory, significantly boosting long Prompt handling throughput.

• Bandwidth-Adaptive Scheduling: Dynamically detects actual PCIe bandwidth and CPU instantaneous compute power during runtime, calculating split ratios in real time. Experts not hit in GPU cache are intelligently split according to real-time bandwidth - some transferred via PCIe to GPU, others computed in parallel on CPU - pushing hardware throughput to the theoretical maximum for this topology.

• Agent State Reuse: Lightweight checkpoints are established at special Token boundaries. When context is edited, only the incremental Prefill of newly added suffixes is required from the most recent valid anchor point, without recalculating the full history. This drastically reduces redundant computation overhead in multi-round interactions for Coding Agents and tool calling scenarios.

• Elastic GPU Memory Hot Scaling: When background processes preempt GPU memory, the system seamlessly contracts GPU Cache without restarting the Serving service, offloading more uncached Experts to CPU computation. This achieves smooth degradation with zero downtime and avoids triggering CUDA Out of Memory (OOM) crashes, ensuring continuous stable inference service.

• Ultra-Low Latency: First Token latency reduced by 42–58%, Agent multi-round interaction TTFT (Time to First Token) reduced by 65–80%. Overall inference speed is 2–4× faster than Ollama and 1.46× faster than llama.cpp. Real-world testing shows Qwen3.6-35B running on RTX 4060 achieves 39.3 tok/s.

3. How to Use

1. Desktop App Installation: Visit the FreeToken official website (Link to be updated after official release)

2. Command Line Quick Installation: Execute uv pip install "freetoken[accel]" via command line to complete CLI tool installation in one step. This method leverages Python's ecosystem for dependency management, facilitating integration of FreeToken into existing workflows or automation scripts for developers.

3. Model Loading and Automatic Scheduling: After loading supported MoE model weights, the system automatically performs CPU-GPU heterogeneous scheduling and adaptive bandwidth configuration. Users need not manually adjust parameters - the system continuously probes hardware status during runtime and dynamically optimizes traffic distribution strategies.

4. Hardware Configuration Recommendations: When running ultra-large models like DeepSeek-V4-Flash, it's recommended to configure 32GB GPU memory paired with 192GB+ system memory to ensure sufficient space for Expert weight loading and GPU caching. For medium-scale models like Qwen3.6-35B, RTX 4060-level GPU cards meet requirements.

5. Elastic GPU Memory Management: If background applications (games, 3D rendering, etc.) preempt GPU memory during usage, FreeToken will automatically execute hot scaling operations, seamlessly contracting GPU cache and transferring computation to CPU - requiring no manual intervention or service restarts.

Important Notes: During first-time execution of ultra-large models, the system needs to pre-load Expert weights into system memory, which may take longer than usual. It's recommended to close unnecessary background applications to free up memory bandwidth, ensuring accurate PCIe bandwidth detection results and achieving optimal scheduling performance.

4. Pros and Cons Analysis

5. Comparison with Similar Tools

Selection Recommendations: For users aiming to run ultra-large MoE models on consumer-grade hardware while pursuing peak inference performance, FreeToken is currently the preferred edge-side inference solution, particularly in scenarios requiring local execution of Coding Agents or multi-turn dialogue services where its state reuse and elastic memory mechanisms deliver stable, smooth experiences. For general-purpose local model inference needs, Ollama remains the most accessible option due to its simple command-line interface and extensive model library.

When dealing with legacy hardware or CPU-only environments, llama.cpp's lightweight characteristics offer better compatibility, though its performance in MoE model and Agent scenarios lags significantly behind FreeToken. In data center or server-side high-concurrency contexts, vLLM's throughput optimization and production-grade stability represent more mature choices, but vLLM is not designed for consumer-grade edge deployment. These two solutions complement each other in application scenarios rather than directly competing.

6. Editor's Summary

FreeToken demonstrates a system-level innovative approach in the field of on-device large model inference. Unlike frameworks such as Ollama and llama.cpp, which focus on model loading and basic inference, FreeToken expands optimization perspectives to full-stack collaboration across CPU, memory, PCIe, and GPU. By implementing cross-layer dual-buffer prefetching and bandwidth-adaptive scheduling, it resolves I/O bottlenecks for MoE models running on consumer-grade hardware. The Agent state reuse mechanism specifically optimizes for typical workloads involving frequent context editing in AI Agent applications, with data showing a 65–80% reduction in TTFT validating its practical effectiveness. Its elastic GPU memory hot-scaling addresses the real-world challenge of limited GPU memory on consumer hardware from a system stability perspective.

From a practical value standpoint, FreeToken lowers the operational threshold for running ultra-large MoE models from data centers to personal computers, offering tangible benefits for AI developers, researchers, and Agent application enthusiasts. Jointly developed by institutions such as UC Berkeley and MIT, its academic foundation provides technical support for future iterations. While the current version still has room for improvement in areas such as Dense model support, multi-GPU parallelism, and ecosystem development, its technical approach for on-device MoE inference presents significant reference value and warrants continued attention.

7. Application Scenarios

• Local AI Coding Agent: Run Claude Code, OpenClaw, and other programming agents locally on notebooks or gaming PCs, supporting multi-round tool calling and chain-of-thought iterations. Enable intelligent code completion and debugging without requiring an internet connection. The agent state reuse mechanism significantly reduces multi-round interaction latency, enhancing coding efficiency.

• Edge-side Ultra-large MoE Model Inference: Personal users can run state-of-the-art models like DeepSeek-V4-Flash (284B) and GLM-5.2 (753B) at full capacity on consumer-grade single GPUs such as RTX 4060/5090, eliminating reliance on data centers to achieve fully localized model usage and ensure data privacy.

• High-concurrency Multi-round Dialogue Services: For dialogue scenarios requiring frequent context modifications (e.g., customer service, knowledge Q&A), leverage Token boundary checkpoints and state reuse to avoid redundant full-history computations, significantly reducing multi-round response latency while improving service throughput and user experience.

• Elastic AI Assistant for Game/Work Parallelism: When running 3D rendering, games, or development tools in the background, FreeToken automatically shrinks GPU cache and offloads computation to the CPU, ensuring uninterrupted and crash-free AI inference services during multi-task parallel execution.

• Research and Prototype Validation on Edge Devices: Researchers can rapidly deploy and test the latest open-source MoE models on ordinary lab workstations without needing to apply for server resources, lowering the barrier to entry for large model research and accelerating experimental iteration and algorithm validation.

8. FAQ

Q: Which models does FreeToken support?
A: FreeToken primarily optimizes for MoE architecture models, enabling full-performance operation for over 20 MoE models including Qwen3.6-35B, DeepSeek-V4-Flash (284B), GLM-5.2 (753B), etc. For Dense model compatibility, please refer to the official repository's published information.

Q: What hardware configuration is required to run FreeToken?
A: For Qwen3.6-35B, we recommend RTX 4060 or higher GPUs; for ultra-large models like DeepSeek-V4-Flash, 32GB VRAM paired with 192GB+ system memory is advised. The system unifies CPU, memory, and GPU scheduling—higher hardware specifications yield better inference performance.

Q: How does FreeToken differ from Ollama?
A: Ollama is a general-purpose local model execution framework primarily relying on GPU inference. FreeToken is a MoE model-specific edge-side inference system that unifies CPU, memory, PCIe, and GPU scheduling. It achieves 2–4× faster inference speeds than Ollama while supporting elastic VRAM scaling and Agent state reuse. The two systems exhibit significant architectural and functional differences.

Q: How to install FreeToken?
A: Two installation options are available:

• Download the Windows/Linux desktop application from the official website and launch services via GUI interface
• Execute uv pip install "freetoken[accel]" to install CLI tools, suitable for developers integrating into existing workflows or automation scripts.

Q: Will FreeToken crash when VRAM is insufficient?
A: No. FreeToken supports elastic VRAM scaling. When background applications preempt VRAM, the system seamlessly shrinks GPU Cache and offloads unhit Experts to CPU computation, achieving smooth degradation without service interruption or CUDA Out of Memory errors.

Q: What open-source license does FreeToken use?
A: The project has been open-sourced on GitHub. Please consult the official repository's LICENSE file for specific license type details. Authorization terms for academic and commercial use require review of the official documentation. We recommend confirming license specifics before deployment.

9. Project Links

1. Product Official Website: https://www.flashml.ai/
2. GitHub Repository: https://github.com/FlashML-org/FreeToken
3. arXiv Technical Paper: https://arxiv.org/pdf/2608.16157