Mooncake
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
Last verified:
What is Mooncake?
Mooncake is a KVCache-centric disaggregated architecture for LLM serving, originally developed as the serving platform for Kimi, a leading LLM service provided by Moonshot AI. Both the Transfer Engine and Mooncake Store are now open-sourced. The system separates prefill and decoding clusters, leveraging underutilized CPU, DRAM, and SSD resources of GPU clusters to implement a disaggregated KVCache pool.
The core components include the Mooncake Transfer Engine, which enables zero-copy GPUDirect RDMA data transfers at full line rate, and Mooncake Store, a high-performance distributed key-value cache storage engine designed specifically for LLM inference scenarios. Mooncake Store provides object-level storage operations (Put, Get, Remove), supports multi-replica data storage, ensures strong consistency, and offers multi-layer storage support with RAM-to-SSD offloading.
Mooncake is designed for ML engineers, LLM service providers, and organizations running large-scale LLM inference workloads. It integrates with mainstream frameworks like vLLM, SGLang, and LMDeploy, supporting PD (prefill-decode) disaggregation. The system excels in long-context scenarios and has been deployed across thousands of nodes, processing over 100 billion tokens daily in production.
Key features include up to 525% throughput increase in simulated scenarios, 75% more request handling capacity under real workloads, Prometheus-compatible metrics endpoints, dynamic resource scaling, fault tolerance, and support for both TCP and RDMA protocols. The system received the Best Paper Award at FAST 2025.
Mooncake pricing
Pricing model: Freemium
Mooncake is fully open-source and free to use. Both the Transfer Engine and Mooncake Store are open-sourced with no paid tiers. The Python package is available via PyPI (pip install mooncake-transfer_engine). No licensing fees or subscription costs are required.
Mooncake pros
- Open-source Transfer Engine and Mooncake Store available
- Zero-copy GPUDirect RDMA transfers at full line rate
- Up to 525% throughput increase in long-context scenarios
- Handles 75% more requests under real workloads
- Processes over 100 billion tokens daily in production
- Multi-replica support alleviates access hotspots
- Strong consistency guarantees for Get operations
- Dynamic resource scaling with elastic node addition/removal
- Multi-layer storage with RAM-to-SSD offloading
- Built-in Prometheus-compatible metrics endpoints
- Integrates with vLLM, SGLang, and LMDeploy
- Supports PD disaggregation for LLM inference
- Robust fault tolerance for master and client failures
- High availability mode with etcd-based leader election
- Object-level Put/Get/Remove APIs for simplicity
- Best Paper Award winner at FAST 2025
Mooncake cons
- Single master node in default mode creates single point of failure
- Replication operates on best-effort basis, not guaranteed
- Requires separate metadata service (etcd/Redis/HTTP) for Transfer Engine
- CXL strategy only supports single-replica allocation
- Random allocation strategy has limited load balancing
- CachelibBufferAllocator deprecated for variable object sizes
- Master crash prevents serving until restored in default mode
- Metadata changes after last snapshot may be lost on master failure
Frequently asked questions about Mooncake
What is Mooncake and what does it do?
Mooncake is a KVCache-centric disaggregated architecture for LLM serving. It serves as the platform for Kimi (Moonshot AI's LLM service) and separates prefill and decoding clusters while leveraging underutilized CPU, DRAM, and SSD resources to create a distributed KVCache pool. The core components are the Transfer Engine for zero-copy data transfers and Mooncake Store for distributed KV cache storage.
Is Mooncake open source and free to use?
Yes, both the Transfer Engine and Mooncake Store are fully open-sourced and free to use. The package is available on PyPI via pip install mooncake-transfer-engine. There are no licensing fees or paid tiers.
What frameworks does Mooncake integrate with?
Mooncake officially integrates with vLLM (since December 2024), SGLang (since April 2025), LMDeploy (as PD disaggregation backend), LMCache (as remote connector), NIXL (as transfer backend), and supports vLLM Ascend for KV cache registration on Ascend NPUs.
How does Mooncake achieve high performance?
Mooncake uses zero-copy GPUDirect RDMA technology to transfer data directly from initiator's DRAM/VRAM to target's DRAM/SSD, saturating network bandwidth while keeping CPU overhead negligible. It supports striping and parallel I/O for large objects, fully utilizing multi-NIC aggregated bandwidth. The KVCache-centric scheduler balances throughput with latency SLOs.
What deployment modes does Mooncake Store support?
Mooncake Store supports three client usage modes: Embedded mode (runs in same process as LLM inference), Embedded mode with dummy-real clients (each rank holds dummy client, instance has real client), and Standalone store service. It also supports two availability modes: Default (single master) and High availability (multiple masters with etcd leader election).
What allocation strategies are available?
Mooncake Store provides three allocation strategies: random (default, maximum throughput for stable clusters), free_ratio_first (balanced utilization for dynamic scaling), and cxl (CXL memory hardware only, single-replica). Users select via --allocation_strategy flag when starting the master service.
How does Mooncake handle fault tolerance?
Failures of any number of master and client nodes won't result in incorrect data being read. As long as at least one master and one client remain operational, Mooncake Store continues functioning. In high availability mode, multiple masters coordinated through etcd automatically elect a new leader if the current one fails.
What HTTP endpoints does Mooncake Store provide?
Mooncake Store HTTP Service provides: /metrics (Prometheus-compatible metrics), /metrics/summary (human-readable KPIs), /query_key (replica information), /get_all_keys (list all stored keys), /get_all_segments (list mounted segments), /query_segment (segment capacity details), and /health (service availability check).
What performance improvements does Mooncake deliver?
Mooncake achieves up to 525% throughput increase in simulated long-context scenarios while adhering to SLOs. Under real workloads, it enables Kimi to handle 75% more requests (115% more on A800, 107% more on H800 clusters). SGLang integration achieved 7x faster weight updates for the 1T-parameter Kimi-K2 model (53s to 7.2s).
How do I get started with Mooncake?
Install via pip install mooncake-transfer-engine. For Transfer Engine, initialize with HOSTNAME, METADATA_SERVER (etcd/Redis/P2PHANDSHAKE), PROTOCOL (rdma/tcp), and DEVICE_NAME. For Mooncake Store, start mooncake_master with --enable_http_metadata_server=true, then create MooncakeDistributedStore instance and call setup() with node address, metadata server URL, segment size, local buffer size, protocol, and master service address.