Lmcache
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
Last verified:
What is Lmcache?
LMCache is the first open-source Knowledge Delivery Network (KDN) designed to accelerate Large Language Model (LLM) applications up to 8x faster at 8x lower cost. It functions as an LLM serving engine extension that reduces Time to First Token (TTFT) and increases throughput, especially in long-context scenarios. By storing KV (Key-Value) caches of reusable texts across various locations including GPU, CPU DRAM, local disk, and even S3, LMCache reuses KV caches of any reused text (not necessarily prefix) in any serving engine instance, saving precious GPU cycles and reducing user response delay.
Key features include Prompt Caching for fast, uninterrupted interactions with AI chatbots and document processing tools by caching long conversational histories; Fast RAG that enhances speed and accuracy of RAG queries by dynamically combining stored KV caches from various text chunks; KV cache offloading and sharing across engines; disaggregated prefill-decode architecture; P2P KVCache sharing; and support for both prefix caching and non-prefix KV caches. The system uses novel compression techniques including CacheGen for KV cache compression and streaming, and CacheBlend for cached knowledge fusion.
LMCache is designed for enterprises and developers running LLM applications at scale, including those using multi-round question answering systems, retrieval-augmented generation (RAG) for enterprise search engines and AI-based document processing, AI chatbots, document processing tools, and production LLM deployments. It integrates seamlessly with popular LLM serving engines like vLLM and TGI (Text Generation Inference), and has officially become a PyTorch Ecosystem project.
Lmcache pricing
Pricing model: Freemium
LMCache is completely free and open-source under Apache License 2.0. There are no paid plans or licensing fees. The tool can be installed through pip install lmcache and comes with pre-built vLLM docker images for deployment. All features including KV cache offloading, disaggregated prefill, P2P KVCache sharing, and compression techniques are available without cost.
Lmcache pros
- Accelerates LLM applications up to 8x faster
- Reduces costs by up to 8x compared to without caching
- 3-10x delay savings when combined with vLLM
- Reduces Time to First Token (TTFT) significantly
- Increases throughput by 2-3x per GPU
- Open-source with Apache License 2.0
- Scales effortlessly without complex GPU request routing
- Novel compression techniques reduce storage costs
- Unique streaming and decompression methods minimize latency
- Seamless integration with vLLM and TGI engines
- Supports KV cache offloading to CPU, disk, and S3
- Enables prefix reuse across queries and engines
- Supports prefill-decode disaggregation for cross-engine cache transfer
- Enhances LLM inference quality through offline content upgrades
- Near-constant TTFT regardless of conversation length
- Part of the PyTorch Ecosystem
- Weekly community meetings with active developer support
Lmcache cons
- Documentation currently under construction with some sections incomplete
- Requires integration with compatible serving engines (vLLM, SGLang, TGI)
- Setup complexity for production-scale deployments
- Additional infrastructure needed for cache storage backends
- Learning curve for KV cache management concepts
- P2P KVCache sharing requires specific network configuration
- Controller configuration requires understanding of multiple ports
- Best performance requires carefully designed prompt structures for text reuse
Frequently asked questions about Lmcache
What is LMCache?
LMCache is the first open-source Knowledge Delivery Network (KDN) that accelerates LLM applications up to 8x faster at 8x lower cost. It is an LLM serving engine extension that reduces Time to First Token (TTFT) and increases throughput by storing and reusing KV (Key-Value) caches of reusable texts across GPU, CPU DRAM, local disk, and S3.
How much faster is LMCache?
LMCache accelerates LLM applications up to 8x faster overall. When combined with vLLM, developers achieve 3-10x delay savings and GPU cycle reduction. For prompt caching, it provides 8-10x faster response times. For Fast RAG applications, it delivers 4-10x faster response times.
What LLM engines does LMCache support?
LMCache seamlessly integrates with popular LLM serving engines including vLLM, SGLang, and TGI (Text Generation Inference). It has official vLLM integration with features like high performance CPU KVCache offloading, disaggregated prefill, and P2P KVCache sharing.
Is LMCache free to use?
Yes, LMCache is completely open-source under Apache License 2.0. There are no licensing fees or paid plans. It can be installed through pip install lmcache and is available with pre-built vLLM docker images at no cost.
What is KV cache and why does LMCache need it?
KV (Key-Value) cache is the attention memory computed during LLM inference. Traditional serving engines treat KV caches as disposable, recomputing them for every request. LMCache extracts and stores these KV caches out of GPU memory and shares them across engines and queries, allowing LLMs to prefill each text only once, which saves GPU cycles and reduces latency.
What use cases benefit most from LMCache?
LMCache delivers the most significant benefits in multi-round question answering (3-5x improvement), retrieval-augmented generation (RAG) for enterprise search and document processing (4-10x improvement), AI chatbots with long conversational histories, and any application with high text reuse across requests. It is particularly valuable for long-context scenarios.
How do I install LMCache?
LMCache can be installed through pip: pip install lmcache. It integrates with vLLM and comes with pre-built vLLM docker images for easier deployment. Detailed documentation is available at docs.lmcache.ai for both LMCache V1 and LMCache V0.
What storage backends does LMCache support?
LMCache stores KV caches across various locations including GPU memory, CPU DRAM (L1 cache), local disk (L2 cache), and even Amazon S3. It implements a complete memory hierarchy treating GPU, CPU, and SSD as a unified pool, similar to how modern CPUs treat L1, L2, L3 caches and RAM.
What is the difference between prefix caching and LMCache?
Traditional prefix caching only reuses cached data when requests share the same prefix. LMCache goes beyond this by reusing KV caches of ANY reused text, not necessarily prefix, in ANY serving engine instance. This makes KV cache composable like LEGO blocks, providing more flexibility and better cache utilization.
Is there community support for LMCache?
Yes, LMCache has an active community with weekly meetings hosted on Tuesdays at 9:00 AM PT and 6:30 PM PT (alternating weekly). The team welcomes contributions and collaborations. Community channels include LinkedIn, Twitter, YouTube, and an interest form for those wanting to connect. Documentation and developer guides are available online.