KVBoost
chunk-level KV cache reuse for HuggingFace, 5–48x faster TTFT
Last verified:
What is KVBoost?
KVBoost is an open-source inference engine for HuggingFace causal language models that makes LLM inference faster with chunk-level KV cache reuse. It achieves 5–48x faster time-to-first-token (TTFT) by keeping a content-addressed KV cache across requests, so prompts sharing chunk-aligned prefixes skip computation entirely. The tool requires no model changes and works directly with HuggingFace Transformers in drop-in mode.
Key features include: cross-request KV cache reuse with CacheBlend seam repair for identical output quality, custom FlashAttention-2 CUDA kernel reducing memory traffic from O(N²) to O(N), AWQ layer streaming enabling models larger than VRAM to run, speculative decoding stacked on streaming for 3x decode speedup, quantized KV cache at 8-bit, CPU paged decoding, async prefix-grouped batching, and an OpenAI-compatible API server. It supports tool calling for agentic coding workflows and long-context optimized scheduling.
KVBoost is designed for AI developers, ML engineers, and startups deploying LLMs who need faster inference and lower VRAM usage. It enables running 32B models on consumer hardware like an RTX 3060 12GB, supports agentic coding workflows locally, and works as a drop-in replacement for the OpenAI SDK, LangChain, LlamaIndex, Instructor, and the Vercel AI SDK. It's particularly useful for developers wanting to run larger models locally without expensive GPU hardware.
KVBoost pricing
Pricing model: Freemium
KVBoost is完全 open-source and free. It is available via pip install kvboost and on GitHub at pythongiant/KVBoost. There are no paid tiers or commercial licensing fees mentioned. The tool is released under open-source terms for personal and commercial use.
KVBoost pros
- 5–48x faster time-to-first-token (TTFT)
- Chunk-level KV cache reuse skips recomputation
- No model changes required – drop-in for HuggingFace
- Runs 32B models on consumer GPUs like RTX 3060 12GB
- AWQ layer streaming streams INT4 weights from host RAM
- Custom FlashAttention-2 CUDA kernel reduces HBM traffic O(N²) to O(N)
- CacheBlend seam repair produces identical output quality to full prefill
- Speculative decoding gives 3.07x decode speedup
- Quantized KV cache at 8-bit saves VRAM
- CPU paged decoding works when GPU is unavailable
- OpenAI-compatible API works with LangChain and LlamaIndex
- Tool calling support for agentic coding workflows
- Async prefix-grouped batching loads shared K/V once
- No measurable accuracy loss (99.2% WARM = 99.2% COLD)
- TTFT stays flat even with longer conversations
- Supports float16/bfloat16, head dims 64/96/128
- Works on Volta through Hopper GPUs (sm_70–sm_90)
- Partial resident execution with configurable resident layers
KVBoost cons
- Only supports AWQ quantization, not other quantization formats
- Requires CUDA-compatible GPU for best performance
- Speculative decoding only 0.5 tok/s on Turing GPUs
- Open-source with no official enterprise support
- No nested JSON support in related KvJson library
- May need compilation for custom FlashAttention kernel
- Limited to HuggingFace causal LMs, not all model types
- Cache warm-up needed for best performance on repeat prompts
- Complex configuration with many command-line arguments
Frequently asked questions about KVBoost
What is KVBoost and what problem does it solve?
KVBoost is an open-source inference engine for HuggingFace causal language models that makes LLM inference faster with chunk-level KV cache reuse. It solves the problem of slow time-to-first-token (TTFT) and high VRAM requirements by reusing cached K/V tensors across requests, streaming AWQ weights from host RAM, and using optimized CUDA kernels. It achieves 5–48x faster TTFT and enables running 32B models on consumer GPUs.
How does chunk-level KV cache reuse work?
Prompts are split into fixed-size chunks and content-addressed by hash. On a cache hit, stored K/V tensors are loaded instead of recomputed. CacheBlend seam repair selectively recomputes the ~15% most-deviated tokens at chunk boundaries, so stitched K/V produces output quality identical to a full prefill. This keeps TTFT flat even across long conversations with 99.2% accuracy retention.
What is AWQ layer streaming and how does it help?
AWQ layer streaming streams INT4 layer weights from pinned host RAM into two CUDA staging slots, overlapping PCIe transfer with compute. Embeddings, layernorms, and configurable head/tail decoder layers stay resident while the rest are DMA'd on demand. This allows running models bigger than VRAM—for example, Qwen2.5-32B-Instruct-AWQ (~19GB packed weights) runs on RTX 3060 12GB with peak VRAM of only 9.58GB.
What speedup does speculative decoding provide?
Speculative decoding stacked on streaming uses a small resident draft model to propose K tokens, which the streamed target verifies in a single multi-token forward. With Qwen2.5-32B target + 1.5B draft on RTX 3060 12GB with gamma=5, it achieves 3.07x decode speedup (0.91 tok/s to 2.79 tok/s), 40% acceptance rate, and averages 3.0 committed tokens per round. Greedy mode is bit-for-bit identical to non-speculative greedy.
What GPUs and hardware does KVBoost support?
KVBoost supports Volta through Hopper GPUs (sm_70–sm_90), including Ampere, Turing, and Hopper architectures. It works with float16/bfloat16 precision and head dims 64/96/128. On Ampere+ GPUs it achieves ~2–5 tok/s speculatively, while Turing GPUs get ~0.5 tok/s (GEMM-bound). It also supports CPU paged decoding when GPU is unavailable.
Is KVBoost compatible with existing AI frameworks?
Yes, KVBoost has an OpenAI-compatible server that is a drop-in for the OpenAI SDK, LangChain, LlamaIndex, Instructor, and the Vercel AI SDK. It supports async prefix-grouped batching where requests sharing a prompt prefix are dispatched as a single batch, loading shared K/V once and broadcasting zero-copy.
Does KVBoost have any accuracy loss from cache reuse?
No measurable accuracy loss occurs. Benchmarks show 99.2% WARM = 99.2% COLD, meaning cached responses maintain identical quality to full prefill. CacheBlend seam repair selectively recomputes the ~15% most-deviated tokens at chunk boundaries to ensure stitched K/V produces output quality identical to a full prefill.
Can I use KVBoost for agentic coding workflows?
Yes, KVBoost is agentic coding ready with tool calling support and enable-auto-tool-choice flag with tool-call-parser auto. It was built specifically to support agentic coding workflows locally, especially as alternatives like Claude get more expensive. It includes long-context optimized scheduling for extended coding sessions.
How do I install and run KVBoost?
Install via pip install kvboost. Run with: python -m kvboost.server --model Qwen/Qwen3-32B-AWQ --awq-streaming --streaming-mode partial_resident --resident-layers 10 --keep-first-k 8 --keep-last-k 8 --streaming-quant-kernel marlin --recompute-strategy cacheblend --chunk-size 64 --prefill-chunk-size 256 --kv-cache-bits 8 --max-cache-bytes 1500000000 --sink-tokens 32 --overlap-k 16 --max-batch-size 1 --max-queue-size 16 --workers 1 --enable-auto-tool-choice --tool-call-parser auto --host HOST --port PORT
What is the FlashAttention-2 kernel in KVBoost?
KVBoost includes a custom FlashAttention-2 CUDA kernel with tiled-softmax that reduces HBM memory traffic from O(N²) to O(N) during KV encoding. It supports float16/bfloat16, head dims 64/96/128, any sequence length, and causal masking. It covers Volta through Hopper (sm_70–sm_90) and falls back gracefully to torch.nn.functional.scaled_dot_product_attention if not compiled.