Llm D
Achieve state of the art inference performance with modern accelerators on Kubernetes
Last verified:
What is Llm D?
Llm D is a Kubernetes-native distributed LLM inference framework designed for production-grade deployment at scale. It provides intelligent scheduling, KV-cache optimization, and state-of-the-art performance across diverse hardware accelerators including NVIDIA GPUs, AMD GPUs, Google TPUs, and Intel XPUs. The framework serves as a
Llm D pricing
Pricing model: Freemium
llm-d is completely free and open source under the Apache-2.0 license. There are no paid plans, subscription tiers, or usage-based pricing. The project is a CNCF Sandbox project donated by IBM Research, Red Hat, and Google Cloud. Users only pay for their own infrastructure costs (Kubernetes cluster, GPUs/TPUs, storage, networking). The framework itself has no licensing fees, no per-request charges, and no hidden costs.
Llm D pros
- Kubernetes-native deployment with standard Kubernetes APIs
- Open source under Apache-2.0 license - completely free to use
- Supports diverse hardware: NVIDIA GPUs, AMD GPUs, Google TPUs, Intel XPUs
- Prefix-cache aware routing maximizes KV-cache hit rates
- 57x faster response times with precise prefix-cache scheduling
- Up to 3x improvement in output tokens per second
- 2x reduction in time-to-first-token (TTFT)
- Disaggregated serving separates prefill and decode phases
- Hierarchical KV offloading to CPU and SSD extends cache capacity
- Predicted latency-based routing using online XGBoost model
- SLO-aware autoscaling with HPA, KEDA, and Workload Variant Autoscaler
- Scale-to-zero capabilities for cost-efficient intermittent workloads
- OpenAI-compatible Batch Gateway for batch inference workloads
- LoRA adapter support with active-active high availability
- UCCL transport backend improves stability in congested networks
- Well-lit paths provide tested, benchmarked deployment blueprints
- Intelligent Endpoint Picker scores pods based on real-time metrics
- Wide expert parallelism scales to 2.2k tokens per second per H200 GPU
- vLLM and SGLang inference engine support
- Active community with Red Hat, Google, IBM, and Tesla contributors
Llm D cons
- Requires Kubernetes infrastructure - not for simple deployments
- Steep learning curve for teams new to Kubernetes
- Complex architecture with multiple components to manage
- Only works with vLLM or SGLang inference engines
- KV-cache sharing limited to same node without offloading
- Prefill/decode disaggregation adds operational complexity
- Requires GPU memory for model execution - CPU-only not supported
- Network congestion can impact distributed performance
- Benchmarking and tuning requires specialized knowledge
- Not suitable for edge devices with limited resources
- Slack/GitHub community support only - no enterprise SLA
- Documentation assumes Kubernetes and LLM serving expertise
- Scale-to-zero may introduce cold-start latency spikes
- Active-active HA adds resource overhead
- 预言 latency predictor requires online training overhead
Frequently asked questions about Llm D
What is llm-d?
llm-d is a Kubernetes-native distributed LLM inference framework that provides high-performance, scalable serving of large language models in production. It features intelligent scheduling, KV-cache optimization, and state-of-the-art performance across diverse hardware accelerators. The framework is built around three core components: the Router (intelligent entry point with LLM-aware load balancing), InferencePool (API defining groups of Model Server Pods), and Model Server (inference engine like vLLM or SGLang).
What hardware does llm-d support?
llm-d supports a comprehensive set of hardware accelerators including NVIDIA GPUs (H100, H200), AMD GPUs (MI300X), Google TPUs, and Intel XPUs. This hardware-agnostic design allows users to deploy across different accelerator families without vendor lock-in, with validated support for TCP and RDMA over RoCE for disaggregation.
How does prefix-cache aware routing work?
Prefix-cache aware routing intelligently routes requests to vLLM instances that already have the relevant KV-cache, maximizing cache hit rates. The llm-d Router's Endpoint Picker scores and selects model server pods based on real-time metrics and KV-cache affinity. Precise prefix-cache aware scheduling provides direct introspection into distributed vLLM caches, delivering 57x faster response times and double the throughput on identical hardware.
What is disaggregated serving in llm-d?
Disaggregated serving splits a single inference request into multiple phases (prefill and decode) handled by specialized workers. The llm-d Router orchestrates this flow by selecting both a prefill and decode endpoint and coordinating KV-cache transfer between them. This approach reduces time-to-first-token latency when traffic is unpredictable and is now supported on Google TPU and Intel XPU.
How does KV-cache offloading work?
llm-d provides hierarchical KV offloading with a tiered storage hierarchy including CPU memory and filesystem (SSD) tiers. The llm-d filesystem backend is a KV cache storage connector for vLLM that offloads KV blocks to shared storage. This significantly enlarges cache space, enables KV reuse across multiple replicas and nodes, preserves stable throughput at scale, and reduces tail latency as concurrency and context lengths grow.
Is llm-d free to use?
Yes, llm-d is completely free and open source under the Apache-2.0 license. There are no paid plans, subscription tiers, or usage-based pricing. The project was donated to the CNCF as a Sandbox project by IBM Research, Red Hat, and Google Cloud. Users only pay for their own infrastructure costs (Kubernetes cluster, GPUs/TPUs, storage, networking).
What is the Workload Variant Autoscaler?
The Workload Variant Autoscaler (WVA) provides globally optimized scaling that minimizes cost by routing traffic across different model variants (such as different hardware or quantization levels) while meeting latency targets. Unlike standard HPA/KEDA which uses queue depth metrics, WVA considers multiple deployment variants to optimize cost-efficiency while maintaining SLOs.
How do I get started with llm-d?
To get started with llm-d, join the Slack workspace at llm-d.slack.com, explore the code at the GitHub Organization, browse the well-lit paths documentation for tested deployment blueprints, and review the architecture guide. The project provides dedicated benchmark support allowing users to validate each well-lit path with a single command. Weekly standup meetings occur every Wednesday at 12:30pm ET for project updates and open discussion.
What batch inference capabilities does llm-d offer?
llm-d handles batch and offline inference workloads through two modules: the Batch Gateway provides an OpenAI-compatible Batch API for job management, while the Async Processor dispatches queued requests with flow-control gating. When composed, the Batch Gateway delegates dispatch to the Async Processor. These modules can be deployed independently or together for flexible batch processing.
How does llm-d improve cost efficiency?
llm-d improves cost efficiency through multiple mechanisms: KV-cache hit rates directly impact bottom line with 10x cost differences between cached and uncached tokens; intelligent routing maximizes GPU utilization allowing clusters to run hotter before scaling out; scale-to-zero capabilities reduce costs for intermittent workloads; the Workload Variant Autoscaler routes traffic across variants to minimize cost while meeting SLOs; and prefix-cache aware scheduling delivers order-of-magnitude gains reducing required hardware.