Lmdeploy
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
Last verified:
What is Lmdeploy?
LMDeploy is a Python toolkit for compressing, deploying, and serving Large Language Models (LLMs) and Vision-Language Models (VLMs), developed by the MMRazor and MMDeploy teams. It provides two core inference engines: TurboMind Engine (written in C++ and CUDA for maximum performance optimization) and PyTorch Engine (pure Python for lower developer barriers and rapid experimentation). The toolkit supports deployment on both Linux and Windows platforms with NVIDIA GPUs ranging from Volta (V100) to Ada Lovelace (40 series), requiring minimum CUDA 11.3.
Key features include efficient inference delivering up to 1.8x higher request throughput than vLLM through persistent batch (continuous batching), blocked KV cache, dynamic split&fuse, tensor parallelism, and high-performance CUDA kernels. It offers effective quantization supporting weight-only and k/v quantization, with 4-bit inference performance 2.4x higher than FP16. The toolkit also provides an effortless distribution server for multi-model, multi-machine, multi-card inference services, and an interactive inference mode that caches attention k/v during multi-round dialogues to avoid repetitive processing of historical sessions.
LMDeploy supports an extensive range of models including Llama series (7B-70B), InternLM series, Qwen series (0.5B-110B), DeepSeek-V3/R1, Baichuan, Code Llama, Mistral, Mixtral, Gemma, Phi-3/4, and numerous VLMs like LLaVA, InternVL series, Qwen-VL, CogVLM, and MiniCPM-V. It supports OpenAI-compatible RESTful APIs (/v1/chat/completions, /v1/models, /v1/completions) and includes AWQ quantization, Automatic Prefix Caching, and KV Cache Quant capabilities.
The toolkit is designed for ML engineers, researchers, and developers who need to deploy LLMs and VLMs in production environments, particularly those working with NVIDIA GPUs who require high-performance inference. It supports integration with Docker, Kubernetes, ModelScope, and various cloud platforms.
Lmdeploy pricing
Pricing model: Freemium
LMDeploy is completely free and open-source under the Apache 2.0 license. There are no paid plans or tiers. The toolkit can be installed via pip (pip install lmdeploy) or built from source. Ready-to-use 4-bit models are available on HuggingFace Hub at no cost. Docker images are available at openmmlab/lmdeploy on Docker Hub for free.
Lmdeploy pros
- Delivers up to 1.8x higher request throughput than vLLM
- 4-bit inference performance is 2.4x higher than FP16
- TurboMind Engine written in C++ and CUDA for ultimate optimization
- PyTorch Engine in pure Python lowers developer barriers
- Supports persistent batch (continuous batching) for efficient inference
- Blocked KV cache reduces memory usage
- Dynamic split&fuse optimizes batch processing
- Tensor parallelism supports multi-GPU deployment
- Supports weight-only and k/v quantization
- AWQ quantization algorithm support
- Automatic Prefix Caching (APC) capability
- OpenAI-compatible RESTful APIs for easy integration
- Supports both Linux and Windows platforms
- Docker and Kubernetes deployment support
- Interactive inference mode caches dialogue history
- Supports DeepSeek V3, R1, and FP8 MoE models
- MXFP4 support on NVIDIA GPUs starting from V100
- Extensive model support including Llama, Qwen, InternLM, InternVL
- Ready-to-use 4-bit models available on HuggingFace Hub
- FlashAttention-2 and FlashMLA support
Lmdeploy cons
- Requires NVIDIA CUDA with minimum version 11.3
- Primary optimization focused on NVIDIA GPUs only
- Windows support limited to tensor parallelism of 1
- AMD GPU support requires ROCm Docker container setup
- Ascend/Cambricon/MACA support needs environment variable configuration
- 4-bit quantization may reduce model quality for small models
- Stop words support limited to characters encoding into single index
- Session length limitations can cause finish_reason: length errors
- OOM errors possible without adjusting cache_max_entry_count
- Not all models support structured output capability
Frequently asked questions about Lmdeploy
What is LMDeploy and what does it do?
LMDeploy is a toolkit for compressing, deploying, and serving Large Language Models (LLMs) and Vision-Language Models (VLMs). It provides two inference engines (TurboMind and PyTorch), supports quantization (including 4-bit AWQ), delivers up to 1.8x higher throughput than vLLM, and offers OpenAI-compatible RESTful APIs for model serving.
How do I install LMDeploy?
The recommended installation is using pip in a conda environment with Python 3.10-3.13: conda create -n lmdeploy python=12 -y && conda activate lmdeploy && pip install lmdeploy. The default prebuilt package since v0.3.0 is compiled on CUDA 12. For source installation: pip install git+https://github.com/InternLM/lmdeploy.git.
What GPUs does LMDeploy support?
LMDeploy supports NVIDIA GPUs including Volta (V100, sm70), Turing (20 series, T4, sm75), Ampere (30 series, A10, A16, A30, A100, sm80/sm86), Ada Lovelace (40 series, sm89), and RTX 50 series (requires CUDA 12.8). Minimum CUDA requirement is version 11.3.
What is the difference between TurboMind and PyTorch engines?
TurboMind Engine is developed in C++ and CUDA, striving for ultimate optimization of inference performance with features like Paged Attention, FlashAttention-2, and 4-bit inference. PyTorch Engine is developed purely in Python, aiming to decrease developer barriers and enable rapid experimentation with new features and technologies.
How do I start an OpenAI-compatible API server?
Use the CLI command: lmdeploy serve api_server <model_path> --server-port 23333. For example: lmdeploy serve api_server internlm/internlm2_5-7b-chat --server-port 23333. You can view parameters with lmdeploy serve api_server -h. The server will be accessible at http://0.0.0.0:23333 with Swagger UI.
What models does LMDeploy support?
LMDeploy supports extensive LLMs including Llama (7B-65B), Llama2/3/3.1/3.2, InternLM series, Qwen series (0.5B-110B), DeepSeek-V3/R1, Baichuan, Code Llama, Mistral, Mixtral, Gemma, Phi-3/4, GLM-4, and VLMs like LLaVA, InternVL series (1B-78B), Qwen-VL, CogVLM, MiniCPM-V, and many more.
What happens when I get finish_reason: length?
When you get finish_reason:length, it means the session is too long to be continued. The session length can be increased by passing --session_len parameter to api_server when launching the service.
How do I handle OOM errors on the server side?
When OOM (Out of Memory) appears at the server side, reduce the cache_max_entry_count of backend_config when launching the service. This adjusts the GPU memory ratio allocated for k/v cache.
Does LMDeploy support quantization?
Yes, LMDeploy supports weight-only quantization and k/v quantization. It supports 4-bit inference using AWQ algorithm, with 4-bit performance being 2.4x higher than FP16. TurboMind also supports online int8/int4 KV cache quantization for all supported devices. Quantization quality has been confirmed via OpenCompass evaluation.
How do I integrate LMDeploy with OpenAI client?
Install the openai package (pip install openai), then create a client with base_url pointing to your LMDeploy server: client = OpenAI(api_key='YOUR_API_KEY', base_url='http://0.0.0.0:23333/v1'). Use client.chat.completions.create() with model name and messages, just like calling OpenAI API. LMDeploy's RESTful API is compatible with /v1/chat/completions, /v1/models, and /v1/completions endpoints.