Vllm Ascend
Community maintained hardware plugin for vLLM on Ascend
Last verified:
What is Vllm Ascend?
vLLM Ascend (vllm-ascend) is a community-maintained hardware plugin for running vLLM seamlessly on Huawei's Ascend NPU. It is the recommended approach for supporting the Ascend backend within the vLLM community, providing a hardware-pluggable interface that decouples the integration of the Ascend NPU with vLLM following the RFC: Hardware pluggable principles.
By using the vLLM Ascend plugin, popular open-source models including Transformer-like models, Mixture-of-Experts (MoE), Embedding models, and Multi-modal LLMs can run seamlessly on the Ascend NPU. The plugin supports offline batch inference and can be deployed as a server implementing the OpenAI API protocol for model serving.
The tool is designed for developers and organizations who want to run large language model inference on Huawei Ascend NPUs (such as Atlas 800A2, Atlas A2 Training series, Atlas 800I A3, Atlas A3 Training series, and Atlas 300I Duo). It supports model fine-tuning, evaluation, reinforcement learning, and deployment scenarios.
Key features include support for tensor parallelism, pipeline parallelism, data parallelism, Expert Parallelism (EP), chunked prefill, prefix caching, asynchronous scheduling, and custom operator compilation. The plugin offers both pip installation and pre-built Docker images for easy deployment on Linux systems.
Vllm Ascend pricing
Pricing model: Freemium
vLLM Ascend is completely free and open source under Apache License 2.0. The plugin is hosted on GitHub and available via pip (vllm-ascend package). Pre-built Docker images are hosted on quay.io/ascend/vllm-ascend and are free to use with no per-user rate limits. No paid plans or subscription tiers exist - all features including model inference, serving, fine-tuning, and deployment are available at no cost.
Vllm Ascend pros
- Runs vLLM seamlessly on Ascend NPU hardware
- Community-maintained and officially recommended by vLLM community
- Supports Transformer-like, MoE, Embedding, and Multi-modal LLMs
- Hardware-pluggable interface decouples NPU integration from vLLM core
- Pre-built Docker images available for multiple hardware variants
- Supports OpenAI API protocol for model serving
- Enables offline batch inference capabilities
- Supports large-scale Expert Parallelism (EP) deployment
- Includes chunked prefill and prefix caching for efficiency
- Supports tensor parallelism and pipeline parallelism
- Offers pip installation and Docker deployment options
- Apache 2.0 open source license - free to use
- Active community with 171 contributors and weekly meetings
- Supports ModelScope mirror for faster model downloads in China
- Compatible with Atlas 800A2, A3, and 300I Duo hardware series
Vllm Ascend cons
- Only supports Linux operating system
- Requires specific CANN version (9.0.0 or >= 8.2.rc1)
- Requires specific torch-npu version matching vLLM version
- Python version restricted to >= 3.9 and < 3.12
- Performance not yet on par with Nvidia GPUs
- Multi-card tensor parallel support still under development
- Complex manual installation requiring firmware/driver setup
- Limited to Ascend NPU hardware - no GPU support
- First model download can take 3-5 minutes at 10 MB/s
- Some GPU-specific features reset or ignored on Ascend
- Requires HCCL_OP_EXPANSION_MODE=AIV for better shape support
- Community-maintained may have slower bug fixes than core vLLM
Frequently asked questions about Vllm Ascend
What is vLLM Ascend?
vLLM Ascend (vllm-ascend) is a community-maintained hardware plugin for running vLLM seamlessly on the Ascend NPU. It is the recommended approach for supporting the Ascend backend within the vLLM community, adhering to the RFC: Hardware pluggable principles that provide a hardware-pluggable interface decoupling Ascend NPU integration from vLLM.
What hardware does vLLM Ascend support?
vLLM Ascend supports Atlas 800I A2 Inference series, Atlas A2 Training series, Atlas 800I A3 Inference series, Atlas A3 Training series, and Atlas 300I Duo (experimental). These are Huawei Ascend NPU hardware devices running on Linux OS.
How do I install vLLM Ascend?
There are two installation methods: using pip (after manually preparing the environment or using a CANN image) or using Docker with pre-built images. For pip, install vllm==0.19.1 first, then install vllm-ascend==0.19.1rc1 with the Huawei extra index URL. For Docker, pull the pre-built image from quay.io/ascend/vllm-ascend and run with appropriate device mappings.
What Python versions are supported?
vLLM Ascend supports Python >= 3.9 and < 3.12. The installation requirements specify Python >= 3.10 and < 3.12 for manual installation, with the broader range of >= 3.9 and < 3.12 in the prerequisites.
What CANN version is required?
vLLM Ascend requires CANN == 9.0.0 for version 0.19.1, or CANN >= 8.2.rc1 as a general prerequisite. CANN is required for vllm-ascend and torch-npu. The CANN prebuilt image includes NNAL (Ascend Neural Network Acceleration Library) which provides libatb.so for advanced tensor operations.
Can I use vLLM Ascend for model fine-tuning?
Yes, vLLM Ascend assists Ascend users in enhancing their experience across fine-tuning, evaluation, reinforcement learning (RL), and deployment scenarios. User stories from LLaMA-Factory, verl, TRL, and GPUStack demonstrate these capabilities.
How do I deploy vLLM as a server with Ascend?
Deploy vLLM server by running 'vllm serve Qwen/Qwen3-0.6B &' command. The server implements the OpenAI API protocol and runs on http://0.0.0.0:8000. You can query models with 'curl http://localhost:8000/v1/models' and send completion requests to http://localhost:8000/v1/completions.
What models are supported on vLLM Ascend?
Popular open-source models including Transformer-like models, Mixture-of-Experts (MoE) models, Embedding models, and Multi-modal LLMs run seamlessly on Ascend NPU. Specific models like Qwen/Qwen2.5-0.5B-Instruct, Qwen/Qwen3-0.6B, and Qwen3.5 are confirmed to work.
Is there Docker support for vLLM Ascend?
Yes, vLLM Ascend offers pre-built Docker images for multiple hardware variants: vllm-ascend:v0.19.1rc1 for Atlas A2 (Ubuntu/openEuler), vllm-ascend:v0.19.1rc1-a3 for Atlas A3 (Ubuntu/openEuler), and vllm-ascend:v0.19.1rc1-310p for Atlas 300I (Ubuntu/openEuler). Images are hosted on quay.io/ascend/vllm-ascend.
How does vLLM Ascend performance compare to Nvidia GPUs?
User reports indicate that vLLM-Ascend performance is not yet on par with Nvidia GPUs. For example, a Qwen3-32B model achieved only 10 tokens/s on Ascend 910B3 NPUs compared to 25 tokens/s on Nvidia A10 GPUs. Multi-card scaling and tensor parallel support are still work in progress, though performance improvements are ongoing with each release.