Gpustack
A GPU cluster manager that configures and orchestrates inference engines like vLLM and SGLang for high-performance AI model deployment.
Last verified:
What is Gpustack?
GPUStack is an open-source GPU cluster manager designed for efficient AI model deployment and inference. It configures and orchestrates inference engines like vLLM, SGLang, and llama-box to deliver automated optimization that significantly improves performance out of the box. The tool enables teams to manage models, inference engines, and compute resources while collaborating seamlessly through a unified management interface.
Key features include broad hardware compatibility supporting NVIDIA GPUs, AMD GPUs, Apple Metal, Ascend NPU, and more across Macs, Windows PCs, and Linux servers. It supports diverse model types including LLMs (Qwen, Llama, DeepSeek, Mistral), vision models, diffusion models, embedding models, reranker models, and audio models. GPUStack offers multiple deployment modes (Throughput, Latency, Standard, Custom), real-time GPU metrics monitoring, OpenAI-compatible APIs, team management with user and API key management, failover and autoscaling, SSO integration, access control, and token quotas with rate limits.
GPUStack is ideal for developers, AI engineers, ML teams, and enterprises who need to deploy large language models at scale. It works for on-premise infrastructure, Kubernetes clusters, and multi-cloud environments (AWS, DigitalOcean, Alibaba Cloud). The tool is perfect for batch processing, high-volume APIs, real-time interactive applications, and organizations that need high availability with enterprise-grade security features.
The platform delivers up to 3x performance improvement with automated tuning, achieving +135% throughput on H200 GPUs for GLM-4.6 and -63% latency reduction on H100 GPUs for Qwen3-8B. It scales on demand across any infrastructure anywhere in the world, allowing you to easily add more GPUs or nodes to expand operations.
Gpustack pricing
Pricing model: Freemium
GPUStack is completely free and open-source under the Apache License 2.0. There is no paid tier or subscription model. The software can be self-hosted on your own GPU servers at no cost. You only pay for your own infrastructure (GPU hardware, cloud GPUs from AWS/DigitalOcean/Alibaba Cloud, or on-premise servers). API keys are generated by users within the UI for accessing the OpenAI-compatible endpoints. Enterprise features like SSO integration, access control, token quotas, and high availability are included free with the open-source software.
Gpustack pros
- Open-source with Apache 2.0 license
- Up to 3x performance improvement with automated optimization
- Broad hardware support: NVIDIA, AMD, Apple Metal, Ascend, Hygon DCU, Moore Threads
- Supports Linux, macOS, and Windows platforms
- Multiple inference backends: llama-box, vox-box, vLLM
- Distributed inference across single-node multi-GPU and multi-node setups
- OpenAI-compatible APIs for easy integration
- Real-time GPU metrics monitoring and historical trends
- Four deployment modes: Throughput, Latency, Standard, Custom
- Built-in playground for chatting with deployed models
- User and API key management with rate limits
- SSO integration for enterprise identity management
- Failover and autoscaling for high availability
- Works on Kubernetes, on-premise, and multi-cloud (AWS, DigitalOcean, Alibaba)
- Lightweight Python package with minimal dependencies
- Token usage tracking and organization-wide quota control
- Supports models from Hugging Face, ModelScope, Ollama, and local paths
- Pluggable backend for running state-of-the-art open-source models immediately
Gpustack cons
- Only Linux supported for GPUStack worker nodes (not macOS or Windows native)
- GPUStack server requires Docker installed
- NVIDIA drivers, Docker, and NVIDIA Container Toolkit are prerequisites
- First-time model deployment takes time to download model files and containers
- MThreads GPU, Iluvatar GPU, MetaX GPU, Cambricon MLU marked as experimental
- Intel oneAPI and Qualcomm AI Engine not yet supported (planned for future)
- Requires specific port ranges open for communication (TCP 40000-40063, 41000-41999)
- GPUStack not recommended for direct PyPi installation, Docker images preferred
Frequently asked questions about Gpustack
What is GPUStack?
GPUStack is an open-source GPU cluster manager for running AI models, including large language models (LLMs), diffusion models, embedding models, reranker models, and audio models. It configures and orchestrates inference engines like vLLM, SGLang, and llama-box to automate optimization and deliver up to 3x performance improvement out of the box.
What hardware does GPUStack support?
GPUStack supports NVIDIA GPU (CUDA 12.4+), AMD GPU (ROCm 6.4+), Apple Metal (M-series chips), Ascend NPU (910C, 910B, 310P3 series), Hygon DCU, Moore Threads MUSA, and experimental support for MThreads GPU, Iluvatar GPU, MetaX GPU, and Cambricon MLU. It runs on Apple Macs, Windows PCs, and Linux servers.
How do I install GPUStack?
On Linux or macOS, run: curl -sfL https://get.gpustack.ai | sh -s -. On Windows, run PowerShell as administrator and execute: Invoke-Expression (Invoke-WebRequest -Uri https://get.gpustack.ai -UseBasicParsing).Content. Alternatively, use Docker: sudo docker run -d --name gpustack --restart unless-stopped -p 80:80 --volume gpustack-data:/var/lib/gpustack gpustack/gpustack.
What models does GPUStack support?
GPUStack supports LLMs (Qwen, LLaMA, Mistral, DeepSeek, Phi, Gemma), Vision Language Models (Llama3.2-Vision, Pixtral, Qwen2.5-VL, LLaVA, InternVL2.5), Diffusion Models (Stable Diffusion, FLUX), Embedding Models (BGE, BCE, Jina), Reranker Models (BGE, BCE, Jina), and Audio Models (Whisper for speech-to-text, CosyVoice for text-to-speech). Models come from Hugging Face, ModelScope, Ollama Library, or local file paths.
Does GPUStack support distributed inference?
Yes, GPUStack supports both single-node multi-GPU and multi-node inference and serving. It requires Worker-to-Worker network connectivity for distributed inference across multiple workers. When using distributed vLLM, GPUStack assigns Ray service ports from the TCP 41000-41999 range.
What are the deployment modes in GPUStack?
GPUStack offers four deployment modes: Throughput Mode (optimized for high throughput under high request concurrency, perfect for batch processing and high-volume APIs), Latency Mode (optimized for low latency under low request concurrency, ideal for real-time interactive applications), Standard Mode (runs at full precision prioritizing compatibility for maximum model accuracy and stability), and Custom Mode (fully customizable optimization parameters tailored to specific requirements).
How do I access the GPUStack API?
Create an API key by hovering over the user avatar, navigating to API Keys page, clicking New API Key, filling in a name, and saving. Copy the generated key (visible only once). Use it with curl or the OpenAI Python library: export GPUSTACK_API_KEY=your_api_key, then call http://your_gpustack_server_url/v1-openai/chat/completions with Authorization: Bearer $GPUSTACK_API_KEY header.
Is GPUStack enterprise-ready?
Yes, GPUStack provides enterprise capabilities including SSO integration for seamless enterprise login and identity management, Access Control to manage who can access or modify models within your team, Token Quotas for organization-wide usage control with rate limits, and High Availability to ensure uninterrupted model service with failover during high demand or failures.
What cloud platforms does GPUStack support?
GPUStack supports multi-cloud deployment on AWS, DigitalOcean, Alibaba Cloud, and more. It also works on Kubernetes clusters with seamless orchestration and management, and on on-premise infrastructure where you leverage existing GPU servers with full control over your infrastructure. You can dynamically scale GPU resources on demand anytime, anywhere.
What are the network requirements for GPUStack?
GPUStack requires Server-to-Worker connectivity (server must reach workers to proxy inference requests), Worker-to-Server connectivity (workers must reach server to register and send updates), and Worker-to-Worker connectivity (for distributed inference). Default ports include TCP 80/443 for UI/API, TCP 10161 for server metrics, TCP 5432 for embedded Postgres, TCP 10150 for worker, TCP 40000-40063 for inference services, and TCP 41000-41999 for Ray services.