Lorax
Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs
Last verified:
What is Lorax?
LoRAX (LoRA eXchange) is a multi-LoRA inference server framework that allows users to serve thousands of fine-tuned LLMs on a single GPU, dramatically reducing serving costs without compromising throughput or latency. It combines a pretrained base model with task-specific LoRA adapters that are dynamically loaded just-in-time per request, enabling efficient multi-adapter inference.
Key features include dynamic adapter loading from HuggingFace, Predibase, or any filesystem; heterogeneous continuous batching that packs requests for different adapters together; adapter exchange scheduling that asynchronously prefetches and offloads adapters between GPU and CPU memory; optimized inference with tensor parallelism, flash-attention, paged attention, SGMV kernels, and quantization; production readiness with Docker images, Kubernetes Helm charts, Prometheus metrics, and Open Telemetry tracing; OpenAI-compatible API supporting multi-turn chat; private adapter support with tenant isolation; and structured JSON output via constrained decoding.
LoRAX is designed for ML engineers, AI startups, and organizations that need to serve multiple fine-tuned LLMs cost-effectively. It supports base models including Llama, CodeLlama, Mistral, Zephyr, Mixtral, Gemma, Gemma2, Phi-3/Phi-2, Qwen2/Qwen, Command-R, DBRX, GPT2, Solar, and Bloom. Adapters must be trained using PEFT or Ludwig frameworks and can target various linear layers in the model architecture.
Lorax pricing
Pricing model: Freemium
Free for commercial use under the Apache 2.0 License. LoRAX is open-source with no paid tiers. You can self-host using prebuilt Docker images (ghcr.io/predibase/lorax:main or :latest) at no cost. Predibase also offers managed LoRAX endpoints including serverless endpoints for users who prefer not to self-host, which would involve Predibase pricing (separate from LoRAX itself).
Lorax pros
- Serves thousands of fine-tuned models on a single GPU
- Dramatically reduces serving costs compared to traditional methods
- Dynamic adapter loading without blocking concurrent requests
- Heterogeneous continuous batching maintains constant latency
- Adapter exchange scheduling optimizes aggregate throughput
- Supports tensor parallelism for large models
- Pre-compiled CUDA kernels including flash-attention and paged attention
- Base model quantization with bitsandbytes, GPT-Q, or AWQ
- Token streaming support for real-time responses
- Prebuilt Docker images avoid custom CUDA compilation
- Kubernetes Helm charts for production deployment
- OpenAI-compatible API as drop-in replacement for OpenAI SDK
- Multi-turn chat conversations with HuggingFace chat templates
- Structured JSON output with schema enforcement via Outlines
- Free for commercial use under Apache 2.0 License
- Private adapters with per-request tenant isolation
- Prometheus metrics and Open Telemetry distributed tracing
- Supports adapters from HuggingFace Hub, Predibase, or filesystem
- Merges adapters per request to create powerful ensembles
Lorax cons
- Requires Nvidia GPU with Ampere generation or above
- Requires CUDA 11.8 compatible device drivers
- Linux OS only (no Windows support)
- Dynamic adapter loading only works with supported architectures
- Structured JSON not supported in legacy completions v1 API
- Chat templates required for multi-turn conversations
- Base model must be loaded first before adapters
- Adapter must have tokenizer/chat template or falls back to base
- Quantization only applies to base model, not adapters
- Private HuggingFace models require HUGGING_FACE_HUB_TOKEN
Frequently asked questions about Lorax
What is LoRAX and what does it do?
LoRAX (LoRA eXchange) is a multi-LoRA inference server framework that allows users to serve thousands of fine-tuned LLMs on a single GPU. It dramatically reduces serving costs without compromising throughput or latency by using a base pretrained model shared across all adapters and dynamically loading task-specific LoRA adapters just-in-time per request.
How do I install and run LoRAX?
The recommended approach is using the prebuilt Docker image to avoid compiling custom CUDA kernels. Run: docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/predibase/lorax:main --model-id mistralai/Mistral-7B-Instruct-v0.1. Requirements include Nvidia GPU (Ampere or above), CUDA 11.8+, Linux OS, and Docker.
What base models does LoRAX support?
LoRAX supports Llama (including CodeLlama), Mistral (including Zephyr), Mixtral, Gemma, Gemma2, Phi-3/Phi-2, Qwen2/Qwen, Command-R, DBRX, GPT2, Solar, and Bloom. Base models can be loaded in fp16 or quantized with bitsandbytes, GPT-Q, or AWQ.
What adapters are supported in LoRAX?
LoRAX supports LoRA adapters trained using the PEFT and Ludwig libraries. Any linear layers in the model can be adapted via LoRA. Adapters can be loaded from HuggingFace Hub, Predibase, or any filesystem, and are loaded just-in-time without blocking concurrent requests.
How does heterogeneous continuous batching work?
Heterogeneous continuous batching packs requests for different adapters together into the same batch, keeping latency and throughput nearly constant regardless of the number of concurrent adapters. This is a key optimization that enables serving thousands of adapters efficiently on a single GPU.
Is LoRAX compatible with OpenAI's API?
Yes, LoRAX supports OpenAI Chat Completions v1 compatible endpoints that serve as a drop-in replacement for the OpenAI SDK. It supports multi-turn chat conversations while retaining dynamic adapter loading. Just replace the base_url with your LoRAX endpoint with /v1 appended and set api_key to EMPTY.
How does structured JSON output work in LoRAX?
LoRAX enforces valid JSON output adhering to a provided JSON schema through structured generation (constrained decoding) using the Outlines library. It modifies next token likelihoods to set invalid tokens to -infinity, guaranteeing only valid JSON tokens can be selected. This supports arbitrary JSON, JSON with schema, or plain text via response_format parameter.
Can I use private models with LoRAX?
Yes, you can access private base models from HuggingFace by setting the HUGGING_FACE_HUB_TOKEN environment variable. For private adapters, LoRAX supports per-request tenant isolation for production use cases requiring model privacy.
How do I deploy LoRAX in production?
LoRAX is production-ready with prebuilt Docker images, Helm charts for Kubernetes deployment with high availability and load balancing, Prometheus metrics for monitoring, and distributed tracing with Open Telemetry. Use helm install with the provided charts/lorax template for Kubernetes deployments.
What is the license for LoRAX?
LoRAX is free for commercial use under the Apache 2.0 License. It is built on top of HuggingFace's text-generation-inference (forked from v0.9.4, also Apache 2.0) and incorporates the SGMV kernel from Punica for multi-adapter inference speedup.