Paddler
Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.
Last verified:
What is Paddler?
Paddler is an open-source LLM load balancer and serving platform that enables teams to run inference and deploy large language models on their own infrastructure. Originally started by Intentee, it provides a self-contained solution for organizations that need privacy, reliability, cost control, and independence from closed-source model providers. The platform uses llama.cpp for inference and is built as a single binary with two main deployable components: the balancer and agents.
The platform features LLM-specific load balancing, dynamic agent addition for autoscaling integration, request buffering enabling scaling from zero hosts, dynamic model swapping, a built-in web admin panel for management/monitoring/testing, and built-in observability metrics. The balancer exposes an Inference service for tokens and embeddings (accessible via WebSocket or REST) and a Management service for setup configuration (accessible via REST). Agents register with the balancer via WebSocket and distribute requests to slots that maintain their own context and KV cache.
Paddler is designed for product teams needing reliable LLM inference and embeddings in their features, DevOps/LLMOps teams deploying models at scale, organizations handling sensitive data with high compliance requirements (medical, financial), and anyone wanting predictable AI costs instead of per-token pricing. It supports function calling, multi-agent fleet creation, and can run on a single local machine or go beyond a single device for distributed deployments.
The platform is self-contained in a single binary file, available through GitHub releases or building from source (MSRV 1.88.0). It's licensed under Apache-2.0, built primarily in Rust (65.9%) with TypeScript (24.9%), and has garnered 1.3k stars on GitHub. The web admin panel provides a GUI for monitoring your fleet, adding/updating models, customizing chat templates and inference parameters, and testing inference without command-line interaction.
Paddler pricing
Pricing model: Freemium
Paddler is completely free and open-source under the Apache-2.0 license. There are no paid tiers or subscription plans. The platform is available through GitHub releases as a downloadable binary or can be built from source. For enterprises and startups wanting to learn more about using Paddler at their company, they can reach out to [email protected] or join the Discord server, but the core platform itself has no cost.
Paddler pros
- Open-source with Apache-2.0 license
- Self-contained single binary deployment
- Built-in llama.cpp engine for inference
- LLM-specific load balancing
- Dynamic agent addition for autoscaling
- Request buffering enables scaling from zero
- Dynamic model swapping capability
- Built-in web admin panel for management
- Supports both WebSocket and REST for inference
- Built-in observability metrics
- Slots maintain own context and KV cache
- Supports function calling
- Multi-agent fleet creation support
- Works on local machine or distributed
- Predictable AI costs without per-token pricing
- Data privacy through self-hosting
- Extensive model support via llama.cpp
- Easy setup with command-line interface
- Agent naming for easy identification
Paddler cons
- Requires Rust 1.88.0 minimum for building from source
- Command-line tool requires terminal comfort
- Needs understanding of Git, system commands, networking
- Self-hosting requires infrastructure management
- Only two deployable components limit flexibility
- Management WebSocket reserved for internal use only
- Web admin panel is optional (requires extra flag)
- No pre-built packages beyond binary
- Smaller community (10 watchers, 64 forks)
Frequently asked questions about Paddler
What is Paddler?
Paddler is an open-source LLM load balancer and serving platform that lets teams run inference and deploy LLMs on their own infrastructure. It uses llama.cpp for inference and provides cost control, data privacy, reliability, and extensive model support.
Who made Paddler?
Paddler is an open-source project originally started by Intentee. It is maintained by the Intentee team and the open-source community, with 10 contributors on GitHub.
Who is Paddler for?
Paddler is for product teams needing reliable LLM inference and embeddings, DevOps/LLMOps teams deploying models at scale, organizations handling sensitive data with high compliance requirements (medical, financial), and anyone wanting predictable AI costs instead of per-token pricing from closed-source providers.
How does Paddler work?
Paddler consists of two main components: the balancer and agents. The balancer exposes an Inference service (receives prompts, sends back tokens) and Management service (manages setup, load balances requests). Agents register via WebSocket and distribute requests to slots that generate tokens using llama.cpp while maintaining their own context and KV cache.
What are the main components of Paddler?
Paddler has two deployable components: the balancer (exposes Inference service, Management service, and optional web admin panel) and agents (deployed on separate instances, distribute requests to slots). Slots are responsible for generating tokens and embeddings using the llama.cpp engine.
How do I install Paddler?
Paddler is self-contained in a single binary. You can either download the latest release from GitHub releases or build from source (MSRV 1.88.0). Once the binary is available in your system, use the paddler command to access all functionality.
How do I start a basic cluster?
Start the balancer with 'paddler balancer' command (optionally with --web-admin-panel-addr for the web interface), then start an agent with 'paddler agent --management-addr <address> --slots <number>'. You can run everything on your local machine without multiple devices.
What APIs does Paddler provide?
Paddler has two APIs: the Inference service API (for tokens and embeddings, accessible via WebSocket or REST) and the Management service API (for managing setup, getting registered agents, model metadata, specifying balancer state, accessible via REST).
What is the web admin panel used for?
The web admin panel lets you monitor your Paddler fleet, add and update models, customize chat templates and inference parameters, and test inference through a GUI. It's enabled by starting the balancer with --web-admin-panel-addr flag followed by address and port.
Why is it called Paddler?
The name originated from wanting to use the Raft consensus algorithm (Paddler because it paddles on a Raft), but that idea was dropped. The name stayed. Later, people started sending the 'that's a paddlin'' clip from The Simpsons, and the team embraced it.