Litellm
Python SDK, Proxy Server (AI Gateway) to call 100+ LLM APIs in OpenAI (or native) format, with cost tracking, guardrails, loadbalancing and logging. [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, VLLM, NVIDIA NIM]
Last verified:
What is Litellm?
LiteLLM is an open-source Python library and self-hosted LLM gateway (Proxy Server) that provides a unified OpenAI-compatible interface to call 100+ LLMs from providers including OpenAI, Anthropic, Vertex AI, AWS Bedrock, Azure OpenAI, Google Gemini, Ollama, and many more. The library lets developers use a single completion() function regardless of which LLM provider they're using, with consistent output formatting and OpenAI exception mapping.
Key features include built-in retry and fallback logic via the Router, load balancing across multiple deployments, virtual keys with per-key/team/user budget tracking, RPM/TPM rate limiting, LLM guardrails for content filtering, and observability integrations with Langfuse, MLflow, Helicone, LangSmith, Arize Phoenix, and OpenTelemetry. The Proxy Server acts as a drop-in replacement for OpenAI's API, requiring no code changes to existing clients.
LiteLLM is designed for developers building AI applications who need to access multiple LLM providers without learning different APIs for each one. It's particularly useful for platform teams managing LLM access across many developers and projects, teams needing spend tracking and cost optimization, and organizations requiring self-hosted infrastructure for data residency or compliance requirements.
The library supports streaming responses, structured JSON output, function calling, prompt caching, and the OpenAI /responses API. It maps all provider-specific errors to OpenAI exception types, making error handling consistent across providers.
Litellm pricing
Pricing model: Freemium
LiteLLM is open-source software with an MIT license, so the software itself is $0. The free tier includes 100+ LLM provider integrations, Langfuse/Arize Phoenix/Langsmith/OTEL logging, virtual keys with budgets and teams, load balancing with RPM/TPM limits, and LLM guardrails. You only pay for your infrastructure (servers, databases, monitoring) which typically costs $200-$500/month for moderate traffic, plus LLM provider API costs at their standard rates. Enterprise Basic is $250/month adding Prometheus metrics, JWT auth, SSO, and audit logs. Enterprise Premium is $30,000/year ($2,500/month) adding priority support, dedicated account management, custom feature development, and compliance assistance. A 30-day trial is available for enterprise features.
Litellm pros
- Unified OpenAI-compatible API for 100+ LLM providers
- Single completion() interface works across all providers
- Consistent output format regardless of provider
- Built-in retry and fallback logic via Router
- Load balancing across multiple model deployments
- Virtual keys with per-key/team/user budget tracking
- RPM and TPM rate limiting out of the box
- LLM guardrails for content filtering
- Observability integrations (Langfuse, MLflow, Helicone, LangSmith)
- Self-hosted gateway gives complete infrastructure control
- Open-source with MIT license, no software licensing fees
- Maps all provider errors to OpenAI exception types
- Supports streaming responses and structured JSON output
- Function calling support across providers
- Prompt caching capabilities for cost reduction
Litellm cons
- Slow cold start: 3-4 second import time due to 1,200+ line init file
- Performance degrades significantly above 500 requests per second
- P99 latency can reach 90+ seconds under heavy load
- Memory usage grows over time (300-400MB baseline)
- Database slows down after ~1 million stored logs
- Main.py file is 5,500+ lines, making debugging difficult
- Uses global variables for configuration, limiting multi-tenant apps
- Frequent breaking changes with multiple releases per day
- Documentation often doesn't match actual code behavior
- No native Model Context Protocol (MCP) support for agentic AI
- No SLA guarantees for community version
- Requires dedicated DevOps expertise for production deployment
Frequently asked questions about Litellm
What is LiteLLM and what does it do?
LiteLLM is an open-source library that gives you a single, unified interface to call 100+ LLMs — OpenAI, Anthropic, Vertex AI, Bedrock, and more — using the OpenAI format. You can use it through either the Proxy Server (self-hosted LLM gateway) or Python SDK. Both give you a unified interface to access multiple LLMs with consistent output format, built-in retry/fallback logic, and spend tracking.
How do I install LiteLLM?
For the Python SDK, run uv add litellm. To run the full Proxy Server (LLM Gateway), run uv tool install 'litellm[proxy]'. New users can also run litellm --setup which walks through an interactive setup wizard that detects your OS, installs LiteLLM, and lets you pick LLM providers and enter API keys.
Which LLM providers does LiteLLM support?
LiteLLM supports 100+ LLM providers including OpenAI (GPT-4o, GPT-4o-mini, o1), Anthropic (Claude Opus, Sonnet, Haiku), Azure OpenAI, Google Vertex AI (Gemini 2.0 Flash, 1.5 Pro), AWS Bedrock (Claude, Llama), Ollama (local models), Fireworks AI, Groq, Mistral AI, Cohere, Together AI, Replicate, HuggingFace, and many more.
Is LiteLLM really free?
The software license is free (MIT licensed), but you own the entire operational burden. You pay for servers, databases, monitoring tools, and load balancing. For a production deployment handling moderate traffic, typical infrastructure costs range from $200-$500 monthly. You also pay LLM providers directly at their standard API rates. Total cost of ownership for production typically ranges from $2,000-$3,500/month when including DevOps labor.
How do I make my first LLM call with LiteLLM?
Import completion from litellm, set your API key in environment variables, and call completion with the model prefix. For example: from litellm import completion; import os; os.environ['OPENAI_API_KEY'] = 'your-api-key'; response = completion(model='openai/gpt-4o', messages=[{'role': 'user', 'content': 'Hello'}]); print(response.choices[0].message.content)
What is the LiteLLM Proxy Server?
The Proxy Server is a self-hosted OpenAI-compatible gateway. Any client that works with OpenAI works with the proxy — no code changes needed. It's running on http://0.0.0.0:4000 by default. You can call it with the OpenAI client by setting base_url='http://0.0.0.0:4000'. The proxy provides virtual keys, spend tracking, load balancing, and an admin UI.
How does LiteLLM handle errors from different providers?
LiteLLM maps every provider's errors to OpenAI exception types, so your existing error handling works out of the box. You can catch litellm.AuthenticationError for bad API keys, litellm.RateLimitError for rate limiting, and litellm.APIError for general API errors. It also maps all provider-specific finish_reason values to OpenAI-compatible values.
Can I use LiteLLM with streaming responses?
Yes, add stream=True to receive chunks as they are generated. Streaming responses yield ModelResponseStream chunks. Example: for chunk in completion(model='openai/gpt-4o', messages=[...], stream=True): print(chunk.choices[0].delta.content or '', end='')
How do I track costs with LiteLLM?
Use a callback to capture cost per response. Set litellm.success_callback = [track_cost] where your callback function accesses kwargs.get('response_cost', 0). You can also use the Pricing Calculator in the LiteLLM UI to estimate costs based on expected token usage and request volume for budget planning.
What are the enterprise features and when should I use them?
Enterprise Basic ($250/month) adds Prometheus metrics, LLM guardrails, JWT authorization, SSO integration (Okta, Azure AD), and audit logs. Enterprise Premium ($30,000/year) adds priority support, dedicated account management, custom feature development, and compliance assistance (SOC 2, HIPAA). Enterprise tiers make sense for teams needing compliance features, production SLAs, or who want enterprise support while still managing their own infrastructure.