Lmring
Open-source, self-hostable LLM arena with model compare, voting, and leaderboards
Last verified:
What is Lmring?
LLMRing is an open-source, provider-agnostic Python library that provides a unified interface for interacting with large language models (LLMs) from multiple providers including OpenAI, Anthropic, Google Gemini, and Ollama. It allows developers to map tasks to aliases backed by ordered model lists with automatic fallbacks, use a single interface for all providers, and optionally track usage and cost via a self-hosted server. Configuration lives in a version-controlled llmring.lock file that keeps model choices explicit, reviewable, and easy to share across teams.
Key features include unified structured output across all providers using JSON Schema, first-class support for reasoning models (OpenAI's o1, o3, gpt-5) with automatic token budget handling, detailed cost breakdowns including prompt caching and long-context tiers, full Model Context Protocol (MCP) support for tool orchestration, CLI commands for lockfile management, and environment-specific profiles for dev/staging/prod configurations. The library works standalone with just the lockfile or connects to an optional server for receipts, usage telemetry, and MCP persistence.
LLMRing is designed for Python developers and teams building AI applications who need reproducible deployments, cost transparency, and resilience against provider rate limits or outages. It's particularly useful for organizations using multiple LLM providers, those requiring strict cost tracking for finance teams, developers working with reasoning models, and teams implementing MCP-based tool orchestration. The MIT license allows unrestricted use.
Lmring pricing
Pricing model: Freemium
LLMRing library and server are completely free and open source under MIT license. The Python package is available on PyPI as 'llmring'. You pay only for the underlying LLM provider APIs (OpenAI, Anthropic, Google, Ollama) which are billed directly to you. The library calculates and tracks costs using the open registry's pricing data, providing detailed cost breakdowns including input/output tokens, cache reads/writes, reasoning tokens, and long-context tiers. No subscription fees, no per-call charges from LLMRing itself.
Lmring pros
- Open source with MIT license for unrestricted use
- Unified interface for OpenAI, Anthropic, Google, and Ollama
- Automatic model fallback when primary is rate-limited or unavailable
- Version-controlled llmring.lock file for reproducible deployments
- Unified JSON Schema structured output across all providers
- First-class reasoning model support with automatic token budgeting
- Detailed cost breakdown including cache reads/writes and thinking tokens
- Full Model Context Protocol (MCP) support for tool orchestration
- Environment-specific profiles for dev, staging, and production
- CLI for interactive lockfile configuration with llmring lock chat
- Long-context pricing automatically kicks in at provider thresholds
- Registry-aware with human-validated model capabilities and pricing
- No secrets in lockfile - API keys via environment variables only
- Automatic API parameter switching for OpenAI reasoning models
- Signed receipts with Ed25519 for audit trails and finance
Lmring cons
- Requires self-hosting server for usage tracking and receipts
- Learning curve for lockfile configuration and alias system
- Ollama support limited to best-effort JSON without guaranteed repair
- No built-in user management - project-scoped via API keys only
- Registry versions track what registry knew, not actual model behavior
- Providers can change prices and limits anytime despite registry
- CORS and TLS must be configured manually in production
- No SaaS hosting option - server must be self-hosted
- Initial setup requires multiple environment variables for providers
Frequently asked questions about Lmring
What is LLMRing and what does it do?
LLMRing is an open-source Python library that provides a unified interface for talking to LLMs from multiple providers (OpenAI, Anthropic, Google Gemini, Ollama). It lets you use task-oriented aliases instead of model IDs, automatically handle model fallbacks when providers rate-limit or degrade, track usage and cost with detailed breakdowns, and version-control your AI configuration through llmring.lock files. API calls go directly to providers while metadata can optionally be logged to a self-hosted server.
How do automatic model fallbacks work?
Aliases in llmring.lock are backed by ordered model lists. LLMRing resolves aliases to the first available model in the list; if the primary model is unavailable or rate-limited, it automatically tries the next entry in priority order. For example, an alias with models ['anthropic:claude-3-5-sonnet', 'openai:gpt-4o'] will use Claude first and fall back to GPT-4o only when needed. This provides instant failover for rate limits or regional outages without changing code.
What is the llmring.lock file?
The llmring.lock file is your authoritative, version-controlled configuration that makes your AI stack reproducible. It contains registry version pinning per provider, alias bindings that map task names to provider:model IDs (with fallback support), and optional profiles for different environments. The file is safe to commit to VCS since it contains no secrets - API keys are passed via environment variables only. Use llmring lock init to create it and llmring lock chat for intelligent conversational configuration.
How does LLMRing handle reasoning models like o1 and gpt-5?
LLMRing 1.1 adds first-class reasoning model support. The registry tags reasoning-capable models with is_reasoning_model and min_recommended_reasoning_tokens. The library automatically applies the recommended split between reasoning and completion tokens, and for OpenAI reasoning calls it automatically swaps to max_completion_tokens for correct API semantics. You can override reasoning_tokens when you need more headroom, and usage/receipts include the extra reasoning budget for accurate cost reports.
What cost breakdown features does LLMRing provide?
LLMRing 1.2 provides detailed cost breakdowns including prompt caching (cache_read_input_tokens billed at discounted rates, cache_creation_5m_tokens and cache_creation_1h_tokens for cache writes), long-context tiers that automatically kick in when crossing provider thresholds, and dedicated thinking tokens priced separately using registry metadata. The response.usage contains cost_breakdown with input, cache_read, cache_write_5m, cache_write_1h, long_context_input, output, and reasoning costs so finance sees exactly where spend goes.
Does LLMRing support structured output across all providers?
Yes, LLMRing provides one interface for JSON Schema structured output across all providers using response_format with json_schema. OpenAI uses native JSON Schema strict mode, Anthropic uses tool-based extraction with validation, Google Gemini uses FunctionDeclaration with schema mapping, and Ollama uses best-effort JSON with automatic repair. You get valid JSON and a parsed dict regardless of provider, with the library adapting the schema per provider under the hood.
What is the optional server and when do I need it?
The llmring-server is an optional self-hosted backend that adds signed receipts (Ed25519 over RFC 8785 JCS), usage logging and statistics, conversation persistence, and MCP tool/resource/prompt management. You don't need it for basic functionality - the library works completely standalone with just llmring.lock. Use the server when you need audit trails for finance, usage dashboards, conversation history, or persistent MCP sessions. It defaults to SaaS at https://api.llmring.ai when LLMRING_SERVER_URL is unset.
How do I configure multiple environments like dev and prod?
LLMRing supports profiles in llmring.lock for environment-specific configurations. You can define [profiles.prod] with high-quality models and fallbacks, and [profiles.dev] with cheaper models for development. Switch profiles via the LLMRING_PROFILE environment variable (e.g., export LLMRING_PROFILE=prod) or the --profile flag in CLI commands. Each profile has its own bindings, allowing you to use different models per environment while keeping everything in one version-controlled file.
What MCP (Model Context Protocol) features does LLMRing support?
LLMRing has full MCP support built into the library and server. You can connect to any MCP server for interactive chat via llmring mcp chat, list and register MCP servers, list available tools, and execute tools with llmring mcp execute. The server provides MCP persistence for tools, resources, and prompts with a database schema containing servers, tools, resources, prompts, and tool_executions tables. All MCP operations are project-scoped via the X-API-Key header.