Ai Gateway
Manages Unified Access to Generative AI Services built on Envoy Gateway
Last verified:
What is Ai Gateway?
Envoy AI Gateway is an open source project built on Envoy Gateway that handles request traffic from application clients to Generative AI (GenAI) services. It provides a unified layer for routing and managing LLM/AI traffic with backend rate limiting, policy control, and automatic failover mechanisms to ensure service reliability. The gateway addresses the complexity of connecting applications to GenAI services by leveraging Envoy's flexibility and Kubernetes-native features.
Key features include support for 16+ LLM providers (OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, Google Gemini, Vertex AI, Mistral, Cohere, Groq, Together AI, DeepInfra, DeepSeek, Hunyuan, SambaNova, Grok, and Tetrate Agent Router Service), model virtualization, provider fallback, usage-based rate limiting with token awareness, prompt caching, inference optimization with InferencePool support, Model Context Protocol (MCP) support for AI agent tool integrations, upstream authentication, comprehensive metrics and observability, header and body mutations, and horizontal pod autoscaling for production deployments.
Envoy AI Gateway is designed for enterprises and developers who need secure, scalable AI/LLM traffic management in Kubernetes environments. It targets teams building GenAI applications who require resilient connectivity across providers and self-hosted models, comprehensive observability for performance and cost management, enterprise-grade security features, and an extensible architecture built on battle-tested Envoy Proxy technology. Adopters include Bloomberg, Tencent Cloud, Nutanix, Tetrate, LY Corporation, and National Research Platform.
Ai Gateway pricing
Pricing model: Freemium
Envoy AI Gateway is completely free and open source under the Apache-2.0 license. There are no licensing fees, per-request charges, or paid tiers. The project follows the open source pricing model where users only pay for their own compute infrastructure costs (Kubernetes cluster, hosting, etc.). No managed service or enterprise tier is available - it is self-hosted only. Helm charts are available for installation at oci://docker.io/envoyproxy/ai-gateway-helm.
Ai Gateway pros
- Open source with Apache-2.0 license - no licensing fees
- Supports 16+ major LLM providers out of the box
- Built on battle-tested Envoy Proxy technology
- Kubernetes-native with Gateway API integration
- Automatic failover between AI providers for reliability
- Token-aware usage-based rate limiting
- Model virtualization abstracts AI models
- Prompt caching with unified cache_control API
- InferencePool support for intelligent load balancing
- Model Context Protocol (MCP) support with OAuth authentication
- Upstream authentication for secure egress to external providers
- Comprehensive metrics and observability for AI workloads
- Header and body mutations per backend or route
- Horizontal pod autoscaling for production deployments
- Two-tier gateway pattern for centralized and fine-grained control
- Active community with weekly Monday meetings
- Vendor-specific fields support for provider-unique capabilities
Ai Gateway cons
- Requires Kubernetes cluster - not standalone deployment
- Still at v0.5 version - relatively early in development
- Learning curve for Envoy Gateway and Gateway API concepts
- No managed cloud offering - self-hosted only
- You pay for your own compute infrastructure costs
- Requires Helm installation and Kubernetes expertise
- Limited documentation compared to mature enterprise gateways
- Controller replicas needed for production scaling adds complexity
Frequently asked questions about Ai Gateway
What is Envoy AI Gateway?
Envoy AI Gateway is an open source project built on Envoy Gateway that handles request traffic from application clients to Generative AI (GenAI) services. It provides a secure, scalable, and efficient way to manage LLM/AI traffic with backend rate limiting and policy control, addressing the complexity of connecting applications to GenAI services by leveraging Envoy's flexibility and Kubernetes-native features.
Which LLM providers does Envoy AI Gateway support?
Envoy AI Gateway supports 16+ LLM providers out of the box: Anthropic, AWS Bedrock, Azure OpenAI, Cohere, DeepInfra, DeepSeek, Google Gemini, Grok, Groq, Hunyuan, Mistral, OpenAI, SambaNova, Tetrate Agent Router Service, Together AI, and Vertex AI. Vendor-specific fields can be used to access provider-unique capabilities in OpenAI-compatible requests.
Is Envoy AI Gateway free to use?
Yes, Envoy AI Gateway is completely free and open source under the Apache-2.0 license. There are no licensing fees, per-request charges, or paid tiers. Users only pay for their own compute infrastructure costs such as Kubernetes cluster hosting.
How do I install Envoy AI Gateway?
Envoy AI Gateway is installed on Kubernetes using Helm. You need a Kubernetes cluster first, then install Envoy Gateway with 'helm install eg oci://docker.io/envoyproxy/gateway-helm', followed by installing Envoy AI Gateway with 'helm upgrade -i aieg oci://docker.io/envoyproxy/ai-gateway-helm'. A basic configuration YAML file is then applied with kubectl.
What is the two-tier gateway pattern in Envoy AI Gateway?
Envoy AI Gateway uses a two-tier gateway pattern. The Tier One Gateway functions as a centralized entry point handling authentication, top-level routing, and global rate limiting. The Tier Two Gateway handles ingress traffic to self-hosted model serving clusters and provides fine-grained control over self-hosted model access with endpoint picker support for LLM inference optimization.
Does Envoy AI Gateway support automatic failover?
Yes, Envoy AI Gateway supports automatic failover mechanisms between AI providers to ensure service reliability. This is part of its resilient connectivity across providers and self-hosted models, ensuring high availability through intelligent routing when providers experience issues.
What is Model Context Protocol (MCP) support in Envoy AI Gateway?
Envoy AI Gateway now supports Model Context Protocol (MCP), bringing enterprise-grade security, routing, and observability to AI agent tool integrations. Features include server multiplexing, tool routing, OAuth authentication, and full spec compliance with zero-friction deployment for connecting AI agents to external tools and data sources.
How does rate limiting work in Envoy AI Gateway?
Envoy AI Gateway provides usage-based rate limiting with token-aware rate limiting specifically designed for AI workloads. This allows organizations to limit usage based on token consumption rather than just request counts, providing more granular control over GenAI usage and cost management.
Can I use Envoy AI Gateway with self-hosted models?
Yes, Envoy AI Gateway supports both external LLM providers and self-hosted models. The Tier Two Gateway specifically handles ingress traffic to self-hosted model serving clusters and provides fine-grained control over self-hosted model access with endpoint picker support for LLM inference optimization.
How do I get help or contribute to Envoy AI Gateway?
You can join the community on Slack in the #envoy-ai-gateway channel, file issues on GitHub, attend weekly community meetings every Monday to discuss updates and address issues, or submit feature requests and pull requests via the GitHub repository. Contributing guides are available for setting up your environment.