Agents
A framework for building realtime voice AI agents 🤖🎙️📹
Last verified:
What is Agents?
LiveKit Agents is a realtime framework for building production-grade multimodal and voice AI agents. The framework lets you add any Python or Node.js program to LiveKit rooms as full realtime participants, enabling agents to communicate through voice, video, and text. You can build agents with code using the Python and Node.js SDKs, or use LiveKit Agent Builder to prototype and deploy agents directly in your browser without writing any code.
Key features include support for streaming audio through an STT-LLM-TTS pipeline, reliable turn detection with a custom model for lifelike conversation flow, handling interruptions, and LLM orchestration. The framework supports tool use with function tools and provider tools, multi-agent handoffs for complex workflows, and extensive integrations with nearly every AI provider for LLMs, speech-to-text, text-to-speech, and more. It includes built-in agent server orchestration, automatic load balancing, and Kubernetes compatibility for production deployments.
LiveKit Agents is designed for developers building voice AI applications including multimodal assistants, telehealth applications, call centers with inbound and outbound call support, realtime translation services, lifelike NPCs for games, and robotics applications where the robot's AI runs in the cloud. The framework is fully open source under the Apache 2.0 license with an active community.
You can deploy agents to LiveKit Cloud for managed deployment with built-in observability including transcripts and traces, or deploy to any custom environment. LiveKit Cloud provides LiveKit Inference for running AI models without API keys, and the platform supports telephony integration via SIP for phone calls.
Agents pricing
Pricing model: Freemium
LiveKit Agents is open source under Apache 2.0 license with no cost to use the SDK. You can deploy to LiveKit Cloud which has a free tier with allowances including adaptive interruption credits. Agent Builder has no additional cost beyond standard LiveKit Cloud quotas. LiveKit Cloud provides managed deployment with automatic scaling up to plan limits. Paid plans include concurrency limits and usage-based pricing for LiveKit Inference credits used for STT-LLM-TTS pipelines. Contact sales for enterprise pricing and higher concurrency limits.
Agents pros
- Open source under Apache 2.0 license
- Supports both Python and Node.js SDKs
- Agent Builder for no-code prototyping in browser
- Managed deployment on LiveKit Cloud infrastructure
- Built-in observability with transcripts and traces
- LiveKit Inference for AI models without API keys
- WebRTC-based reliable realtime communication
- State-of-the-art custom turn detection model
- Support for voice, video, and text modality
- Function tools and provider tools support
- Multi-agent handoffs for complex workflows
- Extensive integrations with major AI providers
- Automatic load balancing and scaling
- Kubernetes compatible for production
- SIP telephony integration for phone calls
- Preemptive generation reduces latency
- Background voice cancellation improves comprehension
- agent dispatch for routing to specific agents
- Secrets management for API keys and credentials
- Data collection mode for structured field extraction
Agents cons
- Agent Builder doesn't support workflows and handoffs
- Agent Builder doesn't support virtual avatars
- Agent Builder doesn't support vision capabilities
- Agent Builder doesn't support realtime models
- Agent Builder doesn't support writing tests
- Requires Python or Node.js knowledge for SDK
- Learning curve for framework concepts
- Concurrency limits on LiveKit Cloud plans
- Build-size limit on each deployment
- Free-tier allowance restrictions for adaptive interruption
Frequently asked questions about Agents
What is LiveKit Agents?
LiveKit Agents is a realtime framework for building production-grade multimodal and voice AI agents. It lets you add any Python or Node.js program to LiveKit rooms as full realtime participants, with tools and abstractions for feeding realtime media and data through an AI pipeline that works with any provider, and publishing realtime results back to the room.
Do I need to write code to use LiveKit Agents?
No, you can use LiveKit Agent Builder to prototype and deploy voice agents directly in your browser without writing any code. Agent Builder produces best-practice Python code using the LiveKit Agents SDK and deploys to LiveKit Cloud. You can also build with code using the Python or Node.js SDKs for full control.
What AI providers does LiveKit Agents support?
The framework supports extensive integrations with nearly every AI provider for LLMs, speech-to-text (STT), text-to-speech (TTS), and more. Supported providers include OpenAI, Gemini, Anthropic, Mistral AI, xAI, and many others. Plugins for major AI providers are available and more are continually added.
How do I deploy my agent to production?
You can deploy agents to LiveKit Cloud with a single LiveKit CLI command for managed deployment on LiveKit's global infrastructure with automatic scaling and load balancing. Alternatively, you can deploy to any custom environment including your own Kubernetes cluster. Agent Builder also deploys directly to LiveKit Cloud with one click.
What is LiveKit Inference?
LiveKit Inference is a service in LiveKit Cloud that lets you run AI models without API keys. It provides access to models for STT, LLMs, and TTS to construct high-performance pipelines. You access models directly through LiveKit Cloud and pay through usage credits rather than managing individual provider API keys.
Can agents handle phone calls?
Yes, LiveKit includes full support for telephony via SIP integration. Your agent can place and receive phone calls, and users can join calls from phones instead of frontend apps. This enables inbound and outbound call support for call center applications.
What is multi-agent handoff?
Multi-agent handoff allows you to break down complex workflows into simpler tasks by transferring control between agents. For example, in a medical office triage system, agents can transfer callers between each other while preserving context. This enables more manageable agent logic and specialized agents for different tasks.
How does turn detection work?
LiveKit Agents uses a custom turn detection model for state-of-the-art conversational behavior. This ensures lifelike conversation flow with reliable detection of when a user is done speaking. The framework also handles interruptions naturally, allowing users to interrupt agent speech.
What is data collection mode?
Data collection mode is an Agent Builder feature where the agent extracts specific fields you define (names, preferences, answers to questions) and returns them as structured results at the end of the call. Fields can be string, number, boolean, object, or list, and can be marked as required or optional. Results are sent to your configured endpoint.
Can Iitomatically generate call summaries?
Yes, when summary settings are enabled, the agent automatically generates a summary of the conversation using the selected LLM and includes it in the request to your configured endpoint. You can provide custom summary instructions and choose which LLM generates the summary. You can preview summaries during live test calls.