Vision Agents
Open Vision Agents by Stream. Build voice and vision agents quickly with any model or video provider. Uses Stream's edge network for ultra-low latency.
Last verified:
What is Vision Agents?
Vision Agents is an open-source Python framework for building low-latency real-time voice and video AI agents. It allows developers to plug in any LLM, speech, or vision model from 25+ providers including OpenAI, Gemini, Anthropic, Deepgram, ElevenLabs, and YOLO. The framework enables shipping agents for telehealth, voice support, live coaching, customer service, and any application that can be wired up with audio/video capabilities.
Key features include sub-500ms latency running on Stream's global edge network, two operational modes (Realtime APIs via WebRTC/WebSocket or custom STT→LLM→TTS pipelines), video processing with YOLO/Roboflow/custom models on every frame, Twilio integration for phone calls with bi-directional audio, RAG support via TurboPuffer vector search and Gemini FileSearch, and production-ready deployment with HTTP server, Prometheus metrics, Docker, and Kubernetes support. The framework includes 30+ plugins with consistent interfaces for swapping providers without rewriting agent logic.
Vision Agents is designed for Python developers, AI engineers, and teams building real-time conversational AI applications. It supports use cases like AI golf coaches with YOLO pose detection, phone support agents with RAG-backed knowledge bases, smart security cameras with face recognition, live sports commentators with object detection, virtual try-on agents, and interactive avatars that see, hear, and respond with real-time voice and video.
The framework offers both Realtime models (speech-to-speech native via WebRTC/WebSocket with built-in STT/TTS) and custom pipeline mode (full control over separate STT, LLM, and TTS components). It includes function calling with MCP server support, built-in telemetry and metrics, turn detection options, avatar integrations, and the ability to create custom plugins for additional providers.
Vision Agents pricing
Pricing model: Freemium
Vision Agents is open-source and free to use. Stream provides 333,000 free participant minutes monthly for real-time transport, plus additional credits through the Maker Program for indie developers. Most AI providers (OpenAI, Gemini, Deepgram, ElevenLabs, Anthropic) offer free tiers to get started. You only pay for the API usage of the specific providers you choose: Stream API keys (STREAM_API_KEY, STREAM_API_SECRET), LLM provider API keys (Google AI Studio for Gemini, OpenAI, Anthropic, xAI), STT provider keys (Deepgram, ElevenLabs), and TTS provider keys (ElevenLabs, Cartesia, Inworld). No subscription fees for the framework itself—costs are purely pay-per-use based on your chosen providers' pricing.
Vision Agents pros
- Open-source Python framework with active development
- Sub-500ms latency on Stream's global edge network
- 25+ AI provider integrations (OpenAI, Gemini, Anthropic, Deepgram, ElevenLabs, YOLO)
- Two modes: Realtime APIs and custom STT/LLM/TTS pipelines
- Real-time video processing with YOLO pose detection and object detection
- Twilio integration for inbound and outbound phone calls
- RAG support with TurboPuffer vector search and Gemini FileSearch
- Production-ready with Docker, Kubernetes, and Prometheus metrics
- 30+ plugins with consistent swappable interface
- Function calling with MCP server support
- Built-in HTTP server with session management and authentication
- Free tier available: 333,000 participant minutes monthly from Stream
- Multiple STT options: Deepgram, ElevenLabs, Fast-Whisper, Fish, Wizper
- Multiple TTS options: ElevenLabs, Cartesia, Deepgram, OpenAI, Inworld, AWS Polly
- Realtime avatars with lip-sync (Anam, HeyGen, LemonSlice)
- Smart Turn detection and Vogent neural turn completion
- Horizontal scaling with Redis-backed session management
- Local model support with GPU Dockerfile for CUDA
- Built-in telemetry and per-session metrics debugging
- Quickstart builds first agent in under 5 minutes with 18 lines of Python
Vision Agents cons
- Requires Stream account for real-time transport (not fully self-hostable)
- Python-only framework (no native JavaScript/TypeScript SDK)
- Need separate API keys for multiple providers (Stream, LLM, STT, TTS)
- GPU required only for local models adds complexity and cost
- Stateful sessions require sticky sessions for horizontal scaling
- Documentation spread across many pages can be overwhelming
- US East region recommended for lowest latency limits global deployment
- Realtime models limit provider choice compared to custom pipeline
Frequently asked questions about Vision Agents
What is Vision Agents?
Vision Agents is an open-source Python framework for building low-latency real-time voice and video AI agents. It lets you plug in any LLM, speech, or vision model from 25+ providers and ship agents for telehealth, voice support, live coaching, phone support, security cameras, sports commentary, virtual try-on, and more. The framework runs on Stream's global edge network with sub-500ms latency.
How do I get started with Vision Agents?
Install Vision Agents with uv: 'uv init --python 3.12 my-agent && cd my-agent && uv add vision-agents[getstream,gemini] python-dotenv'. Create a .env file with STREAM_API_KEY, STREAM_API_SECRET (from getstream.io), and GOOGLE_API_KEY (from aistudio.google.com). Create main.py with the Agent class, then run 'uv run main.py run'. The CLI prints a join link to talk to your agent in the browser. You can build your first agent in under 5 minutes with about 18 lines of Python.
What AI providers does Vision Agents support?
Vision Agents supports 30+ plugins across categories: Language Models (Anthropic/Claude, Gemini, OpenAI/GPT-5+, xAI/Grok, OpenRouter, Kimi AI, Qwen), Realtime (Gemini Realtime, Inworld Realtime, OpenAI Realtime, Qwen Realtime, xAI Realtime, AWS Bedrock/Nova), STT (Deepgram Nova-3, ElevenLabs Scribe v2, AssemblyAI, Fish Audio, Mistral Voxtral, Fast-Whisper, Wizper), TTS (ElevenLabs, Cartesia Sonic, Deepgram Aura-2, OpenAI gpt-4o-mini-tts, Fish Audio, Inworld, Kokoro local, Pocket TTS, xAI, AWS Polly), Vision & Video (Moondream, NVIDIA Cosmos, Roboflow, Ultralytics YOLO, Decart), and Avatars (Anam, LiveAvatar/HeyGen, LemonSlice).
What are the two operational modes in Vision Agents?
Vision Agents has two modes: Realtime Models and Custom Pipeline. Realtime Models (openai.Realtime(), gemini.Realtime()) handle speech-to-speech natively via WebRTC or WebSocket with built-in STT/TTS—this is the fastest path with simplest setup. Custom Pipeline lets you mix and match providers: Deepgram for STT, any LLM (Gemini, OpenAI, Anthropic), ElevenLabs for TTS, with configurable turn detection (Deepgram built-in, ElevenLabs built-in, Smart Turn, Vogent). Custom pipeline gives full control over each component.
Can Vision Agents handle phone calls?
Yes, Vision Agents has Twilio integration for voice calls with bi-directional audio. You can build agents that answer inbound calls (Phone Support Agent example) with RAG-backed knowledge bases via TurboPuffer. The framework supports both inbound and outbound calls through the Phone Calling guide. The Phone & RAG example shows a Twilio-powered agent that answers calls with knowledge retrieval.
How does video processing work in Vision Agents?
Vision Agents supports video processing in two ways: (1) WebRTC sends realtime video at full FPS to LLM models over WebRTC with no intervals—ideal for games and applications where advanced image processing isn't needed before the model. (2) Interval-based processing with Video Processors that intercept video frames at set intervals, run them through custom ML models (YOLO Pose Detection, Roboflow object detection), and forward input to LLMs. The Agent class automatically handles this logic. Processors can be chained together and can also process audio.
How do I deploy Vision Agents to production?
Vision Agents provides Dockerfiles for CPU (~150MB, fast build) and GPU (~8GB, for local model inference). Build with 'docker buildx build --platform linux/amd64 -t vision-agent .'. Deploy to Kubernetes with Helm using provided probes: GET /health for liveness and GET /ready for readiness. Use the HTTP server for multi-session deployments with horizontal scaling via multiple replicas behind a load balancer. Enable Prometheus metrics for monitoring and set alerts for error rates and latency spikes. Use sticky sessions since sessions are stateful.
Is Vision Agents free to use?
Yes, Vision Agents is open-source and free to use. Stream provides 333,000 free participant minutes monthly for real-time transport, plus additional credits through the Maker Program for indie developers. Most AI providers offer free tiers to get started (Deepgram, ElevenLabs, Google AI Studio for Gemini). You only pay for API usage of your chosen providers—there are no subscription fees for the framework itself. Costs are purely pay-per-use based on each provider's pricing.
Can I create custom plugins for Vision Agents?
Yes, Vision Agents supports creating custom plugins. If you don't see your provider in the 30+ existing plugins, you can build your own plugin to connect additional services. The framework uses a consistent interface—plugins of the same type (STT, TTS, LLM) share common interfaces, so you can swap providers in one line of code. See the 'Create Your Own Plugin' guide in the documentation for instructions on building custom plugins.
What examples and use cases are available for Vision Agents?
Vision Agents includes multiple examples: AI Golf Coach (YOLO pose detection watches swing, Gemini gives real-time coaching), Phone Support Agent (Twilio-powered with RAG-backed knowledge bases), Smart Security Camera (face recognition and package detection with YOLO, automated alerts), Live Sports Commentator (Roboflow object detection tracks players/ball, LLM delivers play-by-play), Live Video Try-On (Decart's Lucy-2 real-time model for virtual try-on), Interactive Avatar (Anam avatars that see/hear/respond), Expressive Voice Narrator (Cartesia Sonic 3 TTS storytelling), AI Meeting Copilot (real-time sales assistant with coaching suggestions), Voice Agent Starter (minimal conversational agent), and Video Call Moderator (detects, censors, escalates with verbal warnings).