Semantic Router
Superfast AI decision making and intelligent processing of multi-modal data.
Last verified:
What is Semantic Router?
Semantic Router is a superfast decision-making layer for LLMs and agents that routes requests based on semantic meaning instead of relying on slow, unreliable LLM generations. Rather than waiting seconds for LLM outputs to make tool-use or safety decisions, it uses semantic vector space to make routing decisions in milliseconds (around 100ms vs 5,000ms with gpt-3.5-turbo). The library is fully open source under an MIT license and can run locally or via APIs from providers like OpenAI, Cohere, Hugging Face, and Pinecone.
Key features include a simple API that sets up routes with just a few lines of code, dynamic routes that can generate parameters and trigger function calls, multiple integrations with encoder providers (Cohere, OpenAI, Hugging Face, FastEmbed), vector store support for persistence (Pinecone, Qdrant, Postgres), multi-modal capabilities that route based on image content using CLIP and ViT encoders, and fully local execution with models like Llama and Mistral via Llama.cpp. The system consists of three core components: encoders that transform inputs into vector representations, routes that define patterns with example utterances, and indexing systems that store and retrieve route vectors efficiently.
Semantic Router is designed for developers building chatbots, classification systems, AI agents, and voice assistants that need to understand user intent and make fast routing decisions. It's particularly useful for creating fuzzy if-else statements based on embedding space, scaling agent tools to thousands of options without context window limitations, implementing safety guardrails with deterministic rules, and reducing hallucinations in AI applications. The library supports cloud-based, hybrid, and fully local execution modes.
Semantic Router pricing
Pricing model: Freemium
Fully free and open source under MIT license. No paid plans or tiers - you can use it however and wherever you want. Cost comes only from API usage if using cloud encoders like OpenAI or Cohere, or from running local models on your own hardware.
Semantic Router pros
- Superfast decision making in approximately 100ms versus 5000ms with LLMs
- Fully open source with MIT license for unlimited use
- Simple API requiring only a few lines of code to set up routes
- Dynamic routes that generate parameters and trigger function calls
- Multiple encoder integrations including OpenAI, Cohere, Hugging Face, FastEmbed
- Vector store persistence support with Pinecone, Qdrant, and Postgres
- Multi-modal routing capabilities for images using CLIP and ViT encoders
- Fully local execution option with Llama.cpp and HuggingFace
- Hybrid mode combining local embeddings with API-based LLMs
- Score threshold configuration for controlling matching sensitivity
- Handles semantic similarity so exact keyword matching not required
- Scales to thousands of tools without context window limitations
- Deterministic safety rules prevent AI hallucinations and unsafe behavior
- Async support available for high-concurrency environments
- Batch processing for higher throughput applications
- Returns similarity scores with route choices for debugging
Semantic Router cons
- Requires API keys for OpenAI or Cohere when not running fully local
Frequently asked questions about Semantic Router
What is Semantic Router?
Semantic Router is a superfast decision-making layer for LLMs and agents. Instead of waiting for slow LLM generations to make tool-use decisions, it uses semantic vector space to route requests based on meaning. It makes routing decisions in milliseconds (around 100ms) rather than seconds (around 5000ms with gpt-3.5-turbo).
Is Semantic Router free?
Yes, Semantic Router is fully open source under an MIT license. This means you can use it however and wherever you want with no restrictions, no paid plans, and no tiers. The only costs come from API usage if you choose cloud encoders like OpenAI or Cohere.
How do routes work in Semantic Router?
Routes are defined with example utterances that serve as semantic reference points. Each Route has a name, utterances (example inputs), optional function_schemas for function calling, and optional score_threshold. The encoder converts text into vectors, and the router finds the closest semantic match to incoming queries.
What encoders does Semantic Router support?
Semantic Router supports CohereEncoder and OpenAIEncoder initially, with more coming soon. For local execution, it supports HuggingFaceEncoder and LlamaCppLLM. It also supports dense encoders (OpenAI, HuggingFace), sparse encoders (BM25, TFIDF, AurelioSparse), and multimodal encoders (CLIP, ViT).
Can I run Semantic Router locally without APIs?
Yes, you can run Semantic Router fully locally using HuggingFaceEncoder and LlamaCppLLM with models like Llama and Mistral. This requires installing semantic-router[local]. You can also use hybrid mode combining local embeddings with API-based LLMs.
What happens when no route matches a query?
When no semantic match is found, the route layer returns None. This is useful for creating fallback behavior or passthroughs in your applications when no intent is clearly matched. The function returns RouteChoice with name=None, function_call=None, and similarity_score=None.
Can Semantic Router handle multiple routes?
Yes, you can retrieve multiple routes using retrieve_multiple_routes() which returns a list of RouteChoice objects with their similarity scores. For queries that don't match any routes, it returns an empty list.
Does Semantic Router support vector databases?
Yes, Semantic Router integrates with Pinecone and Qdrant for persistence through PineconeIndex and QdrantIndex. It also supports PostgresIndex for SQL-based vector storage and LocalIndex for in-memory storage.
What are the key use cases for Semantic Router?
Key use cases include building chatbots with intent recognition, creating safety guardrails with deterministic rules, scaling agent tools to thousands of options, voice assistants for task routing without keyboard shortcuts, classification systems, filtering based on embedding space, and reducing LLM hallucinations in AI applications.