semantic-router

Free

Superfast AI decision making and intelligent processing of multi-modal data.

FreeFree tier

Inputs: text

Type

Open Source

Company

aurelio-labs

Visit Website GitHub (38 stars)

About semantic-router

Semantic Router is an open-source Python library that provides a superfast decision-making layer for LLMs and agents. Instead of relying on slow LLM generations to decide which tool or prompt to use, it leverages semantic vector space to route requests based on meaning. It allows developers to define routes (decision paths) with example utterances, then uses an encoder (e.g., Cohere, OpenAI, or local HuggingFace models) to embed input and find the closest route. The library also supports hybrid routing and local LLMs via LlamaCpp, making it suitable for offline or privacy-sensitive applications. Its design enables rapid, lightweight classification for conversational AI, multi-agent orchestration, and content moderation.

Key Features

Superfast decision-making using semantic vector space, avoiding slow LLM generations

Define custom routes with example utterances for flexible intent classification

Supports multiple encoders: OpenAI, Cohere, HuggingFace (local), and hybrid mode

Supports local LLM inference with LlamaCpp for fully offline routing

Auto-sync routes to local storage for persistence

Compatible with Python, easily integrated into LLM pipelines and agent frameworks

Pros & Cons

Pros

Significantly faster than using LLM generations for routing decisions
Open source and free to use, with permissive license
Easy to set up with only a few lines of code
Flexible: supports cloud-based and local encoders/LLMs
Actively maintained with growing community (3.7k stars on GitHub)
Allows offline operation when using local models

Cons

Requires external API keys for OpenAI or Cohere encoders (cost and internet needed)
Accuracy of routing depends on quality of example utterances and encoder model
Not a full chatbot framework; only handles decision layer, not response generation
Local encoder/LLM setup may require additional dependencies and compute resources
Limited to text-based input; multimodal routing mentioned but not fully detailed

Best For

Route user queries to appropriate LLM agents or prompts based on semantic meaningFilter out unwanted topics (e.g., politics) in chatbot conversationsSwitch between conversational modes (chitchat vs. professional) without re-promptingMulti-agent orchestration where different agents handle different types of requestsContent moderation by routing sensitive inputs to safety checksBuilding lightweight, offline intent classification for edge devices

FAQ

How does Semantic Router work?

Semantic Router uses semantic vector space to embed user input (query) and compare it against predefined route examples (utterances). The route whose embeddings are most similar to the input is selected, enabling fast decision-making without needing an LLM call.

What encoders are supported?

Currently supported encoders include OpenAIEncoder, CohereEncoder, and HuggingFaceEncoder for local usage. Hybrid mode combines dense and sparse embeddings.

Can I use Semantic Router offline?

Yes. By using HuggingFaceEncoder for embeddings and LlamaCppLLM for local LLM inference, Semantic Router can run fully offline with no external API calls.

Is Semantic Router free?

Yes, the library is open source and free to use. You only pay for API keys if you choose to use cloud-based encoders like OpenAI or Cohere.

How do I install Semantic Router?

Install via pip: pip install -qU semantic-router. For local models, use pip install -qU "semantic-router[local]". For hybrid routing, use pip install -qU "semantic-router[hybrid]".