Litellm
FreePython SDK, Proxy Server (AI Gateway) to call 100+ LLM APIs in OpenAI (or native) format, with cost tracking, guardrails, loadbalancing and logging. [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, VLLM, NVIDIA NIM]
FreeFree tier
About Litellm
LiteLLM is an open-source library that provides a single, unified interface to call over 100 LLMs (OpenAI, Anthropic, Vertex AI, Bedrock, and more) using the OpenAI Chat Completions format. It includes a Python SDK and a self-hosted AI Gateway (Proxy) with virtual keys, cost tracking, an admin UI, built-in retry/fallback logic, and consistent output formatting across all providers.
Key Features
Single completion() interface for all providers
Consistent OpenAI-format output regardless of provider
Built-in retry and fallback logic via Router
Self-hosted LLM Gateway (Proxy) with virtual keys and cost tracking
Admin UI for managing the proxy
Support for 100+ LLMs including OpenAI, Anthropic, Vertex AI, Bedrock, Ollama, Azure OpenAI
Streaming support with ModelResponseStream chunks
Open-source with Python SDK and containerized deployment
Pros & Cons
Pros
- Single interface eliminates need to learn multiple LLM APIs
- Consistent output format across all providers reduces parsing complexity
- Built-in retry and fallback improves application reliability
- Self-hosted proxy enables virtual keys, cost tracking, and admin management
- Open-source and free to use
Best For
Simplifying multi-provider LLM integration in applicationsBuilding AI tools with automatic fallback across modelsDeploying an internal AI gateway with usage tracking and access controlRapid prototyping with different LLMs without changing codeUnified logging and cost management for enterprise LLM usage
FAQ
What is LiteLLM?
LiteLLM is an open-source library that provides a single, unified interface to call over 100 LLMs using the OpenAI Chat Completions format. It also includes a self-hosted AI Gateway with cost tracking, virtual keys, and an admin UI.
How do I install LiteLLM?
You can install LiteLLM via pip using 'uv add litellm'. To deploy the full AI Gateway with Admin UI, follow the Quickstart guide which runs as a container.
Does LiteLLM support streaming?
Yes, LiteLLM supports streaming responses. When stream=True is set, it yields ModelResponseStream chunks in the OpenAI Chat Completions format.
What providers are supported?
LiteLLM supports 100+ LLMs including OpenAI, Anthropic, Vertex AI, Bedrock, Ollama, Azure OpenAI, and many more.