Infinity
FreeInference for text-embeddings in Python
FreeFree tier
Inputs: text, image, audio
About Infinity
Infinity is an open-source, high-throughput, low-latency REST API server for deploying text-embedding, reranking, CLIP, CLAP, and ColPali models from HuggingFace. Built on PyTorch, Optimum (ONNX/TensorRT), and CTranslate2 with FlashAttention, it supports NVIDIA CUDA, AMD ROCm, CPU, AWS Inferentia2, and Apple MPS accelerators. Dynamic batching and dedicated tokenization worker threads maximize throughput. The API follows OpenAI's embedding specs and provides a CLI (v2) and Docker images for easy deployment in production.
Key Features
Deploy any embedding, reranking, CLIP, CLAP, or ColPali model from HuggingFace
Fast inference backends: PyTorch, Optimum (ONNX/TensorRT), CTranslate2 with FlashAttention
Supports NVIDIA CUDA, AMD ROCm, CPU, AWS INF2, and Apple MPS accelerators
Dynamic batching and tokenization in dedicated worker threads
Multi-modal and multi-model orchestration
OpenAPI compatible with OpenAI's embedding API specs
CLI v2 with environment variable and argument configuration
Pre-built Docker images for CPU, ROCm, and GPU deployments
Tested with unit and end-to-end tests for correctness
MIT licensed, fully open source
Pros & Cons
Pros
- High throughput and low latency due to dynamic batching and efficient backends
- Supports a wide range of hardware accelerators (CUDA, ROCm, CPU, INF2, MPS)
- Drop-in replacement for OpenAI embedding API (OpenAPI compatible)
- Easy deployment via CLI, Docker, or pip install
- Actively maintained with frequent updates and community support
- Multi-model and multi-modal support in a single server
- Open source with MIT license – free to use and modify
Cons
- Documentation may require navigating GitHub README and docs site
- No built-in graphical user interface; primarily CLI/API based
- Occasional breaking changes as project evolves rapidly
- Requires some familiarity with model hosting and containerization for advanced deployments
Best For
Production serving of text embedding models for semantic search and RAG pipelinesDeploying reranking models to improve search result relevanceMulti-modal search using CLIP or CLAP for image/audio-text embeddingsDocument retrieval with ColPali-style late interaction modelsSelf-hosted embedding API with OpenAI-compatible endpointsHigh-throughput batch embedding generation for large datasets
FAQ
What models can I deploy with Infinity?
You can deploy any text-embedding, reranking, CLIP, CLAP, and ColPali model from HuggingFace.
What hardware accelerators does Infinity support?
Infinity supports NVIDIA CUDA, AMD ROCm, CPU, AWS Inferentia2 (INF2), and Apple MPS.
Is Infinity compatible with OpenAI's embedding API?
Yes, Infinity's API is aligned to OpenAI's API specs, making it a drop-in replacement.
How can I install Infinity?
You can install it via pip (`pip install infinity-emb[all]`) or use pre-built Docker images.
How do I run multiple models at once?
Use the v2 CLI with multiple `--model-id` flags to load and orchestrate multiple models.
Is Infinity free?
Yes, Infinity is open source under the MIT license and completely free to use.