Text-Embeddings-Inference
FreeInference for text-embeddings in Rust, HFOIL Licence.
FreeFree tier
Inputs: text
About Text-Embeddings-Inference
Text Embeddings Inference (TEI) is a toolkit for deploying and serving open source text embeddings and sequence classification models. TEI enables high-performance extraction for popular models including FlagEmbedding, Ember, GTE, and E5. It features no model graph compilation step, Metal support for local execution on Macs, small Docker images with fast boot times, token-based dynamic batching, optimized transformers code using Flash Attention, Candle, and cuBLASLt, Safetensors and ONNX weight loading, and production readiness with distributed tracing (OpenTelemetry) and Prometheus metrics.
Key Features
No model graph compilation step
Metal support for local execution on Macs
Small Docker images and fast boot times
Token-based dynamic batching
Optimized transformers code using Flash Attention, Candle, and cuBLASLt
Safetensors weight loading
ONNX weight loading
Production ready with distributed tracing (OpenTelemetry) and Prometheus metrics
Supports multiple model architectures: BERT, CamemBERT, XLM-RoBERTa, JinaBERT, Mistral, Alibaba GTE, Qwen2, MPNet, ModernBERT, Qwen3, Gemma3
Support for re-rankers, sequence classification, and SPLADE pooling models
Pros & Cons
Pros
- Blazing fast inference with optimized transformers and Flash Attention
- Lightweight Docker images with rapid startup
- Supports a wide range of popular open-source embedding models
- Works on multiple hardware platforms (CPU, Metal, CUDA, AMD ROCm)
- Production-ready features: monitoring, tracing, metrics
- Open source with permissive license (Apache 2.0 inferred from license file)
Cons
- Limited to embedding and sequence classification models; not a general-purpose LLM inference server
- Requires familiarity with Docker and command-line deployment for self-hosting
- Hardware acceleration limited to supported backends (CUDA, Metal, ROCm)
Best For
Deploying text embeddings models for semantic searchBuilding retrieval-augmented generation (RAG) pipelinesSequence classification tasks (e.g., sentiment analysis, topic labeling)Re-ranking search results with cross-encoder modelsRunning high-performance embedding extraction on CPU, GPU, or Apple SiliconServerless inference with fast container boot times
FAQ
What models does TEI support?
TEI supports Nomic, BERT, CamemBERT, XLM-RoBERTa models with absolute positions, JinaBERT with Alibi positions, Mistral, Alibaba GTE, Qwen2/3 with Rope positions, MPNet, ModernBERT, and Gemma3. Examples include WhereIsAI/UAE-Large-V1, intfloat/multilingual-e5-large-instruct, Snowflake/snowflake-arctic-embed-l-v2.0, and nomic-ai/nomic-embed-text-v1.5.
Is TEI free and open source?
Yes, Text Embeddings Inference is open source and licensed under the Apache 2.0 license (as indicated by the LICENSE file in the repository). The Docker images and code are freely available.
Can I run TEI on Apple Silicon?
Yes, TEI includes Metal support for local execution on Macs with Apple Silicon. It can be installed via Homebrew on macOS.
How do I deploy TEI with Docker?
TEI provides Docker images for CPU, CUDA (GPU), Intel, and ARM64. You can pull the image and run with the appropriate command. The repository includes example commands and a detailed README.