Chroma: Open-Source Embedding Database for RAG and Semantic Search

Chroma is an open-source embedding database built for LLM applications that need memory. You would reach for it when you are building a retrieval-augmented generation (RAG) pipeline, prototyping in a notebook, or running a self-hosted vector search service. It stores embeddings alongside metadata, so you can filter results by source, date, category, or any other attribute you attach to a document.

What it does

Chroma gives you a local or server-based vector store that accepts documents, computes embeddings (using a pluggable embedding function), and returns the most similar results for a query. You can add, update, delete, and retrieve documents by ID or by metadata filter. The database persists to disk when you use the persistent client, so data survives a restart. It also runs in server mode for multi-user production setups.

Before you start

• A working Python install is required.
• Install the Python package with pip install chromadb.
• For JavaScript/TypeScript, install npm install chromadb @chroma-core/default-embed.
• No external database service is needed for local use. The persistent client writes to a directory you specify.
• If you plan to use OpenAI or HuggingFace embedding functions, you need an API key for those services.
• The default embedding model is all-MiniLM-L6-v2 from sentence-transformers, which downloads on first use.

Quick start

Installation

# Python
pip install chromadb

# JavaScript/TypeScript
npm install chromadb @chroma-core/default-embed

Basic usage (Python)

import chromadb

# Create client
client = chromadb.Client()

# Create collection
collection = client.create_collection(name="my_collection")

# Add documents
collection.add(
    documents=["This is document 1", "This is document 2"],
    metadatas=[{"source": "doc1"}, {"source": "doc2"}],
    ids=["id1", "id2"]
)

# Query
results = collection.query(
    query_texts=["document about topic"],
    n_results=2
)

print(results)

This creates an in-memory client, adds two documents with metadata, and queries for the closest matches. The output is a dictionary with keys documents, metadatas, distances, and ids.

Core operations

1. Create collection

# Simple collection
collection = client.create_collection("my_docs")

# With custom embedding function
from chromadb.utils import embedding_functions

openai_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key="your-key",
    model_name="text-embedding-3-small"
)

collection = client.create_collection(
    name="my_docs",
    embedding_function=openai_ef
)

# Get existing collection
collection = client.get_collection("my_docs")

# Delete collection
client.delete_collection("my_docs")

A collection is a named container for embeddings and documents. You can attach a custom embedding function at creation time. If you do not provide one, Chroma uses the default sentence-transformers model. Use get_collection to retrieve an existing collection and delete_collection to remove it entirely.

2. Add documents

# Add with auto-generated IDs
collection.add(
    documents=["Doc 1", "Doc 2", "Doc 3"],
    metadatas=[
        {"source": "web", "category": "tutorial"},
        {"source": "pdf", "page": 5},
        {"source": "api", "timestamp": "2025-01-01"}
    ],
    ids=["id1", "id2", "id3"]
)

# Add with custom embeddings
collection.add(
    embeddings=[[0.1, 0.2, ...], [0.3, 0.4, ...]],
    documents=["Doc 1", "Doc 2"],
    ids=["id1", "id2"]
)

You can add documents with or without precomputed embeddings. When you supply embeddings, Chroma skips the embedding step and stores the vectors you provide. Metadata is optional but recommended for filtering later.

3. Query (similarity search)

# Basic query
results = collection.query(
    query_texts=["machine learning tutorial"],
    n_results=5
)

# Query with filters
results = collection.query(
    query_texts=["Python programming"],
    n_results=3,
    where={"source": "web"}
)

# Query with metadata filters
results = collection.query(
    query_texts=["advanced topics"],
    where={
        "$and": [
            {"category": "tutorial"},
            {"difficulty": {"$gte": 3}}
        ]
    }
)

# Access results
print(results["documents"])      # List of matching documents
print(results["metadatas"])      # Metadata for each doc
print(results["distances"])      # Similarity scores
print(results["ids"])            # Document IDs

Query returns the n_results most similar documents. The where clause filters by metadata before similarity search, narrowing the candidate pool. The result dictionary gives you the documents, their metadata, distance scores (lower is more similar), and IDs.

4. Get documents

# Get by IDs
docs = collection.get(
    ids=["id1", "id2"]
)

# Get with filters
docs = collection.get(
    where={"category": "tutorial"},
    limit=10
)

# Get all documents
docs = collection.get()

Use get to retrieve documents without a similarity search. You can fetch by ID, by metadata filter, or everything at once. The limit parameter caps the number of returned documents.

5. Update documents

# Update document content
collection.update(
    ids=["id1"],
    documents=["Updated content"],
    metadatas=[{"source": "updated"}]
)

Update replaces the document text and metadata for existing IDs. The embedding is recomputed automatically if you change the document text.

6. Delete documents

# Delete by IDs
collection.delete(ids=["id1", "id2"])

# Delete with filter
collection.delete(
    where={"source": "outdated"}
)

Delete removes documents by ID or by metadata filter. The operation is immediate and permanent.

Persistent storage

# Persist to disk
client = chromadb.PersistentClient(path="./chroma_db")

collection = client.create_collection("my_docs")
collection.add(documents=["Doc 1"], ids=["id1"])

# Data persisted automatically
# Reload later with same path
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_collection("my_docs")

Use PersistentClient instead of Client to save data to disk. The database writes to the directory you specify. On restart, point to the same path and call get_collection to recover your data.

Embedding functions

Default (Sentence Transformers)

# Uses sentence-transformers by default
collection = client.create_collection("my_docs")
# Default model: all-MiniLM-L6-v2

OpenAI

from chromadb.utils import embedding_functions

openai_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key="your-key",
    model_name="text-embedding-3-small"
)

collection = client.create_collection(
    name="openai_docs",
    embedding_function=openai_ef
)

HuggingFace

huggingface_ef = embedding_functions.HuggingFaceEmbeddingFunction(
    api_key="your-key",
    model_name="sentence-transformers/all-mpnet-base-v2"
)

collection = client.create_collection(
    name="hf_docs",
    embedding_function=huggingface_ef
)

Custom embedding function

from chromadb import Documents, EmbeddingFunction, Embeddings

class MyEmbeddingFunction(EmbeddingFunction):
    def __call__(self, input: Documents) -> Embeddings:
        # Your embedding logic
        return embeddings

my_ef = MyEmbeddingFunction()
collection = client.create_collection(
    name="custom_docs",
    embedding_function=my_ef
)

You can swap the embedding model at collection creation time. The built-in options cover OpenAI, HuggingFace, and the default sentence-transformers. For any other model, implement the EmbeddingFunction interface.

Metadata filtering

# Exact match
results = collection.query(
    query_texts=["query"],
    where={"category": "tutorial"}
)

# Comparison operators
results = collection.query(
    query_texts=["query"],
    where={"page": {"$gt": 10}}  # $gt, $gte, $lt, $lte, $ne
)

# Logical operators
results = collection.query(
    query_texts=["query"],
    where={
        "$and": [
            {"category": "tutorial"},
            {"difficulty": {"$lte": 3}}
        ]
    }  # Also: $or
)

# Contains
results = collection.query(
    query_texts=["query"],
    where={"tags": {"$in": ["python", "ml"]}}
)

Metadata filtering supports exact match, comparison operators ($gt, $gte, $lt, $lte, $ne), logical operators ($and, $or), and the $in operator for list membership. Filters apply before the similarity search, so they can significantly reduce latency on large collections.

LangChain integration

from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter

# Split documents
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000)
docs = text_splitter.split_documents(documents)

# Create Chroma vector store
vectorstore = Chroma.from_documents(
    documents=docs,
    embedding=OpenAIEmbeddings(),
    persist_directory="./chroma_db"
)

# Query
results = vectorstore.similarity_search("machine learning", k=3)

# As retriever
retriever = vectorstore.as_retriever(search_kwargs={"k": 5})

LangChain wraps Chroma as a vector store. You can split documents, create the store, and use it as a retriever in a chain. The persist_directory parameter mirrors Chroma's PersistentClient path.

LlamaIndex integration

from llama_index.vector_stores.chroma import ChromaVectorStore
from llama_index.core import VectorStoreIndex, StorageContext
import chromadb

# Initialize Chroma
db = chromadb.PersistentClient(path="./chroma_db")
collection = db.get_or_create_collection("my_collection")

# Create vector store
vector_store = ChromaVectorStore(chroma_collection=collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# Create index
index = VectorStoreIndex.from_documents(
    documents,
    storage_context=storage_context
)

# Query
query_engine = index.as_query_engine()
response = query_engine.query("What is machine learning?")

LlamaIndex uses ChromaVectorStore to connect to a Chroma collection. You create the index with a storage context, then build a query engine on top.

Server mode

# Run Chroma server
# Terminal: chroma run --path ./chroma_db --port 8000

# Connect to server
import chromadb
from chromadb.config import Settings

client = chromadb.HttpClient(
    host="localhost",
    port=8000,
    settings=Settings(anonymized_telemetry=False)
)

# Use as normal
collection = client.get_or_create_collection("my_docs")

For production, run Chroma as a server with chroma run. The client connects via HTTP. The Settings(anonymized_telemetry=False) line disables telemetry. After connecting, the API is identical to the local client.

Best practices

1. Use persistent client - Don't lose data on restart
2. Add metadata - Enables filtering and tracking
3. Batch operations - Add multiple docs at once
4. Choose right embedding model - Balance speed/quality
5. Use filters - Narrow search space
6. Unique IDs - Avoid collisions
7. Regular backups - Copy chroma_db directory
8. Monitor collection size - Scale up if needed
9. Test embedding functions - Ensure quality
10. Use server mode for production - Better for multi-user

Performance

Operation	Latency	Notes
Add 100 docs	~1-3s	With embedding
Query (top 10)	~50-200ms	Depends on collection size
Metadata filter	~10-50ms	Fast with proper indexing

These are rough estimates. Actual latency depends on embedding model speed, collection size, and hardware.

When not to use it

Chroma is not the best choice for every scenario. Consider alternatives:

• Pinecone: Managed cloud service with auto-scaling, no infrastructure to maintain.
• FAISS: Pure similarity search without metadata filtering. Faster for raw vector search.
• Weaviate: Production-grade ML-native database with built-in vector and scalar indexing.
• Qdrant: High-performance Rust-based vector database, good for large-scale deployments.

If you need a fully managed service, want to avoid self-hosting, or require sub-millisecond queries at millions of vectors, one of these alternatives may fit better.

Limits and gotchas

• The default embedding model (all-MiniLM-L6-v2) is a good balance of speed and quality, but it may not capture domain-specific semantics. Test your embedding function on your data.
• Metadata filtering is fast, but complex nested $and/$or queries can slow down on very large collections.
• The in-memory client (chromadb.Client()) loses all data when the process exits. Always use PersistentClient for anything you want to keep.
• Collection names must be unique within a client. Deleting a collection is irreversible.
• The server mode requires the chroma CLI to be running. It does not start automatically.

What pairs with this

Chroma integrates directly with LangChain and LlamaIndex, two popular frameworks for building LLM applications. The upstream tags on this skill include RAG, Vector Database, Embeddings, Semantic Search, Document Retrieval, and Metadata Filtering. For a full Hermes Agent workflow, combine Chroma with document loaders, text splitters, and an LLM to build a complete RAG pipeline.