ColPali

Free

Layout-rich PDF retrieval

FreeFree tier

Inputs: image, text

Type

Open Source

Company

illuin-tech

Visit Website GitHub (38 stars)

About ColPali

ColPali is an open-source model for efficient document retrieval using vision-language models (VLMs). It constructs multi-vector embeddings directly from the visual content of documents, such as PDFs, by feeding ViT output patches from a VLM (e.g., PaliGemma-3B) through a linear projection, following the ColBERT late-interaction method. This approach eliminates the need for separate OCR and layout analysis pipelines, as the single model can understand both textual and visual elements (text, layout, charts, images). The repository includes multiple model variants (ColPali, ColQwen2, ColSmol) with different backbones and licenses, all evaluated on the ViDoRe benchmark, achieving top scores up to 90.9. The code supports training and inference for visual document retrieval.

Key Features

Leverages vision-language models (PaliGemma, Qwen2-VL) for document retrieval

Creates multi-vector embeddings in visual space using ColBERT-style late interaction

Eliminates need for OCR and layout recognition pipelines

Supports multiple model variants with different sizes and licenses (Gemma, Apache 2.0)

State-of-the-art performance on ViDoRe benchmark (up to 90.9 score)

Open-source code for training and inference

Dynamic resolution support in ColQwen2 models

Pros & Cons

Pros

Eliminates brittle OCR and layout analysis steps with a single end-to-end model
Handles both textual and visual document content (layout, charts, images)
High accuracy on the ViDoRe benchmark, outperforming text-only retrievers
Multiple model variants allow trade-offs between speed, size, and accuracy
Fully open-source with permissive licenses for many variants

Best For

Retrieval of layout-rich documents such as PDFs, reports, and invoicesSearch over documents containing charts, tables, and imagesBuilding retrieval-augmented generation (RAG) systems for visual documentsEnterprise document search and knowledge management without OCR preprocessing

FAQ

What is ColPali?

ColPali is an open-source model for visual document retrieval. It uses vision-language models to create multi-vector embeddings from document images, enabling efficient search without OCR.

How does ColPali differ from traditional OCR-based retrieval?

Traditional retrieval often requires separate OCR and layout analysis, which can be brittle. ColPali uses a single vision-language model that directly processes document images, capturing both text and visual layout in one step.

What models are available in this repository?

The repository includes ColPali (based on PaliGemma), ColQwen2 (based on Qwen2-VL), and ColSmol (based on SmolVLM). Each has multiple versions with different licenses and performance levels, as listed in the README.