SmolDocling

Analysis

Why This Paper Matters

Document conversion remains a critical bottleneck in digitizing workflows across industries. Traditional approaches rely on multi-stage pipelines—OCR, layout analysis, table extraction—each introducing errors that compound. Large vision-language models can handle these tasks end-to-end but require substantial computational resources, limiting deployment. SmolDocling addresses this gap by packing powerful document understanding into a 256M parameter model that runs efficiently on consumer hardware. Its novel DocTags format unifies content, structure, and spatial information into a single sequence, simplifying training and inference. This work matters because it makes state-of-the-art document conversion practical for real-world applications like automated form processing, patent analysis, and code documentation.

How It Works

SmolDocling builds on Hugging Face's SmolVLM architecture, which combines a SigLIP visual backbone (93M parameters) with a lightweight SmolLM-2 language model (135M parameters). The visual encoder processes 512x512 image patches, compressing each into 64 visual tokens via radical pixel shuffle, achieving a high pixel-to-token ratio of 4096 pixels per token. The training data is rebalanced to emphasize document understanding (41%) and image captioning (14%), using datasets like The Cauldron, Docmatix, and MathWriting.

DocTags Format

DocTags is the core innovation—an XML-like markup that explicitly separates text content from document structure. Each element includes location tags using a fixed grid coordinate system (0–500), enabling precise spatial awareness. The format supports diverse elements: text, captions, footnotes, formulas, titles, headers, code, lists, tables, and pictures. For tables, DocTags integrates OTSL (Optimized Table Structure Language) to encode cell spans and headers. Code elements preserve formatting and include a classification tag for the programming language (57 supported). Picture elements include image class tags (e.g., pie_chart, bar_chart, natural_image). This uniform representation ensures consistent visual cues whether processing full pages or cropped elements.

Curriculum Learning

Training follows a three-stage curriculum. First, DocTags tokens are added to the tokenizer, and the vision encoder is frozen while the rest of the network adapts to the new output format. Second, the vision encoder is unfrozen, and the model trains on pretraining datasets alongside task-specific conversion datasets for tables, code, equations, and charts. Finally, fine-tuning uses all available datasets. This progressive alignment ensures faster convergence and better performance.

Data Generation

Existing datasets like OCR-IDL, IIT-CDIP, CCpdf, and WordScape lack comprehensive annotations. To overcome this, the authors create DocLayNet-PT by extracting 1.4 million pages from DocFM and adding weak annotations via PDF parsing and enrichment. Docmatix is similarly annotated to preserve DocVQA capabilities. Task-specific datasets are generated: 2.5 million charts from FinTabNet tables rendered with three visualization libraries, 9.3 million code snippets in 56 languages rendered via LaTeX and Pygments, and 5.5 million unique formulas rendered at 120 dpi. For instruction tuning, rule-based techniques and the Granite-3.1-2b-instruct LLM generate tasks like OCR at bounding boxes and element type identification.

Results

SmolDocling outperforms larger models (GOT, Nougat, Qwen2.5-VL) in full-page text transcription and establishes benchmark results for code listing recognition. In formula recognition, it matches GOT and surpasses Nougat and Qwen2.5-VL.

For layout analysis, SmolDocling significantly outperforms Qwen2.5-VL-7b, though both fall below human baseline due to dataset complexity.

Table structure recognition is competitive against larger models, with stronger performance on structure-only evaluation (omitting text content). Challenges arise from low-resolution images in FinTabNet and PubTables-1M.

Chart extraction achieves competitive performance despite dataset inconsistencies.

Significance

SmolDocling demonstrates that compact models can rival much larger ones in document understanding when equipped with the right output format and training strategy. The DocTags format is a key contribution—it disentangles content from structure, reducing confusion for image-to-sequence models and providing a clean interface for downstream tasks. By generating diverse synthetic datasets for charts, code, and equations, the work addresses critical data scarcity issues. The curriculum learning approach offers a blueprint for training efficient multimodal models. Practically, SmolDocling enables document conversion on edge devices, opening possibilities for real-time document processing in mobile apps, IoT, and privacy-sensitive environments. Future work could extend DocTags to more complex layouts, improve handling of low-resolution inputs, and explore distillation for even smaller models.