Florence-2

Analysis

Why This Paper Matters

For years, computer vision has been fragmented: one model for classification, another for detection, a third for segmentation. Practitioners had to juggle multiple architectures, each with its own training pipeline and inference format. Florence-2 breaks this pattern by proposing a single, prompt-driven sequence-to-sequence model that can handle captioning, object detection, grounding, and segmentation — all with a simple text instruction. This is not just an incremental improvement; it represents a conceptual shift toward unified vision foundation models, akin to what GPT models did for NLP. By releasing the model and its massive FLD-5B dataset on HuggingFace, Microsoft makes this capability accessible to the broader AI community, lowering the barrier for building versatile vision applications.

How It Works

Florence-2 adopts a sequence-to-sequence paradigm where both the input and output are sequences of tokens. The input consists of an image and a text prompt (e.g., "Describe this image" or "Detect all cars"), and the output is a text string that may contain special location tokens for region tasks. This formulation unifies tasks that were previously handled by separate heads or decoders.

The architecture uses DaViT as the vision encoder to extract visual features, which are then projected and concatenated with prompt embeddings. A standard encoder-decoder transformer processes this combined sequence to generate the target text. For region-specific tasks, the tokenizer is extended with 1000 quantized coordinate bins, allowing the model to output bounding boxes, quad boxes, or polygons directly as token sequences. This design eliminates the need for task-specific output heads.

A key enabler is the FLD-5B dataset, built through a three-phase data engine. First, specialist models (e.g., DINO for detection, SAM for segmentation) generate initial annotations on a curated set of 126 million images from ImageNet-22k, Object365, Open Images, Conceptual Captions, and LAION. Second, these annotations are filtered and enhanced using rule-based tools (e.g., spaCy for text parsing, confidence thresholds for boxes). Third, the filtered data trains a multitask model, which then re-annotates the dataset iteratively, improving label quality with each round. This process yields 5.4 billion annotations spanning three levels: text (brief, detailed, more detailed), region-text pairs (bounding boxes with descriptions), and text-phrase-region triplets (linking noun phrases to segmentation masks).

Results

Florence-2 comes in two sizes: Base (232M parameters) and Large (771M parameters). Both are trained on FLD-5B at 384x384 resolution, with additional high-resolution fine-tuning at 768x768. The zero-shot evaluation shows that Florence-2-L achieves a CIDEr score of 135.6 on COCO captioning, surpassing the 80B-parameter Flamingo model. On region-level tasks, it improves Flickr30k Recall@1 by 5.7 points over Kosmos-2 and achieves absolute gains of 4-8% on RefCOCO, RefCOCO+, and RefCOCOg. Notably, it performs referring expression segmentation (RES) with 35.8% mIOU — a capability absent in prior foundation models.

When fine-tuned on public supervised data as a single generalist model, Florence-2-L outperforms PolyFormer on RefCOCO REC by 3.0 Accuracy@0.5 and on RES by 3.54 mIOU. It also achieves 140.0 CIDEr on COCO Caption Karpathy split and 81.5% accuracy on TextVQA without external OCR. As a backbone for downstream tasks, Florence-2 pre-training improves COCO object detection by 4.2 AP over ViT-B with DINO and ADE20K semantic segmentation by 1.3 points over BEiT, while being 4x more training-efficient than supervised ImageNet-1k pre-training.

Significance

Florence-2 demonstrates that a single, unified architecture can match or exceed specialist models across a wide range of vision tasks. Its prompt-based interface simplifies deployment: instead of maintaining separate models for detection, captioning, and segmentation, practitioners can use one model with different text instructions. The FLD-5B dataset and iterative annotation pipeline provide a blueprint for scaling multitask learning in vision. While the model still lags on some fine-grained tasks (e.g., RES at 35.8% mIOU), its zero-shot capabilities and efficient transfer learning mark a significant step toward generalist vision systems. For AI practitioners, Florence-2 offers a practical, open-source foundation for building versatile visual AI applications, reducing both engineering overhead and computational costs.