Conference Paper
Computer Vision

From Show to Tell: A Survey on Deep Learning-Based Image Captioning

Matteo Stefanini(University of Modena and Reggio Emilia), Marcella Cornia(University of Modena and Reggio Emilia), Lorenzo Baraldi(University of Modena and Reggio Emilia), Silvia Cascianelli(University of Modena and Reggio Emilia), Giuseppe Fiameni, Rita Cucchiara(University of Modena and Reggio Emilia)
February 7, 2022IEEE Transactions on Pattern Analysis and Machine Intelligence396 citations
arXivPDFDOI

396

Citations

13

Influential Citations

IEEE Transactions on Pattern Analysis and Machine Intelligence

Venue

2022

Year

Abstract

Connecting Vision and Language plays an essential role in Generative Intelligence. For this reason, large research efforts have been devoted to image captioning, i.e. describing images with syntactically and semantically meaningful sentences. Starting from 2015 the task has generally been addressed with pipelines composed of a visual encoder and a language model for text generation. During these years, both components have evolved considerably through the exploitation of object regions, attributes, the introduction of multi-modal connections, fully-attentive approaches, and BERT-like early-fusion strategies. However, regardless of the impressive results, research in image captioning has not reached a conclusive answer yet. This work aims at providing a comprehensive overview of image captioning approaches, from visual encoding and text generation to training strategies, datasets, and evaluation metrics. In this respect, we quantitatively compare many relevant state-of-the-art approaches to identify the most impactful technical innovations in architectures and training strategies. Moreover, many variants of the problem and its open challenges are discussed. The final goal of this work is to serve as a tool for understanding the existing literature and highlighting the future directions for a research area where Computer Vision and Natural Language Processing can find an optimal synergy.

Analysis

Why This Paper Matters

Image captioning sits at the intersection of computer vision and natural language processing, a critical area for generative AI. This survey, published in IEEE TPAMI with 396 citations, offers a timely and comprehensive review of the field from 2015 onward. As models have evolved from region-based CNNs to fully-attentive Transformers and BERT-like early fusion, the literature has grown fragmented. This paper provides a structured map of the landscape, making it invaluable for both newcomers and seasoned researchers.

The survey's quantitative comparison of many state-of-the-art methods is particularly valuable. By systematically evaluating architectures and training strategies on standard benchmarks, the authors identify which innovations truly drive performance. This helps the community focus on the most promising directions rather than chasing incremental gains.

Technical Contributions

  • Comprehensive taxonomy: Categorizes approaches by visual encoder (object regions, grid features, attributes), language model (RNN, Transformer, BERT-like), and training strategy (cross-entropy, reinforcement learning, pretraining).
  • Quantitative comparison: Evaluates many methods on COCO Captions using standard metrics (BLEU, METEOR, ROUGE, CIDEr, SPICE), highlighting the impact of fully-attentive models and early fusion.
  • Discussion of variants: Covers dense captioning, video captioning, and multilingual captioning, showing the breadth of the problem space.
  • Open challenges: Identifies issues like bias, evaluation limitations, and the need for more diverse datasets.

Results

The survey reports that fully-attentive models (e.g., Transformers) and BERT-like early fusion strategies consistently outperform earlier region-based approaches on COCO Captions. For example, models using object detectors as encoders achieve CIDEr scores around 120, while Transformer-based models reach over 130. The paper also notes that reinforcement learning-based training (e.g., CIDEr optimization) yields significant gains over standard cross-entropy.

Significance

This survey consolidates a decade of rapid progress in image captioning, providing a clear reference for the community. By identifying the most impactful innovations, it helps guide future research toward underexplored areas like compositional generalization, bias mitigation, and real-world deployment. The work underscores the synergy between vision and language, a cornerstone of multimodal AI.

image-captioningdeep-learningcomputer-visionnatural-language-processingsurveyattentionvision-language
Computer Vision

Related Papers