Journal Article
Computer Vision

Toward Remote Sensing Image Retrieval Under a Deep Image Captioning Perspective

Genc Hoxha(University of Trento), Farid Melgani(University of Trento), Begüm Demir(Technische Universität Berlin)
January 1, 2020IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing94 citations
PDFDOI

94

Citations

3

Influential Citations

IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing

Venue

2020

Year

Abstract

The performance of remote sensing image retrieval (RSIR) systems depends on the capability of the extracted features in characterizing the semantic content of images. Existing RSIR systems describe images by visual descriptors that model the primitives (such as different land-cover classes) present in the images. However, the visual descriptors may not be sufficient to describe the high-level complex content of RS images (e.g., attributes and relationships among different land-cover classes). To address this issue, in this article, we present an RSIR system that aims at generating and exploiting textual descriptions to accurately describe the relationships between the objects and their attributes present in RS images with captions (i.e., sentences). To this end, the proposed retrieval system consists of three main steps. The first step aims to encode the image visual features and then translate the encoded features into a textual description that summarizes the content of the image with captions. This is achieved based on the combination of a convolutional neural network with a recurrent neural network. The second step aims to convert the generated textual descriptions into semantically meaningful feature vectors. This is achieved by using the recent word embedding techniques. Finally, the last step estimates the similarity between the vectors of the textual descriptions of the query image and those of the archive images, and then retrieve the most similar images to the query image. Experimental results obtained on two different datasets show that the description of the image content with captions in the framework of RSIR leads to an accurate retrieval performance.

Analysis

Why This Paper Matters

Traditional remote sensing image retrieval (RSIR) systems rely on visual descriptors that model low-level primitives such as land-cover classes. However, these descriptors often fail to capture high-level semantic content like object attributes and inter-class relationships. This paper addresses a critical gap by introducing a deep image captioning perspective to RSIR, where textual descriptions are generated and used as retrieval features. This shift from purely visual to multimodal (visual+textual) representation is significant because it aligns retrieval with human-like understanding of image semantics, potentially improving accuracy and interpretability in applications like environmental monitoring, urban planning, and disaster response.

By leveraging advances in image captioning and word embeddings, the authors propose a pipeline that not only retrieves images but also provides a natural language summary of their content. This dual output (image + caption) adds a layer of explainability that is often missing in black-box retrieval systems. The work is timely given the explosion of remote sensing data and the need for more intelligent search tools.

Technical Contributions

  • Novel RSIR Framework: Integrates deep image captioning (CNN+RNN) directly into the retrieval pipeline, moving beyond visual-only features.
  • Caption Generation: Uses a CNN to extract visual features and an RNN to decode them into descriptive sentences, capturing object relationships and attributes.
  • Semantic Feature Extraction: Applies word embedding techniques (e.g., Word2Vec or similar) to convert generated captions into fixed-length feature vectors that encode semantic meaning.
  • Similarity Matching: Computes cosine or other similarity metrics between query caption vectors and archive caption vectors for retrieval.
  • End-to-End Pipeline: The system is designed as a coherent three-step process (encoding, captioning, embedding, retrieval) that can be trained jointly or sequentially.

Results

The paper reports experimental results on two remote sensing datasets (not named in the abstract). The key finding is that caption-based retrieval achieves accurate performance, outperforming traditional visual descriptor-based methods. While specific numerical metrics (e.g., mean average precision) are not provided in the abstract, the authors claim that describing image content with captions leads to more accurate retrieval. This suggests that the semantic richness of captions compensates for the loss of low-level visual detail. The use of two datasets adds robustness to the evaluation.

Significance

This research opens a new direction for RSIR by demonstrating that textual captions can serve as effective retrieval features. It bridges the gap between computer vision and natural language processing in the remote sensing domain, encouraging further work on multimodal retrieval systems. The approach also has practical implications: users could query images using natural language descriptions, making retrieval more intuitive. Future work could explore attention mechanisms, transformer-based captioning, or joint embedding spaces to further improve performance. The paper's impact extends to any field where semantic understanding of aerial or satellite imagery is critical.

remote-sensingimage-retrievalimage-captioningdeep-learningcnnrnnword-embedding
Retrieval Augmented GenerationNeural Networks

Related Papers