Conference Paper
Computer Vision
Featured

SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation

Vijay Badrinarayanan(University of Cambridge), Alex Kendall(University of Cambridge), Roberto Cipolla(University of Cambridge)
January 2, 2017IEEE Transactions on Pattern Analysis and Machine Intelligence19,608 citations
PDFDOI

20k

Citations

0

Influential Citations

IEEE Transactions on Pattern Analysis and Machine Intelligence

Venue

2017

Year

Abstract

We present a novel and practical deep fully convolutional neural network architecture for semantic pixel-wise segmentation termed SegNet. This core trainable segmentation engine consists of an encoder network, a corresponding decoder network followed by a pixel-wise classification layer. The architecture of the encoder network is topologically identical to the 13 convolutional layers in the VGG16 network [1] . The role of the decoder network is to map the low resolution encoder feature maps to full input resolution feature maps for pixel-wise classification. The novelty of SegNet lies is in the manner in which the decoder upsamples its lower resolution input feature map(s). Specifically, the decoder uses pooling indices computed in the max-pooling step of the corresponding encoder to perform non-linear upsampling. This eliminates the need for learning to upsample. The upsampled maps are sparse and are then convolved with trainable filters to produce dense feature maps. We compare our proposed architecture with the widely adopted FCN [2] and also with the well known DeepLab-LargeFOV [3] , DeconvNet [4] architectures. This comparison reveals the memory versus accuracy trade-off involved in achieving good segmentation performance. SegNet was primarily motivated by scene understanding applications. Hence, it is designed to be efficient both in terms of memory and computational time during inference. It is also significantly smaller in the number of trainable parameters than other competing architectures and can be trained end-to-end using stochastic gradient descent. We also performed a controlled benchmark of SegNet and other architectures on both road scenes and SUN RGB-D indoor scene segmentation tasks. These quantitative assessments show that SegNet provides good performance with competitive inference time and most efficient inference memory-wise as compared to other architectures. We also provide a Caffe implementation of SegNet and a web demo at http://mi.eng.cam.ac.uk/projects/segnet.

Analysis

Why This Paper Matters

SegNet addresses a core challenge in computer vision: semantic pixel-wise segmentation, where each pixel in an image must be classified into a category. Prior to SegNet, fully convolutional networks (FCNs) had shown promise, but they often required large memory and computational resources, especially during inference. SegNet's primary motivation was scene understanding for applications like autonomous driving and indoor robotics, where efficiency in memory and time is critical. By designing an architecture that is both accurate and resource-efficient, SegNet bridged the gap between high-performance segmentation and practical deployment.

The paper's significance is further amplified by its widespread adoption and citation (over 19,600 citations). It introduced a simple yet effective idea: using max-pooling indices from the encoder to guide upsampling in the decoder. This eliminated the need for learning to upsample, reducing parameters and memory footprint. The architecture's compatibility with VGG16 weights also enabled transfer learning, making it accessible to researchers and practitioners. SegNet's release of a Caffe implementation and a web demo further democratized access, solidifying its role as a foundational model in the field.

Technical Contributions

  • Encoder-Decoder Architecture: SegNet consists of a VGG16-based encoder (13 convolutional layers) and a corresponding decoder, followed by a pixel-wise classification layer. The decoder mirrors the encoder's structure but uses pooling indices for upsampling.
  • Pooling Index Upsampling: The key innovation is the use of max-pooling indices computed during encoding to perform non-linear upsampling in the decoder. This avoids learning upsampling filters, reducing parameters and memory.
  • Memory Efficiency: By storing only pooling indices (rather than full feature maps) and using them for upsampling, SegNet achieves significant memory savings during inference compared to FCN and DeconvNet.
  • End-to-End Training: The entire network can be trained end-to-end using stochastic gradient descent, with no need for post-processing or separate stages.
  • Benchmarking: The paper provides a controlled comparison with FCN, DeepLab-LargeFOV, and DeconvNet on road scenes and SUN RGB-D indoor scenes, highlighting the memory-accuracy trade-off.

Results

The paper reports that SegNet achieves good segmentation performance with competitive inference time and the most efficient inference memory among the compared architectures. While specific accuracy numbers are not detailed in the abstract, the controlled benchmark on road scenes and SUN RGB-D indoor scenes demonstrates that SegNet provides a favorable balance between accuracy and resource usage. Notably, SegNet has significantly fewer trainable parameters than competing models, which contributes to its efficiency. The comparison reveals that SegNet's memory footprint during inference is substantially lower than that of FCN and DeconvNet, making it suitable for embedded and real-time applications.

Significance

SegNet's impact on the AI field is profound. It popularized the use of encoder-decoder architectures with skip connections for dense prediction tasks, influencing subsequent models like U-Net and DeepLab variants. The pooling-index upsampling technique inspired further research on efficient upsampling methods. Moreover, SegNet's emphasis on inference efficiency paved the way for real-time semantic segmentation in autonomous driving and robotics. Its open-source implementation and web demo accelerated adoption in both academia and industry. Today, SegNet remains a standard baseline and a teaching tool for understanding segmentation architectures, underscoring its lasting legacy in computer vision.

semantic-segmentationencoder-decoderdeep-learningcomputer-visionpooling-indicesvgg16scene-understandingefficient-inference
BenchmarksNeural Networks

Related Papers