Conference Paper
Machine Learning
Featured

Squeeze-and-Excitation Networks

Jie Hu(Chinese Academy of Sciences), Li Shen(University of Oxford), Samuel Albanie(University of Oxford), Gang Sun(Chinese Academy of Sciences), Enhua Wu(Chinese Academy of Sciences)
May 2, 2019IEEE Transactions on Pattern Analysis and Machine Intelligence12,140 citations
DOI

12k

Citations

0

Influential Citations

IEEE Transactions on Pattern Analysis and Machine Intelligence

Venue

2019

Year

Abstract

The central building block of convolutional neural networks (CNNs) is the convolution operator, which enables networks to construct informative features by fusing both spatial and channel-wise information within local receptive fields at each layer. A broad range of prior research has investigated the spatial component of this relationship, seeking to strengthen the representational power of a CNN by enhancing the quality of spatial encodings throughout its feature hierarchy. In this work, we focus instead on the channel relationship and propose a novel architectural unit, which we term the "Squeeze-and-Excitation" (SE) block, that adaptively recalibrates channel-wise feature responses by explicitly modelling interdependencies between channels. We show that these blocks can be stacked together to form SENet architectures that generalise extremely effectively across different datasets. We further demonstrate that SE blocks bring significant improvements in performance for existing state-of-the-art CNNs at slight additional computational cost. Squeeze-and-Excitation Networks formed the foundation of our ILSVRC 2017 classification submission which won first place and reduced the top-5 error to 2.251 percent, surpassing the winning entry of 2016 by a relative improvement of ∼ 25 percent. Models and code are available at https://github.com/hujie-frank/SENet.

Analysis

Why This Paper Matters

Squeeze-and-Excitation Networks (SENet) introduced a simple yet powerful architectural unit that fundamentally changed how practitioners think about channel attention in convolutional neural networks. Prior to this work, most research focused on enhancing spatial encoding within CNNs, but Hu et al. shifted the spotlight to the channel dimension, demonstrating that explicitly modelling interdependencies between channels can yield substantial performance gains. This paper matters because it provided a plug-and-play module that consistently improves existing architectures with minimal additional computational cost, making it highly practical for real-world deployment.

The significance of SENet is further underscored by its victory in the ILSVRC 2017 classification challenge, where it achieved a top-5 error of 2.251%, a relative improvement of approximately 25% over the 2016 winner. This result not only validated the effectiveness of the SE block but also cemented its place as a foundational technique in the deep learning toolbox. Since its publication, the SE block has been widely adopted and inspired numerous follow-up works on attention mechanisms, such as CBAM and ECANet, highlighting its lasting impact on the field.

Technical Contributions

  • Squeeze-and-Excitation Block: A novel architectural unit that performs two operations: squeeze (global average pooling to aggregate spatial information) and excitation (a bottleneck fully connected network to learn channel-wise weights).
  • Channel-wise Recalibration: The block adaptively rescales each channel's feature response based on the learned weights, allowing the network to emphasize informative channels and suppress less useful ones.
  • Architectural Flexibility: SE blocks can be seamlessly integrated into existing CNN architectures (e.g., ResNet, Inception) with negligible additional parameters and computation.
  • SENet Architectures: Stacking SE blocks yields SENets, which generalize well across datasets and achieve state-of-the-art results.

Results

The paper reports extensive experiments on ImageNet, CIFAR-10, and CIFAR-100, showing consistent improvements over baseline models. On ImageNet, SENet achieved a top-5 error of 2.251%, winning the ILSVRC 2017 challenge. When SE blocks were added to ResNet-50, the top-1 error decreased from 24.7% to 23.3%, and top-5 error from 7.8% to 6.7%. Similar gains were observed across different depths and architectures, with only a slight increase in computational cost (e.g., ~10% additional FLOPs for ResNet-50). The improvements were also consistent on smaller datasets like CIFAR-10 and CIFAR-100, demonstrating the generalizability of the approach.

Significance

The introduction of the SE block has had a profound impact on the computer vision community and beyond. It popularized the concept of channel attention, leading to a wave of research on attention mechanisms in CNNs. The paper's code and models were made publicly available, facilitating widespread adoption and further innovation. With over 12,000 citations, SENet is considered a milestone in deep learning architecture design, and its principles have been extended to other domains such as natural language processing and speech recognition. The work also underscores the importance of architectural innovations that are both effective and efficient, setting a benchmark for future research in neural network design.

deep-learningcomputer-visionattention-mechanismconvolutional-neural-networksimage-classificationchannel-attentionsenetfeature-recalibration
Neural Networks

Related Papers