Feature Pyramid Network

Analysis

Why This Paper Matters

Object detection at multiple scales has long been a challenge. Traditional methods used image pyramids—processing the same image at different resolutions—which is computationally expensive and memory-intensive. Deep convolutional networks naturally produce a feature hierarchy with decreasing spatial resolution and increasing semantic strength, but earlier detectors often used only the topmost feature map, discarding fine-grained details needed for small objects. Feature Pyramid Network (FPN) elegantly solves this by constructing a feature pyramid within the network itself, leveraging the inherent multi-scale hierarchy with minimal extra cost. This makes it a foundational building block for many modern detection systems.

How It Works

FPN takes a single-scale image as input and outputs proportionally sized feature maps at multiple levels in a fully convolutional manner. The architecture consists of three key components:

Bottom-up pathway: This is the standard feedforward computation of a backbone ConvNet (e.g., ResNet). It produces feature maps at several scales, with each scale step reducing spatial dimensions by a factor of 2. The output of the last layer of each network stage is chosen as the reference feature map for that pyramid level, as it contains the strongest features.

Top-down pathway and lateral connections: The top-down pathway upsamples spatially coarser but semantically stronger feature maps from higher pyramid levels to hallucinate higher-resolution features. These upsampled features are then merged via lateral connections with the corresponding bottom-up feature maps of the same spatial size. The bottom-up maps have lower-level semantics but better localization because they have been subsampled fewer times. This combination yields features that are both semantically rich and spatially precise at every scale.

Integration with RPN and Fast R-CNN: For region proposal (RPN), a small network head (3×3 conv followed by two 1×1 convs for classification and regression) is attached to each pyramid level. Anchors of a single scale are assigned to each level, eliminating the need for multi-scale anchors. For Fast R-CNN, RoIs of different scales are assigned to pyramid levels using a formula: k = k0 + log2(√(wh)/224), where k0 is typically 4. This assigns larger RoIs to higher pyramid levels and smaller ones to lower levels.

Results

In ablation experiments on COCO, FPN significantly outperforms a strong single-scale Faster R-CNN baseline on ResNets. For bounding box proposals, Average Recall (AR) increases by 8.0 points. For object detection, COCO-style Average Precision (AP) improves by 2.3 points and PASCAL-style AP by 3.8 points. These gains come with minimal extra computational cost, as the pyramid is built within the network rather than by processing multiple image scales.

Significance

FPN introduced a simple yet powerful paradigm for multi-scale feature extraction that has become a standard component in many subsequent object detectors (e.g., Mask R-CNN, RetinaNet). Its design principle—combining top-down semantic features with bottom-up spatial details via lateral connections—has been widely adopted in segmentation and other dense prediction tasks. By enabling efficient and accurate multi-scale representation, FPN helped push the state of the art on benchmarks like COCO and influenced the development of later architectures such as NAS-FPN and BiFPN.