ConvNeXt V2

Analysis

Why This Paper Matters

ConvNeXt V2 tackles a critical gap in modern computer vision: while transformers have dominated self-supervised learning with masked autoencoders (MAE), convolutional networks have struggled to benefit from similar pretraining. The original ConvNeXt showed that pure ConvNets could rival transformers in supervised settings, but when combined with MAE, performance gains were minimal. ConvNeXt V2 closes this gap by co-designing the pretraining framework and the network architecture itself, proving that ConvNets can be just as effective in self-supervised regimes when properly adapted.

How It Works

The paper introduces two key innovations: a Fully Convolutional Masked Autoencoder (FCMAE) and a Global Response Normalization (GRN) layer.

Fully Convolutional Masked Autoencoder

During pretraining, 60% of 32×32 patches are randomly masked from the input image. Only random resized cropping is used for augmentation. The encoder is a ConvNeXt model, but standard convolutions are replaced with submanifold sparse convolutions so that computation only occurs on visible patches. This is critical for efficiency—masking would otherwise waste compute on masked regions. After pretraining, these sparse convolutions are seamlessly converted back to standard convolutions for fine-tuning. The decoder is a lightweight plain ConvNeXt block. The reconstruction target is the mean squared error between the reconstructed masked patches and patch-wise normalized original pixel values.

Global Response Normalization

GRN is designed to increase channel contrast and selectivity. It operates in three steps: global feature aggregation (computing L2 norm per channel), feature normalization (dividing each channel by its aggregated norm), and feature calibration (scaling the normalized values). In ConvNeXt V2, GRN is inserted after the dimension expansion MLP layer, and the previously used LayerScale is removed as it becomes redundant.

The co-design is essential: using FCMAE alone without GRN yields limited improvement, and GRN alone in supervised training has minor effect. Only together do they unlock significant gains.

Results

On ImageNet-1K, the combination of FCMAE and GRN leads to substantial fine-tuning improvements over both supervised baselines and FCMAE-only models. Scaling behavior is strong across model sizes from 3.7M to 650M parameters.

Compared to prior masked image modeling approaches, ConvNeXt V2 outperforms Swin+SimMIM across all sizes and matches ViT+MAE up to the large model regime with fewer parameters. In the huge regime, it slightly lags behind ViT, likely because large ViTs benefit more from self-supervised pretraining.

On ImageNet-21K fine-tuning, ConvNeXt V2 Huge achieves 88.9% top-1 accuracy, a new state-of-the-art among methods using only public data.

On COCO object detection and instance segmentation with Mask-RCNN, FCMAE pretraining consistently improves over supervised baselines, and the GRN layer further boosts performance. ConvNeXt V2 outperforms Swin transformers across all model sizes.

On ADE20K semantic segmentation with UPerNet, ConvNeXt V2 surpasses its supervised V1 counterpart and competes favorably with Swin, especially in the huge model regime.

Significance

ConvNeXt V2 demonstrates that convolutional networks are not obsolete for self-supervised learning. By carefully co-designing the pretraining objective (FCMAE) and the network architecture (GRN), the authors achieve results that rival or exceed transformer-based methods on major benchmarks. This work encourages the community to revisit ConvNet designs for modern pretraining paradigms, potentially leading to more efficient and scalable vision models. The open-source release of pretrained models also enables further research and practical applications.