FastViT

Analysis

Why This Paper Matters

Vision transformers have revolutionized image understanding, but their high computational cost and memory footprint hinder deployment on resource-constrained devices like smartphones and embedded systems. FastViT addresses this by introducing a hybrid architecture that combines the strengths of convolutions and transformers while dramatically improving inference speed. Its key innovation—RepMixer—reparameterizes skip connections into single depthwise convolutions at inference time, slashing memory access overhead without sacrificing accuracy. This makes FastViT a strong candidate for real-time applications where latency is critical.

How It Works

FastViT is a four-stage hybrid transformer. Each stage processes features at a different spatial resolution, similar to ConvNeXt or PoolFormer. The core novelty is the RepMixer token mixing block.

RepMixer: Reparameterizing Skip Connections

Traditional ConvMixer blocks apply depthwise convolution followed by a nonlinear activation and a skip connection. RepMixer rearranges these operations: it removes the nonlinearity and reorders the skip connection so that at inference time, the entire block collapses into a single depthwise convolution. This is achieved by noting that a depthwise convolution followed by an elementwise addition of the input (skip connection) is equivalent to a depthwise convolution with a kernel that has a 1 in the center position for the skip path. Formally, the training-time computation is:

which at inference becomes:

This reparameterization eliminates the extra memory access for the skip connection, reducing latency.

Architecture Details

FastViT uses a convolutional stem (instead of a patch embedding) to process input images. The stem employs factorized depthwise+pointwise convolutions with train-time overparameterization to maintain representational capacity while reducing FLOPs. In later stages, large kernel depthwise convolutions (e.g., 7×7) are inserted into the feedforward network (FFN) and patch embedding layers to enlarge the receptive field without the quadratic cost of self-attention. Conditional positional encodings, generated by a depthwise convolution, are added to patch embeddings to provide spatial information.

Variants

FastViT comes in several sizes: S (small embedding dimensions), M (medium), T (tiny with reduced MLP expansion ratio), and SA (with self-attention layers). The number indicates total blocks.

Results

ImageNet Classification

FastViT-MA36 achieves 84.9% top-1 accuracy, matching LITv2-B while being 49.3% smaller and using 55.4% fewer FLOPs. On mobile devices, FastViT-S12 is 26.3% faster than MobileOne-S4. Compared to ConvNeXt-B, FastViT-MA36 is 1.9× faster on iPhone 12 Pro and 2.0× faster on GPU at 83.9% accuracy.

Knowledge Distillation

FastViT-SA24 matches EfficientFormer-L7 performance with 3.8× fewer parameters, 2.7× fewer FLOPs, and 2.7× lower latency.

Robustness

FastViT shows strong robustness on ImageNet-C (corruptions), ImageNet-A (adversarial), ImageNet-R (renditions), and ImageNet-Sketch, outperforming ConvNeXt-S on clean accuracy and corruption robustness despite fewer parameters.

Downstream Tasks

• 3D Hand Mesh Estimation: FastViT is 1.9× faster than MobileHand and 2.8× faster than MobRecon on FreiHAND.
• Semantic Segmentation: FastViT-MA36 outperforms PoolFormer-M36 by 5.2% mIoU on ADE20K.
• Object Detection: FastViT-MA36 matches CMT-S performance but is 2.4× faster on GPU and 4.3× faster on mobile.

Significance

FastViT demonstrates that careful architectural reparameterization can yield substantial speedups without compromising accuracy. By merging the skip connection into the convolution at inference, it reduces memory bandwidth—a key bottleneck on mobile hardware. The hybrid design (convolutional stem + transformer stages + large kernel depthwise convs) offers a practical recipe for building efficient vision transformers. Its strong performance across multiple tasks (classification, segmentation, detection, hand mesh estimation) suggests broad applicability. Future work could explore scaling to larger models or integrating with other efficient operators like attention with linear complexity.