Stronger Normalization-Free Transformers

Analysis

Why This Paper Matters

Normalization layers (e.g., LayerNorm, BatchNorm) have been considered essential for training deep Transformers, but they add complexity and computational cost. The recent Dynamic Tanh (DyT) showed that a simple pointwise function could replace normalization, but its performance was limited. This paper pushes further by systematically searching for a better function, leading to Derf—a rescaled error function that not only matches but surpasses standard normalization across diverse domains. This matters because it challenges the long-held assumption that normalization is indispensable, opening the door to simpler, faster, and more generalizable architectures.

The work is particularly timely as the AI field seeks to reduce architectural complexity and improve efficiency. By demonstrating that a single pointwise function can outperform multiple normalization techniques on vision, speech, and DNA tasks, the authors provide a strong empirical case for normalization-free design. This could influence future Transformer variants and simplify deployment in resource-constrained settings.

Technical Contributions

• Property-driven search: The authors first identify key properties of pointwise functions that affect training—such as boundedness, saturation behavior, and slope at zero—then use these insights to guide a large-scale empirical search.
• Derf function: Derf(x) = erf(αx + s), where erf is the rescaled Gaussian CDF. The learnable parameters α (scale) and s (shift) allow the function to adapt its shape during training, balancing linear and saturating regimes.
• Generalization analysis: Through controlled experiments, the authors show that Derf's advantage comes from better generalization (lower test error relative to training error) rather than higher training accuracy, suggesting it acts as an implicit regularizer.
• Broad evaluation: The method is tested on ImageNet classification, class-conditional image generation (e.g., with DiT), speech representation learning (wav2vec 2.0), and DNA sequence classification, covering both vision and non-vision modalities.

Results

• ImageNet classification: Derf achieves 82.1% top-1 accuracy with DeiT-S, outperforming LayerNorm (81.8%), RMSNorm (81.7%), and DyT (81.9%).
• Image generation: On class-conditional ImageNet 256x256 with DiT-S, Derf yields FID=19.2 vs. LayerNorm's 19.8 and DyT's 19.5.
• Speech representation: On LibriSpeech ASR, Derf reduces word error rate (WER) to 3.8% compared to LayerNorm's 4.0% and DyT's 3.9%.
• DNA modeling: On GenomicBenchmarks, Derf achieves 96.3% accuracy vs. LayerNorm's 95.8% and DyT's 96.0%.
• All improvements are statistically significant and consistent across multiple seeds.

Significance

This paper provides a strong empirical demonstration that normalization layers can be replaced by a carefully designed pointwise function without sacrificing—and often improving—performance. The simplicity of Derf (a single function with two learnable parameters) makes it an attractive option for practitioners seeking to reduce architectural complexity. The finding that Derf improves generalization rather than fitting capacity suggests it may offer regularization benefits, which could be particularly valuable in data-limited regimes. Future work could explore theoretical underpinnings, scaling to larger models, and integration with other architectural innovations.