Transformers without Normalization

Analysis

Why This Paper Matters

Normalization layers like Layer Normalization (LN) and RMSNorm are considered essential components in modern Transformer architectures. They stabilize training, improve convergence, and enable deeper networks. However, they also introduce computational overhead and complexity. This paper challenges that orthodoxy by showing that a simple element-wise operation—Dynamic Tanh (DyT)—can replace normalization entirely without sacrificing performance. The finding is significant because it suggests that the core function of normalization may be more about non-linear squashing of extreme values than about statistical standardization, opening the door to simpler, more efficient architectures.

How It Works

The authors first empirically study the behavior of normalization layers in trained Transformers. They sample mini-batches from Vision Transformer (ViT), wav2vec 2.0, and Diffusion Transformer (DiT) models and plot the input-output relationship of LN layers before the affine transformation. The key observation is that in deeper layers, this relationship resembles an S-shaped curve, similar to a tanh function. While most points fall in a linear range near zero, extreme values (e.g., x > 50 or x < -50) are squashed non-linearly, which is the critical effect of normalization.

Inspired by this, the authors propose Dynamic Tanh (DyT):

Here, α is a learnable scalar that dynamically scales the input based on its range, and γ and β are per-channel affine parameters identical to those in normalization layers. DyT is applied as a drop-in replacement for normalization layers, without altering activation functions like GELU or ReLU.

Results

The paper evaluates DyT across a wide range of tasks and architectures:

• Supervised Vision: ViT and ConvNeXt models trained on ImageNet-1K show DyT slightly outperforming LN.

• Self-Supervised Vision: MAE and DINO pretraining followed by fine-tuning show DyT performing on par with LN.

• Diffusion Models: DiT models of sizes B, L, and XL achieve comparable or better FID scores with DyT.

• Large Language Models: LLaMA models (7B to 70B) trained on The Pile with 200B tokens show DyT matching RMSNorm on training loss and zero-shot tasks.

• Speech and DNA: wav2vec 2.0 and DNA sequence models (HyenaDNA, Caduceus) also show comparable performance.

Significance

This work is a refreshing challenge to the deep learning community's reliance on normalization. By showing that a simple tanh-based operation can replace complex statistical computations, it suggests that the primary role of normalization in Transformers is non-linear squashing of outliers, not mean-variance standardization. This could lead to simpler, more efficient models, especially in resource-constrained settings. The paper also provides practical initialization guidelines for α, noting that non-LLM models are robust to α₀ while LLMs require careful tuning based on model width. Future work may explore DyT in even larger models and alternative architectures, potentially reshaping how we think about normalization in deep learning.