TransformerXL

Analysis

Why This Paper Matters

Standard Transformer models are limited by a fixed-length context window, which forces them to process text in chunks and lose information across segment boundaries. This is a critical bottleneck for tasks like document-level language modeling, where long-range dependencies (e.g., across paragraphs) are essential. TransformerXL directly addresses this by introducing a recurrence mechanism that allows the model to carry information across segments, effectively extending the context length without increasing computational cost proportionally. This innovation made it a foundational work for subsequent long-context models.

How It Works

The core architecture of TransformerXL retains the standard encoder-decoder stack with multi-head self-attention and feedforward layers. The key modification is the addition of segment-level recurrence. During training, the input sequence is divided into fixed-length segments. For each segment, the hidden states from the previous segment are cached and reused as additional context in the self-attention computation. This means that when processing segment t, the model can attend to both the current segment and the hidden states from segment t-1, enabling a much larger receptive field.

To make this recurrence effective, the authors introduce relative positional embeddings. Instead of encoding absolute positions (e.g., token 5 in the sequence), they encode the relative distance between tokens (e.g., token at position 5 is 3 positions away from token at position 2). This is crucial because when attending to cached states from a previous segment, absolute positions would be meaningless—the model needs to know how far apart tokens are within the current context, not their global indices. The relative embeddings are integrated into the self-attention mechanism, allowing the model to generalize to sequences longer than those seen during training.

Additionally, TransformerXL uses adaptive input embeddings. The embedding matrix size is dynamically adjusted based on the input sequence length, which reduces memory usage for shorter sequences and scales efficiently for longer ones. The training objective is also modified: a targeted cased perplexity loss emphasizes rare and difficult words, improving robustness on challenging datasets.

Results

TransformerXL was evaluated on five standard language modeling benchmarks: WikiText-103, Text8, enwik8, One Billion Word, and Penn Treebank. On all datasets, it achieved new state-of-the-art perplexity scores, often by a significant margin. For example, on WikiText-103, it reached a perplexity of 18.3, compared to the previous best of 20.5. These results demonstrate that the recurrence mechanism and relative positional encoding effectively capture long-range dependencies without sacrificing performance on shorter contexts.

Significance

TransformerXL's impact extends beyond language modeling. Its recurrence mechanism inspired the design of XLNet, which combines autoregressive and autoencoding objectives for pretraining. The relative positional encoding has become a standard component in many modern Transformer variants (e.g., T5, GPT-2/3). By showing that Transformers can handle arbitrarily long sequences with minimal architectural changes, TransformerXL opened the door to applications in document-level generation, long-form question answering, and code generation. Its memory-efficient design also influenced later models like Longformer and BigBird. Overall, TransformerXL remains a key reference for anyone working on long-context sequence modeling.