Resurrecting Recurrent Neural Networks for Long Sequences
FreeRecovering RNN performance on long sequences via careful design.
FreeFree tier
Inputs: text
About Resurrecting Recurrent Neural Networks for Long Sequences
This paper introduces the Linear Recurrent Unit (LRU), a carefully designed deep recurrent neural network block that matches the performance and computational efficiency of deep state-space models (SSMs) on long-range reasoning tasks. Through systematic analysis of signal propagation, the authors show that linearizing and diagonalizing the recurrence, along with improved parameterizations and initializations, allows RNNs to recover the impressive performance of SSMs on benchmarks like the Long Range Arena, while maintaining fast parallelizable training and RNN-like fast inference.
Key Features
Linear Recurrent Unit (LRU) block
Linearized and diagonalized recurrence
Improved parameterizations and initializations
Proper normalization of forward pass
Fast parallelizable training
Fast inference on long sequences
Matches deep SSM performance on Long Range Arena
Pros & Cons
Pros
- Matches state-of-the-art deep SSM performance on long sequences
- Fast inference due to recurrent nature
- Parallelizable training for efficiency
- Based on rigorous signal propagation analysis
- Open source (code available via paper links)
Cons
- Research paper; not yet a production-ready library
- Requires understanding of SSM and RNN design principles
- May not outperform all SSM variants in every scenario
Best For
Long-range reasoning tasksLong sequence modelingTime series analysisNatural language processingScientific sequence modeling
FAQ
What is the Linear Recurrent Unit (LRU)?
The LRU is a recurrent neural network block introduced in this paper that achieves competitive performance with deep state-space models on long sequence tasks through careful design of the recurrence, parameterization, and initialization.
How does the LRU compare to standard RNNs?
Standard RNNs are hard to optimize and slow to train on long sequences. The LRU linearizes and diagonalizes the recurrence, uses better parameterizations and initializations, and ensures proper normalization, matching the performance and training speed of deep SSMs.
What are the main results of the paper?
The paper shows that deep RNNs can recover the impressive performance of deep SSMs on long-range reasoning tasks, while also matching their computational efficiency. The LRU block achieves this on the Long Range Arena benchmark.
Is the code available?
The paper links to code, data, and media resources via services like CatalyzeX, DagsHub, Hugging Face, and Replicate, suggesting the implementation is available for use.