Maglev: Sliding Recurrent Memory

Analysis

Why This Paper Matters

Maglev addresses a critical challenge in scaling Transformers to long sequences: the quadratic cost of full attention. While sliding-window attention reduces complexity, it loses access to distant context. Recurrent models like linear attention or RWKV offer fixed memory but often sacrifice expressiveness. Maglev bridges this gap by introducing a recurrent memory that is trained to mimic a full-attention prefiller, achieving both efficiency and strong performance.

The key insight is to decouple the training-time teacher (prefiller Q) from the inference-time student (decoder P). By using a memory consistency loss, Maglev distills the ability to compress history into a fixed-size memory that P can use autoregressively. This is a novel training paradigm that could be applied to other recurrent architectures.

Technical Contributions

• Two-model architecture: Prefiller Q (full attention) generates memory targets; decoder P (sliding-window + recurrent K/V injection) produces memories for next-token prediction.
• Memory consistency loss: Aligns P's memory with Q's memory, enabling P to be used alone at inference.
• Generalization of sliding-window attention: Maglev's recurrent memory extends sliding-window attention by allowing information from beyond the window to be carried forward.
• Parameter sharing: Sharing parameters between P and Q reduces memory footprint while retaining most performance gains.
• Parallelizable training: Despite being recurrent, training remains parallelizable because Q can process the full sequence in parallel, and P can be trained with teacher forcing.

Results

The paper reports that Maglev improves validation loss and downstream pretraining benchmarks compared to sliding-window and latent recurrent transformer baselines. While specific numbers are not provided in the abstract, the consistent improvement across metrics suggests the memory consistency training is effective. The parameter-sharing variant also preserves most gains, making it a practical choice for memory-constrained settings.

Significance

Maglev offers a new paradigm for efficient long-context modeling. By training a recurrent decoder to emulate a full-attention teacher, it achieves a balance between expressiveness and efficiency. This could inspire further research into distillation-based memory training and hybrid architectures. The ability to share parameters also makes it attractive for deployment on edge devices. As long-context capabilities become increasingly important, Maglev provides a promising step toward scalable Transformers.