Maglev: Sliding Recurrent Memory
Bo Liu, Qiang Liu
Maglev is a recurrent Transformer architecture with fixed-size memory that generalizes sliding-window attention, trains with a memory consistency loss, and improves validation loss and downstream benchmarks.