MSA: Memory Sparse Attention to 100M Tokens (2026)

Free

End-to-end trainable sparse attention with linear complexity — scales to 100M tokens on 2×A800 GPUs with <9% degradation vs 16K baseline; Memory Interleaving enables multi-hop reasoning across scattered segments

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About MSA: Memory Sparse Attention to 100M Tokens (2026)

Memory Sparse Attention (MSA) is an end-to-end trainable sparse attention framework that achieves linear complexity in both training and inference, enabling scaling to 100 million tokens. Core innovations include scalable sparse attention and document-wise Rotary Position Embedding (RoPE), which maintain exceptional stability with less than 9% performance degradation when scaling from a 16K-token baseline to 100M tokens. MSA leverages KV cache compression and Memory Parallel to run 100M-token inference on just two A800 GPUs. Additionally, Memory Interleaving facilitates complex multi-hop reasoning across scattered memory segments. MSA significantly surpasses frontier LLMs, state-of-the-art RAG systems, and leading memory agents on long-context benchmarks, providing a scalable foundation for endowing general-purpose models with intrinsic, lifetime-scale memory.

Key Features

Scalable sparse attention with linear complexity in training and inference

Document-wise Rotary Position Embedding (RoPE) for stable long-context scaling

KV cache compression combined with Memory Parallel for efficient 100M-token inference on 2x A800 GPUs

Memory Interleaving enabling multi-hop reasoning across scattered memory segments

Less than 9% performance degradation when scaling from 16K to 100M tokens

End-to-end trainable, unlike external memory methods such as RAG

Pros & Cons

Pros

Linear complexity enables scaling to 100M tokens with minimal computational overhead
Achieves state-of-the-art results on long-context benchmarks, outperforming frontier LLMs and RAG systems
End-to-end trainable, allowing dynamic memory updates without external retrieval
Low degradation ( <9%) from 16K to 100M tokens, demonstrating high precision retention

Cons

Requires 2x A800 GPUs for full 100M-token inference, still resource-intensive
As a research paper, no immediate production-ready implementation or code is provided
Performance degradation, while small, is not zero; some precision loss occurs at extreme lengths

Best For

Large-corpus summarizationDigital Twins and long-history agent reasoningLong-context processing for general LLM applicationsComplex reasoning tasks requiring integration of scattered information

FAQ

What is the main innovation of MSA?

MSA introduces scalable sparse attention with document-wise RoPE to achieve linear complexity and minimal degradation over extremely long contexts, combined with KV cache compression for efficient deployment.

How does MSA handle memory-intensive tasks?

Using KV cache compression and Memory Parallel, MSA can perform 100M-token inference on just 2x A800 GPUs, while Memory Interleaving enables multi-hop reasoning across scattered segments.

What are the compared baselines?

MSA is compared against frontier LLMs, state-of-the-art RAG systems, and leading memory agents, and it significantly surpasses them on long-context benchmarks.