SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation

Analysis

Why This Paper Matters

SANA-Video 2.0 addresses a critical bottleneck in video generation: the quadratic cost of softmax attention in diffusion transformers. As video lengths and resolutions increase, full-softmax attention becomes prohibitively expensive, limiting practical deployment. This paper demonstrates that a carefully designed hybrid of linear and softmax attention can recover the expressiveness of full softmax while maintaining linear complexity, enabling 720p video generation on a single GPU. This is a significant step toward democratizing high-quality video generation, as it reduces hardware requirements from multi-GPU clusters to a single consumer card.

The paper also introduces Block Attention Residuals, a novel mechanism to propagate high-rank representations through deep networks, addressing a known weakness of linear attention models. Combined with full-stack engineering optimizations (Sol-Engine), the system achieves a 120x speedup over a comparable model (Wan 2.2-A14B) on the same hardware. This work is likely to influence both academic research and industry product development in video generation and efficient transformer architectures.

Technical Contributions

• Hybrid Linear-Softmax Attention: Combines gated linear attention (O(N) complexity) with periodic gated-softmax anchors at a 3:1 ratio. The softmax anchors restore full-rank token interactions that pure linear attention lacks, while the linear layers maintain favorable scaling for long sequences.
• Block Attention Residuals (AttnRes): Routes completed block summaries into later linear layers, enabling anchor-feature reuse and boosting deep-layer effective rank by ~12%. This helps propagate the refreshed representations from softmax anchors across depth.
• From-Scratch Training: The hybrid architecture is trained from scratch rather than linearizing pretrained models, allowing the model to learn the optimal interaction between linear and softmax components. Proxy studies at reduced resolution established 25% softmax as the optimal trade-off.
• Sol-Engine Optimization: A full-stack optimization pipeline including kernel fusion, caching, and sparse attention, accelerating the hardware-friendly backbone by an additional 3.58x.

Results

• Quality: VBench score of 84.30 at 480p with 40-step sampling on a single H100, competitive with far larger softmax video DiTs.
• Speed: Compiled DiT forward pass is 3.2x faster than a matched full-softmax baseline at 720p/60s, with the gap widening for longer videos.
• End-to-End: Full pipeline (5B model) achieves 13.06s at 720p/5s on one H100, making it 120x faster than Wan 2.2-A14B.
• Scaling: The hybrid design retains favorable long-sequence scaling of linear attention, enabling efficient generation of long, high-resolution videos.

Significance

SANA-Video 2.0 demonstrates that hybrid attention can match the quality of full-softmax attention at a fraction of the computational cost, opening the door to practical, long-duration video generation on consumer hardware. This work is likely to accelerate the adoption of video generation in applications such as content creation, simulation, and education. The hybrid attention approach may also inspire similar designs in other domains (e.g., long-document NLP, high-resolution image generation) where quadratic attention is a bottleneck. By releasing the model and optimizations, the authors provide a strong baseline for future efficient video generation research.