W.A.L.T

W.A.L.T (Window Attention Latent Transformer) is a transformer-based approach for photorealistic video generation using diffusion models, developed by researchers at Stanford, Google Research, and Georgia Tech. It employs a causal encoder to jointly compress images and videos into a unified latent space, enabling training and generation across modalities. A window attention architecture is used for memory and training efficiency, combining spatial and spatiotemporal modeling. The model achieves state-of-the-art performance on video generation benchmarks (UCF-101, Kinetics-600) and image generation (ImageNet) without classifier-free guidance. For text-to-video, a cascade of three models—a base latent video diffusion model and two video super-resolution models—generates 512×896 resolution videos at 8 frames per second.