DistillAlign: Coordinating Mode Covering and Mode Seeking in Autoregressive Video Distillation

Analysis

Why This Paper Matters

Autoregressive video generation has made significant strides, but distilling large teacher models into efficient students remains a critical challenge. Existing methods typically use a multi-stage pipeline with Distribution Matching Distillation (DMD), but they often decouple initialization and DMD stages, leading to misaligned target distributions. This paper identifies a fundamental issue: the mode-seeking nature of DMD can cause the student to collapse to high-probability regions of the teacher, sacrificing coverage and diversity. By introducing a distributional evaluation protocol, the authors expose hidden differences that visual scores like VBench fail to capture. This work matters because it provides a principled framework to diagnose and fix distributional misalignment, which is crucial for building high-quality, diverse video generation models.

Technical Contributions

• Distributional Evaluation Protocol: The paper introduces a method to measure precision and coverage between student and teacher distributions in a shared latent space, revealing that some initializations achieve high precision but low coverage, leading to suboptimal refinement.
• Joint Distillation Objective: The key innovation is combining DMD's mode-seeking objective with a Consistency Distillation-based mode-covering constraint. This joint approach balances the trade-off between generating high-probability samples and maintaining diversity.
• Empirical Validation: The authors show that even when target distributions are aligned, DMD's reverse-KL objective can reduce coverage in late training, and their joint distillation mitigates this issue.

Results

The paper reports that their method improves generation quality, coverage, and diversity. A striking result is that using a Wan-1.3B DMD teacher with joint distillation outperforms baselines that were refined with a much larger Wan-14B teacher. This underscores the importance of distributional alignment over sheer model size. The distributional evaluation protocol likely shows that the joint distillation achieves better precision-coverage trade-offs compared to standard DMD pipelines.

Significance

This work has broad implications for AI practitioners working on video generation and model distillation. It challenges the common practice of relying solely on visual quality scores and highlights the need for distributional metrics. The joint distillation framework could be applied to other autoregressive models beyond video, such as language or audio generation, where mode collapse is a known issue. By enabling smaller teachers to outperform larger ones through better alignment, this method could reduce computational costs and democratize access to high-quality generative models.