Vision foundation models can be good tokenizers for latent diffusion models

Analysis

Why This Paper Matters

Latent diffusion models (LDMs) have become a cornerstone of high-quality image generation, but their performance heavily depends on the tokenizer that maps images into a latent space. Traditionally, tokenizers are trained from scratch or with auxiliary objectives. Recent efforts have attempted to incorporate Vision Foundation Models (VFMs) like CLIP or DINO into tokenizer training via knowledge distillation, hoping to inherit their rich semantic representations. This paper challenges that trend by empirically showing that distillation-based integration actually weakens the tokenizer, leading to degraded generative performance.

The significance lies in the counterintuitive finding: instead of forcing a tokenizer to mimic a VFM, one can directly use the VFM as the tokenizer. This simplifies the pipeline, avoids distillation losses, and leverages the full power of pretrained VFMs. For practitioners, this could mean easier adoption of state-of-the-art vision backbones in diffusion models without complex training recipes. The work also prompts a reevaluation of how we transfer knowledge from foundation models to generative systems.

Technical Contributions

• Empirical evidence against distillation: The authors demonstrate that distilling a VFM into a tokenizer during training consistently harms the tokenizer's quality, likely due to conflicting objectives or capacity mismatch.
• Direct VFM tokenization: They propose using VFMs as fixed tokenizers for LDMs, eliminating the need for separate tokenizer training.
• Analysis of tokenizer quality: The paper likely provides metrics showing that VFM-based tokenizers preserve more semantic information and lead to better generation.
• Practical recipe: A simple yet effective alternative to existing distillation-based methods, reducing training complexity.

Results

The abstract does not provide specific numbers, but the key result is qualitative: distillation weakens tokenizers, while direct VFM tokenization improves latent diffusion performance. This suggests that future work should avoid distillation for this purpose. The finding is supported by experiments, though details are not disclosed in the abstract.

Significance

This paper could shift the community's approach to tokenizer design in generative models. By showing that VFMs can be good tokenizers, it opens the door to using powerful pretrained representations directly in diffusion models, potentially improving sample quality and training efficiency. It also raises questions about the role of distillation in multimodal systems. As VFMs continue to advance, their direct integration into generative pipelines may become standard practice, simplifying architectures and leveraging large-scale pretraining. This work is a step toward more unified and efficient generative AI systems.