KVAE: Family of Tokenizers for Multimodal Generative Models

Analysis

Why This Paper Matters

Tokenizers are a critical component in latent diffusion models (LDMs), as they determine the compression and representation of input signals, directly affecting learning speed and generation quality. While many tokenizers exist for individual modalities, a unified family that performs well across audio, image, and video is rare. KVAE addresses this gap by introducing a series of tokenizers designed specifically for text-conditioned generation, with a focus on matching or surpassing the performance of frontier open-source tokenizers.

The paper's significance lies in its comprehensive evaluation and open-source contribution. By providing training details, model selection methods, and ablations, the authors enable the community to reproduce and build upon their work. This is particularly valuable in a field where tokenizer design is often underreported, yet crucial for downstream generation quality.

Technical Contributions

• KVAE-Audio: A continuous full-band 48 kHz audio tokenizer with a 50 Hz latent of 64 channels, designed for high-fidelity audio generation.
• KVAE-3D: Two causal video tokenizers with compression factors of 4x16x16 and 4x8x8, enabling efficient video generation.
• KVAE-2D: An image tokenizer that compresses input by a factor of 8 with 32 channels, balancing reconstruction fidelity and latent size.
• Unified design: All tokenizers are tailored for text-conditioned generation, ensuring compatibility with latent diffusion models.
• Open-source release: Code is available on GitHub, along with training details and ablation studies, facilitating reproducibility and further research.

Results

The paper reports that KVAE tokenizers match or surpass frontier open-source tokenizers, including VAEs from Wan-2.2, HunyuanVideo-1.5, FLUX.2, MovieGen, StableAudio, and MMAudio. Evaluation metrics include reconstruction quality (PSNR, LPIPS, PESQ) and generation quality (Frechet Distance, CLIP score, CLAP score), as well as subjective side-by-side evaluations. While specific numerical values are not provided in the abstract, the claim of matching or surpassing these strong baselines indicates competitive performance.

Significance

KVAE's contribution extends beyond individual tokenizers by offering a unified family that can simplify multimodal generation pipelines. The open-source nature and detailed documentation lower the barrier for practitioners to adopt and adapt these tokenizers. This could lead to more consistent quality across modalities and enable new applications in text-to-audio, text-to-image, and text-to-video generation. The paper also sets a precedent for transparent reporting of tokenizer design, which may encourage similar practices in the field.