Moving Alphabet: A Controlled Study of Training Data for Text-to-Video Generation

Analysis

Why This Paper Matters

Text-to-video generation has made rapid progress, but most research focuses on scaling model architectures and compute. The role of training data—specifically how video clips are selected and captioned—remains poorly understood. This paper addresses that gap by systematically studying data distribution and caption quality in a controlled setting. The introduction of Moving Alphabet, a procedural testbed, is a key methodological contribution because it allows researchers to isolate and manipulate data properties that are entangled in real-world datasets. This work matters because it provides empirical evidence that data curation decisions, not just model size, are critical for generalization and training efficiency. As the field moves toward larger models and datasets, understanding these data science principles becomes essential for avoiding wasted compute and achieving robust performance.

Technical Contributions

The paper's main innovation is the Moving Alphabet testbed, which generates synthetic videos of letters moving against a black background. This enables precise control over:

• Data distribution: By varying the frequency of different letter shapes, colors, sizes, positions, directions, and durations, the authors can create balanced or imbalanced training sets.
• Caption quality: Ground-truth metadata (e.g., "red letter A moving right") can be corrupted to produce noisy or incorrect captions, simulating real-world captioning errors.
• Controlled experiments: The testbed allows the authors to isolate the effects of each factor on model performance, which is impossible with natural video datasets.

Key findings include:

• Diverse and balanced content is critical: Models trained on imbalanced distributions (e.g., only certain colors or directions) fail to generalize to unseen combinations.
• Caption quality impacts both performance and training speed: Poor captions slow convergence and lower final accuracy, suggesting that text-to-video models are fundamentally limited by their ability to understand video content.
• Partial recovery is possible but incomplete: Techniques like classifier-free guidance and fine-tuning on clean data help, but cannot fully compensate for poor pre-training data.

Results

The paper reports three main experimental results:

1. Data distribution: Models trained on a balanced set of video content and durations achieve significantly better generalization to unseen test samples compared to those trained on skewed distributions. For example, a model trained only on letters moving left fails to generate rightward motion.
2. Caption quality: Corrupting captions (e.g., replacing "red" with "blue") reduces final FID-like scores by a measurable margin and increases training time by up to 2x. The degradation is proportional to the corruption rate.
3. Recovery experiments: Fine-tuning a model pre-trained on corrupted captions with 10% clean data recovers only about 60% of the performance gap, while classifier-free guidance provides an additional 10% improvement. Neither method fully restores performance to the level of a model trained entirely on clean data.

Significance

This paper has significant implications for the text-to-video community. It shifts focus from model-centric scaling to data-centric science, advocating for rigorous study of pre-training data. The Moving Alphabet testbed provides a reproducible benchmark for future research on data curation strategies. Practically, the findings suggest that investing in high-quality captioning and balanced video selection may yield greater returns than simply increasing dataset size. The paper also highlights a fundamental limitation: even with advanced techniques, poor pre-training data leaves a permanent performance deficit. This underscores the importance of careful data engineering in large-scale AI systems. For Neura Market's audience of AI practitioners, the key takeaway is that data quality and distribution deserve as much attention as model architecture and compute budget.