CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversity

Analysis

Why This Paper Matters

Post-training of large language models (LLMs) typically improves their quality on benchmarks but often reduces output diversity and creativity. This is problematic for tasks that explicitly require creativity, such as story generation, and also for implicit creative tasks like reinforcement learning (RL), where diverse exploration is crucial. CreativeInstruct addresses this critical trade-off by teaching LLMs to balance creativity and quality through a simple yet effective instruction-tuning approach.

The paper's significance lies in its scalability and practicality. Unlike methods that require multiple models at inference time or complex distillation, CreativeInstruct uses a single model with special [StartCreativity] spans, making it easy to integrate into existing pipelines. The introduction of a structural diversity metric based on graph edit distance also provides a more nuanced evaluation of narrative variation, which is often missed by lexical and semantic metrics.

Technical Contributions

• CreativeInstruct Training Method: The method trains LLMs to emit [StartCreativity] tokens that bias generation toward creativity, allowing the model to switch between creative and standard modes. This is done via instruction tuning, making it scalable and model-agnostic.
• Structural Diversity Metric: A novel metric based on graph edit distance that captures narrative-level structural differences, complementing existing lexical and semantic diversity measures.
• Empirical Validation: The paper provides extensive experiments on narrative generation, comparing against multi-model baselines and distilled variants, and includes human evaluations.
• RL Substrate: Demonstrates that creative checkpoints serve as better starting points for RL, improving performance on AMC and MATH benchmarks.

Results

CreativeInstruct matches or exceeds the diversity of multi-model baselines and distilled variants without sacrificing quality. In human evaluation, annotators rated CreativeInstruct generations as more creative than post-trained LLM generations in 70.3% of cases. When used as a substrate for RL, GRPO applied to a CreativeInstruct checkpoint improved by ~4% on AMC and ~5% points on MATH over the same training applied to the post-trained checkpoint. These results highlight the method's effectiveness in restoring creativity while maintaining quality.

Significance

The broader impact of CreativeInstruct is substantial. It offers a practical solution to the creativity-diversity trade-off in LLMs, which is a known limitation of post-training. This could enable more creative AI applications, from storytelling to content generation, and also improve RL by providing more diverse exploration. The structural diversity metric also provides a better tool for evaluating generative models, potentially influencing future research on diversity metrics. Overall, CreativeInstruct paves the way for more balanced and versatile LLMs.