Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking
FreeLanguage Models Can Teach Themselves to Think Before Speaking
About Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking
Quiet-STaR is a generalization of the Self-Taught Reasoner (STaR) that enables language models to learn to generate internal rationales—'thoughts'—at each token position to explain and predict future text, improving their overall predictions. It addresses key challenges such as the high computational cost of generating continuations, the model's initial inability to generate or use internal thoughts, and the need to predict beyond individual next tokens. The method introduces a tokenwise parallel sampling algorithm, learnable start and end tokens that demarcate thoughts, and an extended teacher-forcing technique. After continued pretraining on a corpus of internet text, Quiet-STaR yields zero-shot improvements on GSM8K (from 5.9% to 10.9%) and CommonsenseQA (from 36.3% to 47.2%) without any fine-tuning on these tasks, along with reduced perplexity on difficult tokens in natural text. This approach marks a step toward more general and scalable reasoning in language models.
Key Features
Pros & Cons
- Improves prediction of difficult-to-predict tokens in natural text
- Achieves zero-shot improvements on reasoning tasks without fine-tuning
- Provides a general and scalable framework for learning to reason from arbitrary text
- Addresses key challenges in generating and using internal thoughts during autoregressive generation
- High computational cost due to generating continuations and thoughts at every token
- Requires continued pretraining on a large corpus of internet text for effective learning
- May not be directly applicable to all language model architectures or domains without adaptation