Thinking LLMs: General Instruction Following with Thought Generation
FreeEquipping LLMs with explicit thinking for general instruction following.
FreeFree tier
Inputs: textOutputs: text
About Thinking LLMs: General Instruction Following with Thought Generation
This paper proposes a training method to equip existing large language models (LLMs) with explicit thinking abilities for general instruction following, without requiring additional human data. The approach uses an iterative search and optimization procedure that explores multiple possible thought generations, scores the resulting responses using a judge model, and optimizes thought generation via preference optimization. The method improves performance on AlpacaEval and Arena-Hard benchmarks, showing gains not only on traditional reasoning and problem-solving tasks but also on non-reasoning categories such as marketing, health, and general knowledge.
Key Features
Iterative search and optimization to explore thought generations
Uses a judge model to score responses of thought candidates
Preference optimization for thought generation
No additional human data required
Applicable to both reasoning and non-reasoning tasks
Pros & Cons
Pros
- Explicit thinking ability improves response quality on complex instructions
- No need for additional human-annotated data
- Demonstrates gains on non-reasoning categories like marketing, health, general knowledge
- Superior performance on AlpacaEval and Arena-Hard benchmarks
Cons
- Computational overhead from iterative thought generation and optimization
- Dependency on a judge model for scoring thought candidates
Best For
General instruction followingComplex reasoning and planning tasksMarketing content generationHealth-related queriesGeneral knowledge tasks
FAQ
What does the training method involve?
It uses an iterative search and optimization procedure that explores possible thought generations and optimizes via preference scoring by a judge model.
On which benchmarks does the method show improvement?
The method shows superior performance on AlpacaEval and Arena-Hard.
Does the method require additional human data?
No, the method equips LLMs with thinking abilities without using additional human data.