Neura Market
Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planning logo

Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planning

Free

Improve LLM multi-step reasoning with deliberative planning and no fine-tuning

FreeFree tier
Type
Open Source
Visit Website GitHub (40 stars)

About Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planning

Q* is a general, versatile, and agile framework that improves multi-step reasoning in Large Language Models (LLMs) by casting the reasoning process as a heuristic search problem. It introduces a plug-and-play Q-value model that serves as a heuristic function to estimate expected future rewards, guiding the LLM's decoding process to select the most promising next reasoning step. Unlike fine-tuning approaches, Q* avoids significant computational overhead and the risk of performance degeneration on other tasks. The method is evaluated on mathematical reasoning benchmarks (GSM8K, MATH) and a code generation benchmark (MBPP), demonstrating consistent improvements in reasoning accuracy for open-source LLMs.

Key Features

Casts multi-step reasoning as a heuristic search problem
Learns a plug-and-play Q-value model as a heuristic function for estimating expected future rewards
Guides LLM decoding to select the most promising next reasoning step
No fine-tuning required for the current task, avoiding computational overhead and performance degeneration on other tasks
Evaluated on GSM8K, MATH, and MBPP benchmarks
Improves reasoning performance of existing open-source LLMs

Pros & Cons

Pros
  • Plug-and-play approach requires no fine-tuning of the base LLM
  • Avoids computational overhead and risk of catastrophic forgetting on other tasks
  • General framework applicable to various multi-step reasoning tasks
  • Demonstrated improvements across math and code benchmarks
  • Compatible with open-source LLMs without modification
Cons
  • Requires training a separate Q-value model, which may need additional data and computation
  • Effectiveness may depend on the quality of the Q-value model and the search strategy used
  • Not evaluated on a wide range of non-reasoning tasks; generality beyond math and code is unproven

Best For

Improving mathematical reasoning in LLMs (e.g., solving grade-school math word problems)Enhancing code generation capabilities of LLMs (e.g., solving programming problems)General multi-step reasoning tasks in natural languageBenchmarking and comparing LLM reasoning abilities

FAQ

What is Q*?
Q* is a framework for improving multi-step reasoning in LLMs by treating the reasoning process as a heuristic search and using a learned Q-value model to guide the LLM's decoding.
Does Q* require fine-tuning the LLM?
No, Q* is a plug-and-play method that works with existing open-source LLMs without fine-tuning, avoiding computational overhead and performance degeneration on other tasks.
On which benchmarks was Q* evaluated?
Q* was evaluated on GSM8K (grade-school math), MATH (complex math), and MBPP (Python code generation), showing consistent improvements in reasoning performance.
What types of LLMs can benefit from Q*?
Q* is designed for open-source LLMs and has demonstrated improvements in multi-step reasoning tasks, particularly in math and code domains.
How does Q* differ from other reasoning improvement methods?
Q* avoids fine-tuning the base LLM and instead learns a separate heuristic function to guide decoding, making it computationally efficient and task-agnostic.