Neura Market
Uncertainty of Thoughts: Uncertainty-Aware Planning Enhances Information Seeking in Large Language Models logo

Uncertainty of Thoughts: Uncertainty-Aware Planning Enhances Information Seeking in Large Language Models

Free

Uncertainty-aware planning for active information seeking in LLMs

FreeFree tier
Type
Open Source
Visit Website GitHub (40 stars)

About Uncertainty of Thoughts: Uncertainty-Aware Planning Enhances Information Seeking in Large Language Models

Uncertainty of Thoughts (UoT) is an algorithm designed to enhance large language models' ability to actively seek information by asking effective follow-up questions. It integrates three key components: uncertainty-aware simulation to predict possible future scenarios and their likelihood, uncertainty-based rewards motivated by information gain, and a reward propagation scheme to select optimal questions. Evaluated on medical diagnosis, troubleshooting, and the 20 Questions game, UoT achieves an average 38.1% improvement in task success rate across multiple LLMs while reducing the number of questions needed. The code is publicly available and the work was presented at NeurIPS 2024.

Key Features

Uncertainty-aware simulation to model future scenarios and their likelihoods
Uncertainty-based rewards motivated by information gain
Reward propagation scheme to select optimal questions
Active information seeking via follow-up questions
Compatible with multiple large language models

Pros & Cons

Pros
  • Average 38.1% improvement in task success rate across multiple LLMs
  • Reduces the number of questions needed to complete a task (improves efficiency)
  • Works with various LLMs without requiring fine-tuning
  • Open-source code available for reproduction and extension
  • Published at NeurIPS 2024, peer-reviewed research
Cons
  • Requires additional computational overhead for simulation and reward propagation
  • May not perform well on tasks with very limited possible queries or deterministic information

Best For

Medical diagnosis (asking patients for more details about symptoms)Troubleshooting technical problemsInteractive question-answering games like 20 QuestionsAny task requiring sequential information gathering

FAQ

What is Uncertainty of Thoughts (UoT)?
UoT is an algorithm that augments large language models with the ability to actively seek information by asking effective questions. It combines uncertainty-aware simulation, uncertainty-based rewards, and reward propagation to select the best question to ask.
What tasks is UoT designed for?
UoT is designed for tasks where the information needed to solve a problem is not initially given and must be actively sought through follow-up questions, such as medical diagnosis, troubleshooting, and the 20 Questions game.
How much does UoT improve performance?
In experiments across multiple LLMs, UoT achieved an average performance improvement of 38.1% in the rate of successful task completion compared with direct prompting.
Is the code for UoT available?
Yes, the code has been released and is linked from the arXiv paper page.
Where was UoT published?
UoT was presented at NeurIPS 2024.