Neura Market
UQLM logo

UQLM

Free

UQLM: Uncertainty Quantification for Language Models, is a Python library for LLM hallucination detection using state-of-the-art uncertainty quantification techniques ![GitHub Repo stars](https://img.shields.io/github/stars/cvs-health/uqlm?style=social)

FreeFree tier
Type
Open Source
Company
CVS Health
Visit Website GitHub (40 stars)

About UQLM

UQLM (Uncertainty Quantification for Language Models) is an open-source Python library developed by CVS Health for detecting hallucinations in Large Language Model (LLM) outputs. It provides a comprehensive suite of response-level scorers that quantify uncertainty, returning a confidence score between 0 and 1 where higher scores indicate lower likelihood of errors or hallucinations. The library supports multiple scorer types: black-box (consistency-based), white-box (token probability based), LLM-as-a-Judge, ensemble, and long-text scorers. It is compatible with any LLM through LangChain and offers easy installation via pip. UQLM is designed to help developers build more reliable LLM applications by identifying uncertain or hallucinated responses.

Key Features

Provides response-level scorers for uncertainty quantification
Supports black-box (consistency-based) scorers using multiple generations
Supports white-box scorers using token probabilities (minimal latency)
Supports LLM-as-a-Judge scorers using additional judge calls
Supports ensemble scorers that combine multiple methods
Supports long-text scorers for claim-level analysis
Returns confidence scores between 0 and 1 for each response
Compatible with any LLM via LangChain Chat Models
Includes mitigation option (use_best) to select uncertainty-minimized response
Easy installation via pip (pip install uqlm)

Pros & Cons

Pros
  • Open-source and free to use
  • Comprehensive suite of UQ techniques in one library
  • Works with any LLM through LangChain integration
  • Easy installation via pip
  • Offers both low-latency (white-box) and high-accuracy (black-box) options
  • Includes mitigation strategies to reduce hallucination impact
  • Well-documented with examples and demos
Cons
  • Black-box and long-text scorers require multiple LLM calls, increasing cost and latency
  • White-box scorers limited to models that expose token probabilities
  • LLM-as-a-Judge scorers add extra latency and may incur additional API costs
  • Requires LangChain as a dependency (may add complexity for some projects)
  • Primarily focused on detection, not prevention of hallucinations

Best For

Detecting hallucinations in LLM-generated textQuantifying uncertainty of LLM outputs for reliability assessmentImproving trustworthiness of AI chatbots and assistantsSelecting the most confident response from multiple generationsEvaluating LLM performance in production environments

FAQ

What is UQLM?
UQLM (Uncertainty Quantification for Language Models) is an open-source Python library from CVS Health for detecting hallucinations in LLM outputs using state-of-the-art uncertainty quantification techniques.
How do I install UQLM?
Install via pip: pip install uqlm
What types of scorers does UQLM provide?
It provides five types: Black-Box (consistency-based), White-Box (token probability), LLM-as-a-Judge, Ensemble, and Long-Text scorers.
Does UQLM work with any LLM?
Yes, it works with any LLM that can be used via LangChain Chat Models. Black-box, LLM-as-a-Judge, ensemble, and long-text scorers are universal; white-box scorers require access to token probabilities.
What does the confidence score mean?
Each scorer returns a confidence score between 0 and 1, where higher scores indicate a lower likelihood of errors or hallucinations.