Chain-of-Thought Hub

Free

Measuring LLMs' Reasoning Performance

FreeFree tier

Type

Open Source

Visit Website GitHub (38 stars)

About Chain-of-Thought Hub

Chain-of-Thought Hub is an open-source benchmark suite that measures the complex reasoning performance of large language models (LLMs) using chain-of-thought prompting. It compiles a carefully curated set of challenging tasks spanning math (GSM8K, MATH), science (TheoremQA), symbolic reasoning (BBH), knowledge (MMLU, C-Eval), coding (HumanEval), factual reasoning (SummEdits), and long-context understanding (RepoBench, Qspr, QALT, BkSS). The project categorizes benchmarks into main, experimental, and long-context tracks, providing scripts, results, and analysis to compare models from different organizations. It is actively maintained by researchers from the University of Edinburgh, University of Washington, Allen Institute for AI, and University of Waterloo, and is referenced by leading industrial and academic labs.

Key Features

Covers a wide range of reasoning tasks: math, science, symbolic, knowledge, coding, factual, long-context

Categorizes benchmarks into main, experimental, and long-context tracks

Provides scripts and results for reproducible evaluation

Includes datasets like GSM8K, MATH, MMLU, BBH, HumanEval, TheoremQA, SummEdits, and more

Actively updated with community contributions via pull requests and issue discussions

Referenced by leading industrial and academic organizations in LLM development

Pros & Cons

Pros

Carefully curated and regularly updated set of challenging benchmarks
Covers diverse reasoning domains beyond simple conversation
Open-source and freely available on GitHub with detailed documentation
Used and cited by leading industry and academic groups
Includes long-context tasks for future LLM capabilities

Cons

Limited to the specific datasets included; may not cover all reasoning types
Requires technical setup to run benchmarks and evaluate models
Results depend on prompting strategy; chain-of-thought may not suit all tasks
Not a standalone tool; users need access to LLMs and Python environment

Best For

Benchmarking LLMs on complex reasoning tasks to evaluate model capabilitiesComparing reasoning performance between different models (e.g., GPT-3.5 vs GPT-4)Research on chain-of-thought prompting and prompt engineering techniquesTesting long-context reasoning for next-generation LLM applicationsCommunity-driven evaluation for open-source and proprietary LLMs

FAQ

What is Chain-of-Thought Hub?

Chain-of-Thought Hub is an open-source project that compiles a list of complex reasoning tasks to benchmark the reasoning performance of large language models using chain-of-thought prompting. It provides datasets, scripts, and results for evaluation.

What datasets are included?

The hub includes main datasets (GSM8K, MATH, MMLU, BBH, HumanEval, C-Eval), experimental datasets (TheoremQA, SummEdits), and long-context datasets (Qspr, QALT, BkSS). These cover math, science, symbolic reasoning, knowledge, coding, factual reasoning, and long-context understanding.

How can I contribute?

You can contribute by sending a pull request to fill in missing benchmark results, raising an issue to suggest new tasks, or brainstorming benchmarks for long-context reasoning. The project welcomes community involvement.