CompMix

Free

a benchmark evaluating QA methods that operate over a mixture of heterogeneous input sources (KB, text, tables, infoboxes).

FreeFree tier

Inputs: textOutputs: text

Type

Open Source

Company

Max Planck Institute for Informatics

Visit Website GitHub (38 stars)

About CompMix

CompMix is a benchmark for evaluating question answering (QA) methods that operate over a mixture of heterogeneous input sources: knowledge bases (Wikidata), free text (Wikipedia), tables, and infoboxes. It consists of 9,410 questions (4,966 train, 1,680 dev, 2,764 test) derived from conversational questions in ConvMix. The questions exhibit complex phenomena such as multiple entities, relations, temporal conditions, comparisons, and aggregations. Answers are grounded to the Wikidata KB, with normalization for dates and other literals. The dataset was created by master crowdworkers on Amazon Mechanical Turk, who selected entities of interest and wrote conversational questions, then provided completed intent-explicit versions. CompMix is licensed under Creative Commons Attribution 4.0 and is available on Hugging Face. It includes a leaderboard for model comparison.

Key Features

9,410 questions with train/dev/test splits

Questions exhibit complex phenomena: multiple entities, relations, temporal conditions, comparisons, aggregations

Answers grounded to Wikidata KB

Derived from real human conversational questions (ConvMix)

Crowdworker-created via Amazon Mechanical Turk

Available on Hugging Face

Leaderboard for benchmarking models

Pros & Cons

Pros

Large-scale dataset with 9,410 questions
Covers heterogeneous input sources: KB, text, tables, infoboxes
Real human-generated questions from crowdworkers
Includes a public leaderboard for model comparison
Free, open-source under Creative Commons 4.0 license
Available on Hugging Face for easy integration

Cons

Primary focus on English and Wikidata/Wikipedia sources may limit domain coverage

Best For

Evaluating heterogeneous question answering modelsBenchmarking retrieval-augmented generation (RAG) over heterogeneous dataResearch on conversational question answering

FAQ

How was CompMix created?

CompMix collates completed versions of conversational questions from ConvMix. Crowdworkers on Amazon Mechanical Turk selected entities and wrote conversational questions, then provided completed intent-explicit versions. Answers were found in Wikidata, Wikipedia text, tables, or infoboxes.

What types of questions are in CompMix?

Questions exhibit complex phenomena like multiple entities, relations, temporal conditions, comparisons, and aggregations.

What is the size of the dataset?

The dataset has 9,410 questions split into train (4,966), dev (1,680), and test (2,764) sets.

How are answers provided?

All answers are grounded to the Wikidata KB, except for dates which are normalized and other literals like names.

Is the dataset publicly available?

Yes, under Creative Commons Attribution 4.0 International License. It is also available on Hugging Face at https://huggingface.co/datasets/pchristm/CompMix.