FELM
Freea meta-benchmark that evaluates how well factuality evaluators assess the outputs of large language models (LLMs).
About FELM
FELM (Factuality Evaluation of Large Language Models) is a meta-benchmark designed to evaluate how well factuality evaluators assess the outputs of LLMs. It addresses the under-explored area of evaluating the evaluators themselves. The benchmark collects responses generated by ChatGPT across five domains: World Knowledge, Science/Technology, Writing/Recommendation, Reasoning, and Math. Each response is annotated with fine-grained factuality labels, resulting in 847 instances with 4,427 labeled segments. FELM provides a leaderboard comparing various evaluators, including GPT-4, Vicuna-33B, and ChatGPT, using metrics such as F1 Score and Balanced Accuracy. The dataset and code are publicly available, and the paper was presented at NeurIPS 2023 Datasets and Benchmarks Track.
Key Features
Pros & Cons
- Fine-grained annotation allows detailed analysis of evaluator performance
- Multi-domain coverage increases benchmark diversity and applicability
- Public leaderboard facilitates transparent comparison of evaluators
- Open source dataset and code enable reproducibility and extension
- Responses are generated only by ChatGPT, limiting diversity of LLM outputs evaluated
- Moderate dataset size (847 instances) may not capture all factuality challenges
- Focuses solely on English text, not tested on multilingual scenarios