G-Eval
FreeNLG evaluation using GPT-4 with better human alignment
FreeFree tier
About G-Eval
G-Eval is an open-source implementation of the paper 'G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment'. It provides a framework for evaluating natural language generation (NLG) outputs, such as text summaries, using GPT-4 as an evaluator. The repository includes prompts for evaluating fluency on the SummEval dataset, scripts for running the evaluation, and meta-evaluation tools to assess the quality of the evaluations themselves. Licensed under MIT.
Key Features
Uses GPT-4 for NLG evaluation with improved human alignment
Supports fluency evaluation on the SummEval dataset
Provides customizable prompts for different evaluation dimensions
Includes meta-evaluation tools to validate evaluation results
Open source under MIT license
Written in Python
Pros & Cons
Pros
- Leverages GPT-4 for more human-aligned evaluations
- Provides detailed prompts and results for reproducibility
- Open source and free to use
- Includes meta-evaluation to measure evaluation quality
Cons
- Requires GPT-4 API access, which can be costly
- Only provides evaluation for fluency dimension by default (other dimensions depend on custom prompts)
- Limited to the SummEval dataset in the provided examples
- Not actively maintained (last commit not recent)
Best For
Evaluating fluency of text summarization outputsAssessing NLG systems for research and developmentComparing evaluation methods against human judgments