Prometheus
FreeOpen-source evaluator LM for fine-grained, reproducible evaluation.
FreeFree tier
Inputs: textOutputs: text
About Prometheus
Prometheus is an open-source evaluator language model designed for fine-grained evaluation of language model responses based on customized score rubrics. It offers a reproducible and inexpensive alternative to human evaluation and GPT-4 evaluation. The model was presented at ICLR 2024 and NeurIPS 2023 Workshop. The repository includes training code, a Feedback Collection dataset, and a specific input/output format to enable structured evaluation.
Key Features
Open-source evaluator language model
Fine-grained evaluation on customized score rubrics
Reproducible and inexpensive
Alternative to human and GPT-4 evaluation
Includes Feedback Collection dataset for training
Training code provided for custom evaluator LMs
Specific input/output format for structured evaluation (instruction, response, reference answer, score rubric)
Outputs detailed feedback and a score between 1 and 5
Pros & Cons
Pros
- Open-source and freely available
- Reproducible evaluation results
- Low cost compared to API-based evaluators like GPT-4
- Allows customization of score rubrics for specific evaluation criteria
- Provides both detailed feedback and numeric score
- Backed by academic research (ICLR 2024, NeurIPS 2023)
Best For
Evaluating language model responses for quality and adherence to custom criteriaReplacing or augmenting human evaluation in research and productionInexpensive evaluation alternative to GPT-4 for fine-grained scoringTraining custom evaluator LMs for specific domains or rubrics
FAQ
What is Prometheus?
Prometheus is an open-source evaluator language model designed for fine-grained evaluation of language model responses based on customized score rubrics, offering a reproducible and inexpensive alternative to human evaluation and GPT-4 evaluation.
How do I use Prometheus?
Prometheus is used by providing an instruction, a response to evaluate, a reference answer that scores 5, and a score rubric with descriptions for scores 1-5. The model outputs feedback and a score between 1 and 5.
What is the input format?
The input format includes fields: ###Task Description, ###The instruction to evaluate, ###Response to evaluate, ###Reference Answer (Score 5), ###Score Rubrics with descriptions for scores 1-5, and ###Feedback.
What is the output format?
The output format is: 'Feedback: (write a feedback for criteria) [RESULT] (an integer number between 1 and 5)' with no other text.
Can I train my own evaluator LM using Prometheus?
Yes, the repository provides training code in the train directory, using the Feedback Collection dataset, and instructions for fine-tuning a language model.