AutoEvals
FreeQuickly and easily evaluate AI model outputs
FreeFree tier
Inputs: text
About AutoEvals
AutoEvals is an open-source evaluation framework developed by Braintrust that provides a quick and easy way to evaluate AI model outputs. It bundles multiple automatic evaluation methods including LLM-as-a-judge, heuristic techniques (e.g., Levenshtein distance), and statistical metrics (e.g., BLEU). The tool supports model-graded evaluation for subjective tasks such as fact checking and safety, and allows users to run evaluations on individual examples, tweak prompts, and debug outputs. It is compatible with OpenAI's API (Python SDK v0.x and v1.x) and offers both Python and TypeScript libraries. Users can also create custom model-graded evaluations with flexible prompts and output parsing.
Key Features
LLM-as-a-judge evaluation for subjective tasks
Heuristic evaluation methods (e.g., Levenshtein distance)
Statistical evaluation methods (e.g., BLEU)
Model-graded evaluations adapted from OpenAI evals
Custom evaluation creation with flexible prompts and output parsing
Support for both synchronous and asynchronous evaluation APIs
Compatible with OpenAI Python SDK v0.x and v1.x
Available in Python and TypeScript
Pros & Cons
Pros
- Quick and easy setup with minimal code
- Broad range of built-in evaluation methods
- Open-source and free to use
- Based on best practices from OpenAI's evals project
- Supports both Python and TypeScript ecosystems
- Allows custom prompts and output parsing for advanced use
Cons
- Requires an OpenAI API key (or other LLM provider) for model-graded evals
- Subjective evaluations depend on the judge model's quality
- Heuristic and statistical methods may not capture all semantic nuances
Best For
Fact checking AI model outputsSafety evaluation of LLM responsesDebugging and iterating on LLM completionsModel-graded testing for custom subjective metricsRapid prototyping of evaluation pipelines
FAQ
What programming languages does AutoEvals support?
AutoEvals supports both Python (3.9+) and TypeScript.
How do I install AutoEvals?
For TypeScript: npm install autoevals. For Python: pip install autoevals.
Does AutoEvals require an API key for evaluation?
By default, AutoEvals uses the OPENAI_API_KEY environment variable to authenticate with OpenAI's API for model-graded evaluations.
Can I create custom evaluation prompts?
Yes, AutoEvals allows you to create custom model-graded evaluations with your own prompts and output parsing logic.
What evaluation methods are included?
AutoEvals includes LLM-as-a-judge, heuristic methods like Levenshtein distance, and statistical metrics like BLEU.