Neura Market
Auto-evaluator logo

Auto-evaluator

Free

a lightweight evaluation tool for question-answering using Langchain ![GitHub Repo stars](https://img.shields.io/github/stars/PineappleExpress808/auto-evaluator?style=social)

FreeFree tier
Inputs: text
Type
Open Source
Visit Website GitHub (40 stars)

About Auto-evaluator

Auto-evaluator is a lightweight, open-source evaluation tool for question-answering (QA) chains built with LangChain. It allows users to input a set of documents, automatically generates question-answer pairs using an LLM (GPT-3.5-turbo), and evaluates the responses from a configurable QA chain. Users can explore scoring across different chain configurations (e.g., text splitting, embeddings, retrieval methods) and grade prompts. The tool runs as a Streamlit app and requires API keys for OpenAI and Anthropic to access all default models, though additional models can be added. It provides a dashboard for comparing performance and is also available as a hosted app and a Hugging Face Space.

Key Features

Auto-generates question-answer pairs from user-provided documents using GPT-3.5-turbo
Supports configurable text splitting methods (chunk size, overlap) and embedding methods
Allows selection of retrieval type and number of neighbors for context retrieval
Uses LLM self-grading (GPT-3.5-turbo) to score response quality against ground truth answers
Provides a Streamlit dashboard to compare scores across multiple chain configurations
Open source and free to use (requires OpenAI/Anthropic API keys for full functionality)

Pros & Cons

Pros
  • Lightweight and easy to set up as a Streamlit app
  • Automates the creation of evaluation datasets from any document set
  • Provides quantitative scoring to objectively compare chain settings
  • Extensible: additional LLMs and models can be integrated
  • Fully open source with a permissive license
Cons
  • Requires external API keys (OpenAI, Anthropic) for default models, incurring usage costs
  • Self-grading relies on LLM judgment, which may have biases or inconsistencies
  • Limited built-in metrics beyond LLM-based scoring; no support for human evaluation workflows
  • Documentation is primarily the README; no formal user guide or API docs

Best For

Evaluating and comparing different LLM chain configurations for QA tasksBenchmarking retrieval-augmented generation (RAG) pipelinesTesting the impact of chunking and embedding strategies on answer accuracyDeveloping and refining custom QA systems in research or production

FAQ

What is Auto-evaluator?
Auto-evaluator is a lightweight evaluation tool for LLM-based question-answering chains. It auto-generates QA pairs from documents, runs a configurable QA chain, and uses an LLM to score the responses.
How do I run Auto-evaluator?
Install the requirements with `pip install -r requirements.txt` and then run `streamlit run auto-evaluator.py`. You will need an OpenAI API key (and optionally an Anthropic key) to access default models.
What models does Auto-evaluator support?
By default it uses GPT-3.5-turbo for QA generation and grading. It also supports Anthropic models if an API key is provided. Additional models (e.g., from Hugging Face) can be easily added.
Is Auto-evaluator free?
The tool itself is free and open source. However, using it requires API keys from OpenAI and/or Anthropic, which have their own usage costs.
Can I contribute to Auto-evaluator?
Yes, the repository is open on GitHub at https://github.com/rlancemartin/auto-evaluator. Contributions such as bug fixes, new features, and documentation improvements are welcome.