Neura Market
instruct-eval logo

instruct-eval

Free

This repository contains code to quantitatively evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out tasks.

FreeFree tier
Type
Open Source
Company
Declare Lab
Visit Website GitHub (38 stars)

About instruct-eval

InstructEval is a comprehensive evaluation framework for instruction-tuned large language models (LLMs). It provides code to quantitatively assess models like Alpaca, Flan-T5, and Vicuna on a suite of held-out academic benchmarks including MMLU, BBH, DROP, and HumanEval. The tool supports a wide range of model architectures from HuggingFace Transformers (AutoModelForCausalLM, AutoModelForSeq2SeqLM, LlamaForCausalLM, ChatGLM) and simplifies benchmarking across multiple tasks. Additionally, it integrates Red-Eval for safety evaluation and the IMPACT dataset for evaluating writing capabilities. The project is open-source and actively maintained by the Declare Lab.

Key Features

Supports multiple benchmarks: MMLU, BBH, DROP, HumanEval
Compatible with HuggingFace model architectures (CausalLM, Seq2SeqLM, Llama, ChatGLM)
Includes Red-Eval for safety evaluation and jailbreak testing
Includes IMPACT dataset for evaluating writing capabilities (Informative, Professional, Argumentative, Creative)
Simple and convenient evaluation with a unified interface
Open-source with active development and a leaderboard

Pros & Cons

Pros
  • Supports a wide variety of popular LLMs out-of-the-box
  • Covers multiple challenging benchmarks for holistic evaluation
  • Includes dedicated safety evaluation (Red-Eval) and writing evaluation
  • Actively maintained with regular updates and new features
  • Easy to use with HuggingFace integration
Cons
  • Requires familiarity with command-line tools and Python
  • Evaluation may need significant GPU resources for large models
  • Limited to benchmarks and datasets included in the repository

Best For

Evaluating instruction-tuned models on academic benchmarksComparing model generalization across unseen tasksAssessing safety and robustness of LLMs using jailbreaking promptsMeasuring writing quality in four aspectsBenchmarking custom models before deployment

FAQ

What models does InstructEval support?
It supports models from HuggingFace Transformers including AutoModelForCausalLM (e.g., GPT-2, GPT-J, OPT-IML, BLOOMZ), AutoModelForSeq2SeqLM (e.g., Flan-T5, Flan-UL2, TK-Instruct), LlamaForCausalLM (e.g., LLaMA, Alpaca, Vicuna), and ChatGLM.
What benchmarks are included?
InstructEval evaluates on held-out tasks including MMLU, BBH, DROP, HumanEval, and additionally offers safety evaluation via Red-Eval and writing capability evaluation via the IMPACT dataset.
Is InstructEval free and open-source?
Yes, the repository is open-source under an MIT license and can be used freely.