APE: Human-Level Prompt Engineers (2023)

Free

LLM auto-generates and selects instructions — beats human prompts

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About APE: Human-Level Prompt Engineers (2023)

Automatic Prompt Engineer (APE) is a method for automatically generating and optimizing instructions for large language models (LLMs). Inspired by classical program synthesis and human prompt engineering, APE treats the instruction as a 'program' and searches over a pool of candidate instructions proposed by an LLM to maximize a chosen score function. The selected instruction is then evaluated by measuring the zero-shot performance of another LLM. In experiments across 24 NLP tasks, APE-generated instructions outperformed prior LLM baselines by a large margin and achieved better or comparable performance to human-written instructions on 19 out of 24 tasks. The method can also steer models toward truthfulness and/or informativeness and improve few-shot learning performance by prepending generated prompts to standard in-context learning prompts.

Key Features

Automatic instruction generation and selection using LLMs

Treats instructions as 'programs' optimized by searching over candidate pool

Candidates proposed by an LLM, selected to maximize a score function

Zero-shot evaluation of generated instructions on another LLM

Outperforms prior LLM baseline and matches/exceeds human annotators on most tasks

Can steer models toward truthfulness and informativeness

Improves few-shot learning by prepending prompts to in-context learning prompts

Pros & Cons

Pros

Generates prompts that outperform prior automated baselines
Achieves comparable or better results than human-written prompts on most tasks
Fully automatic, no manual prompt tuning required
Can be used to steer model behavior (truthfulness, informativeness)
Improves few-shot learning with minimal additional cost

Cons

Requires access to a large language model for candidate generation and evaluation
Computational cost may be high due to iterative search and LLM calls
Performance depends on the quality and diversity of initial candidate proposals
Not designed for multimodal or non-text tasks

Best For

Automated prompt engineering for any NLP taskImproving truthfulness and informativeness of LLM outputsEnhancing few-shot learning performance via better initial promptsBenchmarking and comparing prompt quality across models

FAQ

What is Automatic Prompt Engineer (APE)?

APE is a method that automatically generates and selects instruction prompts for large language models by treating the instruction as a program and searching over candidates proposed by an LLM.

How does APE work?

APE starts by having an LLM propose a pool of instruction candidates. It then searches this pool to find the instruction that maximizes a given score function (e.g., task accuracy). The best instruction is evaluated on a separate LLM in a zero-shot setting.

Does APE outperform human-written prompts?

Yes, on 19 out of 24 NLP tasks, APE-generated instructions achieved better or comparable performance to those written by human annotators, and outperformed previous automated baselines.

Can APE be used for few-shot learning?

Yes, APE can improve few-shot learning performance by generating effective instructions that are prepended to standard in-context learning prompts.