Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision

Free

Self-aligning LLMs with minimal human supervision

FreeFree tier

Type

Open Source

Visit Website GitHub (38 stars)

About Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision

SELF-ALIGN is a novel approach for aligning large language models (LLMs) with human intentions using minimal human supervision. It combines principle-driven reasoning and the generative power of LLMs to self-align AI agents. The method consists of four stages: (1) generating synthetic prompts with a topic-guided method to enhance diversity, (2) using a small set of human-written principles and in-context learning from demonstrations to produce helpful, ethical, and reliable responses, (3) fine-tuning the original LLM with the self-aligned responses so that the model can generate desirable responses directly without the principle set or demonstrations, and (4) a refinement step to address overly brief or indirect responses. Applied to LLaMA-65b, SELF-ALIGN produced the Dromedary AI assistant, which surpasses Text-Davinci-003 and Alpaca on benchmark datasets with fewer than 300 lines of human annotations (200 seed prompts, 16 generic principles, and 5 exemplars). Accepted at NeurIPS 2023 as a Spotlight paper.

Key Features

Principle-driven reasoning for self-alignment

Minimal human supervision (fewer than 300 lines of annotations)

Synthetic prompt generation with topic-guided diversity

In-context learning from human-written principles and demonstrations

Fine-tuning the base LLM on self-aligned responses

Refinement step to fix overly brief or indirect responses

Produces the Dromedary AI assistant from LLaMA-65b

Outperforms Text-Davinci-003 and Alpaca on benchmarks

Pros & Cons

Pros

Requires only 200 seed prompts, 16 principles, and 5 exemplars (under 300 lines of human annotations)
Significantly surpasses state-of-the-art systems like Text-Davinci-003 and Alpaca on benchmarks
Eliminates the need for expensive reinforcement learning from human feedback (RLHF)
Uses the generative power of LLMs for self-alignment, enhancing scalability
Publicly available as an open-source research paper with code links

Cons

Based on LLaMA-65b, a large model requiring substantial compute resources
Still requires some human-written principles and seed prompts (though minimal)
The refinement step is needed to handle overly brief or indirect responses
Evaluated only on benchmark datasets; real-world deployment performance may vary
The method may not fully eliminate biases present in the base LLM or seed data

Best For

Aligning large language models with human intentions (helpfulness, ethics, reliability)Reducing the cost and reliance on human annotations for supervised fine-tuningCreating AI assistants that can generate high-quality responses without RLHFSelf-supervised improvement of LLM outputs for conversational AI

FAQ

What is SELF-ALIGN?

SELF-ALIGN is a principle-driven approach to align large language models with human intentions using minimal human supervision. It combines synthetic prompt generation, in-context learning from human principles, fine-tuning, and a refinement step to create an AI assistant that produces helpful, ethical, and reliable responses.

How does SELF-ALIGN differ from RLHF?

Unlike RLHF which requires extensive human feedback on model outputs, SELF-ALIGN uses a small set of human-written principles and demonstrations to guide the model through in-context learning, then fine-tunes the model on self-generated high-quality responses, significantly reducing human effort.

What model was used to create Dromedary?

Dromedary was developed by applying SELF-ALIGN to the LLaMA-65b base language model.

How much human annotation is needed?

The method requires fewer than 300 lines of human annotations, including 200 seed prompts, 16 generic principles, and 5 exemplars for in-context learning.

What benchmarks were used to evaluate Dromedary?

Dromedary significantly surpassed Text-Davinci-003 and Alpaca on various benchmark datasets, though specific benchmarks are not listed in the abstract.