promptfoo-action

# Github Action for LLM Prompt Evaluation

This Github Action uses [promptfoo](https://www.promptfoo.dev) to produce a before/after view of edit prompts.

When you change a prompt, an eval will automatically be posted on the pull request:

<img width="650" alt="pull request llm eval" src="https://github.com/typpo/promptfoo-action/assets/310310/ec75fb39-c6b1-4395-9e41-6d66a7bf8657"/>

The provided link opens the promptfoo web viewer, which allows you to interactively explore the before vs. after:

<img width="650" alt="promptfoo web viewer" src="https://github.com/typpo/promptfoo-action/assets/310310/d0ef0497-0c1a-4886-b115-1ee92680891b"/>

## Supported Events

This action supports multiple GitHub event types:
- **Pull Request** (`pull_request`, `pull_request_target`) - Compares changes between base and head branches
- **Push** (`push`) - Compares changes between commits *(requires v1.1.0+)*
- **Manual Trigger** (`workflow_dispatch`) - Allows manual evaluation with custom inputs *(requires v1.1.0+)*

> **Note:** Version v1.0.0 only supports `pull_request` events. To use `push` or `workflow_dispatch` events, please use `@v1` (which now points to v1.1.0+) or explicitly use `@v1.1.0`.

## Configuration

The action can be configured using the following inputs:

| Parameter            | Description                                                                                                                                               | Required |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| `config`             | The path to the configuration file. This file contains settings for the action.                                                                           | Yes      |
| `github-token`       | The Github token. Used to authenticate requests to the Github API.