promptfoo
FreeTest your prompts. Evaluate and compare LLM outputs, catch regressions, and improve prompt quality.
FreeFree tier
About promptfoo
promptfoo is an open-source CLI and library for evaluating and red-teaming LLM applications. It enables developers to test prompts, agents, and RAGs, compare outputs across multiple models (GPT, Claude, Gemini, DeepSeek, Azure, Bedrock, Ollama, and more), and perform automated vulnerability scanning. Using simple declarative configuration files, it integrates seamlessly into CI/CD pipelines for continuous quality and security checks. Used by OpenAI and Anthropic, promptfoo is MIT licensed and actively maintained. It supports features like side-by-side model comparison, regression detection, and code scanning for pull requests.
Key Features
Evaluate and compare LLM outputs across multiple models (GPT, Claude, Gemini, DeepSeek, etc.)
Red teaming and vulnerability scanning for LLM applications
Support for agents, RAGs, and prompts
Simple declarative configuration files (YAML)
CLI and programmable library integration
CI/CD integration for automated checks in workflows
Side-by-side model comparison and regression detection
Used by OpenAI and Anthropic
Open source (MIT license)
Package available via npm, brew, pip
Pros & Cons
Pros
- Open source and free (MIT licensed) with no usage limitations
- Supports a wide range of major LLM providers out of the box
- Built-in red teaming and vulnerability scanning capabilities
- Easy integration with CI/CD pipelines for automated quality assurance
- Simple declarative configuration lowers the learning curve for evaluations
- Used by leading AI companies like OpenAI and Anthropic, indicating trust and reliability
- Actively maintained and now part of OpenAI with continued open-source development
- Multiple installation methods (npm, brew, pip)
Cons
- Requires Node.js (^20 or >=22) for direct npm usage; may need additional setup for other environments
- Requires API keys and often paid accounts for most LLM providers to run evaluations
- Primarily CLI-based, which may have a learning curve for non-developers
- Documentation is extensive but may require initial investment to navigate effectively
Best For
Testing and improving prompt quality for LLM appsCatching regressions in LLM outputs during developmentSecurity red teaming and penetration testing for AI systemsComparing LLM performance to select the best model for a taskAutomating LLM evaluation in CI/CD pipelinesReviewing pull requests for AI-related security and compliance issuesSharing evaluation results with team members
FAQ
What is promptfoo?
Promptfoo is an open-source CLI and library for evaluating and red-teaming LLM applications. It allows you to test prompts, agents, and RAGs, compare outputs across multiple models, and perform vulnerability scanning. It uses simple declarative configs and integrates with CI/CD.
Is promptfoo free?
Yes, promptfoo is completely free and open source under the MIT license. There are no usage fees or paid tiers.
What LLM models does promptfoo support?
Promptfoo supports a wide range of providers including OpenAI (GPT), Anthropic (Claude), Google (Gemini), DeepSeek, Azure, Bedrock, Ollama, and many more. You can configure multiple models in your evaluation configs.
How do I install promptfoo?
Promptfoo can be installed globally via npm with `npm install -g promptfoo`, via Homebrew with `brew install promptfoo`, or via pip with `pip install promptfoo`. You can also use `npx promptfoo@latest` without installing.