Neura Market
Prompttools logo

Prompttools

Free

Open-source Python tools for testing and evaluating models, vector DBs, and prompts.

FreeFree tier
Inputs: textOutputs: text
Type
Open Source
Company
Hegel AI
Visit Website GitHub (38 stars)

About Prompttools

PromptTools is an open-source, self-hostable Python toolkit for experimenting with, testing, and evaluating large language models (LLMs), vector databases, and prompts. Developed by Hegel AI, it provides a simple, code-first interface that allows developers to test prompts and parameters across multiple models (including OpenAI, Anthropic, LLaMA, HuggingFace, Mistral, Gemini, PaLM, Vertex AI, Azure OpenAI, and Replicate) and vector databases (such as Chroma, Weaviate, and LanceDB). Users can run experiments in Jupyter notebooks, Python scripts, or a local Streamlit playground, and visualize results with a single method call. The project aims to make prompt engineering and model evaluation accessible through familiar coding workflows, with support for fine-tuned models and retrieval accuracy testing.

Key Features

Support for multiple LLM APIs: OpenAI (ChatCompletion, Completion, fine-tuned), LLaMA.Cpp, HuggingFace, Anthropic, Mistral, Google Gemini, Google PaLM, Google Vertex AI, Azure OpenAI, Replicate, and Ollama (in progress)
Support for vector databases: Chroma, Weaviate, LanceDB
Experiment interfaces: Python code, Jupyter notebooks, and local Streamlit playground
Built-in visualization to compare prompt variations and model outputs
Open-source and self-hostable with pip install
Hosted playground available on Streamlit Community Cloud (limited functionality)
Easy testing of temperature and other parameters across models

Pros & Cons

Pros
  • Open-source and free to use with no hidden costs
  • Self-hostable, giving full control over data and environment
  • Supports a wide range of popular LLM APIs and vector databases
  • Simple, Pythonic API that integrates easily into existing workflows
  • Includes both code/notebook and GUI playground interfaces
  • Active development with ongoing support for new models and databases
Cons
  • Requires programming knowledge (Python) to use effectively
  • Lacks a fully managed cloud service; hosted playground is limited
  • Some integrations (e.g., Ollama) are still in progress
  • Documentation may not cover all edge cases for advanced use cases
  • Not designed for production-scale load testing or continuous monitoring

Best For

Comparing model responses across different LLMs (e.g., GPT-3.5 vs GPT-4) with the same promptEvaluating retrieval accuracy of vector databases for RAG pipelinesIterating on prompt engineering by testing variations of system and user messagesEducational experimentation to understand behavior of different models and parametersQuick prototyping and debugging before deploying prompts to production

FAQ

How do I install prompttools?
You can install it via pip: pip install prompttools.
Which LLMs does prompttools support?
It supports OpenAI (Completion, ChatCompletion, fine-tuned models), LLaMA.Cpp, HuggingFace (Hub API and Inference Endpoints), Anthropic, Mistral AI, Google Gemini, Google PaLM, Google Vertex AI, Azure OpenAI Service, Replicate, and Ollama (in progress).
Can I use prompttools with vector databases?
Yes, it supports Chroma, Weaviate, and LanceDB for evaluating retrieval accuracy.
Is there a hosted version of the playground?
Yes, a hosted version of the playground is available on Streamlit Community Cloud, but it does not support LlamaCpp.
Do I need a GPU to use prompttools?
No, prompttools works with cloud-based APIs (OpenAI, Anthropic, etc.) and local CPU models via LLaMA.Cpp, but GPU may accelerate local inference.