Neura Market
datasetGPT logo

datasetGPT

Free

A command-line interface to generate textual and conversational datasets with LLMs. ![GitHub Repo stars](https://img.shields.io/github/stars/radi-cho/datasetGPT?style=social)

FreeFree tier
Type
Open Source
Visit Website GitHub (40 stars)

About datasetGPT

datasetGPT is an open-source command-line interface and Python library for generating textual and conversational datasets using large language models (LLMs). It supports multiple backends including OpenAI, Cohere, and Petals, allowing users to generate diverse text samples by specifying prompts, options, and generation parameters. The tool outputs structured JSON datasets, making it suitable for tasks such as constructing AI content detection corpora, collecting LLM conversations for research, automating text generation over large input sets, and creating data for fine-tuning smaller models. Installation is via pip install datasetGPT, and users must provide API keys for the chosen backends. The project is maintained on GitHub by radi-cho and is distributed freely without restrictions on downstream use, though users should comply with backend API terms.

Key Features

Command-line interface and Python library for LLM-based dataset generation
Supports multiple backends: OpenAI, Cohere, Petals
Customizable prompts with template variables and options
Generates structured JSON datasets with sample metadata
Batch generation across multiple backend/model combinations
Open source and freely distributable

Pros & Cons

Pros
  • Open source with no usage restrictions
  • Simple CLI and Python API for flexible integration
  • Multi-backend support allows comparing outputs from different LLMs
  • Structured JSON output with full generation parameters for reproducibility
  • Easy installation via pip
Cons
  • Requires API keys for third-party backends (OpenAI, Cohere, etc.) which may incur costs
  • Limited to text generation; no support for multi-modal or other data types
  • Documentation may be sparse beyond README; depends on GitHub community support
  • Privacy concerns when sending data to external LLM APIs

Best For

Constructing textual corpora to train/fine-tune detectors for AI-generated contentCollecting datasets of LLM-produced conversations for research and analysis of AI performance, impact, and ethicsAutomating LLM tasks over large input texts, e.g., summarizing 1000 paragraphs with a single CLI commandLeveraging large LLMs to produce diverse texts for a task, then fine-tuning a smaller model with the generated data

FAQ

What backends does datasetGPT support?
datasetGPT supports OpenAI, Cohere, and Petals as backend LLM providers.
Is datasetGPT free to use?
Yes, datasetGPT is open source and free to use. However, usage of third-party backends may require payment for API access.
How do I install datasetGPT?
You can install datasetGPT via pip: `pip install datasetGPT`. Additionally, install backend-specific packages like `openai`, `cohere`, or `petals` as needed.
Can I use datasetGPT for conversational datasets?
Yes, datasetGPT is designed to generate both textual and conversational datasets.
What format are the generated datasets in?
The datasets are output as JSON arrays, each entry containing fields such as sample_id, backend, prompt, options, generation parameters, and the generated output text.