About gpt-crawler
GPT Crawler is an open-source tool from BuilderIO that crawls websites to generate knowledge files in JSON format. These files can be uploaded to OpenAI to create custom GPTs or custom assistants based on site content. It offers flexible configuration including URL start point, URL match pattern, content selector, and page limit. The tool can be run locally with Node.js, in a Docker container, or as an API. It is designed to simplify the process of turning any website documentation or content into a specialized AI assistant.
Key Features
Crawls websites and generates a JSON knowledge file from page content
Configurable start URL, URL match pattern, CSS selector, and maximum pages to crawl
Supports local execution (Node.js), Docker containers, or as an API
Output can be directly uploaded to OpenAI to create custom GPTs or custom assistants
Open-source and free to use
Includes example configuration for crawling documentation sites
Pros & Cons
Pros
- Fully open-source and free with no usage limits
- Easy to configure with simple TypeScript/JavaScript configuration
- Multiple deployment options: local, Docker, or API mode
- Outputs structured JSON ready for OpenAI’s custom GPT/assistant upload
- Active GitHub repository with many stars and forks, indicating community support
Cons
- Requires basic familiarity with Node.js and command-line tools to set up
- No built-in authentication or login handling for crawling private/password-protected sites
- Crawling can be slow on large sites and may require careful configuration to avoid rate limiting
- The resulting custom GPT feature requires a paid ChatGPT plan to use
Best For
Create a custom GPT that answers questions about your product documentationBuild an AI assistant trained on specific website contentGenerate knowledge bases from multiple web pages for RAG applicationsQuickly turn a public website into a private AI chatbot knowledge source
FAQ
What is GPT Crawler?
GPT Crawler is an open-source tool that crawls one or more URLs, scrapes content based on a configurable CSS selector, and outputs a JSON file. That file can be uploaded to OpenAI to create a custom GPT or custom assistant with knowledge from the crawled site.
How do I get started with GPT Crawler?
Clone the repository, install dependencies with npm i, edit config.ts to set the URL and other options, then run npm start. The output is saved as output.json.
Can I run GPT Crawler in a Docker container?
Yes, the repository includes a Dockerfile. You can build and run the container using Docker commands as described in the README.
Is GPT Crawler free to use?
Yes, it is completely free and open-source, released under a license included in the repository.