Website Crawler & AI Chatbot with Google Sheets Memory

This n8n workflow automates website crawling and turns static sites into interactive AI chatbots. In initial mode, it validates URLs with GPT-5 nano, discovers sitemaps from robots.txt, selects relevant page types (pages, posts, etc.), crawls each page, converts HTML to Markdown, and uses AI to extract language, headings, links, and summaries. All data is structured and stored in Google Sheets, marked ready when complete.

Once indexed, switch to agent mode: a LangChain-style agent queries the Sheets database to answer questions as if it's the website itself. It can fetch live data via HTTP requests for updates, making it ideal for dynamic interactions without recrawling.

Benefits include massive time savings for SEO analysts, content creators, and marketers—index once, query endlessly. Use cases: competitor analysis, content research, FAQ bots from sites, or monitoring changes. Note high initial AI token costs for large sites; perfect for targeted crawling.

Integrates GPT-5 nano/GPT-4o, Google Sheets, HTTP, and looping logic for scalable extraction.