ScrapingBee Sitemap Extractor to Google Sheets (Recursive)

This advanced n8n workflow automates comprehensive sitemap scraping for any domain. Triggered by a webhook with the domain (e.g., www.example.com), it first checks robots.txt for sitemap links, falling back to sitemap.xml if needed. It intelligently handles various response types: parsing plain text XML for links and sub-sitemaps, extracting text from binary files, and decompressing gzip content before processing. Website links are immediately appended to a 'links' column in your Google Sheet, while new XML sitemaps trigger recursive execution to uncover all nested links.

Key benefits include time-saving automation of manual sitemap crawling, robust error handling for real-world web files, and seamless integration with ScrapingBee for reliable scraping (bypassing blocks) and Google Sheets for easy data export/analysis. No coding required beyond API keys—ideal for SEO audits, competitor analysis, content inventory, or lead generation from public sites.

Use cases: SEO professionals mapping site structures, marketers collecting competitor URLs, researchers building datasets, or agencies offering scraping services. Outputs a complete link list in Sheets, ready for further processing like CSV export or CRM import. Compatible only with self-hosted n8n due to community nodes.