Web Scraper: Extract Content from Sitemaps to Google Drive

This workflow automates comprehensive web scraping by starting with a website's XML sitemap to discover all pages systematically. It fetches the sitemap, parses URLs, and then scrapes each page using HTTP requests with browser-like headers, robust error handling, timeouts, and 3-second rate limiting to respect servers and avoid blocks.

Content extraction is sophisticated: JavaScript code identifies titles from tags, Open Graph, or H1s; strips unnecessary elements; detects frameworks like WordPress or Divi; and compiles clean text with metadata including word count, HTML size, response codes, and tech indicators. Output is formatted as markdown files with YAML frontmatter for easy analysis.

Benefits include no external dependencies, consistent results across sites, and valuable insights for audits or migrations. Use cases: competitive research, SEO analysis, content inventory before redesigns, or archiving site data. Saves hours of manual work with production-ready reliability.