Extract Clean Web Content with Anti-Bot Fallback for AI Agents

This n8n workflow enables reliable extraction of clean, readable content from public webpages by accepting a URL and optional fulltext flag. It first attempts to use the n8n-nodes-webpage-content-extractor community node for direct scraping. If the site employs anti-bot measures (e.g., Cloudflare), it seamlessly falls back to Scrape.do API, which offers a generous free tier.

Output modes include full page content ({title, text}) when fulltext is true, or summarized excerpts ({title, url, content}) when false. Designed exclusively for self-hosted n8n instances due to the required community node, it's optimized as a reusable sub-workflow via Execute Sub-workflow node or as a tool for AI Agents with Call n8n Workflow Tool.

Key benefits: Bypasses common scraping blocks, delivers structured data ready for AI processing, and saves hours of manual content curation. Ideal use cases encompass chatbot knowledge enrichment, automated summarization pipelines, RSS feed enhancement, and empowering AI agents to browse/extract web content dynamically.

Setup is straightforward: Install the community node, provide API keys if needed for fallback, and integrate via simple JSON input.