Autonomous AI Web Crawler for Data Extraction

This n8n workflow features an advanced AI agent designed to crawl websites starting from an input URL, retrieving targeted information such as social media profile links. Equipped with two key tools—a text retrieval tool to extract all page content and a URLs tool to identify and navigate links—the agent intelligently explores subpages to find relevant data. Users can customize the agent's prompt and JSON schema to target different information, like contact details or product listings, making it highly versatile.

The workflow operates by taking a starting website URL from a database like Supabase (easily swappable), where the agent iteratively uses its tools to gather and parse content until the desired data is obtained. It leverages OpenAI for decision-making and processing, ensuring autonomous operation without manual intervention. A demo video showcases it in action, highlighting its efficiency in real-world scenarios.

Benefits include significant time savings for data collection tasks, scalability for bulk processing multiple sites, and adaptability across industries. Use cases span lead generation (scraping contacts), market research (gathering social profiles), competitive analysis (extracting links), and content aggregation. Its agentic AI approach handles dynamic sites better than traditional scrapers, reducing maintenance needs.

Setup is straightforward: connect a database for input URLs, configure the agent's prompt/schema, set OpenAI credentials, and optionally modularize tools into sub-workflows for reusability.