AI Website Scraper & Custom Prompt Analyzer with Apify, Gemini & LangChain

This n8n workflow automates website scraping and AI-driven analysis using Apify for content extraction in Markdown format, Gemini (via OpenRouter) for intelligent prompt execution, and LangChain for structured processing. Users input a URL, maximum pages to scrape, and a custom prompt (e.g., 'collect all contact info'), triggering the workflow to fetch content from multiple pages, process each with AI, and return aggregated JSON results.

Key benefits include time-saving automation for data extraction tasks, flexibility for any prompt-based analysis, and scalability for handling multiple pages without manual intervention. It integrates seamlessly with Apify's firescraper actor for robust scraping, avoiding common issues like JavaScript rendering or anti-bot measures.

Use cases span market research (summarizing news sites), lead generation (extracting contacts), job hunting (parsing listings), and content curation (news aggregation). Perfect for IT, development, and data collection in lifestyle or news categories, reducing hours of manual work to minutes.

The workflow supports enqueueing for async processing, GET method defaults, and handles metadata like titles and images alongside content, ensuring comprehensive outputs for downstream automations like CRM integration or reporting.