AI-Powered Web Scraping to JSON with ScrapeNinja

This n8n workflow revolutionizes web scraping by combining ScrapeNinja's robust HTML fetching with Google Gemini AI to dynamically generate JavaScript extraction code. Traditional scrapers fail when websites update layouts, but this solution uses AI to analyze the HTML and create custom extractor functions on-the-fly, ensuring reliable data extraction.

The process starts with inputting a URL into the ScrapeNinja node, which retrieves clean HTML. This HTML is then passed to the Google Gemini Chat Model with a prompt to write a JS function that parses and structures the data (e.g., titles, prices, articles). The generated code runs in a secure sandbox, producing clean JSON output without manual coding or maintenance.

Ideal for e-commerce monitoring, news aggregation, lead generation, or research, it saves hours of debugging selectors. Self-hosted only; requires ScrapeNinja community node (v0.3.0+) and API credentials. Benefits include adaptability, no brittle XPath/CSS, and scalability for batch scraping.