Generate AI-Ready llms.txt Files from Screaming Frog Crawls

This n8n workflow transforms Screaming Frog's internal_html CSV exports into optimized llms.txt files, which are structured for easy ingestion by large language models (LLMs). Users input the website name, a short description, and upload the CSV via a form-trigger node. The workflow processes the data, extracts relevant HTML content from URLs, and compiles it into a downloadable llms.txt file directly from the n8n UI.

Key features include an optional AI-powered text classifier node (deactivated by default) for intelligent URL filtering based on custom descriptions, ensuring only pertinent pages are included. This reduces noise and enhances data quality for AI training or analysis. The final 'Convert to File' node enables seamless downloads, with easy extensibility to cloud storage like Google Drive.

Benefits include significant time savings for SEO professionals, content analysts, and AI developers by automating tedious data extraction and formatting. Use cases span website audits, LLM fine-tuning datasets, competitive analysis, and healthcare site crawling for patient management resources (e.g., extracting protocol pages). Activate the classifier for domain-specific filtering, such as medical content, to boost relevance.