Generate AI-Ready XML Sitemaps from Screaming Frog Website Crawls - n8n Workflow | Neura Market
Generate AI-Ready XML Sitemaps from Screaming Frog Website Crawls
This workflow helps you generate an **llms.txt** file (if you're unfamiliar with it, check out [this article](https://towardsdatascience.com/llms-txt-414d5121bcb3/)) using a **Screaming Frog export**. [Screaming Frog](https://www.screamingfrog.co.uk/seo-spider/) is a well-known website crawler. You can easily crawl a website. Then, export the **internal_html** section in CSV format.
## How It Works:
A **form** allows you to enter:
- The **name of the website**
- A **short description**
- The **internal_html.csv** file from your Screaming Frog export
Once the form is submitted, the **workflow is triggered automatically**, and you can **download the llms.txt file directly from n8n**.
## Downloading the File
Since the last node in this workflow is **Convert to File**, you will need to **download the file directly from the n8n UI**. However, you can easily **add a node** (e.g., Google Drive, OneDrive) to automatically upload the file **wherever you want**.
## AI-Powered Filtering (Optional):
This workflow includes a **text classifier node**, which is **deactivated by default**.
- You can **activate it** to apply a more **intelligent filter** to select URLs for the `llms.txt` file.
- Consider modifying the **description** in the classifier node to specify the type of URLs you want to include.
## How to Use This Workflow
1. **Crawl the website** you want to generate an `llms.txt` file for using **Screaming Frog**.
2. **Export the internal_html** section in CSV format.
3. In **n8n**, click **Test Workflow**, fill in the form, and **upload** the `internal_html.csv` file.
4. Once the workflow is complete, go to the **Export to File** node and **download the output**.
**That's it! You now have your llms.txt file!**
**Recommended Usage:**
Use this workflow **directly in the n8n UI by clicking** Test Workflow and uploading the file in the form.
Platform
n8n
Category
AI
Price
Free
Creator
Dataki
set
noOp
filter
summarize
stickyNote
formTrigger
convertToFile
extractFromFile
lmChatOpenAi
textClassifier
How to import this workflow into n8n
1Purchase or download the workflow to get the n8n workflow JSON file.
2In your n8n instance, open Workflows and choose "Import from File" (or paste the JSON with Ctrl+V on the canvas).
3Open each node marked with a credential warning and connect your own accounts and API keys.
4Run the workflow once manually to verify the data flow, then toggle it to Active.