Custom API & Web Scraping with HTTP Requests

This workflow showcases how to integrate with services lacking n8n pre-built nodes by leveraging the versatile HTTP Request node. It demonstrates three key techniques: splitting API responses into manageable items, web scraping HTML content, and implementing pagination for complete data retrieval. Ideal for developers needing flexible data extraction without custom coding.

The first branch fetches mock album data from JSONPlaceholder API via HTTP Request, then uses Item Lists to split the JSON array into individual items for streamlined processing. This approach is perfect for handling bulk API responses efficiently.

The second branch performs web scraping by requesting a random Wikipedia page and extracting the article title using the HTML Extract node. This enables quick data pulls from websites without dedicated integrations.

The third branch manages pagination for GitHub's starred repositories API, dynamically setting page parameters with Set nodes, checking completion via If conditions, and looping until all data is retrieved. This ensures comprehensive collection from paginated endpoints.

Benefits include time-saving custom integrations, broad applicability for data collection in any domain, and scalability for production use. Use cases span web scraping reviews, aggregating API data for analytics, or syncing non-native services in automation pipelines.