Indeed Job Scraper with AI Filtering & Company Research Using Apify and Avily

This workflow contains community nodes that are only compatible with the self-hosted version of n8n. This workflow **scrapes job listings** on Indeed via Apify, **automatically gets that dataset**, extracts information about the listing, **filters jobs for relevance**, finds a decision maker at the company, and updates a database (Google Sheets) with that info for outreach. **All you need to do is run the Apify actor** then the database will update with the processed data. #### Benefits: - Complete Job Search Automation - A webhook monitors the Apify actor which sends an integration and starts the process. - AI-Powered Filter - Uses ChatGPT to analyze content/context, identify company goals, and filters based on job description. - Smart Duplicate Prevention - Automatically tracks processed job listings in a database to avoid redundancy. - Multi-Platform Intelligence - Combines Indeed scraping, web research via Avily, and enriches each listing. - Niche Focus - Processes content from multiple niches, currently 6 (hardcoded) but can be changed to fit other niches (just prompt the job filter node). #### How It Works: 1. **Indeed Job Discovery:** - Search and apply filter for relevant job listings, copy and use URL in Apify. - Uses Apify's Indeed job scraper to scrape job listings from the URL of interest. - Automatically scrapes the information, stores it in a dataset, and initiates an integration. 2. **Oncoming Data Processing:** - Loops over 500 items (can be changed) with a batch size of 55 items (can be changed) to avoid running into API timeouts. - Multiple filters to ensure all fields are scrapped with our required metrics (website must exist and number of employees < 250). - Duplicate job listings are removed from the oncoming batch to be processed. 3. **Job Analysis & Filter:** - An additional filter to remove any job listing from the oncoming batch if it already exists in the Google Sheets database. - Then all new job listings get passed to ChatGPT which uses information about the job post/description to determine if it is relevant to us. - All relevant jobs get a new field verdict which is either true or false and we keep the ones where verdict is true. 4. **Enrich & Update Database:** - Uses Avily to search for a decision maker (doesn't always find one) and populate a row in Google Sheet with information about the job listing, the company, and a decision maker at that company. - Waits for 1 minute and 30 seconds to avoid Google Sheets and ChatGPT API timeouts then loops back to the next batch to start filtering again until all job listings are processed. ### Required Google Sheets Database Setup: Before running this workflow, create a Google Sheets database with these exact column headers: **Essential Columns:** - jobUrl - Unique identifier for job listings - title - Position title - descriptionText - Description of job listing - hiringDemand/isHighVolumeHiring - Are they hiring at high volume? - hiringDemand/isUrgentHire - Are they hiring with urgency? - isRemote - Is this job remote? - jobType/0 - Job type: In person, Remote, Part-time, etc. - companyCeo/name - CEO name collected from Avily's search - icebreaker - Column for holding custom icebreakers for each job listing (Not completed in the workflow. I will upload another that does this called Personalized IJSFE) - scrapedCeo - CEO name collected from Apify Scraper - email - Email listed for job listing - companyName - Name of company that posted the job - companyDescription - Description of the company that posted the job - companyLinks/corporateWebsite - Website of the company that posted the job - companyNumEmployees - Number of employees the company listed that they have - location/country - Location of where the job is to take place - salary/salaryText - Salary on job listing **Setup Instructions:** - Create a new Google Sheet with these column headers in the first row. - Name the sheet whatever you please. - Connect your Google Sheets OAuth credentials in n8n. - Update the document ID in the workflow nodes. The merge logic relies on the id column to prevent duplicate processing, so this structure is essential for the workflow to function correctly. Feel free to reach out for additional help or clarification at my Gmail: terflix45@gmail.com, and I'll get back to you as soon as I can. ### Set Up Steps: 1. **Configure Apify Integration:** - Sign up for an Apify account and obtain an API key. - Get the Indeed job scraper actor and use Apify's integration to send an HTTP request to your n8n webhook (if the test URL doesn't work, use the production URL). - Use the Apify node with Resource: Dataset, Operation: Get items, and use your API key as your credentials. 2. **Set Up AI Services:** - Add OpenAI API credentials for job filtering. - Add Avily API credentials for company research. - Set up appropriate rate limiting for cost control. 3. **Database Configuration:** - Create a Google Sheets database with the provided column structure. - Connect Google Sheets OAuth credentials. - Configure the merge logic for duplicate detection. 4. **Content Filtering**

Platform

n8n

Category

Data & Analytics

Price

Free

Creator

Adrian Bent

• Lead Gen
• if
• set
• wait
• merge
• filter
• webhook
• stickyNote
• googleSheets
• apify