CONTEXT

1. Project Overview

Build a robust Node.js web‑scraping tool using Puppeteer to extract CVE data from the Wiz vulnerability database. The app should:

• Navigate to the CVE search page
• Handle infinite scroll / “Load more” pagination
• Scrape CVE IDs and metadata
• Visit each CVE's detail page
• Extract "Additional resources" links
• Output structured JSON data

---

2. Core Requirements

• Target URL: https://www.wiz.io/vulnerability-database/cve/search
• Infinite‑scroll handling: Auto-click the “Load more” button until all entries are loaded
• Data extraction: Scrape CVE IDs, severity, score, technologies, component, publish date
• Detail‑page navigation: Click into each CVE detail page for extra info
• Additional resources: Extract titles + URLs from each CVE’s "Additional resources" section
• JSON output: Return a structured JSON, e.g.:

{
  "scrapeDate": "2025-07-07T00:32:00.000Z",
  "totalCVEs": 140558,
  "cveData": [
    {
      "cveId": "CVE-2025-6926",
      "severity": "HIGH",
      "score": 8.8,
      "technologies": ["Linux","Debian"],
      "component": "mediawiki",
      "publishedDate": "Jul 03, 2025",
      "detailUrl": "...",
      "additionalResources": [
        { "title": "NVD CVE", "url": "..." },
        { "title": "Wordfence Analysis", "url": "..." }
      ]
    }
  ]
}

---

3. Technical Implementation

3.1. Project Setup

mkdir wiz-cve-scraper
cd wiz-cve-scraper
npm init -y
npm install puppeteer fs-extra

3.2. Dependencies

• puppeteer: Browser automation + scraping
• fs‑extra: JSON read/write utilities
• Headless browser setup with viewport config and user‑agent

3.3. Core Code Structure (e.g. app.js)

const puppeteer = require('puppeteer');
const fs = require('fs-extra');

class WizCVEScraper {
  constructor(){ this.browser = null; this.page = null; this.cveData = []; }
  async initialize(){
    this.browser = await puppeteer.launch({
      headless: false,
      args: ['--no-sandbox','--disable-setuid-sandbox'],
      defaultViewport: { width: 1920, height: 1080 }
    });
    this.page = await this.browser.newPage();
    await this.page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64)');
  }
  async scrapeAllCVEs(){
    await this.page.goto('https://www.wiz.io/vulnerability-database/cve/search', {
      waitUntil: 'networkidle2', timeout: 30000
    });
    await this.loadAllCVEs();
    const cveList = await this.extractCVEList();
    for(const c of cveList){
      const details = await this.processCVEDetails(c);
      this.cveData.push(details);
    }
    return this.cveData;
  }
  async loadAllCVEs(){
    // Repeatedly click “Load more” until no more to load
  }
  async extractCVEList(){
    // Scrape table rows → CVE IDs, scores, metadata + URLs
  }
  async processCVEDetails(cve){
    // Open CVE detail page, extract links from “Additional resources”
  }
}

---

4. Handle Infinite Scroll

• Detect “Load more” button via CSS selectors
• Click until disabled or hidden
• Wait for new entries to render (e.g. networkidle, DOM changes)
• Retry logic + timeouts

---

5. Data Extraction Strategy

5.1. CVE Table

• Use page.$$eval() to collect rows
• Extract fields: ID, severity, score, technologies, component, publish date, URL

5.2. CVE Detail Pages

• Visit each CVE URL
• Query "Additional resources" section
• Return list of link titles + URLs

---

6. JSON Output Structure

Top-level JSON should include:

• scrapeDate (ISO timestamp)
• totalCVEs (count)
• cveData (array of detailed objects, each with fields as above)

---

7. Error Handling & Robustness

• Use try/catch for all navigation & scraping steps
• Retry on network failures or timeouts
• Respect rate limiting: use delays (e.g. await page.waitForTimeout(...))
• Gracefully log errors; skip problematic entries

---

8. Performance Optimizations

• Fetch detail pages concurrently (with a concurrency cap)
• Use a pool of browser pages
• Cache processed CVEs to avoid duplicates
• Optimize DOM queries (minimal selectors)
• Show progress (e.g. via console logs or a progress bar)

---

9. Configuration / Customization

• Accept config options: max concurrency, delay between loads, filters (date range, severity)
• Support resume functionality via checkpoints
• Allow output filename customization

---

10. Testing & Validation

• Unit tests for helper functions (e.g. extractors)
• Validate output JSON schema
• Include sample run results
• Monitor performance (time, memory)

---

11. Bonus Features

• API Endpoint: Use Express.js to trigger scraping
• Scheduler: Cron-based triggering
• Analytics: Generate stats & trends

---

✅ Deliverables Checklist

• Fully functional Node.js app as per specs
• README.md with installation, usage & troubleshooting
• package.json with scripts (e.g. start, test)
• Example JSON output files
• Logging and robust error handling
• (Optional) Bonus features if implemented

---

References

1. Wiz CVE search: https://www.wiz.io/vulnerability-database/cve/search
2. Puppeteer infinite scroll patterns, etc.
3. JSON parsing & storage guides

---