Neura Market Logo

Firecrawl Integration: Search, Scrape, and Web Fetch Fallback

This page explains how OpenClaw integrates with Firecrawl for web search, scraping, and fallback extraction. It is intended for developers needing to configure Firecrawl with or without an API key.

Read this when

  • You want Firecrawl-backed web extraction
  • You want keyless Firecrawl Search (Free) or keyless web_fetch
  • You need a Firecrawl API key for search or higher limits
  • You want Firecrawl as a web_search provider
  • You want anti-bot extraction for web_fetch

OpenClaw integrates with Firecrawl through three distinct mechanisms:

  • acting as the web_search provider
  • offering explicit plugin tools: firecrawl_search and firecrawl_scrape
  • serving as a fallback extractor for web_fetch

This hosted extraction and search service bypasses bot detection and caches responses, making it useful for JavaScript-heavy sites or pages that reject basic HTTP requests.

Install plugin

After installing the official plugin, restart Gateway:

openclaw plugins install @openclaw/firecrawl-plugin
openclaw gateway restart

Keyless access and API keys

Firecrawl provides two web_search providers:

  • Firecrawl Search (firecrawl), accesses the hosted /v2/search API using your key; automatically activated when a key exists.
  • Firecrawl Search (Free) (firecrawl-free), uses the hosted starter tier without a key, requiring no API credentials. It must be manually chosen and never activates automatically, since selecting it forwards your search queries to Firecrawl's free service.

The explicitly configured Firecrawl web_fetch fallback also operates without a key. The explicit firecrawl_search and firecrawl_scrape tools need an API key. Add FIRECRAWL_API_KEY to the gateway environment or configure it to increase rate limits.

{
  tools: {
    web: {
      search: {
        provider: "firecrawl",
      },
    },
  },
  plugins: {
    entries: {
      firecrawl: {
        enabled: true,
        config: {
          webSearch: {
            apiKey: "FIRECRAWL_API_KEY_HERE",
            baseUrl: "https://api.firecrawl.dev",
          },
        },
      },
    },
  },
}

Notes:

  • Selecting Firecrawl during onboarding or via openclaw configure --section web automatically activates the installed Firecrawl plugin.
  • To run without an API key, choose Firecrawl Search (Free) in onboarding or set provider: "firecrawl-free". The keyed Firecrawl Search provider routes plugins.entries.firecrawl.config.webSearch.apiKey or FIRECRAWL_API_KEY.
  • web_search with Firecrawl supports query and count.
  • For Firecrawl-specific options such as sources, categories, or result scraping, rely on firecrawl_search.
  • baseUrl points to the hosted Firecrawl instance at https://api.firecrawl.dev by default. Self-hosted overrides are permitted only for private or internal endpoints; HTTP is allowed exclusively for those private targets.
  • FIRECRAWL_BASE_URL serves as the shared environment fallback for Firecrawl search and scrape base URLs.
  • Firecrawl search requests time out after 30 seconds by default; firecrawl_search's timeoutSeconds parameter adjusts this per request.

Configure Firecrawl web_fetch fallback

{
  tools: {
    web: {
      fetch: {
        provider: "firecrawl", // explicit selection enables keyless fallback
      },
    },
  },
  plugins: {
    entries: {
      firecrawl: {
        enabled: true,
        config: {
          webFetch: {
            baseUrl: "https://api.firecrawl.dev",
            onlyMainContent: true,
            maxAgeMs: 172800000,
            timeoutSeconds: 60,
          },
        },
      },
    },
  },
}

Notes:

  • The explicitly selected Firecrawl web_fetch fallback does not require an API key. When configured, OpenClaw sends plugins.entries.firecrawl.config.webFetch.apiKey or FIRECRAWL_API_KEY for elevated limits.
  • Choosing Firecrawl during onboarding or through openclaw configure --section web activates the plugin and assigns Firecrawl for web_fetch unless another fetch provider is already set.
  • firecrawl_scrape mandates an API key.
  • maxAgeMs determines the maximum age of cached results in milliseconds. Default is 172,800,000 ms (2 days).
  • onlyMainContent defaults to true; timeoutSeconds defaults to 60.
  • Legacy tools.web.fetch.firecrawl.* and tools.web.search.firecrawl.* configuration is automatically migrated by openclaw doctor --fix.
  • Firecrawl scrape and base URL overrides follow the same hosted/private rule as search: public hosted traffic uses https://api.firecrawl.dev; self-hosted overrides must point to private or internal endpoints.
  • firecrawl_scrape blocks obviously private, loopback, metadata, and non-HTTP(S) target URLs before forwarding them to Firecrawl, matching the web_fetch target-safety contract for explicit Firecrawl scrape calls.

firecrawl_scrape uses the same plugins.entries.firecrawl.config.webFetch.* settings and environment variables, including its required API key.

Self-hosted Firecrawl

Set plugins.entries.firecrawl.config.webSearch.baseUrl, plugins.entries.firecrawl.config.webFetch.baseUrl, or FIRECRAWL_BASE_URL when self-hosting Firecrawl. OpenClaw accepts http:// only for loopback, private-network, .local, .internal, or .localhost targets. Public custom hosts are rejected to prevent accidental exposure of Firecrawl API keys to arbitrary endpoints.

Firecrawl plugin tools

Choose this option when you need Firecrawl specific search settings rather than the generic web_search. An API key is mandatory.

Parameters:

  • query
  • count (range 1-100)
  • sources
  • categories
  • includeDomains / excludeDomains (hostnames only; cannot be used together)
  • tbs (time constraint, e.g. qdr:d, qdr:w, sbd:1)
  • location together with country (geographic targeting)
  • scrapeResults
  • timeoutSeconds

firecrawl_scrape

Pick this for pages with heavy JavaScript or bot protection where basic web_fetch performs poorly.

Parameters:

  • url
  • extractMode
  • maxChars
  • onlyMainContent
  • maxAgeMs
  • proxy
  • storeInCache
  • timeoutSeconds

Stealth / bot circumvention

Unless the caller provides overrides, firecrawl_scrape and the web_fetch Firecrawl fallback are set to proxy: "auto" combined with storeInCache: true. No proxy or storeInCache controls exist for firecrawl_search or the web_search Firecrawl provider; stealth proxy mode is only active for scrape and fetch requests.

Firecrawl's proxy mode manages bot avoidance through basic, stealth, or auto. When a basic attempt fails, auto will retry using stealth proxies, which can consume more credits than scraping without stealth.

How web_fetch uses Firecrawl

The web_fetch extraction sequence is:

  1. Readability (local)
  2. Configured fetch provider, for instance Firecrawl (when explicitly chosen, or detected automatically from configured credentials)
  3. Basic HTML cleanup (final fallback)

The tools.web.fetch.provider parameter controls the selection. If you leave it out, OpenClaw automatically picks the first available web-fetch provider based on configured credentials. The official Firecrawl plugin supplies that fallback.

870 words · updated Jul 27, 2026