Web Fetch Tool: HTTP GET with Readable Content Extraction

web_fetch performs a straightforward HTTP GET and converts the response into readable content, turning HTML into markdown or plain text. JavaScript is never executed. For pages that rely heavily on JS or sit behind authentication, turn to the Web Browser tool instead.

Quick start

This tool is on by default and needs no setup:

await web_fetch({ url: "https://example.com/article" });

Tool parameters

url (string, required), The target URL to retrieve. Only http(s) is supported.
extractMode (markdown' | 'text, default: markdown), Determines the output format after the main content has been extracted.
maxChars (number), Caps the output at this character count. The value is clamped to tools.web.fetch.maxCharsCap.

Result

web_fetch yields a closed structured result carrying these fields:

Request details: url, finalUrl, status, extractMode, and extractor
Response details that appear only when available: contentType, title, and warning
Wrapped content details: externalContent, truncated, length, rawLength, fetchedAt, tookMs, and text
cached: true shows up when the response came from cache
spill: { path, chars, truncated? } appears when truncated content was saved to a private temporary file; truncated is included only if that file holds partial source content

length measures the wrapped text size. rawLength measures the extracted content size prior to external-content wrapping.

How it works

Fetch

Issues an HTTP GET using a Chrome-like User-Agent and the Accept-Language header. Private and internal hostnames are blocked, and redirects are re-validated.

Extract

Applies Readability to the HTML response to pull out the main content.

Fallback (optional)

When Readability fails and a fetch provider is configured, the request is retried through that provider, such as Firecrawl's bot-circumvention mode.

Cache

Responses are cached for 15 minutes (adjustable) to avoid fetching the same URL repeatedly.

Progress updates

web_fetch logs a public progress line only if the fetch remains unresolved after five seconds:

Fetching page content...

Fast cache hits and quick network replies complete before the timer triggers, so no progress line is ever shown. Cancelling the call stops the timer. The progress line exists purely as channel UI state and never carries any fetched page content.

Config

{
  tools: {
    web: {
      fetch: {
        enabled: true, // default: true
        provider: "firecrawl", // optional; omit for auto-detect
        maxChars: 20000, // default output chars; capped by maxCharsCap
        maxCharsCap: 20000, // hard cap for maxChars param
        maxResponseBytes: 750000, // max download size before truncation (32000-10000000)
        timeoutSeconds: 30,
        cacheTtlMinutes: 15,
        maxRedirects: 3,
        useTrustedEnvProxy: false, // let a trusted HTTP(S) env proxy resolve DNS
        readability: true, // use Readability extraction
        userAgent: "Mozilla/5.0 ...", // override User-Agent
        headers: {
          // optional; every value is treated as sensitive
          "X-Routing-Target": "staging",
        },
        ssrfPolicy: {
          dangerouslyAllowPrivateNetwork: false, // broad private-network opt-in; keep false by default
          allowedHostnames: ["internal.example"], // narrow exact host exception
          allowRfc2544BenchmarkRange: true, // opt-in for trusted fake-IP proxies using 198.18.0.0/15
          allowIpv6UniqueLocalRange: true, // opt-in for trusted fake-IP proxies using fc00::/7
        },
      },
    },
  },
}

Firecrawl fallback

Should Readability extraction come up short, web_fetch can switch to Firecrawl for bot-circumvention and improved extraction:

{
  tools: {
    web: {
      fetch: {
        provider: "firecrawl", // optional; omit for auto-detect from available credentials
      },
    },
  },
  plugins: {
    entries: {
      firecrawl: {
        enabled: true,
        config: {
          webFetch: {
            // apiKey: "fc-...", // optional; omit for keyless starter access
            baseUrl: "https://api.firecrawl.dev",
            onlyMainContent: true,
            maxAgeMs: 172800000, // cache duration (2 days)
            timeoutSeconds: 60,
          },
        },
      },
    },
  },
}

plugins.entries.firecrawl.config.webFetch.apiKey is optional and works with SecretRef objects. Older tools.web.fetch.firecrawl.* settings are automatically migrated to plugins.entries.firecrawl.config.webFetch through openclaw doctor --fix.

Note

If a Firecrawl API-key SecretRef is configured but cannot be resolved and no FIRECRAWL_API_KEY env fallback exists, the gateway fails fast at startup.

Note

Firecrawl baseUrl overrides are restricted: hosted traffic is locked to https://api.firecrawl.dev; self-hosted overrides must point at private or internal endpoints, and http:// is allowed only for those private targets.

Behavior as of the current runtime:

Setting tools.web.fetch.provider explicitly chooses the fetch fallback provider.
When provider is not provided, OpenClaw picks the first available web-fetch provider automatically based on the credentials it finds. Outside a sandbox, web_fetch may rely on installed plugins that declare contracts.webFetchProviders and register a matching provider during runtime. The official Firecrawl plugin currently serves as that fallback.
Within a sandbox, web_fetch calls support both bundled providers and installed ones whose official npm or ClawHub provenance has been verified. As of now, that includes only the official Firecrawl plugin; third-party external fetch plugins remain disallowed.
With Readability turned off, web_fetch goes directly to the chosen provider fallback. When no provider exists, it fails closed.

Custom request headers

Use tools.web.fetch.headers if your setup needs extra request metadata on outbound fetches, like a routing or service-injection header that directs traffic toward a gateway you manage.

{
  tools: {
    web: {
      fetch: {
        headers: {
          "X-Routing-Target": "${WEB_FETCH_ROUTING_TARGET}",
        },
      },
    },
  },
}

Warning

Any value you configure is considered sensitive and gets redacted from exposed config and debug captures. Those headers are still transmitted to every initial URL that web_fetch requests, and the model decides that URL. Only set credential headers when that matches your intended trust boundary.

Useful behavior to know:

Values are plain strings and accept ${VAR} environment substitution just like any other config string. Structured SecretRef values are not supported.
Headers affect only the direct web_fetch request. Provider fallbacks like Firecrawl hit their own API and never see these headers.
Validation happens when the request is assembled, not at config load, so a single bad entry is discarded while the rest still apply. Config load stays permissive on purpose: a fail-closed validation error from one header-name typo would take down the entire surface. Each dropped entry is logged by name.
Names that get dropped:
- Accept, Accept-Language, and User-Agent belong to the fetch and readability contract. For the user agent, use tools.web.fetch.userAgent.
- Framing and hop-by-hop names like Content-Length, Transfer-Encoding, Connection, and Upgrade, which a request either rejects outright or ignores.
- Names that are not valid HTTP tokens, such as "X Routing Target".
Values that get dropped: bytes a request cannot carry (CR, LF, NUL, or any character above U+00FF). Missing environment variables are flagged by config loading; the global $${VAR} escape stays available when the literal ${VAR} text is meant to appear.
Two entries whose names differ only by case collapse to the later entry, so a request never carries a comma-joined value that the receiving gateway cannot parse. The dropped name is logged without either value. If the later entry is unusable, neither value goes out.
Rejection occurs before the cache key is computed, so the key always matches the bytes actually sent: changing a header that is really sent partitions the fetch cache, while adding one that gets dropped does not.
When a redirect crosses origins, the guarded-fetch safe allowlist applies. Routing headers outside that list are dropped; standard safe headers such as Cache-Control, Content-Type, and Range are kept.

Trusted env proxy

If your deployment needs web_fetch to route through a trusted outbound HTTP(S) proxy, set tools.web.fetch.useTrustedEnvProxy: true.

In this mode, OpenClaw still runs hostname-based SSRF checks before sending the request, but it lets the proxy resolve DNS instead of doing local DNS pinning. Enable this only when the proxy is operator-controlled and enforces outbound policy after DNS resolution.

Note

If no HTTP(S) proxy env var is set, or the target host is excluded by NO_PROXY, web_fetch falls back to the normal strict path with local DNS pinning.

Limits and safety

maxChars is clamped to tools.web.fetch.maxCharsCap (default 20000)
Response body is capped at maxResponseBytes (default 750000, clamped to 32000-10000000) before parsing; oversized responses are truncated with a warning
Private/internal hostnames are blocked
tools.web.fetch.ssrfPolicy.allowedHostnames allows exact trusted hosts while leaving other private/internal targets blocked
tools.web.fetch.ssrfPolicy.dangerouslyAllowPrivateNetwork broadly permits private-network targets; enable it only when model-selected URLs are trusted in this deployment
tools.web.fetch.ssrfPolicy.allowRfc2544BenchmarkRange and tools.web.fetch.ssrfPolicy.allowIpv6UniqueLocalRange are narrow opt-ins for trusted fake-IP proxy stacks; leave them unset unless your proxy owns those synthetic ranges and enforces its own destination policy
Redirects are checked and limited by maxRedirects (default 3)
tools.web.fetch.headers values are redacted from exposed config and debug captures, sent to the initial fetched host, and retained on redirects only when the existing guarded-fetch policy allows them
useTrustedEnvProxy is an explicit opt-in and should only be enabled for operator-controlled proxies that still enforce outbound policy after DNS resolution
web_fetch is best-effort -- some sites need the Web Browser

Tool profiles

If you use tool profiles or allowlists, add web_fetch or group:web:

{
  tools: {
    allow: ["web_fetch"],
    // or: allow: ["group:web"]  (includes web_fetch, web_search, and x_search)
  },
}

Web Search -- search the web with multiple providers
Web Browser -- full browser automation for JS-heavy sites
Firecrawl -- Firecrawl search and scrape tools