HAR-Derived API Client: Capture XHR and Replay It

This skill records a site's XHR traffic into a HAR file and then distills that recording into the private JSON API the page was calling, so you can replay it with plain HTTP instead of driving a browser on every request. It is for anyone who has found themselves looping browser_navigate against the same page, or who needs a CLI or client for a site that has no public API but clearly fetches JSON. The trick is credited to Jared Longster and was popularized by Dax (thdxr).

What it does

The workflow has three stages. First, a real browser visits the site and performs the interaction you care about while its network traffic is written to a HAR file. Second, a derivation script reads that HAR, keeps only the XHR, fetch, and JSON entries, groups them by endpoint, and prints what a client would need: the method, the URL or path template, query parameters, request bodies, and replay hints such as the browser User-Agent and whether cookies or auth headers were present. Third, you write a small HTTP client from that printout and run it without a browser in the loop.

The payoff is cost and speed. Browser automation is expensive and slow compared with a single HTTP request, so once you have the endpoint, you stop paying for the browser. What the skill does not do is bypass authentication, solve CAPTCHAs, or defeat bot detection. It captures and replays. If the site needs a logged-in session, you carry the captured headers and cookies forward; you do not forge them.

The scripts are stdlib plus Playwright. Capture needs Playwright. Derivation is pure stdlib. Replay needs only requests or httpx, or curl.

It covers every Hermes browser pathway: the default local browser_navigate backend, the cloud and remote backends (Browserbase, Browser-Use, Firecrawl), and any /browser connect CDP endpoint. There are two capture scripts, one for a browser you launch and one for a browser you attach to over CDP, because HAR recording works differently in each case.

Before you start

Playwright plus a browser binary, for the capture step only. Install with pip install playwright and then playwright install chromium --no-shell. If a system Playwright already has browsers under ~/.cache/ms-playwright, you can reuse them.
requests or httpx for the replay step. Stdlib urllib also works.
No API keys are required. Any keys or tokens the client needs are the ones the HAR captured.
For the CDP path (har_capture_cdp.py) you need a reachable CDP endpoint. On Hermes, run /browser connect to print the active endpoint, or read BROWSER_CDP_URL or browser.cdp_url in config. Cloud backends expose it as cdpUrl or connectUrl.

In practice the Playwright install is the only heavy prerequisite, and it applies only to capture. Once a HAR exists, derivation and replay run on a bare Python install.

How to run

The scripts live under this skill's scripts/ directory and are invoked through the terminal tool. The part that trips people up is choosing the right capturer for the pathway Hermes used to reach the browser.

Browser pathway	How Hermes reaches it	Capturer
Local `browser_navigate` (default, agent-browser/Playwright)	launched locally	`har_capture.py`
Camofox (`CAMOFOX_URL` set)	local REST/CDP	`har_capture_cdp.py` if it exposes CDP, else drive it yourself
Browserbase / Browser-Use / Firecrawl (cloud)	CDP (`cdpUrl`)	`har_capture_cdp.py`
`/browser connect` / `BROWSER_CDP_URL`	CDP	`har_capture_cdp.py`

The rule of thumb: if Hermes launched the browser, use har_capture.py; if it connected to one over CDP, use har_capture_cdp.py. har_capture.py relies on Playwright's record_har_path, which only works on a locally-owned context. har_capture_cdp.py attaches with connect_over_cdp() and assembles the HAR from page.on("request"/"response") events, because record_har_path is unavailable on a connected browser.

For either path, har_to_client.py then filters the HAR to XHR, fetch, and JSON, groups by endpoint, and prints params, headers, bodies, and replay hints covering User-Agent, cookie, and auth.

Resolve paths against this skill's directory. The canonical loop:

# 1a. Capture, LOCAL browser (Hermes launched it)
python3 scripts/har_capture.py "https://SITE/" out.har \
  --action "fill:input[name=search]:my query" --action "sleep:3" --wait 2

# 1b. Capture, CDP browser (cloud backend or /browser connect)
#     get the endpoint from /browser connect or BROWSER_CDP_URL
python3 scripts/har_capture_cdp.py "ws://HOST/devtools/browser/..." out.har \
  --goto "https://SITE/" --action "fill:input[name=search]:my query" \
  --action "sleep:3" --wait 2

# 2. Derive — read the endpoints out of the HAR
python3 scripts/har_to_client.py out.har --host SITE --max-body 400

# 3. Replay — write a tiny client from the printed endpoint (see Procedure)

Quick reference

har_capture.py <url> <out.har> [--wait S] [--headed] [--action SPEC ...]
  action SPEC:  fill:SELECTOR:TEXT | press:SELECTOR:KEY | click:SELECTOR
                goto:URL | sleep:SECONDS      (run in order after page load)
  use when Hermes LAUNCHED the browser (local browser_navigate default)

har_capture_cdp.py <cdp_url> <out.har> [--goto URL] [--wait S] [--action SPEC ...]
  same action SPEC; attaches to an existing CDP browser and does NOT close it
  use for cloud backends (Browserbase/Browser-Use/Firecrawl) & /browser connect

har_to_client.py <in.har> [--host SUBSTR] [--include-static] [--max-body N]
  default: keeps only XHR/fetch/JSON; --host narrows to one domain
  prints per endpoint: query params, non-boring req headers, req body sample,
                       response status/content-type + body sample
  prints "### Replay hints": the browser User-Agent, cookie/auth presence

Procedure

Pick the capturer by pathway. Launched locally means har_capture.py; reached over CDP means har_capture_cdp.py. On Hermes, /browser connect tells you the CDP endpoint when a cloud or remote backend is active.
Find the interaction. Open the site with browser_navigate, or use a --headed capture, to see which selector to type into or click, and confirm that a JSON XHR fires in devtools or the network panel.
Capture the HAR through the terminal tool. Order the --action flags so they reach the request: fill the box, then sleep long enough for the debounced XHR, and always leave --wait at the end so late responses flush. Both capturers embed response bodies, so the derived client sees real payload shapes.
Derive with har_to_client.py --host. Read off the method, the URL or path template (numeric and UUID segments collapse to {id}), the query params, the request-body JSON, and the ### Replay hints block.
Write the client. Recreate the request exactly: same method, path, query params, body. Send the headers the site actually needs, at minimum the User-Agent from the replay hints. If the hints report cookies or an auth or token header, resend those too.
Test browserless. Run the client with the terminal tool and confirm it returns the same data the browser saw. This is the payoff: no browser in the loop.
(Optional) Wrap as a CLI. A small argparse script over the derived call, for example search.py "frank herbert".

Worked example, a Wikipedia search-title endpoint derived and replayed live:

import requests
r = requests.get(
    "https://en.wikipedia.org/w/rest.php/v1/search/title",
    params={"q": "frank herbert", "limit": 5},
    headers={"accept": "application/json",
             "User-Agent": "Mozilla/5.0 ... Chrome/131 Safari/537.36"},  # from HAR
    timeout=15,
)
for p in r.json()["pages"]:
    print(p["title"], "-", p.get("description"))

When not to use it

If the page is server-rendered, there is no XHR to derive. har_to_client.py prints "No API-looking entries" because the data arrived inside the HTML. In that case you scrape the HTML or find the interaction that does fetch JSON. The skill is also the wrong tool when the site requires a session you cannot legitimately carry forward: it captures and replays credentials, it does not obtain them.

Limits and gotchas

The default library User-Agent gets a 403. Many sites, including Wikipedia and Cloudflare-fronted APIs, reject python-requests/x.y. Always send the browser UA from the replay hints. This is the most common reason a derived client fails where the browser succeeded.
A failed --action aborts before the HAR flushes, so you get no file. If capture errors on a selector, the run produced nothing. Fix the selector, using --headed to watch, and rerun. Do not debug a missing HAR.
Debounced and typeahead XHRs need a real pause. Add --action "sleep:3" after fill; typing alone will not have fired the request by the time the HAR closes.
Auth and session endpoints need the captured Cookie or Authorization header, and those expire. The derived client is only as durable as the credential, so re-capture when it returns a 401. HARs contain live secrets: treat out.har as sensitive and delete it after deriving.
record_har_content="embed" makes big HARs. Use --max-body to cap what is printed; the file itself can be large for media-heavy pages.
Endpoints shift. Sites change private APIs without notice. Re-run the capture and derive loop when a client breaks rather than patching URLs by hand.
The wrong capturer gives you an empty or missing HAR. har_capture.py against a cloud or CDP backend records nothing, because it launches its own local browser instead of the one you meant. har_capture_cdp.py needs the endpoint; on Hermes get it from /browser connect or BROWSER_CDP_URL. Match the capturer to the pathway.
A Headless-Chrome UA is a weak tell. Local or agent-browser capture yields a HeadlessChrome/... User-Agent, and some sites sniff the "Headless" token. Cloud backends such as Browserbase and Browser-Use send a real desktop-Chrome UA, so a client derived from a cloud capture replays more reliably. If a headless-derived client returns a 403 where the browser did not, swap the "Headless" UA for a normal Chrome UA string before assuming the endpoint changed.
CDP capture does not close the browser. har_capture_cdp.py attaches to a browser it does not own and leaves it running, which is correct for cloud or remote sessions Hermes manages. Do not add a close; let the owning backend tear it down.

Verification

An end-to-end proof against a live site with no API key:

python3 scripts/har_capture.py "https://en.wikipedia.org/wiki/Main_Page" ~/.hermes/cache/scratch/wiki.har \
  --action "fill:input[name=search]:dune messiah" --action "sleep:3" --wait 2
python3 scripts/har_to_client.py ~/.hermes/cache/scratch/wiki.har --host wikipedia.org --max-body 200

Expect the derivation to print GET https://en.wikipedia.org/w/rest.php/v1/search/title with q and limit params and a JSON pages response, then replay it with the Procedure snippet and confirm matching titles come back over plain HTTP.

The skill sits in the Web Development category of the optional skills set, alongside the other browser and HTTP tooling that Hermes ships. It pairs naturally with the terminal tool, which is how every script here is invoked, and with the browser backends it supports: the local browser_navigate pathway, the cloud backends Browserbase, Browser-Use, and Firecrawl, and any endpoint reached through /browser connect.

HAR-Derived API Client: Turn Browser Traffic Into HTTP Calls