HAR-Derived API Client: Capture XHR and Replay It

This skill turns a single browser session into a reusable HTTP client. You drive a site once with Playwright while its network traffic is recorded to a HAR file, then distill that HAR into the private JSON endpoints the page was calling. From there you replay those endpoints with plain HTTP, which is far cheaper and faster than browser-controlling the page on every request. Reach for it when a site has no public API but its front end clearly fetches JSON, or when you are about to loop browser navigations for the same query.

The technique is credited to Jared Longster, popularized by Dax (thdxr). It captures and replays. It does not bypass auth, solve CAPTCHAs, or defeat bot-detection. If the site needs a logged-in session, you carry its headers and cookies forward, you do not forge them.

What it does

The skill ships three scripts. Two capture, one derives. The capture scripts drive a browser through a sequence of actions you specify (fill a field, press a key, click, navigate, sleep) and record the resulting network traffic to a HAR file. The derivation script then reads that HAR, keeps only the XHR, fetch, and JSON entries, groups them by endpoint, and prints what you need to rebuild each call: query parameters, the request headers that matter, a sample request body, the response status and content type, and a sample of the response body. It also prints a ### Replay hints block covering the browser User-Agent and whether cookies or an auth header were present.

There are two capture scripts because HAR recording works differently depending on how the browser was reached. har_capture.py uses Playwright's record_har_path, which only works on a locally-owned context. har_capture_cdp.py attaches with connect_over_cdp() and assembles the HAR from page.on("request"/"response") events, because record_har_path is unavailable on a connected browser.

The scripts are stdlib-plus-Playwright: capture needs Playwright, derivation is pure stdlib, replay needs only requests/httpx (or curl).

The skill covers every Hermes browser pathway: the default local Chromium that browser_exec drives, plus the cloud/remote backends (Browserbase, Browser-Use, Firecrawl) and any /browser connect CDP endpoint.

Before you start

Playwright plus a browser binary, for the capture step only. Install with pip install playwright then playwright install chromium --no-shell. If a system Playwright already has browsers under ~/.cache/ms-playwright, you can reuse it.
requests or httpx for the replay step. Stdlib urllib also works.
No API keys. Any keys or tokens the client needs are the ones the HAR captured.
For the CDP path (har_capture_cdp.py), a reachable CDP endpoint. On Hermes, run /browser connect to print the active endpoint, or read BROWSER_CDP_URL / browser.cdp_url in config. Cloud backends expose it as cdpUrl/connectUrl.

The skill is optional and installed on demand. It lives at optional-skills/web-development/har-derived-api-client, is version 0.1.0, authored by Hermes Agent, MIT licensed, and runs on linux, macos, and windows.

In practice the prerequisites mean two things. First, the capture half is the only part that needs a real browser, so you can run it on a machine that has Playwright set up and do the replay anywhere. Second, the CDP path only matters if you are working against a browser Hermes did not launch itself. If you are on the default local setup, you can ignore it entirely.

How to run

The scripts live under this skill's scripts/ directory and are invoked through the terminal tool. Resolve paths against the skill directory. The part that trips people up is picking the right capturer for the pathway:

Browser pathway	How Hermes reaches it	Capturer
Local Chromium (default `browser_exec` browser)	launched locally	`har_capture.py`
Browserbase / Browser-Use / Firecrawl (cloud)	CDP (`cdpUrl`)	`har_capture_cdp.py`
`/browser connect` / `BROWSER_CDP_URL`	CDP	`har_capture_cdp.py`

Rule of thumb: if Hermes launched the browser, use har_capture.py; if it connected to one over CDP, use har_capture_cdp.py.

The canonical loop, for either path:

# 1a. Capture, LOCAL browser (Hermes launched it)
python3 scripts/har_capture.py "https://SITE/" out.har \
  --action "fill:input[name=search]:my query" --action "sleep:3" --wait 2

# 1b. Capture, CDP browser (cloud backend or /browser connect)
#     get the endpoint from /browser connect or BROWSER_CDP_URL
python3 scripts/har_capture_cdp.py "ws://HOST/devtools/browser/..." out.har \
  --goto "https://SITE/" --action "fill:input[name=search]:my query" \
  --action "sleep:3" --wait 2

# 2. Derive — read the endpoints out of the HAR
python3 scripts/har_to_client.py out.har --host SITE --max-body 400

# 3. Replay — write a tiny client from the printed endpoint (see Procedure)

Quick reference

har_capture.py <url> <out.har> [--wait S] [--headed] [--action SPEC ...]
  action SPEC:  fill:SELECTOR:TEXT | press:SELECTOR:KEY | click:SELECTOR
                goto:URL | sleep:SECONDS      (run in order after page load)
  use when Hermes LAUNCHED the browser (local browser_exec default)

har_capture_cdp.py <cdp_url> <out.har> [--goto URL] [--wait S] [--action SPEC ...]
  same action SPEC; attaches to an existing CDP browser and does NOT close it
  use for cloud backends (Browserbase/Browser-Use/Firecrawl) & /browser connect

har_to_client.py <in.har> [--host SUBSTR] [--include-static] [--max-body N]
  default: keeps only XHR/fetch/JSON; --host narrows to one domain
  prints per endpoint: query params, non-boring req headers, req body sample,
                       response status/content-type + body sample
  prints "### Replay hints": the browser User-Agent, cookie/auth presence

Procedure

Pick the capturer by pathway. Launched locally means har_capture.py; reached over CDP means har_capture_cdp.py. On Hermes, /browser connect tells you the CDP endpoint when a cloud or remote backend is active.
Find the interaction. Open the site with browser_exec (new_tab(url) + wait_for_load(), or a --headed capture) to see which selector to type into or click via js(), and confirm a JSON XHR fires: js("performance.getEntriesByType('resource').filter(e => ['xmlhttprequest','fetch'].includes(e.initiatorType)).map(e => e.name)").
Capture the HAR via the terminal tool. Order the --action flags to reach the request: fill the box, then sleep long enough for the debounced XHR, and always leave --wait at the end so late responses flush. Both capturers embed response bodies, so the derived client sees real payload shapes.
Derive with har_to_client.py --host. Read off the method, the URL or path template (numeric and UUID segments collapse to {id}), the query params, the request-body JSON, and the ### Replay hints block.
Write the client. Recreate the request exactly: same method, path, query params, body. Send the headers the site actually needs, at minimum copying the User-Agent from the replay hints. If the hints report cookies or an auth/token header, resend those too.
Test browserless. Run the client with the terminal tool and confirm it returns the same data the browser saw. This is the payoff: no browser in the loop.
(Optional) Wrap as a CLI. A small argparse script over the derived call, for example search.py "frank herbert".

A worked example, derived and replayed live against Wikipedia's search-title endpoint:

import requests
r = requests.get(
    "https://en.wikipedia.org/w/rest.php/v1/search/title",
    params={"q": "frank herbert", "limit": 5},
    headers={"accept": "application/json",
             "User-Agent": "Mozilla/5.0 ... Chrome/131 Safari/537.36"},  # from HAR
    timeout=15,
)
for p in r.json()["pages"]:
    print(p["title"], "-", p.get("description"))

Verification

An end-to-end proof against a live site with no API key:

python3 scripts/har_capture.py "https://en.wikipedia.org/wiki/Main_Page" ~/.hermes/cache/scratch/wiki.har \
  --action "fill:input[name=search]:dune messiah" --action "sleep:3" --wait 2
python3 scripts/har_to_client.py ~/.hermes/cache/scratch/wiki.har --host wikipedia.org --max-body 200

Expect the derivation to print GET https://en.wikipedia.org/w/rest.php/v1/search/title with q and limit params and a JSON pages response. Then replay it with the Procedure snippet and confirm matching titles come back over plain HTTP.

When not to use it

If the page is server-rendered, there is no XHR to derive. har_to_client.py prints "No API-looking entries" because the data arrived in the HTML. In that case scrape the HTML, or find the interaction that does fetch JSON. The skill is also the wrong tool when you need to defeat bot-detection or forge a session: it captures and replays only, so if the site requires a logged-in session you carry its headers and cookies forward rather than manufacturing them.

Limits and gotchas

Default library User-Agent gets 403. Many sites (Wikipedia, Cloudflare-fronted APIs) reject python-requests/x.y. Always send the browser UA from the replay hints. This is the number one reason a derived client fails when the browser succeeded.
A failed --action aborts before the HAR flushes. You get no file. If capture errors on a selector, the run produced nothing. Fix the selector (use --headed to watch) and rerun. Do not debug a missing HAR.
Debounced or typeahead XHRs need a real pause. Add --action "sleep:3" after fill; typing alone will not have fired the request when the HAR closes.
Auth and session endpoints need the captured Cookie/Authorization header, and those expire. The derived client is only as durable as the credential; re-capture when it 401s. HARs contain live secrets, so treat out.har as sensitive and delete it after deriving.
record_har_content="embed" makes big HARs. Use --max-body to cap what is printed; the file itself can be large for media-heavy pages.
Endpoints shift. Sites change private APIs without notice. Re-run the capture-to-derive loop when a client breaks rather than patching URLs by hand.
Wrong capturer means an empty or missing HAR. har_capture.py on a cloud or CDP backend records nothing, because it launches its own local browser instead of the one you meant. har_capture_cdp.py needs the endpoint; on Hermes get it from /browser connect or BROWSER_CDP_URL. Match the capturer to the pathway.
Headless-Chrome UA is a weak tell. Local headless capture yields a HeadlessChrome/... User-Agent, and some sites sniff the "Headless" token. Cloud backends (Browserbase, Browser-Use) send a real desktop-Chrome UA, so a client derived from a cloud capture replays more reliably. If a headless-derived client 403s where the browser did not, swap the "Headless" UA for a normal Chrome UA string before assuming the endpoint changed.
CDP capture does not close the browser. har_capture_cdp.py attaches to a browser it does not own and leaves it running, which is correct for cloud or remote sessions Hermes manages. Do not add a close; let the owning backend tear it down.

What pairs with it

This skill sits inside the Web Development category alongside the other optional skills. It depends on the browser tooling you already use through Hermes: browser_exec for finding the interaction, and /browser connect for the CDP endpoint when a cloud or remote backend is active. The natural next step after deriving an endpoint is wrapping it as a small CLI, which turns a one-off capture into a tool you keep.

HAR-Derived API Client: Record XHR, Build an HTTP Client