Derive an HTTP API Client from a HAR Recording with Hermes Agent

When a website has no public API but its pages clearly fetch JSON, you can stop scripting clicks and start calling the underlying endpoints directly. This skill records a browser session into a HAR file, distills that file into the site's private JSON API, and gives you the exact headers and parameters needed to replay requests with plain HTTP. It is a capture-and-replay technique, not a way to bypass authentication, solve CAPTCHAs, or defeat bot detection. If the site needs a logged-in session, you carry its cookies and headers forward; you do not forge them.

What it does

You drive a real browser once, while the skill records every network request into a HAR file. Then a derivation script filters that HAR down to the XHR, fetch, and JSON calls, groups them by endpoint, and prints the method, URL template, query parameters, request body, and response shape. Finally, you write a small client that replays those requests directly, without launching a browser. The result is far cheaper and faster than browser-controlling the page on every request, especially when you would otherwise loop browser_navigate for the same query repeatedly.

The skill covers every Hermes browser pathway: the default local browser_navigate backend, the cloud and remote backends (Browserbase, Browser-Use, Firecrawl), and any /browser connect CDP endpoint. There are two capture scripts, one for a browser you launch and one for a browser you attach to over CDP, because HAR recording works differently in each case.

Before you start

You need Playwright and a browser binary for the capture step only. Install them with:

pip install playwright
playwright install chromium

If a system Playwright already has browsers under ~/.cache/ms-playwright, you can reuse that instead of installing a fresh copy.

For the replay step, you need requests or httpx; the standard library urllib also works. No API keys are required. Any keys or tokens the client needs are the ones the HAR captured.

For the CDP path (har_capture_cdp.py), you need a reachable CDP endpoint. On Hermes, run /browser connect to print the active endpoint, or read BROWSER_CDP_URL or browser.cdp_url in config. Cloud backends expose it as cdpUrl or connectUrl.

How to Run

Scripts live under this skill's scripts/ directory and are invoked through the terminal tool. The first decision is picking the capturer by pathway, which is the part that trips people up.

Browser pathway	How Hermes reaches it	Capturer
Local `browser_navigate` (default, agent-browser/Playwright)	launched locally	`har_capture.py`
Camofox (`CAMOFOX_URL` set)	local REST/CDP	`har_capture_cdp.py` if it exposes CDP, else drive it yourself
Browserbase / Browser-Use / Firecrawl (cloud)	CDP (`cdpUrl`)	`har_capture_cdp.py`
`/browser connect` / `BROWSER_CDP_URL`	CDP	`har_capture_cdp.py`

The rule of thumb: if Hermes launched the browser, use har_capture.py; if it connected to one over CDP, use har_capture_cdp.py. har_capture.py uses Playwright's record_har_path, which only works on a locally-owned context. har_capture_cdp.py attaches with connect_over_cdp() and assembles the HAR from page.on("request"/"response") events, because record_har_path is unavailable on a connected browser.

Then, for either path, har_to_client.py filters the HAR to XHR/fetch/JSON, groups by endpoint, and prints params, headers, bodies, and replay hints (User-Agent / cookie / auth).

Resolve paths against this skill's directory. The canonical loop looks like this:

# 1a. Capture, LOCAL browser (Hermes launched it)
python3 scripts/har_capture.py "https://SITE/" out.har \
  --action "fill:input[name=search]:my query" --action "sleep:3" --wait 2

# 1b. Capture, CDP browser (cloud backend or /browser connect)
#     get the endpoint from /browser connect or BROWSER_CDP_URL
python3 scripts/har_capture_cdp.py "ws://HOST/devtools/browser/..." out.har \
  --goto "https://SITE/" --action "fill:input[name=search]:my query" \
  --action "sleep:3" --wait 2

# 2. Derive — read the endpoints out of the HAR
python3 scripts/har_to_client.py out.har --host SITE --max-body 400

# 3. Replay — write a tiny client from the printed endpoint (see Procedure)

Quick Reference

har_capture.py <url> <out.har> [--wait S] [--headed] [--action SPEC ...]
  action SPEC:  fill:SELECTOR:TEXT | press:SELECTOR:KEY | click:SELECTOR
                goto:URL | sleep:SECONDS      (run in order after page load)
  use when Hermes LAUNCHED the browser (local browser_navigate default)

har_capture_cdp.py <cdp_url> <out.har> [--goto URL] [--wait S] [--action SPEC ...]
  same action SPEC; attaches to an existing CDP browser and does NOT close it
  use for cloud backends (Browserbase/Browser-Use/Firecrawl) & /browser connect

har_to_client.py <in.har> [--host SUBSTR] [--include-static] [--max-body N]
  default: keeps only XHR/fetch/JSON; --host narrows to one domain
  prints per endpoint: query params, non-boring req headers, req body sample,
                       response status/content-type + body sample
  prints "### Replay hints": the browser User-Agent, cookie/auth presence

Procedure

Pick the capturer by pathway (see How to Run table). Launched-locally → har_capture.py; reached over CDP → har_capture_cdp.py. On Hermes, /browser connect tells you the CDP endpoint when a cloud/remote backend is active.
Find the interaction. Open the site with browser_navigate (or --headed capture) to see which selector to type into or click, and confirm a JSON XHR fires in devtools/network.
Capture the HAR via the terminal tool. Order --action to reach the request: fill the box, then sleep long enough for the debounced XHR, and always leave --wait at the end so late responses flush. Both capturers embed response bodies, so the derived client sees real payload shapes.
Derive with har_to_client.py --host . Read off: the method, the URL/path template (numeric/UUID segments collapse to {id}), query params, request-body JSON, and the ### Replay hints block.
Write the client. Recreate the request exactly, same method, path, query params, body. Send the headers the site actually needs: at minimum copy the User-Agent from the replay hints. If hints report cookies or an auth/token header, resend those too.
Test browserless. Run the client with the terminal tool and confirm it returns the same data the browser saw. This is the payoff: no browser in the loop.
(Optional) Wrap as a CLI, a small argparse script over the derived call, e.g. search.py "frank herbert".

Worked example (Wikipedia search-title, derived + replayed live):

import requests
r = requests.get(
    "https://en.wikipedia.org/w/rest.php/v1/search/title",
    params={"q": "frank herbert", "limit": 5},
    headers={"accept": "application/json",
             "User-Agent": "Mozilla/5.0 ... Chrome/131 Safari/537.36"},  # from HAR
    timeout=15,
)
for p in r.json()["pages"]:
    print(p["title"], "-", p.get("description"))

Pitfalls

Default library User-Agent gets 403. Many sites (Wikipedia, Cloudflare-fronted APIs) reject python-requests/x.y. Always send the browser UA from the replay hints. This is the #1 reason a derived client fails when the browser succeeded.
A failed --action aborts before the HAR flushes, you get no file. If capture errors on a selector, the run produced nothing; fix the selector (use --headed to watch) and rerun. Don't debug a missing HAR.
Server-rendered pages have no XHR to derive, har_to_client.py prints "No API-looking entries". The data came in the HTML; scrape it or find the interaction that does fetch JSON.
Debounced/typeahead XHRs need a real pause. Add --action "sleep:3" after fill; typing alone won't have fired the request when the HAR closes.
Auth/session endpoints need the captured Cookie/Authorization header, and those expire. The derived client is only as durable as the credential; re-capture when it 401s. HARs contain live secrets, treat out.har as sensitive and delete it after deriving.
record_har_content="embed" makes big HARs. Use --max-body to cap what's printed; the file itself can be large for media-heavy pages.
Endpoints shift. Sites change private APIs without notice. Re-run the capture→derive loop when a client breaks rather than patching URLs by hand.
Wrong capturer = empty/no HAR. har_capture.py on a cloud/CDP backend records nothing (it launches its own local browser instead of the one you meant). har_capture_cdp.py needs the endpoint; on Hermes get it from /browser connect or BROWSER_CDP_URL. Match the capturer to the pathway (How to Run table).
Headless-Chrome UA is a weak tell. Local/agent-browser capture yields a HeadlessChrome/... User-Agent; some sites sniff the "Headless" token. Cloud backends (Browserbase/Browser-Use) send a real desktop-Chrome UA, so a client derived from a cloud capture replays more reliably. If a headless-derived client 403s where the browser didn't, swap the "Headless" UA for a normal Chrome UA string before assuming the endpoint changed.
CDP capture doesn't close the browser. har_capture_cdp.py attaches to a browser it doesn't own and leaves it running, correct for cloud/remote sessions Hermes manages. Don't add a close; let the owning backend tear it down.

Verification

End-to-end proof against a live site with no API key:

python3 scripts/har_capture.py "https://en.wikipedia.org/wiki/Main_Page" /tmp/wiki.har \
  --action "fill:input[name=search]:dune messiah" --action "sleep:3" --wait 2
python3 scripts/har_to_client.py /tmp/wiki.har --host wikipedia.org --max-body 200

Expect the derivation to print GET https://en.wikipedia.org/w/rest.php/v1/search/title with q and limit params and a JSON pages response, then replay it with the Procedure snippet and confirm matching titles come back over plain HTTP.

When not to use it

This skill is not for sites that render everything server-side. If the page's data arrives in the HTML rather than through XHR, there is nothing to derive; har_to_client.py will report "No API-looking entries". In that case, scrape the HTML directly or find a different interaction that does fetch JSON. Also avoid this approach when you need to bypass authentication or bot detection; the skill explicitly does not do that. If the site requires a logged-in session, you must carry the captured credentials forward, and those expire, so the derived client is only as durable as the credential.

Limits and gotchas

The main limits are the ones listed in Pitfalls: the default library User-Agent often gets 403, so you must copy the browser UA; a failed action aborts before the HAR flushes, leaving no file; debounced XHRs need a real pause; auth headers expire; and endpoints shift without notice. The HAR file itself contains live secrets, so treat it as sensitive and delete it after deriving. Also, har_capture_cdp.py does not close the browser it attaches to, which is correct for cloud sessions but means you should not add a close call.

What pairs with this

This skill fits naturally with the rest of the Hermes web development toolkit. After deriving a client, you can wrap it as a CLI script and call it from other skills or workflows. The capture step relies on the browser backends, so familiarity with browser_navigate and /browser connect helps. For sites that do render server-side, pair this with a scraping skill instead. The skill's upstream tags point to Browser, HAR, API, Reverse-Engineering, and Playwright, so anything in those categories complements it.