Har Derived API Client

HAR-Derived API Client

Drive a website once with a real browser while recording its network traffic to a HAR file, then distill that HAR into the site's private JSON API so you can call it directly with plain HTTP — far cheaper and faster than browser-controlling the page on every request. Credit: trick by Jared Longster, popularized by Dax (thdxr). This captures and replays; it does NOT bypass auth, solve CAPTCHAs, or defeat bot-detection — if the site needs a logged-in session, you carry its headers/cookies forward, you don't forge them.

The scripts are stdlib-plus-Playwright: capture needs Playwright, derivation is pure stdlib, replay needs only requests/httpx (or curl).

Covers every Hermes browser pathway: the default local browser_navigate backend, plus the cloud/remote backends (Browserbase, Browser-Use, Firecrawl) and any /browser connect CDP endpoint. There are two capture scripts — one for a browser you launch, one for a browser you attach to over CDP — because HAR recording works differently in each case (see How to Run).

When to Use

• "Build a CLI/client for <website>" — derive its API instead of scripting clicks.
• "This site has no public API but the page clearly fetches JSON."
• You're about to loop browser_navigate for the same query repeatedly — stop and derive the endpoint once.
• Reverse-engineering an autocomplete, search, feed, or checkout XHR.
• You captured a session on a cloud backend (Browserbase / Browser-Use / Firecrawl) or via /browser connect and want the API without re-renting the browser.

Prerequisites

• Playwright + a browser binary (capture step only):
  • pip install playwright then playwright install chromium
  • (If a system Playwright already has browsers under ~/.cache/ms-playwright, reuse it.)
• requests or httpx for the replay step (stdlib urllib also works).
• No API keys. Any keys/tokens the client needs are the ones the HAR captured.
• For the CDP path (har_capture_cdp.py): a reachable CDP endpoint. On Hermes, run /browser connect to print the active endpoint, or read BROWSER_CDP_URL / browser.cdp_url in config. Cloud backends expose it as cdpUrl/connectUrl.

How to Run

Scripts under this skill's scripts/, invoked through the terminal tool. Pick the capturer by pathway — this is the part that trips people up:

Rule of thumb: if Hermes launched the browser, use har_capture.py; if it connected to one over CDP, use har_capture_cdp.py. har_capture.py uses Playwright's record_har_path, which only works on a locally-owned context. har_capture_cdp.py attaches with connect_over_cdp() and assembles the HAR from page.on("request"/"response") events, because record_har_path is unavailable on a connected browser.

Then, for either path:

• har_to_client.py — filters the HAR to XHR/fetch/JSON, groups by endpoint, and prints params, headers, bodies, and replay hints (User-Agent / cookie / auth).

Resolve paths against this skill's directory. Canonical loop:

# 1a. Capture, LOCAL browser (Hermes launched it)
python3 scripts/har_capture.py "https://SITE/" out.har \
  --action "fill:input[name=search]:my query" --action "sleep:3" --wait 2

# 1b. Capture, CDP browser (cloud backend or /browser connect)
#     get the endpoint from /browser connect or BROWSER_CDP_URL
python3 scripts/har_capture_cdp.py "ws://HOST/devtools/browser/..." out.har \
  --goto "https://SITE/" --action "fill:input[name=search]:my query" \
  --action "sleep:3" --wait 2

# 2. Derive — read the endpoints out of the HAR
python3 scripts/har_to_client.py out.har --host SITE --max-body 400

# 3. Replay — write a tiny client from the printed endpoint (see Procedure)

Quick Reference

har_capture.py <url> <out.har> [--wait S] [--headed] [--action SPEC ...]
  action SPEC:  fill:SELECTOR:TEXT | press:SELECTOR:KEY | click:SELECTOR
                goto:URL | sleep:SECONDS      (run in order after page load)
  use when Hermes LAUNCHED the browser (local browser_navigate default)

har_capture_cdp.py <cdp_url> <out.har> [--goto URL] [--wait S] [--action SPEC ...]
  same action SPEC; attaches to an existing CDP browser and does NOT close it
  use for cloud backends (Browserbase/Browser-Use/Firecrawl) & /browser connect

har_to_client.py <in.har> [--host SUBSTR] [--include-static] [--max-body N]
  default: keeps only XHR/fetch/JSON; --host narrows to one domain
  prints per endpoint: query params, non-boring req headers, req body sample,
                       response status/content-type + body sample
  prints "### Replay hints": the browser User-Agent, cookie/auth presence

Procedure

0. Pick the capturer by pathway (see How to Run table). Launched-locally → har_capture.py; reached over CDP → har_capture_cdp.py. On Hermes, /browser connect tells you the CDP endpoint when a cloud/remote backend is active.
1. Find the interaction. Open the site with browser_navigate (or --headed capture) to see which selector to type into / click, and confirm a JSON XHR fires in devtools/network.
2. Capture the HAR via the terminal tool. Order --action to reach the request: fill the box, then sleep long enough for the debounced XHR, and always leave --wait at the end so late responses flush. Both capturers embed response bodies, so the derived client sees real payload shapes.
3. Derive with har_to_client.py --host <domain>. Read off: the method, the URL/path template (numeric/UUID segments collapse to {id}), query params, request-body JSON, and the ### Replay hints block.
4. Write the client. Recreate the request exactly — same method, path, query params, body. Send the headers the site actually needs: at minimum copy the User-Agent from the replay hints. If hints report cookies or an auth/token header, resend those too.
5. Test browserless. Run the client with the terminal tool and confirm it returns the same data the browser saw. This is the payoff: no browser in the loop.
6. (Optional) Wrap as a CLI — a small argparse script over the derived call, e.g. search.py "frank herbert".

Worked example (Wikipedia search-title, derived + replayed live):

import requests
r = requests.get(
    "https://en.wikipedia.org/w/rest.php/v1/search/title",
    params={"q": "frank herbert", "limit": 5},
    headers={"accept": "application/json",
             "User-Agent": "Mozilla/5.0 ... Chrome/131 Safari/537.36"},  # from HAR
    timeout=15,
)
for p in r.json()["pages"]:
    print(p["title"], "-", p.get("description"))