Blocked Page Recovery: Fetch 403, 429, and Paywalled Pages

When a fetch returns a 403 or 429, a Cloudflare "Just a moment..." interstitial, a paywall, or a bot-detection wall, the page is not gone. A third party often holds a copy. This skill gives Hermes Agent a fixed order of routes to try, cheapest first, so the agent stops retrying the same URL and starts looking elsewhere. It is aimed at research and scraping workflows where a single blocked source would otherwise stall the whole task.

What it does

The skill encodes a recovery ladder. Each rung is a different way to get the page content, ordered by cost and by how trustworthy the result is. The agent works down the ladder rather than looping on the original URL.

1. Wayback Machine  — archive.org "available" API  (snapshot + timestamp)
2. archive.today    — domain rotation: archive.ph → .md → .li → .is
3. Jina Reader      — only if JINA_API_KEY is set  (live server-side render)
4. API-first pivot  — look for /api/, /graphql, .json, or RSS on the same host
5. Real browser     — browser tool as the last, most expensive resort

A bundled script runs the whole ladder in one shot:

python3 scripts/recover_page.py "https://example.com/blocked-article" --json

The script tries each route in order, validates every body it gets back (see the fake-success section below), and prints the first genuine hit together with its provenance.

Before you start

The skill is bundled with Hermes Agent and installed by default, so there is no separate install step. It lives at skills/web/blocked-page-recovery, version 1.0.0, authored by Hermes Agent under the MIT licence, and it runs on linux, macos, and windows.

Two practical prerequisites follow from the routes themselves. The Jina Reader rung only works if JINA_API_KEY is present in the environment; when it is unset, that route is skipped entirely. The archive.today rung writes to ~/.hermes/cache/scratch/page.html, so that path needs to be writable.

Provenance discipline

Every recovered copy carries a provenance label, and you must preserve it when citing. This is the part practitioners most often get wrong.

Route	Provenance	How to cite
Wayback / archive.today	snapshot	Cite WITH the snapshot date: "as archived 2026-08-06". Never present a snapshot as the live page, it may be stale.
Jina Reader	live	Server-side re-render of the live page; cite normally.
Live fetch / browser	live	Cite normally.

If the user needs current data, such as prices, availability, or breaking news, a snapshot is context rather than an answer. Say so explicitly and state its age.

Manual routes

1. Wayback Machine

Wayback has the best provenance, so it comes first. Discovery uses the available API, which returns the closest snapshot URL and timestamp as JSON:

# Discovery: returns closest snapshot URL + timestamp as JSON
curl -sL "https://archive.org/wayback/available?url={URL}"
# Then fetch archived_snapshots.closest.url

When you need to enumerate many snapshots, or recover a deleted page, use the CDX index instead:

curl -sL "https://web.archive.org/cdx/search/cdx?url={URL}&output=json&limit=10"

CDX intermittently returns 503 under load. When that happens, fall back to the available API rather than retrying CDX in a tight loop.

This route works for any publicly crawled URL. It fails for robots-blocked sites, URLs that were never crawled, and JS-only SPAs, because snapshots do not render JavaScript.

2. archive.today

archive.today holds user-submitted archives and often has paywalled news articles that Wayback lacks. It rate-limits aggressively with 429 responses and rotates domains, so iterate across them:

for d in archive.ph archive.md archive.li archive.is; do
  curl -sL --max-time 20 "https://$d/newest/{URL}" -o ~/.hermes/cache/scratch/page.html \
    -w "%{http_code}" && break
done

Validate the body, not the status code. A 429 still ships several KB of rate-limit HTML, which looks like a success to a size check alone.

3. Jina Reader

r.jina.ai re-renders the live page in a real browser server-side and returns markdown. Anonymous access no longer works: it returns 401 and a Turnstile challenge, so a key is required:

curl -s -H "Authorization: Bearer $JINA_API_KEY" "https://r.jina.ai/{URL}"

This route handles JS SPAs that archives cannot. Skip it entirely when the environment variable is unset.

4. API-first pivot

WAFs guard the HTML surface far more aggressively than the data endpoints behind it. After two or three blocked attempts on a site, stop fighting the HTML and look for:

• /api/..., /graphql, or .json variants of the page URL
• An RSS/Atom feed (/feed, /rss, `` in any copy you did recover)
• A sitemap (/sitemap.xml) that reveals canonical URLs which may not be gated

Fake successes

Some routes return HTTP 200 with a plausible body that is not the page. The script rejects them automatically, and you should reject them manually too.

• Google Cache is dead, and has been since mid-2024. webcache.googleusercontent.com returns 200 plus tens of KB, but the body is a Google Search interstitial with a JS redirect, not a cache. Never use it.
• AMP caches (*.cdn.ampproject.org) mostly return a roughly 300-byte Redirecting meta-refresh stub that points back at the original, blocked URL. Treating that as success creates a fetch loop.
• Rate-limit bodies: archive.today 429 pages are multi-KB HTML. Check for the target's actual content, such as title words or expected strings, not just size.

The script applies these detection heuristics: a body under a per-route byte floor; meta-refresh or JS-redirect stubs whose target is the original host; and interstitial titles such as "Just a moment", "Redirecting", "Google Search", and "Attention Required".

When not to use it

Do not route through generic web proxy relays. They are man-in-the-middle by construction. Never send cookies or Authorization headers through one, and do not use one for anything the user will rely on, because provenance cannot be verified. Archives are preferable: they at least timestamp their copies.

Limits and gotchas

Each rung has a failure mode, and knowing them saves time. Wayback cannot help with robots-blocked sites, never-crawled URLs, or JS-only SPAs. archive.today rate-limits hard and needs body validation. Jina Reader needs a key. The API-first pivot depends on the site exposing data endpoints or feeds. The browser tool is the most expensive option and belongs last.

Provenance is the other trap. A snapshot may be stale, so citing it as the live page is a factual error, not a formatting slip. When currency matters, say the copy is archived and give its date.

What pairs with it

For citation work, pair this skill with grounded-citations, which is listed as a related skill. Recovery gives you a source; grounded-citations keeps the provenance attached when you write it up.