Blocked Page Recovery: Fetch Paywalled and WAF'd Pages via Fallbacks

When a page refuses to load, whether it is a 403, a 429, a Cloudflare interstitial, a paywall, or a bot-detection screen, you have options beyond retrying the same URL. This guide walks through a proven ladder of fallbacks, from the Wayback Machine to a real browser, so you can recover a usable copy without burning time or getting stuck in a loop. It is written for researchers, journalists, and developers who need the content behind a blocked page and want to preserve its provenance.

What it does

This skill gives you a systematic way to recover a blocked page by trying cheaper, more reliable sources first. Instead of hammering the original URL, you work down a ladder: first check if the Wayback Machine has a snapshot, then try archive.today, then Jina Reader if you have an API key, then pivot to the site's own API or feeds, and only as a last resort use a real browser. Each route returns a copy with clear provenance, so you know whether you are looking at a live page or an archived snapshot. The bundled script automates the whole process, validating each response and stopping at the first genuine hit.

Before you start

This skill is bundled with Hermes Agent, so it is installed by default. The script lives at skills/research/blocked-page-recovery and runs on Linux, macOS, and Windows. You need Python 3 to run the script. For the Jina Reader route, you must set the JINA_API_KEY environment variable; without it, that route is skipped entirely. No other API keys are required for the archive routes.

The ladder

The core idea is to try the cheapest, most reliable route first and escalate only when necessary. Here is the ladder as defined by the skill:

1. Wayback Machine  — archive.org "available" API  (snapshot + timestamp)
2. archive.today    — domain rotation: archive.ph → .md → .li → .is
3. Jina Reader      — only if JINA_API_KEY is set  (live server-side render)
4. API-first pivot  — look for /api/, /graphql, .json, or RSS on the same host
5. Real browser     — browser tool as the last, most expensive resort

Run it in one shot with the bundled script:

python3 scripts/recover_page.py "https://example.com/blocked-article" --json

The script tries each route in order, validates every body (see "Fake successes" below), and prints the first genuine hit with its provenance.

Provenance discipline (non-negotiable)

Every recovered copy carries a provenance you MUST preserve when citing:

Route	Provenance	How to cite
Wayback / archive.today	snapshot	Cite WITH the snapshot date: "as archived 2026-08-06". Never present a snapshot as the live page, it may be stale.
Jina Reader	live	Server-side re-render of the live page; cite normally.
Live fetch / browser	live	Cite normally.

If the user needs current data (prices, availability, breaking news), a snapshot is context, not an answer, say so explicitly and note its age.

Manual routes

1. Wayback Machine (best provenance, try first)

The Wayback Machine offers the best provenance because every snapshot carries a timestamp. Start with the available API to find the closest snapshot:

# Discovery: returns closest snapshot URL + timestamp as JSON
curl -sL "https://archive.org/wayback/available?url={URL}"
# Then fetch archived_snapshots.closest.url

For enumerating many snapshots (or recovering deleted pages), the CDX index:

curl -sL "https://web.archive.org/cdx/search/cdx?url={URL}&output=json&limit=10"

CDX intermittently returns 503 under load, if it does, fall back to the available API; don't retry-hammer it.

Works for: any publicly crawled URL. Fails for: robots-blocked sites, never-crawled URLs, JS-only SPAs (snapshots don't render).

2. archive.today (paywalls, deleted content)

archive.today holds user-submitted archives, which often include paywalled news articles that the Wayback Machine lacks. It rate-limits aggressively (429) and rotates domains, so iterate through the known domains:

for d in archive.ph archive.md archive.li archive.is; do
  curl -sL --max-time 20 "https://$d/newest/{URL}" -o /tmp/page.html \
    -w "%{http_code}" && break
done

Validate the body, not the status code, a 429 still ships several KB of rate-limit HTML that looks like a success to a size check alone.

3. Jina Reader (requires JINA_API_KEY)

r.jina.ai re-renders the live page in a real browser server-side and returns markdown. Anonymous access is dead (401 → Turnstile); a key is required:

curl -s -H "Authorization: Bearer $JINA_API_KEY" "https://r.jina.ai/{URL}"

Handles JS SPAs that archives can't. Skip this route entirely when the env var is unset.

4. API-first pivot

WAFs protect the HTML surface far more aggressively than the data endpoints behind it. After 2-3 blocked attempts on a site, stop fighting the HTML and look for:

• /api/..., /graphql, or .json variants of the page URL
• An RSS/Atom feed (/feed, /rss, `` in any copy you did recover)
• A sitemap (/sitemap.xml) revealing canonical URLs that may not be gated

Fake successes, routes that LIE

These return HTTP 200 with a plausible body that is NOT the page. The script rejects them automatically; reject them manually too:

• Google Cache is dead (since mid-2024). webcache.googleusercontent.com returns 200 + tens of KB, but it's a Google Search interstitial with a JS redirect, not a cache. Never use it.
• AMP caches (*.cdn.ampproject.org) mostly return a ~300-byte Redirecting meta-refresh stub pointing back at the original (blocked) URL. Treating that as success creates a fetch loop.
• Rate-limit bodies: archive.today 429 pages are multi-KB HTML. Check for the target's actual content (title words, expected strings), not just size.

Detection heuristics the script applies: body under a per-route byte floor; meta-refresh/JS-redirect stubs whose target is the original host; interstitial titles ("Just a moment", "Redirecting", "Google Search", "Attention Required").

Proxy relays: don't

Generic "web proxy" relays are man-in-the-middle by construction. Never send cookies or Authorization headers through one, and don't use them for anything the user will rely on, provenance is unverifiable. Prefer archives, which at least timestamp their copies.

When not to use it

If the page is not actually blocked, just slow or temporarily down, retrying the original URL is simpler and faster than any fallback. Also, if the user needs live, current data, a snapshot is not an acceptable substitute; say so and note the snapshot's age. For sites that require login, none of these routes will help unless the content is publicly archived.

Limits and gotchas

• The Wayback Machine fails for robots-blocked sites, never-crawled URLs, and JS-only SPAs.
• archive.today rate-limits aggressively; expect 429s and rotate domains.
• Jina Reader requires a valid JINA_API_KEY; without it, skip the route.
• CDX can return 503 under load; fall back to the available API instead of retrying.
• Always validate the response body, not just the status code, because rate-limit pages and redirect stubs can look like successes.

What pairs with this

The related skill grounded-citations complements this one: after you recover a page, use it to ensure every claim you make is backed by a proper citation, including the snapshot date when you rely on an archived copy.