Hermes OSINT: Public Records Investigative Framework

This framework connects public records from government contracts, corporate filings, lobbying disclosures, sanctions lists, offshore leaks, property records, court records, web archives, knowledge bases, and global news. It resolves entities across heterogeneous sources, builds cross-links with explicit confidence, runs statistical timing tests, and produces structured evidence chains.

When to Use This Framework

Use this skill for investigations that follow the money: government contracts, lobbying, and legislation. It supports corporate due diligence by revealing who controls a company, where it is incorporated, who serves on its boards, and what filings it has made. It screens entities against the OFAC SDN sanctions list and ICIJ Offshore Leaks. It investigates pay-to-play connections by matching contractors with offshore ties or lobbying clients who win awards. It retrieves property ownership records from NYC (deeds and mortgages by name or address; for other counties, point users at the relevant recorder). It searches federal and state court opinions and PACER dockets. It resolves entities across sources where naming varies (LLC suffixes, abbreviations). It constructs evidence chains with explicit confidence levels. It gathers what has been said about an entity through international news (GDELT), Wikipedia narratives, and Wayback Machine captures to recover dead URLs.

Do NOT use this skill for general web research (use web_search and web_extract), domain or infrastructure OSINT (use domain-intel), academic literature (use arxiv), or social-media profile discovery (use sherlock optionally). US federal campaign finance is not covered; point users at https://www.fec.gov/data/.

Prerequisites

• Python 3 installed (stdlib only, no additional packages).
• A terminal tool available to run scripts.
• The SKILL_DIR environment variable or knowledge of the skill directory path.
• For OpenCorporates: an optional free token (OPENCORPORATES_API_TOKEN or --token).
• For Wikipedia: recommended to set HERMES_OSINT_UA=your-app/1.0 (your@email).
• For ICIJ Offshore Leaks: sufficient disk space for the ~70 MB bulk CSV (cached 30 days).
• For timing_analysis.py: optional --seed N for reproducibility.

Planning the Investigation

Start by reading the data-source wiki entries to understand which sources apply to your investigation. List the available source files:

ls SKILL_DIR/references/sources/

# Federal financial / regulatory
cat SKILL_DIR/references/sources/sec-edgar.md       # corporate filings
cat SKILL_DIR/references/sources/usaspending.md     # federal contracts
cat SKILL_DIR/references/sources/senate-ld.md       # lobbying
cat SKILL_DIR/references/sources/ofac-sdn.md        # sanctions
cat SKILL_DIR/references/sources/icij-offshore.md   # offshore leaks

# Identity / property / litigation / archives / news
cat SKILL_DIR/references/sources/nyc-acris.md       # NYC property records
cat SKILL_DIR/references/sources/opencorporates.md  # global corporate registry
cat SKILL_DIR/references/sources/courtlistener.md   # court records (federal + state)
cat SKILL_DIR/references/sources/wayback.md         # Wayback Machine archives
cat SKILL_DIR/references/sources/wikipedia.md       # Wikipedia + Wikidata
cat SKILL_DIR/references/sources/gdelt.md           # global news monitoring

Each source file follows a 9-section template: summary, access, schema, coverage, cross-reference keys, data quality, acquisition, legal, and references. Read the cross-reference potential section first to pick the right pair of sources.

Acquiring Data

All scripts are in SKILL_DIR/scripts/. They use Python stdlib only, require zero install, and work on Linux, macOS, and Windows. Most sources work with no API key (OpenCorporates has an optional free token that raises rate limits). All outputs are normalized CSV with a header row. Scripts are idempotent. Rate-limit notes are in each source wiki; default fetchers sleep politely between paginated requests. API keys raise rate limits for supported sources (SEC_USER_AGENT, SENATE_LDA_TOKEN, OPENCORPORATES_API_TOKEN, COURTLISTENER_TOKEN). All scripts surface 429 responses immediately with the upstream's quota message.

When a private individual will not be in a source, the script returns 0 rows with a clear warning. For example, SEC EDGAR flags individual Form 3/4/5 filers versus corporate registrants. USAspending returns 0 rows for non-federal contractors. Senate LDA returns 0 rows for non-lobbying clients.

Federal Financial and Regulatory Sources

# SEC EDGAR filings (corporate disclosures)
python3 SKILL_DIR/scripts/fetch_sec_edgar.py --cik 0000320193 \
    --types 10-K,10-Q --out data/edgar_filings.csv

# USAspending federal contracts
python3 SKILL_DIR/scripts/fetch_usaspending.py --recipient "EXAMPLE CORP" \
    --fy 2024 --out data/contracts.csv

# Senate LD-1 / LD-2 lobbying disclosures
python3 SKILL_DIR/scripts/fetch_senate_ld.py --client "EXAMPLE CORP" \
    --year 2024 --out data/lobbying.csv

# OFAC SDN sanctions list (full snapshot)
python3 SKILL_DIR/scripts/fetch_ofac_sdn.py --out data/ofac_sdn.csv

# ICIJ Offshore Leaks — downloads ~70 MB bulk CSV on first use,
# then searches it locally. Cached for 30 days under
# $HERMES_OSINT_CACHE/icij/ (default: ~/.cache/hermes-osint/icij/).
python3 SKILL_DIR/scripts/fetch_icij_offshore.py --entity "EXAMPLE CORP" \
    --out data/icij.csv

Identity, Property, Litigation, Archives, and News

# NYC property records (deeds, mortgages, liens) — ACRIS via Socrata
python3 SKILL_DIR/scripts/fetch_nyc_acris.py --name "SMITH, JOHN" \
    --out data/acris.csv
python3 SKILL_DIR/scripts/fetch_nyc_acris.py --address "571 HUDSON" \
    --out data/acris_addr.csv

# OpenCorporates — 130+ jurisdiction corporate registry
# (free token required; set OPENCORPORATES_API_TOKEN or pass --token)
python3 SKILL_DIR/scripts/fetch_opencorporates.py --query "Example Corp" \
    --jurisdiction us_ny --out data/opencorporates.csv

# CourtListener — federal + state court opinions, PACER dockets
python3 SKILL_DIR/scripts/fetch_courtlistener.py --query "Smith v. Example Corp" \
    --type opinions --out data/courts.csv

# Wayback Machine — historical web captures
python3 SKILL_DIR/scripts/fetch_wayback.py --url "example.com" \
    --match host --collapse digest --out data/wayback.csv

# Wikipedia + Wikidata — narrative bio + structured facts
# Set HERMES_OSINT_UA=your-app/1.0 (your@email) to identify yourself
python3 SKILL_DIR/scripts/fetch_wikipedia.py --query "Bill Gates" \
    --out data/wp.csv

# GDELT — global news in 100+ languages, ~2015→present
python3 SKILL_DIR/scripts/fetch_gdelt.py --query '"Example Corp"' \
    --timespan 1y --out data/gdelt.csv

GDELT covers global news from approximately 2015 to the present.

Resolving Entities Across Sources

Run entity_resolution.py to match entities across two CSV files. It uses three matching tiers: exact (normalized strings equal after suffix/punctuation strip, high confidence), fuzzy (sorted-token equality/word-bag match, medium confidence), and token_overlap (at least 60% token overlap, at least 2 shared tokens, tokens at least 4 characters, low confidence).

# Match lobbying clients (Senate LDA) against contract recipients (USAspending)
python3 SKILL_DIR/scripts/entity_resolution.py \
    --left  data/lobbying.csv   --left-name-col  client_name \
    --right data/contracts.csv  --right-name-col recipient_name \
    --out data/cross_links.csv

The output cross_links.csv has columns: match_type, confidence, left_name, right_name, left_normalized, right_normalized, left_row, right_row.

This script does NOT use external fuzzy libraries (no rapidfuzz, no jellyfish). Token-bag matching is the upper bound. If you need Levenshtein, transliteration, or phonetic matching, pip-install separately.

Statistical Timing Correlation (Optional)

Run timing_analysis.py to test whether events (such as lobbying filings) are statistically correlated with award dates (such as contract awards). The column flags are generic and work for any (event, payee) time series joined through cross-links.

python3 SKILL_DIR/scripts/timing_analysis.py \
    --donations data/lobbying.csv --donation-date-col filing_date \
        --donation-amount-col income --donation-donor-col client_name \
        --donation-recipient-col registrant_name \
    --contracts data/contracts.csv --contract-date-col award_date \
        --contract-vendor-col recipient_name \
    --cross-links data/cross_links.csv \
    --permutations 1000 \
    --out data/timing.json

The null hypothesis is that event timing is independent of award dates. The one-tailed p-value equals the fraction of permutations with mean nearest-award distance less than or equal to the observed distance. The test requires a minimum of 3 events per (payer, vendor) pair. The script uses Python's random module for permutations; for reproducibility, pass --seed N.

Building the Findings JSON (Evidence Chain)

Run build_findings.py to produce a structured findings JSON with an evidence chain linking each claim to source rows.

python3 SKILL_DIR/scripts/build_findings.py \
    --cross-links data/cross_links.csv \
    --timing data/timing.json \
    --out data/findings.json

Every finding has the fields: id, title, severity, confidence, summary, evidence[], sources[]. Each evidence item points back to a specific row in a source CSV.

Adding a New Source

To add a new data source to the framework, copy the source template and fill in the 9 sections:

cp SKILL_DIR/templates/source-template.md \
    SKILL_DIR/references/sources/<your-source>.md

Parameters and Environment Variables

Parameter	Meaning	Required
SKILL_DIR	Directory holding this SKILL.md and subdirectories references/sources/ and scripts/	Yes
OPENCORPORATES_API_TOKEN	Free token for OpenCorporates API to raise rate limits; can also pass --token	No (but recommended)
SEC_USER_AGENT	User agent string for SEC EDGAR API to raise rate limits	No
SENATE_LDA_TOKEN	Token for Senate LDA API to raise rate limits	No
COURTLISTENER_TOKEN	Token for CourtListener API to raise rate limits	No
HERMES_OSINT_CACHE	Cache directory for ICIJ Offshore Leaks bulk CSV (default: ~/.cache/hermes-osint/icij/)	No
HERMES_OSINT_UA	User agent string for Wikipedia API (recommended: your-app/1.0 (your@email))	No
--seed N	Seed for random number generator in timing_analysis.py for reproducibility	No

Constraints and Caveats

• Python stdlib only; zero install; works on Linux, macOS, Windows.
• Most sources work with no API key (OpenCorporates has optional free token that raises rate limits).
• entity_resolution.py does NOT use external fuzzy libraries; token-bag matching is the upper bound.
• timing_analysis.py uses Python's random for permutations; for reproducibility, pass --seed N.
• fetch_*.py scripts use urllib.request and respect Retry-After; heavy bulk usage may still violate ToS.
• All Phase-1 sources are public records; bulk acquisition is permitted under their respective access terms.
• Some sources rate-limit aggressively; respect their headers.
• Some sources redact registrant info (GDPR on WHOIS, sealed filings).
• Cross-referencing public records to identify private individuals can have ethical implications; the skill produces evidence chains, not accusations.
• Every claim must trace to a record; no naked assertions.
• Confidence tier travels with the claim; match_type=fuzzy is probable, not confirmed.
• Entity resolution produces candidates, NOT conclusions; a fuzzy match between "ACME LLC" and "Acme Holdings Group" is a lead, not a fact.
• Statistical significance does not equal wrongdoing; p < 0.05 means the timing pattern is unlikely under the null; it does not establish corruption.
• All data sources here are public records; they may still contain inaccuracies, stale info, or redactions.

Failure Modes

• Script returns 0 rows with clear warning when a private individual will not be in a source (e.g., SEC EDGAR for non-public-company person, USAspending for non-federal contractor, Senate LDA for non-lobbying client).
• EDGAR flags when the company-name resolver matched an individual Form 3/4/5 filer rather than a corporate registrant.
• Scripts surface 429 responses immediately with the upstream's quota message so the user knows to slow down or supply a key.
• Heavy bulk usage may violate ToS of some sources.
• entity_resolution.py cannot perform Levenshtein, transliteration, or phonetic matching without external libraries.
• timing_analysis.py requires a minimum of 3 events per (payer, vendor) pair to run the test.