OSINT Investigation: Public Records Cross-Reference Guide
Follow the money via public records and sanctions data.
Written by Neura Market from the official Hermes Agent documentation for Osint Investigation. Commands, paths, and version numbers are reproduced from the source unchanged.
Read the official documentationThis skill is for investigators who need to follow the money through public records: government contracts, corporate filings, lobbying disclosures, sanctions lists, offshore leaks, property records, court opinions, web archives, and global news. It is a stdlib-only toolkit that runs on Linux, macOS, and Windows with zero installation, and it is designed to resolve entities across heterogeneous sources, build cross-links with explicit confidence, run statistical timing tests, and produce structured evidence chains. Reach for it when a research question demands more than a single web search, and when you need to trace relationships that span multiple public registries.
What it does
This skill turns a terminal into an investigative workstation. You run Python scripts that fetch normalized CSV data from a dozen public sources, then you join those CSVs by entity name, test whether events cluster suspiciously in time, and finally assemble a findings JSON where every claim points back to a specific row in a source file. The whole pipeline is designed for reproducibility and auditability: nothing is asserted without a record behind it.
The source list covers federal financial and regulatory data (SEC EDGAR filings, USAspending contracts, Senate lobbying disclosures, OFAC sanctions, ICIJ offshore leaks) plus identity, property, litigation, archives, and news sources (NYC ACRIS property records, OpenCorporates, CourtListener, Wayback Machine, Wikipedia/Wikidata, GDELT). Each source has a dedicated fetch script, and each source has a wiki entry that documents access, schema, coverage, and legal notes.
What makes this different from a general web search is the cross-referencing layer. You are not just looking up one record; you are joining records across sources and assigning a confidence level to every match. The skill also includes a permutation test for timing correlation, which can flag whether lobbying filings cluster suspiciously close to contract awards. The output is an evidence chain, not a verdict.
Before you start
This skill is optional and installed on demand. It lives at optional-skills/research/osint-investigation and is version 0.1.0. It is adapted from the MIT-licensed ShinMegamiBoson/OpenPlanter project and is itself MIT licensed. It runs on Linux, macOS, and Windows.
The only hard requirement is Python's standard library. There is no pip install step. Most sources work without an API key; OpenCorporates has an optional free token that raises rate limits. You will need a terminal tool to run the scripts, and the skill expects SKILL_DIR to point at the directory holding this SKILL.md.
Before you start an investigation, read the data-source wiki entries to plan which sources apply. The skill's workflow begins with listing those entries and reading the ones relevant to your question. Each entry follows a 9-section template: summary, access, schema, coverage, cross-reference keys, data quality, acquisition, legal, references. The cross-reference potential section maps join keys between sources, so read those first to pick the right pair.
Workflow
The agent runs scripts via the terminal tool. SKILL_DIR is the directory holding this SKILL.md.
1. Identify which sources apply
Read the data-source wiki entries to plan the investigation:
ls SKILL_DIR/references/sources/
# Federal financial / regulatory
cat SKILL_DIR/references/sources/sec-edgar.md # corporate filings
cat SKILL_DIR/references/sources/usaspending.md # federal contracts
cat SKILL_DIR/references/sources/senate-ld.md # lobbying
cat SKILL_DIR/references/sources/ofac-sdn.md # sanctions
cat SKILL_DIR/references/sources/icij-offshore.md # offshore leaks
# Identity / property / litigation / archives / news
cat SKILL_DIR/references/sources/nyc-acris.md # NYC property records
cat SKILL_DIR/references/sources/opencorporates.md # global corporate registry
cat SKILL_DIR/references/sources/courtlistener.md # court records (federal + state)
cat SKILL_DIR/references/sources/wayback.md # Wayback Machine archives
cat SKILL_DIR/references/sources/wikipedia.md # Wikipedia + Wikidata
cat SKILL_DIR/references/sources/gdelt.md # global news monitoring
Each entry follows a 9-section template: summary, access, schema, coverage, cross-reference keys, data quality, acquisition, legal, references.
The cross-reference potential section maps join keys between sources, read those first to pick the right pair.
2. Acquire data
Each source has a stdlib-only fetch script in SKILL_DIR/scripts/:
Federal financial / regulatory
# SEC EDGAR filings (corporate disclosures)
python SKILL_DIR/scripts/fetch_sec_edgar.py --cik 0000320193 \
--types 10-K,10-Q --out data/edgar_filings.csv
# USAspending federal contracts
python SKILL_DIR/scripts/fetch_usaspending.py --recipient "EXAMPLE CORP" \
--fy 2024 --out data/contracts.csv
# Senate LD-1 / LD-2 lobbying disclosures
python SKILL_DIR/scripts/fetch_senate_ld.py --client "EXAMPLE CORP" \
--year 2024 --out data/lobbying.csv
# OFAC SDN sanctions list (full snapshot)
python SKILL_DIR/scripts/fetch_ofac_sdn.py --out data/ofac_sdn.csv
# ICIJ Offshore Leaks — downloads ~70 MB bulk CSV on first use,
# then searches it locally. Cached for 30 days under
# $HERMES_OSINT_CACHE/icij/ (default: ~/.cache/hermes-osint/icij/).
python SKILL_DIR/scripts/fetch_icij_offshore.py --entity "EXAMPLE CORP" \
--out data/icij.csv
Identity / property / litigation / archives / news
# NYC property records (deeds, mortgages, liens) — ACRIS via Socrata
python SKILL_DIR/scripts/fetch_nyc_acris.py --name "SMITH, JOHN" \
--out data/acris.csv
python SKILL_DIR/scripts/fetch_nyc_acris.py --address "571 HUDSON" \
--out data/acris_addr.csv
# OpenCorporates — 130+ jurisdiction corporate registry
# (free token required; set OPENCORPORATES_API_TOKEN or pass --token)
python SKILL_DIR/scripts/fetch_opencorporates.py --query "Example Corp" \
--jurisdiction us_ny --out data/opencorporates.csv
# CourtListener — federal + state court opinions, PACER dockets
python SKILL_DIR/scripts/fetch_courtlistener.py --query "Smith v. Example Corp" \
--type opinions --out data/courts.csv
# Wayback Machine — historical web captures
python SKILL_DIR/scripts/fetch_wayback.py --url "example.com" \
--match host --collapse digest --out data/wayback.csv
# Wikipedia + Wikidata — narrative bio + structured facts
# Set HERMES_OSINT_UA=your-app/1.0 (your@email) to identify yourself
python SKILL_DIR/scripts/fetch_wikipedia.py --query "Bill Gates" \
--out data/wp.csv
# GDELT — global news in 100+ languages, ~2015→present
python SKILL_DIR/scripts/fetch_gdelt.py --query '"Example Corp"' \
--timespan 1y --out data/gdelt.csv
All outputs are normalized CSV with a header row. Re-run scripts idempotently.
When a private individual won't be in a source (e.g. SEC EDGAR for a non-public- company person, USAspending for someone who isn't a federal contractor, Senate LDA for someone who isn't a lobbying client), the script returns 0 rows with a clear warning rather than silently writing an empty CSV. EDGAR specifically flags when the company-name resolver matched an individual Form 3/4/5 filer rather than a corporate registrant.
Rate-limit notes are in each source's wiki entry. Default fetchers sleep politely between paginated requests. API keys raise rate limits for sources that support them (SEC_USER_AGENT, SENATE_LDA_TOKEN, OPENCORPORATES_API_TOKEN, COURTLISTENER_TOKEN). All scripts surface 429 responses immediately with the upstream's quota message so the user knows to slow down or supply a key.
3. Resolve entities across sources
Normalize names and find matches between two CSV files:
# Match lobbying clients (Senate LDA) against contract recipients (USAspending)
python SKILL_DIR/scripts/entity_resolution.py \
--left data/lobbying.csv --left-name-col client_name \
--right data/contracts.csv --right-name-col recipient_name \
--out data/cross_links.csv
Three matching tiers with explicit confidence:
| Tier | Method | Confidence |
|---|---|---|
exact | Normalized strings equal after suffix/punctuation strip | high |
fuzzy | Sorted-token equality (word-bag match) | medium |
token_overlap | ≥60% token overlap, ≥2 shared tokens, tokens ≥4 chars | low |
Output cross_links.csv columns: match_type, confidence, left_name, right_name, left_normalized, right_normalized, left_row, right_row.
4. Statistical timing correlation (optional)
Test whether two time series cluster suspiciously close together, e.g. lobbying filings near contract awards, using a permutation test:
python SKILL_DIR/scripts/timing_analysis.py \
--donations data/lobbying.csv --donation-date-col filing_date \
--donation-amount-col income --donation-donor-col client_name \
--donation-recipient-col registrant_name \
--contracts data/contracts.csv --contract-date-col award_date \
--contract-vendor-col recipient_name \
--cross-links data/cross_links.csv \
--permutations 1000 \
--out data/timing.json
The script's column flags are intentionally generic, the original tool was written for donations vs awards, but it works for any (event, payee) time series joined through cross-links. Null hypothesis: event timing is independent of award dates. One-tailed p-value = fraction of permutations with mean nearest-award distance ≤ observed. Minimum 3 events per (payer, vendor) pair to run the test.
5. Build the findings JSON (evidence chain)
python SKILL_DIR/scripts/build_findings.py \
--cross-links data/cross_links.csv \
--timing data/timing.json \
--out data/findings.json
Every finding has id, title, severity, confidence, summary, evidence[], sources[]. Each evidence item points back to a specific row in a source CSV. The user (or a follow-up agent) can verify every claim against its source.
Confidence and evidence discipline
This is the load-bearing rule of the skill. Tell the user:
- Every claim must trace to a record. No naked assertions.
- Confidence tier travels with the claim.
match_type=fuzzyis "probable", not "confirmed." - Entity resolution produces candidates, NOT conclusions. A
fuzzymatch between "ACME LLC" and "Acme Holdings Group" is a lead, not a fact. - Statistical significance ≠ wrongdoing. p < 0.05 means the timing pattern is unlikely under the null. It does not establish corruption.
- All data sources here are public records. They may still contain inaccuracies, stale info, or redactions (GDPR, sealed records).
Adding a new data source
Use the template:
cp SKILL_DIR/templates/source-template.md \
SKILL_DIR/references/sources/<your-source>.md
Fill in all 9 sections. Write a fetch_.py script in scripts/ that uses stdlib only and writes a normalized CSV. Update the source list in the "When to use" section above.
Tools and their limits
entity_resolution.pydoes NOT use external fuzzy libraries (no rapidfuzz, no jellyfish). Token-bag matching is the upper bound here. If you need Levenshtein, transliteration, or phonetic matching, pip-install separately.timing_analysis.pyuses Python'srandomfor permutations. For reproducibility, pass--seed N.fetch_*.pyscripts useurllib.requestand respectRetry-After. Heavy bulk usage may still violate ToS, read each source's legal section first.
Legal note
All Phase-1 sources are public records. Bulk acquisition is permitted under their respective access terms (FOIA, public records law, ICIJ explicit publication, OFAC public data). However:
- Some sources rate-limit aggressively. Respect their headers.
- Some redact registrant info (GDPR on WHOIS, sealed filings).
- Cross-referencing public records to identify private individuals can have ethical implications. The skill produces evidence chains, not accusations.
When not to use it
This skill is not for general web research; use web_search / web_extract for that. It is not for domain or infrastructure OSINT; the domain-intel skill covers that. It is not for academic literature; use the arxiv skill. It is not for social-media profile discovery; the optional sherlock skill handles that. And it deliberately does not cover US federal campaign finance: the FEC API is unreliable for ad-hoc contributor-name queries on the free DEMO_KEY tier, so for federal donations point users at https://www.fec.gov/data/ directly.
Limits and gotchas
The entity resolution is intentionally simple. It does not use external fuzzy libraries, so token-bag matching is the upper bound. If you need Levenshtein, transliteration, or phonetic matching, you must pip-install separately. The timing analysis uses Python's random for permutations, so pass --seed N for reproducibility. The fetch scripts use urllib.request and respect Retry-After, but heavy bulk usage may still violate terms of service, so read each source's legal section first.
Data quality varies. Public records can contain inaccuracies, stale info, or redactions due to GDPR or sealed records. The skill produces evidence chains, not accusations, and cross-referencing public records to identify private individuals can have ethical implications.
What pairs with this
This skill sits alongside two related skills in the same category: domain-intel for domain and infrastructure OSINT, and arxiv for academic literature. Use them together when an investigation spans corporate records, technical infrastructure, and scholarly sources.