Web Search and Extract Integration for Hermes Agent

The Hermes Agent web search and extract integration connects the Hermes Agent runtime to multiple web search and content extraction providers, enabling AI agents to search the web and extract readable content from URLs. Once wired up, an agent can call web_search to retrieve ranked search results and web_extract to fetch and summarize page content, using backends such as Firecrawl, SearXNG, Brave Search, DuckDuckGo, Tavily, Exa, Parallel, or xAI (Grok).

What It Does

Hermes Agent includes two model-callable web tools backed by multiple providers:

• web_search: Search the web and return ranked results.
• web_extract: Fetch and extract readable content from one or more URLs.

Both tools are configured through a single backend selection, which can be set via hermes tools or directly in config.yaml. The integration supports per-capability configuration, meaning you can use different providers for search and extract independently. For example, you could use SearXNG (free) for search and Firecrawl for extract.

According to the official documentation, the supported backends and their capabilities are as follows:

• Firecrawl (default): Supports both search and extract. Free tier includes 500 credits per month. Requires FIRECRAWL_API_KEY.
• SearXNG: Search only. Free, self-hosted. Requires SEARXNG_URL.
• Brave Search (free tier): Search only. 2,000 queries per month. Requires BRAVE_SEARCH_API_KEY.
• DDGS (DuckDuckGo): Search only. Free, no API key needed.
• Tavily: Supports both search and extract. 1,000 searches per month on free tier. Requires TAVILY_API_KEY.
• Exa: Supports both search and extract. 1,000 searches per month on free tier. Requires EXA_API_KEY.
• Parallel: Supports both search and extract. Paid. Requires PARALLEL_API_KEY.
• xAI (Grok): Search only. Paid (SuperGrok or per-token). Requires XAI_API_KEY or hermes auth add xai-oauth.

Brave Search, DDGS, and xAI are search-only. If you also need web_extract, you must pair any of them with Firecrawl, Tavily, Exa, or Parallel. DDGS uses the ddgs Python package under the hood; if it is not already installed, run pip install ddgs (or let Hermes lazy-install it on first use). xAI runs Grok's server-side web_search tool on the Responses API. Results are LLM-generated rather than index-backed, so titles, descriptions, and URL choice are all model output (see the trust-model caveat below).

For Nous Subscribers with a paid Nous Portal subscription, web search and extract are available through the Tool Gateway via managed Firecrawl with no API key needed. New installs can run hermes setup --portal to log in and turn on all gateway tools at once; existing installs can flip just web via hermes tools.

Setup

Quick setup via hermes tools

Run hermes tools, navigate to Web Search & Extract, and pick a provider. The wizard prompts for the required URL or API key and writes it to your config.

hermes tools

Firecrawl (default)

Firecrawl is the default backend and is recommended for most users. It provides full-featured search and extract.

# ~/.hermes/.env
FIRECRAWL_API_KEY = fc-your-key-here

Get a key at firecrawl.dev. The free tier includes 500 credits per month.

For self-hosted Firecrawl, point at your own instance instead of the cloud API:

# ~/.hermes/.env
FIRECRAWL_API_URL = http://localhost:3002

When FIRECRAWL_API_URL is set, the API key is optional (disable server auth with USE_DB_AUTHENTICATION=false).

SearXNG (free, self-hosted)

SearXNG is a privacy-respecting, open-source metasearch engine that aggregates results from 70+ search engines. No API key is required. Just point Hermes at a running SearXNG instance. SearXNG is search-only; web_extract requires a separate extract provider.

Option A: Self-host with Docker (recommended)

This gives you a private instance with no rate limits.

1. Create a working directory:

mkdir -p ~/searxng/searxng
cd ~/searxng

2. Write a docker-compose.yml:

# ~/searxng/docker-compose.yml
services :
searxng :
image : searxng/searxng : latest
container_name : searxng
ports :
- "8888:8080"
volumes :
- ./searxng : /etc/searxng : rw
environment :
- SEARXNG_BASE_URL=http : //localhost : 8888/
restart : unless - stopped

3. Start the container:

docker compose up -d

4. Enable the JSON API format:

SearXNG ships with JSON output disabled by default. Copy the generated config and enable it:

# Copy the auto-generated config out of the container
docker cp searxng:/etc/searxng/settings.yml ~/searxng/searxng/settings.yml

Open ~/searxng/searxng/settings.yml. If use_default_settings: true is present, the file only contains your overrides. All other settings are inherited from the built-in defaults. To enable JSON responses for Hermes, add the following override:

search :
formats :
- html
- json

Your settings.yml should look similar to:

# Read the documentation before extending the defaults:
# https://docs.searxng.org/admin/settings/

use_default_settings : true

server :
secret_key : "abcdef12345678"
image_proxy : true

search :
formats :
- html
- json

5. Restart to apply:

docker cp ~/searxng/searxng/settings.yml searxng:/etc/searxng/settings.yml
docker restart searxng

6. Verify it works:

curl -s "http://localhost:8888/search?q=test&format=json" | python3 -c \
"import sys,json; d=json.load(sys.stdin); print(f'{len(d[ " results " ])} results')"

You should see something like 10 results. If you get a 403 Forbidden, JSON format is still disabled. Recheck step 4.

7. Configure Hermes:

# ~/.hermes/.env
SEARXNG_URL = http://localhost:8888

Then select SearXNG as the search backend in ~/.hermes/config.yaml:

web :
search_backend : "searxng"

Or set via hermes tools -> Web Search & Extract -> SearXNG.

Option B: Use a public instance

Public SearXNG instances are listed at searx.space. Filter by instances that have JSON format enabled (shown in the table).

# ~/.hermes/.env
SEARXNG_URL = https://searx.example.com

Public instances have rate limits, variable uptime, and may disable JSON format at any time. For production use, self-hosting is strongly recommended.

Pair SearXNG with an extract provider

SearXNG handles search; you need a separate provider for web_extract. Use the per-capability keys:

# ~/.hermes/config.yaml
web :
search_backend : "searxng"
extract_backend : "firecrawl" # or tavily, exa, parallel

With this config, Hermes uses SearXNG for all search queries and Firecrawl for URL extraction, combining free search with high-quality extraction.

Tavily

Tavily is an AI-optimised search and extract provider with a generous free tier.

# ~/.hermes/.env
TAVILY_API_KEY = tvly-your-key-here

Get a key at app.tavily.com. The free tier includes 1,000 searches per month.

Exa

Exa is a neural search engine with semantic understanding. Good for research and finding conceptually related content.

# ~/.hermes/.env
EXA_API_KEY = your-exa-key-here

Get a key at exa.ai. The free tier includes 1,000 searches per month.

Parallel

Parallel is an AI-native search and extraction provider with deep research capabilities.

# ~/.hermes/.env
PARALLEL_API_KEY = your-parallel-key-here

Get access at parallel.ai.

xAI (Grok)

Routes web_search through Grok's server-side web_search tool on the Responses API. Grok runs the actual searching and returns the top results as structured JSON.

Works with either credential path. No new env vars, no new setup wizard:

# ~/.hermes/.env (env-var path)
XAI_API_KEY = sk-xai-your-key-here

Or for SuperGrok subscribers:

hermes auth add xai-oauth

Then select xAI as the search backend:

# ~/.hermes/config.yaml
web :
backend : "xai"

Optional knobs:

web :
backend : "xai"
xai :
model : grok - build - 0.1 # reasoning model required by web_search (default)
allowed_domains : # optional, max 5, mutex with excluded_domains
- arxiv.org
excluded_domains : # optional, max 5
- example - spam.com
timeout : 90 # seconds (default)

Search-only. Pair with Firecrawl, Tavily, Exa, or Parallel if you also need web_extract. On 401, the provider performs a single forced OAuth-token refresh and retries (covers mid-window revocation and opaque tokens the proactive expiry check cannot decode); env-var credentials skip the retry.

Trust model: Unlike index-backed providers (Brave, Tavily, Exa) which return verbatim search-engine results, xAI is an LLM choosing which URLs to surface and writing the titles and descriptions itself. The content of the query influences the output, so a maliciously crafted query (e.g. injected via untrusted upstream input the agent picked up) can in principle steer Grok into emitting attacker-chosen URLs. Treat returned URLs the same way you would treat any model-generated link. Validate before fetching, especially if the query came from untrusted input.