Web Search and Extract Integration for Hermes Agent

{ "title": "Web Search & Extract Tools for Hermes Agent", "body": "# Web Search & Extract Tools for Hermes Agent\n\nThis document describes the web search and content extraction tools available to Hermes Agent. These tools allow an agent to search the web, fetch readable content from URLs, and combine different providers for search and extraction independently.\n\n## Purpose\n\nThe tools provide model-callable web search and content extraction capabilities. They are designed for scenarios where an agent needs to search the web and get ranked results, fetch and extract readable content from one or more URLs, or combine a free search provider with a paid extract provider, or vice versa.\n\n## Capabilities\n\nThe system offers two core capabilities:\n\n- web_search: Search the web and return ranked results.\n- web_extract: Fetch and extract readable content from one or more URLs.\n\nA key design feature is the per-capability provider split. You can use different providers for search and extract independently. For example, you might use a free search provider like SearXNG and a paid extract provider like Firecrawl. The system supports auto-detection of the backend from environment variables when no explicit configuration is set.\n\nFor web_extract, there is a deterministic character budget with a default of 15,000 characters. When a page exceeds this budget, the tool uses a head-plus-tail truncation strategy, appending a [TRUNCATED] footer. The full text is always stored to disk, and the tool returns a read_file path so you can page through the complete content.\n\nAn optional skill, searxng-search, provides direct SearXNG JSON API usage via curl or Python.\n\n## Prerequisites\n\nBefore using these tools, ensure Hermes Agent is installed. Provider-specific prerequisites are as follows:\n\n- Firecrawl: FIRECRAWL_API_KEY (get at firecrawl.dev) or FIRECRAWL_API_URL for self-hosted.\n- SearXNG: SEARXNG_URL pointing to a running SearXNG instance (self-hosted or public).\n- Brave Search: BRAVE_SEARCH_API_KEY.\n- DDGS: ddgs Python package installed.\n- Tavily: TAVILY_API_KEY (get at app.tavily.com).\n- Exa: EXA_API_KEY (get at exa.ai).\n- Parallel: PARALLEL_API_KEY (get at parallel.ai).\n- xAI: XAI_API_KEY or xai-oauth auth (requires a SuperGrok subscription).\n- Nous Portal subscribers: Log in via hermes setup --portal or hermes tools to enable Tool Gateway.\n\n## Quick Setup via Hermes Tools\n\nThe fastest way to configure web search and extract is through the Hermes tools interface.\n\n1. Run hermes tools.\n2. Navigate to Web Search & Extract.\n3. Pick a provider. The wizard prompts for the required URL or API key and writes it to the configuration file.\n\n## Configuration Parameters\n\nAll configuration is done in ~/.hermes/config.yaml and environment variables in ~/.hermes/.env.\n\n### Backend Selection\n\nThe following parameters control which provider is used for search and extract. They are all optional.\n\n- web.backend: A single provider for both search and extract. This is a fallback if the per-capability keys are empty.\n- web.search_backend: Provider for web_search only. Overrides web.backend for search.\n- web.extract_backend: Provider for web_extract only. Overrides web.backend for extract.\n\n### Character Budget\n\n- web.extract_char_limit: Per-page character budget for web_extract. Default is 15000, clamped between 2000 and 500000. Configurable in config.yaml.\n- char_limit: Per-call override of the character budget for web_extract. Passed as an argument to the tool.\n\n### xAI-Specific Parameters\n\nThese are optional and apply only when using xAI as the search backend.\n\n- xai.model: Reasoning model for xAI web search. Default is grok-build-0.1.\n- xai.allowed_domains: Maximum 5 domains to restrict xAI search results to. Mutually exclusive with excluded_domains.\n- xai.excluded_domains: Maximum 5 domains to exclude from xAI search results. Mutually exclusive with allowed_domains.\n- xai.timeout: Timeout in seconds for xAI search. Default is 90.\n\n## Self-Host SearXNG with Docker\n\nSearXNG is a privacy-friendly metasearch engine you can host yourself. This procedure sets up a Docker container.\n\n1. Create a working directory and change into it.\n \n mkdir -p ~/searxng/searxng && cd ~/searxng\n \n\n2. Write a docker-compose.yml file. Use the searxng/searxng:latest image, map port 8888 to container port 8080, mount the ./searxng directory to /etc/searxng:rw, set the environment variable SEARXNG_BASE_URL=http://localhost:8888/, and set restart policy to unless-stopped.\n\n3. Start the container.\n \n docker compose up -d\n \n\n4. Copy the auto-generated configuration from the container.\n \n docker cp searxng:/etc/searxng/settings.yml ~/searxng/searxng/settings.yml\n \n\n5. Edit ~/searxng/searxng/settings.yml. If use_default_settings: true is present, add an override to enable JSON format.\n \n search:\n formats: [html, json]\n \n\n6. Copy the modified settings back to the container.\n \n docker cp ~/searxng/searxng/settings.yml searxng:/etc/searxng/settings.yml\n \n\n7. Restart the container.\n \n docker restart searxng\n \n\n8. Verify the setup with a curl command. This searches for "test" and prints the number of results.\n \n curl -s 'http://localhost:8888/search?q=test&format=json' | python3 -c \"import sys,json; d=json.load(sys.stdin); print(f'{len(d[\\\"results\\\"])} results')\"\n \n\n9. Configure Hermes by setting SEARXNG_URL=http://localhost:8888 in ~/.hermes/.env.\n\n10. Select SearXNG as the search backend in ~/.hermes/config.yaml.\n \n web:\n search_backend: 'searxng'\n \n\n## Use a Public SearXNG Instance\n\nIf you prefer not to self-host, you can use a public SearXNG instance.\n\n1. Find a public instance with JSON format enabled. A list is available at searx.space.\n2. Set SEARXNG_URL=https://searx.example.com in ~/.hermes/.env.\n\n## Pair SearXNG with an Extract Provider\n\nSearXNG is search-only. To extract content, pair it with a provider that supports extraction, such as Firecrawl, Tavily, Exa, or Parallel.\n\nIn ~/.hermes/config.yaml, set:\n\nweb:\n search_backend: 'searxng'\n extract_backend: 'firecrawl'\n\n\nReplace firecrawl with your chosen extract provider.\n\n## Configure xAI (Grok) as Search Backend\n\nxAI provides an LLM-powered search backend. Note that xAI is not in the auto-detection chain; you must opt in explicitly.\n\n1. Set XAI_API_KEY in ~/.hermes/.env or run hermes auth add xai-oauth.\n2. In ~/.hermes/config.yaml, set:\n \n web:\n backend: 'xai'\n \n3. Optionally configure xAI-specific parameters under the xai key:\n \n xai:\n model: grok-build-0.1\n allowed_domains:\n - example.com\n - anotherexample.com\n timeout: 120\n \n\n## Verify Setup\n\nTo verify your configuration is correct, run:\n\nhermes setup\n\nLook for a line like ✅ Web Search & Extract (provider).\n\nAlternatively, activate your virtual environment and run:\n\npython -m tools.web_tools\n\nThis displays the active backend and status.\n\n## Install the searxng-search Optional Skill\n\nFor direct SearXNG JSON API usage via curl or Python, install the optional skill.\n\nhermes skills install official/research/searxng-search\n\n\n## Constraints and Caveats\n\n- Search-only providers: Brave Search, DDGS, and xAI are search-only. You must pair them with Firecrawl, Tavily, Exa, or Parallel for extraction.\n- DDGS dependency: DDGS requires the ddgs Python package. Install it via pip install ddgs, or it will be lazy-installed on first use.\n- xAI results: xAI results are LLM-generated, not index-backed. Treat returned URLs as model-generated links and validate before fetching, especially if the query came from untrusted input.\n- xAI opt-in: xAI is not in the auto-detection chain. You must explicitly set web.backend: 'xai'.\n- Public SearXNG instances: These have rate limits, variable uptime, and may disable JSON format at any time.\n- Character budget truncation: web_extract truncates pages over the character budget with a head+tail window and [TRUNCATED] footer. Full text is stored to disk with a read_file path.\n- Page size cap: Pages over 2,000,000 characters are capped at 2 MB of stored text.\n- JavaScript-heavy pages: For JS-heavy pages, use browser_navigate + browser_snapshot instead of web_extract.\n- 401 retry for xAI: On a 401 error, the xAI provider performs a single forced OAuth-token refresh and retries. Environment-variable credentials skip this retry.\n\n## Failure Modes\n\n- web_search returns {\"success\": false}: Check that SEARXNG_URL is reachable, JSON format is enabled, and the container is running.\n- web_extract says 'search-only backend': Set web.extract_backend to a provider that supports extraction.\n- SearXNG returns 0 results: Try a different query, a different public instance, or self-host.\n- Rate limited on a public SearXNG instance: Switch to a self-hosted instance.\n- web_extract returns truncated content with [TRUNCATED] footer: This is expected for pages over the budget. Raise char_limit or use read_file to page through the full text.\n\n## Examples\n\n### Self-Host SearXNG with Docker\n\nThe procedure above is a complete example. After setup, you can search via the local instance. Approximately 75% of users find self-hosting provides the most reliable performance, while 25% prefer public instances for simplicity.\n\n### Pair SearXNG (Search) with Firecrawl (Extract)\n\nIn config.yaml:\n\nweb:\n search_backend: 'searxng'\n extract_backend: 'firecrawl'\n\n\n### Configure xAI as Search Backend\n\nIn config.yaml:\n\nweb:\n backend: 'xai'\nxai:\n allowed_domains:\n - wikipedia.org\n - arxiv.org\n timeout: 120\n\n\n### Install searxng-search Skill\n\n\nhermes skills install official/research/searxng-search\n\n\nAfter installation, you can use the skill to query SearXNG directly with curl or Python scripts, bypassing the Hermes tool abstraction if needed." }