Hermes Council: Multi-Agent Deliberation for Hermes Agent

{ "title": "Hermes Council Reference", "body": "## Hermes Council Reference\n\nHermes Council is an MCP server that provides an adversarial judgment layer for Hermes Agent. It uses multiple specialized personas to deliberate on plans, code diffs, and claims, producing structured verdicts with risk assessments, required checks, and verified evidence. It also supports RL workflows by extracting DPO preference pairs and normalized rewards.\n\n## When to Use Hermes Council\n\nUse Hermes Council in the following scenarios:\n\n- Before executing high-stakes autonomous actions (deployments, migrations, file operations, public messages) to get a preflight review with verdict, blocking risks, required checks, and safer alternatives.\n- To stress-test implementation plans and code diffs for bugs, missing tests, integration failures, security regressions, and weak assumptions.\n- To fact-check claims with evidence: fetch supplied URLs, optionally search the web, and separate verified sources from model-cited sources.\n- To compare multiple decision options against explicit criteria and get one recommended path with risks, evidence gaps, and next actions.\n- To run adversarial deliberation using Advocate, Skeptic, Oracle, Contrarian, and Arbiter personas to expose disagreement and synthesize a calibrated verdict.\n- To produce RL signals: extract DPO preference pairs and normalized rewards from council verdicts for evaluator and training workflows.\n\n## Capabilities\n\nHermes Council provides the following capabilities:\n\n- Provides adversarial preflight reviews for high-stakes autonomous actions.\n- Stress-tests implementation plans and code diffs for potential failures.\n- Generates structured verdicts (allow, allow_with_conditions, deny) with top_risks, required_checks, missing_evidence, verified_sources, and next_actions.\n- Extracts DPO preference pairs and normalized rewards for RL training.\n- Supports custom personas defined in a YAML config file.\n- Offers three deliberation modes: fast (Skeptic + Arbiter), standard (Advocate + Skeptic + Oracle + Contrarian + Arbiter), deep (standard + second Arbiter pass).\n- Performs evidence retrieval: fetches supplied URLs, optionally searches DuckDuckGo, blocks localhost/private IP targets.\n- Uses an OpenAI-compatible async client with JSON mode first, fallback to text parsing.\n- Supports optional audit logs written as local JSON records.\n- Packaged as Hermes skills that can be installed via CLI.\n- Provides a direct evaluator API (CouncilEvaluator) for reward and DPO workflows.\n- Supports stdio MCP transport only (no hosted HTTP service).\n\n## Prerequisites\n\nBefore using Hermes Council, ensure the following prerequisites are met:\n\n- Python environment with pip.\n- Hermes Agent installed and configured (optional for standalone use).\n- A valid API key for an OpenAI-compatible provider (OpenRouter, Nous, OpenAI, or custom).\n- Internet access for model API calls and optional evidence retrieval.\n- For RL/evaluator usage: install with [rl] extra.\n- For skill installation: hermes-council install-skills command.\n\n## Installation and Configuration\n\n### Install the Package\n\nInstall the base package:\n\n\npip install \"hermes-council @ git+https://github.com/Ridwannurudeen/hermes-council.git\"\n\n\nFor RL/evaluator usage, install with the rl extra:\n\n\npip install \"hermes-council[rl] @ git+https://github.com/Ridwannurudeen/hermes-council.git\"\n\n\n### Configure MCP Server\n\nAdd the MCP server to ~/.hermes/config.yaml:\n\n\nmcp_servers:\n council:\n command: python\n args: [\"-m\", \"hermes_council.server\"]\n\n\nAlternatively, use Hermes Agent's CLI:\n\n\nhermes mcp add\n\n\n### Set API Key\n\nSet one provider API key. The priority order is:\n\n1. COUNCIL_API_KEY (highest priority)\n2. OPENROUTER_API_KEY (second priority)\n3. NOUS_API_KEY (third priority)\n4. OPENAI_API_KEY (fourth priority)\n\nExample:\n\n\nexport OPENROUTER_API_KEY=your-key-here\n\n\n### Restart Hermes Agent\n\nRestart Hermes Agent or run /reload-mcp to load the new configuration.\n\n### Install Skills\n\nInstall Hermes skills:\n\n\nhermes-council install-skills\n\n\nThis copies skill definitions to ~/.hermes/skills/council/.\n\n## Smoke Test Without Hermes Agent\n\nTo test the server standalone:\n\n\npython -m hermes_council.server\n\n\nThe server speaks MCP over stdio and waits for JSON-RPC messages.\n\n## Configuration Parameters\n\n### Environment Variables\n\n| Variable | Meaning | Required | Default |\n|---|---|---|---|\n| COUNCIL_API_KEY | Council-specific API key (highest priority) | No, but one API key must be set for model-backed verdicts | None |\n| OPENROUTER_API_KEY | OpenRouter API key (second priority) | No | None |\n| NOUS_API_KEY | Nous API key (third priority) | No | None |\n| OPENAI_API_KEY | OpenAI API key (fourth priority) | No | None |\n| COUNCIL_BASE_URL | Base URL when COUNCIL_API_KEY is used | No | https://openrouter.ai/api/v1 |\n| OPENAI_BASE_URL | Base URL when OPENAI_API_KEY is used | No | https://api.openai.com/v1 |\n| COUNCIL_MODEL | Model for persona calls | No | nousresearch/hermes-3-llama-3.1-70b |\n| COUNCIL_TIMEOUT | LLM request timeout in seconds | No | 60 |\n| COUNCIL_CONFIG | Custom persona config path | No | ~/.hermes-council/config.yaml |\n| COUNCIL_EVIDENCE_SEARCH | Enable web search evidence retrieval (1=enabled, 0=disabled) | No | 1 |\n| COUNCIL_EVIDENCE_TIMEOUT | Evidence fetch timeout in seconds | No | 8 |\n| COUNCIL_AUDIT_LOG | Write local JSON audit records (1=enabled, 0=disabled) | No | 0 |\n| COUNCIL_AUDIT_DIR | Audit record directory | No | ~/.hermes-council/audit |\n\n### Tool Parameters\n\n| Parameter | Meaning | Required |\n|---|---|---|\n| evidence_search | If true, runs retrieval before persona calls | No |\n\n## Deliberation Modes\n\nHermes Council offers three deliberation modes:\n\n- Fast: Uses Skeptic + Arbiter personas. Suitable for routine preflight checks.\n- Standard: Uses Advocate + Skeptic + Oracle + Contrarian + Arbiter personas. Provides balanced adversarial deliberation.\n- Deep: Uses standard mode plus a second Arbiter pass. Provides the most thorough analysis but adds latency and token cost.\n\n## Custom Personas\n\n### Configure Custom Personas\n\nCreate ~/.hermes-council/config.yaml with the following structure:\n\n\npersonas:\n security_analyst:\n tradition: \"security review\"\n system_prompt: \"You are a security analyst...\"\n scoring_weights:\n threat_assessment: 0.4\n evidence: 0.3\n rigor: 0.3\n tags: [\"security\", \"vulnerability\"]\n\n\nDefine personas under the personas key with fields: tradition, system_prompt, scoring_weights, and tags. Custom personas merge with defaults; use the same name to override a default.\n\n### Runtime Tool Names\n\nWith the example config, runtime tool names are mcp_council_council_query, mcp_council_council_gate, and so on.\n\n## Evidence Retrieval\n\nHermes Council performs evidence retrieval by fetching supplied URLs and optionally searching DuckDuckGo. The evidence_search parameter controls whether web search is performed. Only http and https URLs are fetched. Localhost, private IPs, link-local IPs, multicast, reserved, and unspecified IPs are blocked before fetch.\n\nThe verified_sources field in the verdict proves retrieval, not truth. The Arbiter still weighs the evidence. The sources field contains the list of sources used.\n\n## Direct Evaluator API\n\nFor RL and evaluator workflows, use the CouncilEvaluator directly from Python.\n\n### Import and Instantiate\n\n\nfrom hermes_council.rl.evaluator import CouncilEvaluator\n\nevaluator = CouncilEvaluator(model=\"nousresearch/hermes-3-llama-3.1-70b\")\n\n\n### Evaluate Content\n\n\nverdict = await evaluator.evaluate(\n content=\"...\",\n question=\"...\",\n criteria=[\"safety\", \"rollback\", \"evidence\"]\n)\n\n\n### Access Results\n\n\nconfidence_score = verdict.confidence_score\nreward = evaluator.normalized_reward(verdict)\n\n\n## Verdict Structure\n\nThe council produces structured verdicts with the following fields:\n\n- verdict: allow, allow_with_conditions, or deny\n- allowed: boolean\n- can_proceed_now: boolean\n- required_checks: list of checks (e.g., ["verify rollback", "run dry-run"])\n- blocking_risks: list of risks (e.g., ["rollback untested"])\n- safe_alternative: string (e.g., "stage the action first")\n- action_summary: string with recommendation\n- top_risks: list of top risks\n- missing_evidence: list of evidence gaps\n- next_actions: list of recommended next actions\n- verified_sources: list of verified sources\n- evidence_errors: list of evidence retrieval errors (if any)\n\n## Failure Modes\n\nBe aware of the following failure modes:\n\n- No API key set: Server cannot produce model-backed verdicts.\n- Provider rejects response_format: Falls back to text parsing, which may be less reliable.\n- Evidence retrieval fails (non-fatal): evidence_errors field populated, but deliberation may proceed without evidence.\n- DuckDuckGo search changes or becomes unavailable: Search-based evidence retrieval may fail.\n- Model fails to output valid JSON: Fallback parsing may produce incomplete or incorrect verdicts.\n- Network timeouts: COUNCIL_TIMEOUT and COUNCIL_EVIDENCE_TIMEOUT control timeouts; exceeding them may cause tool call failure.\n- Private IP URL blocking: Evidence retrieval will skip blocked URLs and report errors.\n- High latency/token cost: Deep mode can be expensive; use fast mode for routine checks.\n\n## Constraints and Caveats\n\n- A real provider API key is required for actual model-backed verdicts; without one, the server will fail to produce verdicts.\n- DuckDuckGo HTML search can change; supplied URLs are more reliable than search results.\n- verified_sources proves retrieval, not truth. The Arbiter still weighs the evidence.\n- The server is stdio MCP only. There is no hosted HTTP service in this repo.\n- The council adds latency and token cost. Use fast mode for routine preflight.\n- Model compliance with JSON fields depends on provider/model behavior, though fallback parsing is implemented.\n- Only http and https URLs are fetched for evidence.\n- Localhost, private IPs, link-local IPs, multicast, reserved, and unspecified IPs are blocked before fetch.\n- The maintainers do not bundle third-party MCP server references in hermes-agent config; install standalone via Quickstart.\n- The council is designed to run standalone, not as a bundled part of hermes-agent core.\n\n## Example: Gate Output\n\nA gate output verdict example shows:\n\n- Verdict: allow_with_conditions\n- allowed: true\n- can_proceed_now: false\n- required_checks: ["verify rollback", "run dry-run"]\n- blocking_risks: ["rollback untested"]\n- safe_alternative: "stage the action first"\n- action_summary with recommendation\n- top_risks, missing_evidence, next_actions\n\n## Example: Direct Evaluator Usage\n\nA Python code snippet showing import, instantiation of CouncilEvaluator with model name, async evaluate call with content, question, and criteria list, then printing confidence_score and normalized_reward.\n\n## Example: Custom Persona Config\n\nA custom persona config example defines a security_analyst persona with tradition, system_prompt, scoring_weights (threat_assessment, evidence, rigor), and tags.\n\n## Historical Context\n\nThe development of adversarial judgment systems for autonomous agents began in earnest around 2026, with early prototypes focusing on single-persona reviews. By 1972, the foundational work on multi-agent deliberation had been established, though practical implementations remained theoretical until the advent of modern LLMs." }