Hermes Agent Credential Pools: Rotating and Sharing API Keys

hermes auth add openrouter --api-key sk-or-v1-your-second-key

hermes auth add anthropic --type api-key --api-key sk-ant-api03-your-second-key

hermes auth add anthropic --type oauth
# Opens browser for OAuth login

hermes auth list

openrouter (2 credentials):
#1 OPENROUTER_API_KEY api_key env:OPENROUTER_API_KEY ←
#2 backup-key api_key manual

anthropic (3 credentials):
#1 hermes_pkce oauth hermes_pkce ←
#2 claude_code oauth claude_code
#3 ANTHROPIC_API_KEY api_key env:ANTHROPIC_API_KEY

hermes auth

What would you like to do?
1. Add a credential
2. Remove a credential
3. Reset cooldowns for a provider
4. Set rotation strategy for a provider
5. Exit

anthropic supports both API keys and OAuth login.
1. API key (paste a key from the provider dashboard)
2. OAuth login (authenticate via browser)
Type [1/2]:

# After setting up a custom endpoint via hermes model:
hermes auth list
# Shows:
# Together.ai (1 credential):
# #1 config key api_key config:Together.ai ←

# Add a second key for the same endpoint:
hermes auth add Together.ai --api-key sk-together-second-key

{
  "credential_pool": {
    "openrouter": [ ... ],
    "custom:together.ai": [ ... ]
  }
}

Configuration Reference

CLI Commands

Command	Description
hermes auth	Interactive pool management wizard
hermes auth list	Show all pools and credentials
hermes auth list <provider>	Show a specific provider's pool
hermes auth add <provider>	Add a credential (prompts for type and key)
hermes auth add <provider> --type api-key --api-key <key>	Add an API key non-interactively
hermes auth add <provider> --type oauth	Add an OAuth credential via browser login
hermes auth remove <provider> <index>	Remove credential by 1-based index
hermes auth reset <provider>	Clear all cooldowns/exhaustion status

Rotation Strategies

Configure via hermes auth → "Set rotation strategy" or in config.yaml:

credential_pool_strategies:
  openrouter: round_robin
  anthropic: least_used

Strategy	Behavior
fill_first (default)	Use the first healthy key until it's exhausted, then move to the next
round_robin	Cycle through keys evenly, rotating after each selection
least_used	Always pick the key with the lowest request count
random	Random selection among healthy keys

Error Recovery

Error	Behavior	Cooldown
429 Rate Limit	Retry same key once (transient). Second consecutive 429 rotates to next key	1 hour
402 Billing/Quota	Immediately rotate to next key	24 hours
401 Auth Expired	Try refreshing the OAuth token first. Rotate only if refresh fails	,
All keys exhausted	Fall through to fallback_model if configured	,

The has_retried_429 flag resets on every successful API call, so a single transient 429 does not trigger rotation.

Storage

Pool state is stored in ~/.hermes/auth.json under the credential_pool key:

{
  "version": 1,
  "credential_pool": {
    "openrouter": [
      {
        "id": "abc123",
        "label": "OPENROUTER_API_KEY",
        "auth_type": "api_key",
        "priority": 0,
        "source": "env:OPENROUTER_API_KEY",
        "secret_source": "bitwarden",
        "secret_fingerprint": "sha256:12ab34cd56ef7890",
        "last_status": "ok",
        "request_count": 142
      }
    ],
    "anthropic": [
      {
        "id": "manual1",
        "label": "personal-api-key",
        "auth_type": "api_key",
        "priority": 0,
        "source": "manual",
        "access_token": "sk-ant-api03-..."
      }
    ]
  }
}

The OpenRouter entry above was borrowed from an external source, so the raw key is not stored in auth.json. The manual Anthropic entry was intentionally added to Hermes' credential store, so its token remains persistable.

Strategies are stored in config.yaml (not auth.json):

credential_pool_strategies:
  openrouter: round_robin
  anthropic: least_used

How It Works

The credential pool integrates at the provider resolution layer. The architecture involves several components:

• agent/credential_pool.py, Pool manager: storage, selection, rotation, cooldowns
• hermes_cli/auth_commands.py, CLI commands and interactive wizard
• hermes_cli/runtime_provider.py, Pool-aware credential resolution
• run_agent.py, Error recovery: 429/402/401 → pool rotation → fallback

For the full data flow diagram, the official documentation references docs/credential-pool-flow.excalidraw in the repository.

Request Flow

Your request
→ Pick key from pool (round_robin / least_used / fill_first / random)
→ Send to provider
→ 429 rate limit?
→ Plan/usage limit reached (e.g. ChatGPT/Codex "usage limit reached")?
→ Rotate to next pool key immediately (no retry, the cap won't clear on retry)
→ Generic / transient 429?
→ Retry same key once (transient blip)
→ Second 429 → rotate to next pool key
→ All keys exhausted → fallback_model (different provider)
→ 402 billing error?
→ Immediately rotate to next pool key (24h cooldown)
→ 401 auth expired?
→ Try refreshing the token (OAuth)
→ Refresh failed → rotate to next pool key
→ Success → continue normally

Auto-Discovery

Hermes automatically discovers credentials from multiple sources and seeds the pool on startup:

Source	Example	Auto-seeded?
Environment variables	OPENROUTER_API_KEY, ANTHROPIC_API_KEY	Yes
OAuth tokens (auth.json)	Codex device code, Nous device code	Yes
Claude Code credentials	~/.claude/.credentials.json	Yes (Anthropic)
Hermes PKCE OAuth	~/.hermes/auth.json	Yes (Anthropic)
Custom endpoint config	model.api_key in config.yaml	Yes (custom endpoints)
Manual entries	Added via hermes auth add	Persisted in auth.json

Auto-seeded entries are updated on each pool load. If you remove an environment variable, its pool entry is automatically pruned. Manual entries added via hermes auth add are never auto-pruned.

Borrowed runtime secrets (for example environment variables, Bitwarden/Vault/keyring/systemd references, and custom config values) are reference-only at the auth.json boundary. Hermes can use the resolved value in memory for the current run, but it persists only metadata such as the source ref, label, status, request counters, and a non-reversible fingerprint. Manual entries and Hermes-owned OAuth/device-code state keep the durable tokens they need to refresh.

Delegation and Subagent Sharing

When the agent spawns subagents via delegate_task, the parent's credential pool is automatically shared with children:

• Same provider: the child receives the parent's full pool, enabling key rotation on rate limits.
• Different provider: the child loads that provider's own pool (if configured).
• No pool configured: the child falls back to the inherited single API key.

This means subagents benefit from the same rate-limit resilience as the parent, with no extra configuration needed. Per-task credential leasing ensures children do not conflict with each other when rotating keys concurrently.

Thread Safety

The credential pool uses a threading lock for all state mutations: select(), mark_exhausted_and_rotate(), try_refresh_current(), mark_used(). This ensures safe concurrent access when the gateway handles multiple chat sessions simultaneously.

Requirements and Limitations

• Credential pools are mainly for API-key providers (OpenRouter, Anthropic). A single Nous Portal OAuth covers 300+ models, so most users do not need a pool when on Portal.
• Key rotation resets the prompt cache. Each rotation incurs a full-price re-read of the conversation context.
• The fill_first strategy is the default. If you want even distribution, you must explicitly configure round_robin, least_used, or random.
• Custom endpoint pools require the endpoint to be set up via hermes model first.
• OAuth credentials for Anthropic require a Claude Max plan plus extra usage credits.
• The has_retried_429 flag resets on every successful API call, so a single transient 429 does not trigger rotation.
• All keys exhausted falls through to fallback_model only if configured.
• The official documentation does not specify a maximum number of keys per pool or any version constraints on the Hermes Agent itself.

Troubleshooting

No failure modes or troubleshooting steps are documented in the source material. If you encounter issues, consider the following based on the documented behavior:

• If keys are not being rotated as expected, verify the rotation strategy in config.yaml or via the interactive wizard.
• If a key is stuck in cooldown, use hermes auth reset <provider> to clear all cooldowns and exhaustion status.
• If a custom endpoint pool does not appear, ensure the endpoint was set up via hermes model first.
• If subagents are not inheriting the pool, confirm they are using the same provider and that delegate_task is being used.