Hermes Agent Credential Pools: Rotating and Sharing API Keys

{ "title": "Hermes Credential Pool Manager Reference", "body": "## Purpose\n\nThe Credential Pool Manager registers multiple API keys or OAuth tokens for the same provider. When one key hits a rate limit or billing quota, the manager automatically rotates to the next healthy key. This keeps the session alive without switching to a different provider.\n\n## When to Use\n\n- You need to handle rate limits or billing quotas for the same provider without switching to a different provider.\n- You are running long conversations where prompt cache resets are acceptable to avoid session interruption.\n- You are using API-key providers like OpenRouter or Anthropic. This is not needed for Nous Portal OAuth, which covers 300+ models.\n\n## Capabilities\n\n- Auto-rotate to the next healthy key on a 429 rate limit, 402 billing error, or 401 auth expired.\n- Support multiple rotation strategies: fill_first (default), round_robin, least_used, random.\n- Interactive CLI wizard for managing credentials.\n- Non-interactive CLI commands for adding, removing, and resetting credentials.\n- Auto-discovery of credentials from environment variables, OAuth tokens, Claude Code credentials, Hermes PKCE OAuth, and custom endpoint configuration.\n- Manual credential entries persisted in auth.json; auto-seeded entries are pruned when the source is removed.\n- Custom endpoint pools keyed by endpoint name (e.g., Together.ai).\n- Delegation and subagent sharing: the parent pool is shared with child subagents for the same provider.\n- Thread safety with a threading lock for all state mutations.\n- Fallback to fallback_model if all pool keys are exhausted.\n- Cooldown periods: 1 hour for 429/402, 5 minutes for 401; provider-supplied reset_at timestamps override defaults.\n- Retry the same key once on a transient 429; a second consecutive 429 triggers rotation.\n- OAuth token refresh attempt on 401 before rotation.\n- Borrowed runtime secrets (env vars, Bitwarden, etc.) are stored as reference-only with a fingerprint; manual entries keep durable tokens.\n- Per-task credential leasing for concurrent subagents.\n\n## Prerequisites\n\n- Hermes installed and configured.\n- At least one API key or OAuth token for the provider.\n- For OAuth: a browser must be available for login.\n- For custom endpoints: the endpoint must be configured via hermes model or in config.yaml.\n\n## Procedures\n\n### Adding a Second OpenRouter Key\n\n1. Run the following command:\n\nbash\nhermes auth add openrouter --api-key sk-or-v1-your-second-key\n\n\n### Adding a Second Anthropic API Key\n\n1. Run the following command:\n\nbash\nhermes auth add anthropic --type api-key --api-key sk-ant-api03-your-second-key\n\n\n### Adding an Anthropic OAuth Credential\n\n1. Run the following command:\n\nbash\nhermes auth add anthropic --type oauth\n\n\n2. A browser opens for OAuth login.\n\n### Checking Credential Pools\n\n1. Run the following command:\n\nbash\nhermes auth list\n\n\nThis shows all providers and their credentials. A ← marks the current credential.\n\n### Interactive Pool Management\n\n1. Run the following command:\n\nbash\nhermes auth\n\n\n2. The menu displays:\n - 1. Add a credential\n - 2. Remove a credential\n - 3. Reset cooldowns for a provider\n - 4. Set rotation strategy for a provider\n - 5. Exit\n\n3. For providers supporting both API keys and OAuth (Anthropic, Nous, Codex), choose the type:\n - 1. API key\n - 2. OAuth login\n\n### Adding a Credential Non-Interactively\n\n1. Run the following command:\n\nbash\nhermes auth add <provider> --type api-key --api-key <key>\n\n\n### Adding an OAuth Credential Non-Interactively\n\n1. Run the following command:\n\nbash\nhermes auth add <provider> --type oauth\n\n\n### Removing a Credential\n\n1. Run the following command:\n\nbash\nhermes auth remove <provider> <index>\n\n\nThe index is a 1-based index of the credential to remove.\n\n### Resetting Cooldowns for a Provider\n\n1. Run the following command:\n\nbash\nhermes auth reset <provider>\n\n\n### Setting Rotation Strategy\n\n1. Run hermes auth and select Set rotation strategy.\n2. Alternatively, set the strategy in config.yaml under credential_pool_strategies.\n\n### Adding a Second Key for a Custom Endpoint\n\n1. Set up the custom endpoint via hermes model. This auto-generates a name like Together.ai.\n2. Run the following command:\n\nbash\nhermes auth add Together.ai --api-key sk-together-second-key\n\n\n## Parameters\n\n| Parameter | Meaning | Required |\n|-------------|-------------------------------------------------------------------------|-----------------------------------------------|\n| provider | Name of the provider (e.g., openrouter, anthropic) or custom endpoint name. | Yes |\n| --type | Credential type: api-key or oauth. | For the add command when the provider supports both types. |\n| --api-key | The API key string. | For non-interactive add of api-key type. |\n| index | 1-based index of the credential to remove. | For the remove command. |\n| strategy | Rotation strategy: fill_first (default), round_robin, least_used, random. | For setting rotation strategy. |\n\n## Rotation Strategies\n\n- fill_first (default): Uses the first available key until it fails, then moves to the next.\n- round_robin: Cycles through keys in order, skipping any that are on cooldown.\n- least_used: Picks the key that has been used the fewest times.\n- random: Picks a random key from the pool.\n\nStrategies are stored in config.yaml, not in auth.json.\n\n## Failure Modes\n\n### 429 Rate Limit\n\n- Retry the same key once. If the second consecutive 429 occurs, rotate to the next key with a 1-hour cooldown.\n\n### 402 Billing/Quota\n\n- Immediately rotate to the next key with a 1-hour cooldown.\n\n### 401 Auth Expired\n\n- Try refreshing the OAuth token. If the refresh fails, rotate to the next key with a 5-minute cooldown.\n\n### All Keys Exhausted\n\n- Fall through to fallback_model if configured.\n\n### Provider-Supplied Timestamps\n\n- Provider-supplied reset_at timestamps override default cooldowns.\n\n## Constraints and Caveats\n\n- Key rotation resets the provider-side prompt cache. Each rotation costs a full-price pass over the context.\n- Credential pools are for same-provider rotation. Fallback providers are cross-provider failover.\n- Pools are tried first. If all keys are exhausted, fallback_model activates.\n- Nous Portal OAuth covers 300+ models, so most users do not need a pool on Portal.\n- Auto-seeded entries are pruned when the source is removed. Manual entries are never auto-pruned.\n- Borrowed runtime secrets (env vars, Bitwarden, etc.) are reference-only. Only metadata and fingerprint are stored in auth.json.\n- Manual entries and Hermes-owned OAuth/device-code state keep durable tokens.\n- Custom endpoint pools are stored under the custom: prefix in auth.json.\n- The has_retried_429 flag resets on every successful API call.\n- Thread safety is enforced via a threading lock for all state mutations.\n\n## Examples\n\n### Adding a Second OpenRouter Key\n\nbash\nhermes auth add openrouter --api-key sk-or-v1-your-second-key\n\n\n### Adding a Second Anthropic API Key\n\nbash\nhermes auth add anthropic --type api-key --api-key sk-ant-api03-your-second-key\n\n\n### Adding an Anthropic OAuth Credential\n\nbash\nhermes auth add anthropic --type oauth\n\n\nThis opens a browser for OAuth login.\n\n### Checking Pools\n\nbash\nhermes auth list\n\n\nOutput shows openrouter (2 credentials) and anthropic (3 credentials) with ← marking the current credential.\n\n### Interactive Wizard\n\nbash\nhermes auth\n\n\nMenu: 1. Add a credential, 2. Remove a credential, 3. Reset cooldowns for a provider, 4. Set rotation strategy for a provider, 5. Exit.\n\n### Adding a Second Key for Custom Endpoint\n\nbash\nhermes auth add Together.ai --api-key sk-together-second-key\n\n\n## Credential Storage Details\n\n- Manual credential entries are persisted in auth.json.\n- Auto-seeded entries are pruned when the source is removed.\n- Borrowed runtime secrets (env vars, Bitwarden, etc.) are stored as reference-only with a fingerprint. Only metadata and fingerprint are kept in auth.json.\n- Manual entries and Hermes-owned OAuth/device-code state keep durable tokens.\n- Custom endpoint pools are stored under the custom: prefix in auth.json.\n- Rotation strategies are stored in config.yaml, not in auth.json.\n\n## Thread Safety\n\nAll state mutations use a threading lock to ensure thread safety. This is critical when multiple subagents are leasing credentials concurrently.\n\n## Delegation and Subagent Sharing\n\nThe parent pool is shared with child subagents for the same provider. This allows concurrent subagents to use credentials from the same pool without conflict. Per-task credential leasing is supported for concurrent subagents.\n\n## Cooldown Management\n\n- 429 rate limit: 1-hour cooldown after rotation.\n- 402 billing/quota: 1-hour cooldown after rotation.\n- 401 auth expired: 5-minute cooldown after rotation.\n- Provider-supplied reset_at timestamps override these defaults.\n- The has_retried_429 flag resets on every successful API call. A single retry is attempted on a 429; a second consecutive 429 triggers rotation.\n\n## OAuth Token Refresh\n\nOn a 401 auth expired error, the system attempts to refresh the OAuth token before rotating to the next key. If the refresh fails, rotation proceeds with a 5-minute cooldown.\n\n## Fallback Model\n\nIf all keys in the pool are exhausted, the system falls through to fallback_model if configured. This is a cross-provider failover, distinct from the same-provider credential pool rotation." }