Model Failover and Auth Profile Rotation in OpenClaw

OpenClaw manages failures through a two-phase approach:

1. Auth profile rotation stays within the current provider.
2. Model fallback moves to the next entry in agents.defaults.model.fallbacks.

Runtime flow

Resolve session state

Determine which model and auth profile should be active for the session.

Build candidate chain

Assemble the list of candidate models from the current selection and the fallback policy tied to that selection's source. Configured defaults, cron job primaries, and auto-selected fallback models may use configured fallbacks; explicit user session selections are strict.

Try the current provider

Attempt the current provider while applying auth-profile rotation and cooldown rules.

Advance on failover-worthy errors

When that provider is depleted with an error that qualifies for failover, advance to the next model candidate.

Use fallback for the current turn

Execute the winning fallback candidate without altering the session's selected provider or model.

Retry safe pure overload exhaustion

When every candidate fails solely due to provider overload, retry the entire turn-local chain up to 10 times using exponential backoff, provided no tool execution or assistant output has begun. After 30 seconds, emit one status notice so the user is not left waiting silently.

Throw FailoverError if exhausted

If all candidates fail, raise a FailoverError containing structured per-attempt details and the earliest cooldown expiry when one is known.

Fallback execution is confined to the current turn. The reply runner persists only fallback notice state so /status and transition notices can tell apart the selected model from the model that responded; the fallback is not saved as the next turn's model selection.

Selection source policy

Whether the fallback chain is permitted depends on the selection source:

• Configured default: agents.defaults.model.primary relies on agents.defaults.model.fallbacks.
• Agent primary: agents.entries.*.model is strict unless that agent's model object carries its own fallbacks. Use fallbacks: [] to make the strict behavior explicit, or a non-empty list to enable model fallback for that agent.
• Runtime fallback: the fallback candidate applies only to the current turn. The next turn starts from the selected primary again. OpenClaw still recognizes previously stored modelOverrideSource: "auto" entries, probes their configured origin every 5 minutes, and clears them once the origin recovers. /new, /reset, and sessions.reset also clear those entries.
• User session override: /model, the model picker, session_status(model=...), and sessions.patch write modelOverrideSource: "user". This is an exact session selection. If the selected provider/model fails before producing a reply, OpenClaw reports the failure instead of answering from an unrelated configured fallback.
• Legacy session override: older session entries may have modelOverride without modelOverrideSource. OpenClaw treats those as user overrides so an explicit old selection is not silently converted into fallback behavior.
• Cron payload model: a cron job payload.model / --model is a job primary, not a user session override. It uses configured fallbacks unless the job provides payload.fallbacks; payload.fallbacks: [] makes the cron run strict.

Outside group and channel conversations, OpenClaw posts a visible notice when a turn moves onto fallback and another notice when a later turn succeeds on the selected primary. Group and channel conversations keep the same fallback state and lifecycle events without posting these notices. Persisted notice state prevents repeated notices when consecutive turns use the same selected/active pair, while model selection itself remains unchanged.

Auth failure skip cache

By default, every new turn keeps the existing fallback retry behavior: OpenClaw retries each configured fallback candidate again, including non-primary candidates that recently failed with auth or auth_permanent.

Opt in to suppress repeat auth failures with:

OPENCLAW_FALLBACK_SKIP_TTL_MS=60000

When enabled, OpenClaw records an in-memory, session-scoped skip marker for a non-primary fallback candidate after an auth-class failure. The key includes the session, provider, model, and selected automatic or explicit profile ID. Switching profiles does not inherit another profile's failure marker. Primary candidates are never skipped, so an explicit user model selection still surfaces the real auth error. The cache is process-local and clears on Gateway restart.

The value is a TTL in milliseconds. 0 or unset disables the cache. Positive values are clamped between 1 second and 10 minutes.

User-visible fallback notices

Outside group and channel conversations, OpenClaw sends a status notice in the same reply surface when a session moves onto an auto-selected fallback:

↪️ Model Fallback: <fallback> (selected <primary>; <reason>)

When a later probe succeeds and the session returns to the selected primary, OpenClaw sends:

↪️ Model Fallback cleared: <primary> (was <fallback>)

These notices are operational messages, not assistant content. They deliver once per state change outside group and channel conversations, including side-effect-only turns when feasible, but repeated turn-local fallback transitions do not repeat them. Group and channel conversations suppress the visible notices while retaining the same fallback state and lifecycle events. Delivery bypasses normal source-reply suppression, does not consume the first assistant reply slot for threaded channels, and is excluded from text-to-speech.

Auth storage (keys + OAuth)

OpenClaw uses auth profiles for both API keys and OAuth tokens.

• Secrets and runtime auth-routing state live in ~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite.
• Config auth.profiles / auth.order are metadata + routing only (no secrets).
• Legacy credentials/oauth.json, auth-profiles.json, auth-state.json, and per-agent auth.json files are imported only by openclaw doctor --fix. Runtime fails closed for the affected agent until credential-bearing legacy files are migrated; it never silently imports or falls back to them.

More detail: OAuth

Credential types:

• type: "api_key" → { provider, key }
• type: "oauth" → { provider, access, refresh, expires, email? } (+ projectId/enterpriseUrl for some providers)
• type: "token" → static bearer-style token, optionally expiring; OpenClaw does not refresh it (used for aws-sdk and other credential-chain auth modes)

Profile IDs

OAuth logins create distinct profiles so multiple accounts can coexist.

• Default: provider:default when no email is available.
• OAuth with email: provider:<email> (for example openai:user@example.com).

Profiles live in the per-agent openclaw-agent.sqlite auth profile store.

Rotation order

When a provider has multiple profiles, OpenClaw picks an order like this:

Explicit config

auth.order[provider] (if set).

Configured profiles

auth.profiles filtered by provider.

Stored profiles

Per-agent SQLite auth profile entries for the provider.

If no explicit order is configured, OpenClaw applies a round-robin order:

• Primary key: profile type (OAuth, then static token, then API key).
• Secondary key for OAuth: profiles with a currently usable access token before profiles whose access token is expired. Expired OAuth profiles stay eligible so the runtime can refresh them when no usable peer is available.
• Next key: usageStats.lastUsed (oldest first, within each type/state tier).
• Cooldown/disabled profiles are moved to the end, ordered by soonest expiry.

Session stickiness (cache-friendly)

OpenClaw pins the automatically chosen auth profile per session to keep provider caches warm. It does not rotate on every request. An automatic pin may rotate or clear when:

• the session is reset (/new / /reset)
• a compaction completes (compaction count increments)
• the profile is in cooldown/disabled

Manual selection via /model …@<profileId> -s sets a user override. A valid user pin survives /new, /reset, session rollover, compaction, and cooldown windows. It remains the first preference when eligible; while that exact profile is in cooldown or disabled, OpenClaw tries the next eligible same-provider profile without replacing the stored pin. OpenClaw clears the pin when the profile disappears, no longer matches the selected provider, or the user selects another explicit profile. /model default -s clears the model override while retaining a compatible auth pin and clearing an incompatible one.

Note

Auto-pinned and user-pinned auth profiles are both retry preferences: OpenClaw tries the selected profile first while it is eligible, then may rotate to another same-provider profile on auth failures, rate limits, billing limits, or timeouts. A user pin stays persisted during that temporary rotation, so new runs prefer it again after its cooldown expires without changing the selected model or runtime. This auth rotation does not loosen model selection: an explicit user provider/model selection remains strict and reports failure after its same-provider auth profiles are exhausted.

OpenAI Codex subscription plus API-key backup

For OpenAI agent models, auth and runtime are separate. openai/gpt-* stays on the Codex harness while auth can rotate between a Codex subscription profile and an OpenAI API-key backup.

Use auth.order.openai for the user-facing order:

{
  auth: {
    order: {
      openai: ["openai:user@example.com", "openai:api-key-backup"],
    },
  },
}

Use openai:* for both ChatGPT/Codex OAuth profiles and OpenAI API-key profiles. When the subscription hits a Codex usage limit, OpenClaw records the exact reset time when Codex provides one, tries the next ordered auth profile, and keeps the run inside the Codex harness. Once the reset time passes, the subscription profile is eligible again and the next automatic selection can return to it.

Use a user-pinned profile to make one account/key the durable first preference for that session. If it becomes unavailable, OpenClaw temporarily rotates through the remaining eligible auth.order.openai profiles and returns to the pinned profile after recovery.

Cooldowns

When a profile fails due to auth/rate-limit errors (or a timeout that looks like rate limiting), OpenClaw marks it in cooldown and moves to the next profile.

CLI-backed runtimes settle profile health only after their resume, fork, and fresh-session recovery attempts finish. A terminal credential failure cools down the exact selected profile before model fallback; a successful run clears stale failure state. Transcript, format, context, pre-provider timeout, and ambient CLI failures without a selected profile do not change shared profile health.

What lands in the rate-limit / timeout bucket

That rate-limit bucket is broader than plain 429: it also includes provider messages such as Too many concurrent requests, ThrottlingException, concurrency limit reached, workers_ai ... quota limit exceeded, throttled, resource exhausted, and periodic usage-window limits such as weekly limit reached or monthly limit exhausted.

Format/invalid-request errors are usually terminal because retrying the same payload would fail the same way, so OpenClaw surfaces them instead of rotating auth profiles. Known retry-repair paths can opt in explicitly: for example Cloud Code Assist tool call ID validation failures are sanitized and retried once through the allowFormatRetry policy.

OpenAI-compatible provider-completed stop/finish reasons such as Unhandled stop reason: error, stop reason: error, reason: error, and Provider finish_reason: error are classified as server_error (HTTP-like status 500), not timeout. They remain failover-eligible for model/profile rotation, but diagnostics keep the provider finish-reason text instead of rewriting the user copy to "LLM request timed out." Transport-shaped finish reasons such as Provider finish_reason: abort, network_error, and malformed_response stay in the timeout/failover bucket (status 408).

Generic server text can also land in that timeout bucket when the source matches a known transient pattern. For example, the bare model runtime stream-wrapper message An unknown error occurred is treated as failover-worthy for every provider because the shared model runtime emits it when provider streams end with stopReason: "aborted" or stopReason: "error" without specific details. JSON api_error payloads with transient server text such as internal server error, unknown error, 520, upstream error, or backend error are also treated as failover-worthy timeouts.

OpenRouter-specific generic upstream text such as bare Provider returned error is treated as timeout only when the provider context is actually OpenRouter. Generic internal fallback text such as LLM request failed with an unknown error. stays conservative and does not trigger failover by itself.

SDK retry-after caps

Some provider SDKs may otherwise sleep for a long Retry-After window before returning control to OpenClaw. For Stainless-based SDKs such as Anthropic and OpenAI, OpenClaw caps SDK-internal retry-after-ms / retry-after waits at 60 seconds by default and surfaces longer retryable responses immediately so this failover path can run. Tune or disable the cap with OPENCLAW_SDK_RETRY_MAX_WAIT_SECONDS; see Retry behavior.

Model-scoped cooldowns

Cooldowns tied to rate limits can also apply per model:

• When the failing model ID is identifiable, OpenClaw records cooldownModel for rate-limit failures.
• If a cooldown is limited to one model, a sibling model on the same provider remains eligible for attempts.
• Billing or disabled windows continue to block the entire profile across all models.

Standard cooldowns, excluding billing and permanent-auth cases, scale with the profile's recent error tally:

• First failure: 30 seconds
• Second failure: 1 minute
• Third and subsequent failures: 5 minutes (capped)

Once the profile's built-in failure window lapses, the counters reset.

The per-agent SQLite auth state holds this data under usageStats:

{
  "usageStats": {
    "provider:profile": {
      "lastUsed": 1736160000000,
      "cooldownUntil": 1736160600000,
      "errorCount": 2
    }
  }
}

Billing disables

Billing or credit failures, such as "insufficient credits" or "credit balance too low", qualify for failover, yet they are rarely temporary. Rather than applying a brief cooldown, OpenClaw flags the profile as disabled with a longer backoff and shifts to the next profile or provider.

Note

Not every response shaped like a billing issue is 402, and not every HTTP 402 reaches this point. Even when a provider sends 401 or 403, OpenClaw keeps explicit billing wording in the billing lane, but provider-specific matchers remain limited to their owning provider, such as OpenRouter 403 Key limit exceeded.

In the meantime, temporary 402 usage-window and organization or workspace spend-limit errors are grouped as rate_limit when the message appears retryable, for instance weekly usage limit exhausted, daily limit reached, resets tomorrow, or organization spending limit exceeded. These follow the short cooldown and failover route rather than the extended billing-disable route.

Permanent-auth failures with high confidence, such as revoked or deactivated keys and deactivated workspaces, receive a similar disabled lane but recover far sooner than billing, because some providers briefly present auth-like payloads during incidents.

The per-agent SQLite auth state stores this information:

{
  "usageStats": {
    "provider:profile": {
      "disabledUntil": 1736178000000,
      "disabledReason": "billing"
    }
  }
}

Overload and rate-limit errors get more aggressive handling than billing cooldowns: by default, OpenClaw permits one same-provider auth-profile retry, then moves to the next configured model fallback without delay.

Model fallback

When every profile on a provider fails, OpenClaw advances to the next model in agents.defaults.model.fallbacks. This covers auth failures, rate limits, and timeouts that have used up profile rotation, while other errors do not trigger fallback. Provider errors lacking sufficient detail are still labeled precisely in fallback state: empty_response indicates the provider returned no usable message or status, no_error_details means the provider explicitly returned Unknown error (no error details in response), and unclassified indicates OpenClaw kept the raw preview but no classifier matched it yet.

Signals of provider busyness, such as ModelNotReadyException, fall into the overloaded bucket and follow the same one-rotation-then-fallback policy as rate limits, as shown in the defaults table above.

If overload failures alone exhaust the entire candidate chain, the reply runner retries the chain up to 10 times within the same turn. Full-turn retry is permitted only before tool execution or assistant output begins, which prevents duplicate mutations or messages if an overload appears after observable work. Backoff starts at 2.5 seconds and doubles up to a 30-second cap. Once the turn has waited 30 seconds, OpenClaw sends one transient status notice: The AI service is temporarily overloaded. I’m still retrying; this may take a few minutes. The retry and any fallback winner stay turn-local, while ordinary transient server errors keep their separate one-retry policy.

When a run begins from the configured default primary, a cron job primary, an agent primary with explicit fallbacks, or an auto-selected fallback override, OpenClaw can traverse the matching configured fallback chain. Agent primaries without explicit fallbacks and explicit user selections, such as /model ollama/qwen3.5:27b, the model picker, sessions.patch, or one-off CLI provider or model overrides, are strict: if that provider or model is unreachable or fails before producing a reply, OpenClaw reports the failure rather than answering from an unrelated fallback.

Candidate chain rules

OpenClaw assembles the candidate list from the currently requested provider/model plus any configured fallbacks.

Rules

• The requested model always comes first.
• Explicit configured fallbacks are deduplicated but not filtered by the model allowlist, as they represent explicit operator intent.
• If the current run is already on a configured fallback within the same provider family, OpenClaw continues using the full configured chain.
• When no explicit fallback override is supplied, configured fallbacks are attempted before the configured primary, even if the requested model uses a different provider.
• When no explicit fallback override is given to the fallback runner, the configured primary is appended at the end so the chain can return to the normal default once earlier candidates are exhausted.
• When a caller supplies fallbacksOverride, the runner uses exactly the requested model plus that override list. An empty list disables model fallback and stops the configured primary from being appended as a hidden retry target.

Which errors advance fallback

Continues on

• auth failures
• rate limits and cooldown exhaustion
• overloaded or provider-busy errors
• timeout-shaped failover errors
• billing disables
• LiveSessionModelSwitchError, which is normalized into a failover path so a stale persisted model does not create an outer retry loop
• other unrecognized errors when candidates remain

Does not continue on

• explicit aborts that are not timeout or failover shaped
• context overflow errors that should stay within compaction or retry logic, such as request_too_large, input token count exceeds the maximum number of input tokens, input exceeds the maximum number of tokens, input too long for the model, or ollama error: context length exceeded
• a final unknown error when no candidates remain
• Claude Fable 5 safety refusals; direct API-key requests handle those at the provider level via Anthropic's server-side fallback to claude-opus-4-8 instead, as described in Anthropic

Cooldown skip vs probe behavior

When every auth profile for a provider is already in cooldown, OpenClaw does not permanently skip that provider. It decides per candidate:

Per-candidate decisions

• Persistent auth failures skip the entire provider immediately.
• Billing disables usually skip, but the primary candidate can still be probed on a throttle so recovery is possible without restarting.
• The primary candidate may be probed near cooldown expiry, with a per-provider throttle.
• Same-provider fallback siblings can be attempted despite cooldown when the failure looks transient, such as rate_limit, overloaded, or unknown. This matters especially when a rate limit is model-scoped and a sibling model may recover immediately.
• Transient cooldown probes are limited to one per provider per fallback run so a single provider does not stall cross-provider fallback.

Session overrides and live model switching

Session model changes are shared state. The active runner, /model command, compaction or session updates, and live-session reconciliation all read or write parts of the same session entry. Fallback execution does not write model-selection fields, so it cannot replace a newer manual selection while retrying.

Live model switching follows these rules:

• Only explicit user-driven model changes mark a pending live switch, including /model, session_status(model=...), and sessions.patch.
• System-driven model changes, such as fallback rotation, heartbeat overrides, or compaction, never mark a pending live switch on their own.
• User-driven model overrides are treated as exact selections for fallback policy, so an unreachable selected provider surfaces as a failure instead of being masked by agents.defaults.model.fallbacks.
• Runtime fallback candidates remain turn-local. The next turn starts from the current selected model, including a manual selection that arrived during the previous run.
• Previously stored auto fallback overrides remain supported: OpenClaw periodically probes their configured origin and clears the override when it recovers; /new, /reset, and sessions.reset clear auto-sourced overrides immediately.
• Outside group and channel conversations, user replies announce fallback transitions and fallback-cleared recovery once per state change. Repeated turns with the same selected or active pair do not repeat the notice; group and channel conversations retain the same fallback state and lifecycle events without posting it.
• /status shows the selected model and, when fallback state differs, the active fallback model and reason.
• Live-session reconciliation prefers persisted session overrides over stale runtime model fields.
• If a live-switch error points at a later candidate in the active fallback chain, OpenClaw jumps directly to that selected model instead of walking unrelated candidates first.

The active run keeps its chosen candidate without alteration. Live reconciliation updates that candidate solely when a user explicitly requests a switch, which removes any need for temporary fallback overrides or rollbacks.

Observability and failure summaries

runWithModelFallback(...) stores per-attempt details that support logging and user-facing cooldown notices:

• provider/model attempted
• reason (rate_limit, overloaded, billing, auth, model_not_found, and other similar failover causes)
• optional status/code
• human-readable error summary

Structured model_fallback_decision logs additionally include flat fallbackStep* fields whenever a candidate fails, gets skipped, or a subsequent fallback succeeds. Those fields make the attempted transition explicit (fallbackStepFromModel, fallbackStepToModel, fallbackStepFromFailureReason, fallbackStepFromFailureDetail, fallbackStepFinalOutcome), allowing log and diagnostic exporters to reconstruct the primary failure even when the terminal fallback also fails.

If every candidate fails, OpenClaw raises FailoverError carrying structured attempt records. The outer reply runner can draw on those records to craft a more precise message, such as "all models are temporarily rate-limited," and include the earliest cooldown expiry when one is available.

That cooldown summary is model-aware:

• unrelated model-scoped rate limits are ignored for the attempted provider/model chain
• if the remaining block is a matching model-scoped rate limit, OpenClaw reports the last matching expiry that still blocks that model

Related config

Refer to Gateway configuration for:

• auth.profiles / auth.order
• agents.defaults.model.primary / agents.defaults.model.fallbacks
• agents.defaults.imageModel routing

For the broader model selection and fallback overview, see Models.