Live Testing Suites: Network-Touching Tests Guide

For quick start guidance, QA runners, unit/integration suites, and Docker workflows, refer to Testing. This page addresses live (network-touching) tests: model matrix, CLI backends, ACP, media providers, and credential handling.

Live tests vs your real gateway

Live suites and ad hoc smokes must never disrupt a gateway that is already handling production traffic (yours or another operator's):

• Supply your own gateway: use the in-process gateway (Layer 2 below) or launch a dev instance with an isolated state directory (OPENCLAW_STATE_DIR=<scratch>) and a free port. Avoid binding the default gateway port (18789) while a real gateway occupies it.
• Never openclaw gateway stop/restart (or launchctl/systemctl/tmux equivalents) a service you did not start in this session, since that is the operator's live instance. Obtain explicit approval beforehand.
• Want realistic data? Copy the live state/DB into your dev state directory and test against the copy. In-place migrations of a live gateway's state likewise require explicit approval.

Live: local smoke commands

Export the required provider key in the process environment before ad hoc live checks.

Safe media smoke:

pnpm openclaw infer tts convert --local --json \
  --text "OpenClaw live smoke." \
  --output /tmp/openclaw-live-smoke.mp3

Safe voice-call readiness smoke:

pnpm openclaw voicecall setup --json
pnpm openclaw voicecall smoke --to "+15555550123"

voicecall smoke acts as a dry run unless --yes is also present; use --yes only when you plan to place an actual call. For Twilio, Telnyx, and Plivo, a successful readiness check demands a public webhook URL, local/private loopback URLs get rejected because those providers cannot reach them.

Live: Android node capability sweep

• Test: src/gateway/android-node.capabilities.live.test.ts
• Script: pnpm android:test:integration
• Goal: run every command currently advertised by a connected Android node and verify command contract behavior.
• Scope:
  • Preconditioned/manual setup (the suite does not install/run/pair the app).
  • Command-by-command gateway node.invoke validation for the selected Android node.
• Required pre-setup:
  • Android app already connected and paired to the gateway.
  • App kept in foreground.
  • Permissions/capture consent granted for capabilities you expect to pass.
• Optional target overrides:
  • OPENCLAW_ANDROID_NODE_ID or OPENCLAW_ANDROID_NODE_NAME.
  • OPENCLAW_ANDROID_GATEWAY_URL / OPENCLAW_ANDROID_GATEWAY_TOKEN / OPENCLAW_ANDROID_GATEWAY_PASSWORD.
• Full Android setup details: Android App

Live: model smoke (profile keys)

Live model tests split into two layers so failures stay isolated:

• "Direct model" reveals whether the provider/model can answer at all with the given key.
• "Gateway smoke" reveals whether the full gateway+agent pipeline works for that model (sessions, history, tools, sandbox policy, etc.).

The curated model lists below reside in src/agents/live-model-filter.ts and evolve over time; treat the arrays there as the source of truth, not this page.

MiniMax M3 uses minimax/MiniMax-M3 as its default provider/model reference.

Layer 1: Direct model completion (no gateway)

• Test: src/agents/models.profiles.live.test.ts
• Goal:
  • Enumerate discovered models
  • Use getApiKeyForModel to select models you have creds for
  • Run a small completion per model (and targeted regressions where needed)
• How to enable:
  • pnpm test:live (or OPENCLAW_LIVE_TEST=1 if invoking Vitest directly)
  • Set OPENCLAW_LIVE_MODELS=modern, small, or all (alias for modern) to actually run this suite; otherwise it skips, so pnpm test:live on its own stays focused on gateway smoke.
• How to select models:
  • OPENCLAW_LIVE_MODELS=modern runs the curated high-signal priority list (see Live: model matrix)
  • OPENCLAW_LIVE_MODELS=small runs the curated small-model priority list
  • OPENCLAW_LIVE_MODELS=all is an alias for modern
  • or OPENCLAW_LIVE_MODELS="openai/gpt-5.6-luna,anthropic/claude-opus-4-6,..." (comma allowlist)
  • Local Ollama small-model runs default to http://127.0.0.1:11434; set OPENCLAW_LIVE_OLLAMA_BASE_URL only for LAN, custom, or Ollama Cloud endpoints.
  • Modern/all and small sweeps default to their curated-list length as a cap; set OPENCLAW_LIVE_MAX_MODELS=0 for an exhaustive selected-profile sweep or a positive number for a smaller cap.
  • Exhaustive sweeps use OPENCLAW_LIVE_TEST_TIMEOUT_MS for the whole direct-model test timeout. Default: 60 minutes.
  • Direct-model probes run with 20-way parallelism by default; set OPENCLAW_LIVE_MODEL_CONCURRENCY to override.
• How to select providers:
  • OPENCLAW_LIVE_PROVIDERS="google,google-gemini-cli" (comma allowlist)
• Where keys come from:
  • By default: profile store and env fallbacks
  • Set OPENCLAW_LIVE_REQUIRE_PROFILE_KEYS=1 to enforce profile store only
• Why this exists:
  • Separates "provider API is broken / key is invalid" from "gateway agent pipeline is broken"
  • Contains small, isolated regressions (example: OpenAI Responses/Codex Responses reasoning replay + tool-call flows)

Layer 2: Gateway + dev agent smoke (what "@openclaw" actually does)

• Test: src/gateway/gateway-models.profiles.live.test.ts
• Objective:
  • Launch a gateway that runs inside the process
  • Establish or update an agent:dev:* session, with a per-run model override
  • Step through models-with-keys and verify:
    • The response carries actual substance, with no tools involved
    • A genuine tool call executes correctly, via the read probe
    • Additional optional tool checks, covering exec plus read probe
    • OpenAI regression scenarios, where a tool-only call is followed by a follow-up, continue to function
• Probe specifics, useful for diagnosing failures on the spot:
  • For read: a nonce file gets placed in the workspace, then the agent is told to read that file and return the nonce in its reply.
  • For exec+read: the agent is directed to exec-write a nonce into a temporary file and then read it back.
  • For the image probe: a generated PNG, built from "CAT" plus a random code, is attached, and the model is expected to respond with cat <CODE>.
  • Code references: see src/gateway/gateway-models.profiles.live.test.ts and test/helpers/live-image-probe.ts.
• Turning it on:
  • Use pnpm test:live, or OPENCLAW_LIVE_TEST=1 when calling Vitest directly.
• Picking models:
  • By default, the curated high-signal list from modern takes priority.
  • OPENCLAW_LIVE_GATEWAY_MODELS=small pushes the curated small-model set through the complete gateway plus agent pipeline.
  • OPENCLAW_LIVE_GATEWAY_MODELS=all acts as a shorthand for modern.
  • Alternatively, set OPENCLAW_LIVE_GATEWAY_MODELS="provider/model", or a comma-separated list, to limit the scope.
  • The modern/all and small gateway sweeps cap at their curated list length by default; to run an exhaustive sweep, set OPENCLAW_LIVE_GATEWAY_MAX_MODELS=0, or use a positive number for a tighter cap.
• Picking providers, to avoid routing everything through OpenRouter:
  • OPENCLAW_LIVE_GATEWAY_PROVIDERS="google,google-gemini-cli,openai,anthropic,zai,minimax" accepts a comma-separated allowlist.
• Tool and image probes stay active in this live test at all times:
  • The read and exec+read probes handle tool stress.
  • The image probe activates whenever the model declares support for image input.
  • High-level flow:
    • A small PNG containing "CAT" plus a random code is produced (test/helpers/live-image-probe.ts).
    • It is transmitted through agent and attachments: [{ mimeType: "image/png", content: "<base64>" }].
    • The gateway converts attachments into images[], which includes src/gateway/server-methods/agent.ts and src/gateway/chat-attachments.ts.
    • The embedded agent sends a multimodal user message onward to the model.
    • The check confirms the reply includes cat along with the code, allowing for minor OCR discrepancies.

Tip

To discover what is testable on your setup, along with the exact provider/model ids, execute:

openclaw models list
openclaw models list --json

Live: CLI backend smoke (Claude, Gemini, or other local CLIs)

• Test: src/gateway/gateway-cli-backend.live.test.ts
• Purpose: exercise the Gateway and agent pipeline against a local CLI backend, leaving your default configuration untouched.
• Backend-specific smoke defaults are defined within the owning plugin's cli-backend.ts definition.
• Activation:
  • pnpm test:live, or OPENCLAW_LIVE_TEST=1 when running Vitest directly.
  • OPENCLAW_LIVE_CLI_BACKEND=1
• Default behavior:
  • The default provider and model is claude-cli/claude-sonnet-4-6.
  • Command, argument, and image handling come from the metadata of the owning CLI backend plugin.
• Optional overrides:
  • OPENCLAW_LIVE_CLI_BACKEND_MODEL="claude-cli/claude-sonnet-4-6"
  • OPENCLAW_LIVE_CLI_BACKEND_COMMAND="/full/path/to/claude"
  • OPENCLAW_LIVE_CLI_BACKEND_ARGS='["-p","--output-format","json"]'
  • Set OPENCLAW_LIVE_CLI_BACKEND_IMAGE_PROBE=1 to attach a real image, with file paths inserted into the prompt. This is disabled by default in Docker recipes.
  • Use OPENCLAW_LIVE_CLI_BACKEND_IMAGE_ARG="--image" to hand image file paths to the CLI as arguments instead of injecting them into the prompt.
  • OPENCLAW_LIVE_CLI_BACKEND_IMAGE_MODE="repeat", or "list", controls how image arguments are delivered when IMAGE_ARG is enabled.
  • OPENCLAW_LIVE_CLI_BACKEND_RESUME_PROBE=1 triggers a second turn to confirm the resume flow works.
  • OPENCLAW_LIVE_CLI_BACKEND_CACHE_PROBE=1 runs a fresh Claude CLI turn, followed by two native resumes, and demands at least 90% prompt-cache reuse on the second resume. This probe switches off the image, MCP, and model-switch probes.
  • OPENCLAW_LIVE_CLI_BACKEND_MODEL_SWITCH_PROBE=1 opts into the Claude Sonnet to Opus same-session continuity probe, provided the selected model has a valid switch target. Disabled by default, including in Docker recipes.
  • OPENCLAW_LIVE_CLI_BACKEND_MCP_PROBE=1 opts into the MCP and tool loopback probe. Docker recipes leave this off by default.

Example:

  OPENCLAW_LIVE_CLI_BACKEND=1 \
  OPENCLAW_LIVE_CLI_BACKEND_MODEL="claude-cli/claude-sonnet-4-6" \
  pnpm test:live src/gateway/gateway-cli-backend.live.test.ts

Cheap Gemini MCP config smoke:

OPENCLAW_LIVE_TEST=1 \
  pnpm test:live src/agents/cli-runner/bundle-mcp.gemini.live.test.ts

No response generation is requested from Gemini here. Instead, the same system settings OpenClaw hands to Gemini are written out, then gemini --debug mcp list runs to confirm a saved transport: "streamable-http" server is normalized into Gemini's HTTP MCP format and can reach a local streamable-HTTP MCP server.

Docker recipe:

pnpm test:docker:live-cli-backend

Single-provider Docker recipes:

pnpm test:docker:live-cli-backend:claude
pnpm test:docker:live-cli-backend:claude:cache
pnpm test:docker:live-cli-backend:claude-subscription
pnpm test:docker:live-cli-backend:gemini

Notes:

• The Docker runner is located at scripts/test-live-cli-backend-docker.sh.
• pnpm test:docker:live-cli-backend:claude:cache depends on Anthropic API-key authentication. It records normalized cache usage for both resumes and errors out when the second resume shows less than 90% reuse.
• Inside the repo's Docker image, it executes the live CLI-backend smoke as the non-root node user.
• It pulls CLI smoke metadata from the owning plugin, then installs the corresponding Linux CLI package (@anthropic-ai/claude-code or @google/gemini-cli) into a cached writable prefix at OPENCLAW_DOCKER_CLI_TOOLS_DIR (default: ~/.cache/openclaw/docker-cli-tools).
• codex-cli is no longer shipped as a CLI backend; instead, pair openai/* with the Codex app-server runtime (see Live: Codex app-server harness smoke).
• pnpm test:docker:live-cli-backend:claude-subscription needs portable Claude Code subscription OAuth via either ~/.claude/.credentials.json combined with claudeAiOauth.subscriptionType or CLAUDE_CODE_OAUTH_TOKEN sourced from claude setup-token. It first verifies direct claude -p in Docker, then executes two Gateway CLI-backend turns without retaining Anthropic API-key env vars. This subscription lane disables the Claude MCP/tool and image probes by default, since it taps into the signed-in subscription's usage limits, and Anthropic may alter Claude Agent SDK / claude -p billing and rate-limit behavior without an OpenClaw release.
• Claude and Gemini share the same probe set (text turn, image classification, MCP automations tool call, model-switch continuity) via the flags above, yet none of those probes activate by default; switch each flag on individually as needed.

Live: APNs HTTP/2 proxy reachability

• Test: src/infra/push-apns-http2.live.test.ts
• Goal: route through a local HTTP CONNECT proxy to Apple's sandbox APNs endpoint, dispatch the APNs HTTP/2 validation request, and confirm Apple's real 403 InvalidProviderToken response arrives back through the proxy path.
• Enable:
  • OPENCLAW_LIVE_TEST=1 OPENCLAW_LIVE_APNS_REACHABILITY=1 pnpm test:live src/infra/push-apns-http2.live.test.ts
• Optional timeout:
  • OPENCLAW_LIVE_APNS_TIMEOUT_MS=30000

Live: ACP bind smoke (/acp spawn ... --bind here)

• Test: src/gateway/gateway-acp-bind.live.test.ts
• Goal: exercise the real ACP conversation-bind flow with a live ACP agent:
  • transmit /acp spawn <agent> --bind here
  • bind a synthetic message-channel conversation in place
  • send a standard follow-up on that same conversation
  • confirm the follow-up shows up in the bound ACP session transcript
• Enable:
  • pnpm test:live src/gateway/gateway-acp-bind.live.test.ts
  • OPENCLAW_LIVE_ACP_BIND=1
• Defaults:
  • ACP agents in Docker: claude,codex,gemini
  • ACP agent for direct pnpm test:live ...: claude
  • Synthetic channel: Slack DM-style conversation context
  • ACP backend: acpx
• Overrides:
  • OPENCLAW_LIVE_ACP_BIND_AGENT=claude
  • OPENCLAW_LIVE_ACP_BIND_AGENT=codex
  • OPENCLAW_LIVE_ACP_BIND_AGENT=droid
  • OPENCLAW_LIVE_ACP_BIND_AGENT=gemini
  • OPENCLAW_LIVE_ACP_BIND_AGENT=opencode
  • OPENCLAW_LIVE_ACP_BIND_AGENTS=claude,codex,gemini
  • OPENCLAW_LIVE_ACP_BIND_AGENT_COMMAND='npx -y @agentclientprotocol/claude-agent-acp@<version>'
  • OPENCLAW_LIVE_ACP_BIND_CODEX_MODEL=gpt-5.6-luna
  • OPENCLAW_LIVE_ACP_BIND_OPENCODE_MODEL=opencode/kimi-k2.6
  • OPENCLAW_LIVE_ACP_BIND_IMAGE_PROBE=1 (or on/true/yes) to force the image probe on; any other value forces it off. Runs by default for every agent except opencode.
  • OPENCLAW_LIVE_ACP_BIND_REQUIRE_CRON=1
  • OPENCLAW_LIVE_ACP_BIND_PARENT_MODEL=openai/gpt-5.6-luna
• Notes:
  • This lane relies on the gateway chat.send surface with admin-only synthetic originating-route fields, letting tests attach message-channel context without pretending to deliver externally.
  • When OPENCLAW_LIVE_ACP_BIND_AGENT_COMMAND is not set, the test uses the embedded acpx plugin's built-in agent registry for the chosen ACP harness agent.
  • Bound-session cron MCP creation is best-effort by default, as external ACP harnesses may cancel MCP calls after the bind/image proof has passed; set OPENCLAW_LIVE_ACP_BIND_REQUIRE_CRON=1 to make that post-bind cron probe strict.

Example:

OPENCLAW_LIVE_ACP_BIND=1 \
  OPENCLAW_LIVE_ACP_BIND_AGENT=claude \
  pnpm test:live src/gateway/gateway-acp-bind.live.test.ts

Docker recipe:

pnpm test:docker:live-acp-bind

Single-agent Docker recipes:

pnpm test:docker:live-acp-bind:claude
pnpm test:docker:live-acp-bind:codex
pnpm test:docker:live-acp-bind:droid
pnpm test:docker:live-acp-bind:gemini
pnpm test:docker:live-acp-bind:opencode

Docker notes:

• The Docker runner is located at scripts/test-live-acp-bind-docker.sh.
• In its default configuration, it executes the ACP bind smoke against the aggregate live CLI agents sequentially: claude, codex, then gemini.
• To restrict the matrix, pass OPENCLAW_LIVE_ACP_BIND_AGENTS=claude, OPENCLAW_LIVE_ACP_BIND_AGENTS=codex, OPENCLAW_LIVE_ACP_BIND_AGENTS=droid, OPENCLAW_LIVE_ACP_BIND_AGENTS=gemini, or OPENCLAW_LIVE_ACP_BIND_AGENTS=opencode.
• It places the matching CLI auth material into the container, then installs the requested live CLI (@anthropic-ai/claude-code, @openai/codex, Factory Droid via https://app.factory.ai/cli, @google/gemini-cli, or opencode-ai) when absent. The ACP backend is the embedded acpx/runtime package from the official acpx plugin.
• For the Droid Docker variant, ~/.factory is staged for settings, FACTORY_API_KEY is forwarded, and the API key is mandatory since local Factory OAuth/keyring auth cannot be carried into the container. ACPX's built-in droid exec --output-format acp registry entry is used.
• The OpenCode Docker variant serves as a strict single-agent regression lane. A temporary OPENCODE_CONFIG_CONTENT default model is written from OPENCLAW_LIVE_ACP_BIND_OPENCODE_MODEL (defaulting to opencode/kimi-k2.6).
• Direct acpx CLI calls are only a manual or workaround route for comparing behavior outside the Gateway. The Docker ACP bind smoke exercises OpenClaw's embedded acpx runtime backend.

Live: Codex app-server harness smoke

• Purpose: exercise the plugin-owned Codex harness by going through the standard gateway agent call:
  • load the codex plugin that ships with the bundle
  • pick an OpenAI model via /model <ref> --runtime codex
  • dispatch the first gateway agent turn at the desired thinking level
  • follow up with a second turn on the same OpenClaw session, confirming the app-server thread is able to pick up where it left off
  • execute /codex status and /codex models using that same gateway command route
  • optionally, run two Guardian-supervised escalated shell checks: one harmless command that should pass and one fake-secret upload that should be refused, prompting the agent to ask for clarification
• Test name: src/gateway/gateway-codex-harness.live.test.ts
• Activation flag: OPENCLAW_LIVE_CODEX_HARNESS=1
• Harness default model: openai/gpt-5.6-luna
• Fresh OpenAI API-key selection default: openai/gpt-5.6-sol
• Default thinking: low
• Model override: OPENCLAW_LIVE_CODEX_HARNESS_MODEL=openai/<model>
• Thinking override: OPENCLAW_LIVE_CODEX_HARNESS_THINKING=<level>
• Non-default model effort assertion: OPENCLAW_LIVE_CODEX_HARNESS_EXPECTED_EFFORT=<level>
• Matrix override: OPENCLAW_LIVE_CODEX_HARNESS_TARGETS=<model>=<thinking>,...
• Auth mode: OPENCLAW_LIVE_CODEX_HARNESS_AUTH=codex-auth (default) relies on the copied Codex login; api-key uses OPENAI_API_KEY through the Codex app-server.
• Optional image probe: OPENCLAW_LIVE_CODEX_HARNESS_IMAGE_PROBE=1
• Optional MCP/tool probe: OPENCLAW_LIVE_CODEX_HARNESS_MCP_PROBE=1
• Optional Guardian probe: OPENCLAW_LIVE_CODEX_HARNESS_GUARDIAN_PROBE=1
• Optional resume stress: OPENCLAW_LIVE_CODEX_HARNESS_RESUME_STRESS=1 appends four history turns, then shuts down and restarts the Gateway and Codex app-server three times, each time demanding the same native thread id and conversation history. Adjust the bounded counts with OPENCLAW_LIVE_CODEX_HARNESS_RESUME_STRESS_HISTORY_TURNS (1-20) and OPENCLAW_LIVE_CODEX_HARNESS_RESUME_STRESS_RESTARTS (1-10).
• Optional fan-out stress: enable OPENCLAW_LIVE_CODEX_HARNESS_SUBAGENT_PROBE=1 and OPENCLAW_LIVE_CODEX_HARNESS_SUBAGENT_COUNT (1-12). Every child is launched concurrently by the harness, which then waits for all terminal runs and confirms each distinct child reply and native thread identity.
• Optional compaction stress: OPENCLAW_LIVE_CODEX_HARNESS_COMPACTION_STRESS=1 produces bounded native tool output, forces automatic compaction events, checks the stored compaction count and hidden-marker recall, restarts the Gateway and the physical Codex app-server, then runs the output and compaction cycle again. Tune the bounded work with OPENCLAW_LIVE_CODEX_HARNESS_COMPACTION_STRESS_TURNS (1-8) and OPENCLAW_LIVE_CODEX_HARNESS_LARGE_OUTPUT_BYTES (100000-800000).
• Full direct-API context: OPENCLAW_LIVE_CODEX_HARNESS_FULL_CONTEXT=1 applies the 922000 context and 700000 total automatic-compaction limits, sends dense bounded user turns without /compact or any other manual checkpoint, and expects a later small turn to trigger native automatic compaction. It needs OPENCLAW_LIVE_CODEX_HARNESS_AUTH=api-key plus an absolute OPENCLAW_LIVE_CODEX_HARNESS_MODEL_CATALOG path. The catalog must show the exact selected model with context_window: 922000, max_context_window: 922000, and auto_compact_token_limit: 700000 so Codex does not pull the override back into its usual catalog window. The regular reduced-threshold stress above retains the stricter automatic-compaction and hidden-marker retention checks.
• Optional loop-relay opt-out probe: OPENCLAW_LIVE_CODEX_HARNESS_DISABLE_LOOP_RELAY=1
• The requested thinking preference may translate to the nearest effort Codex advertises for that model. Luna, for instance, maps minimal to low.
• Known Codex catalog models derive that exact native effort on their own. Unknown model overrides must declare the expected mapped effort.
• The smoke forces provider/model agentRuntime.id: "codex" so a broken Codex harness cannot pass by quietly falling back to OpenClaw.
• Auth: Codex app-server auth from the local Codex subscription login, or OPENAI_API_KEY when OPENCLAW_LIVE_CODEX_HARNESS_AUTH=api-key. Docker can copy ~/.codex/auth.json and ~/.codex/config.toml for subscription runs.

Local recipe:

OPENCLAW_LIVE_CODEX_HARNESS=1 \
  OPENCLAW_LIVE_CODEX_HARNESS_IMAGE_PROBE=1 \
  OPENCLAW_LIVE_CODEX_HARNESS_MCP_PROBE=1 \
  OPENCLAW_LIVE_CODEX_HARNESS_GUARDIAN_PROBE=1 \
  OPENCLAW_LIVE_CODEX_HARNESS_MODEL=openai/gpt-5.6-luna \
  pnpm test:live -- src/gateway/gateway-codex-harness.live.test.ts

Docker recipe:

pnpm test:docker:live-codex-harness

Restart and history stress:

OPENCLAW_LIVE_CODEX_HARNESS_RESUME_STRESS=1 \
pnpm test:docker:live-codex-harness

Fan-out, large-output, compaction, and restart stress:

OPENCLAW_LIVE_CODEX_HARNESS_AUTH=api-key \
  OPENCLAW_LIVE_CODEX_HARNESS_SUBAGENT_PROBE=1 \
  OPENCLAW_LIVE_CODEX_HARNESS_SUBAGENT_COUNT=8 \
  OPENCLAW_LIVE_CODEX_HARNESS_RESUME_STRESS=1 \
  OPENCLAW_LIVE_CODEX_HARNESS_COMPACTION_STRESS=1 \
  pnpm test:docker:live-codex-harness

Full native Codex 922000 input-budget compaction stress:

OPENCLAW_LIVE_CODEX_HARNESS=1 \
  OPENCLAW_LIVE_CODEX_HARNESS_AUTH=api-key \
  OPENCLAW_LIVE_CODEX_HARNESS_FULL_CONTEXT=1 \
  OPENCLAW_LIVE_CODEX_HARNESS_MODEL_CATALOG=/absolute/path/to/models-api-1m.json \
  OPENCLAW_LIVE_CODEX_HARNESS_MODEL=openai/gpt-5.6-sol \
  OPENCLAW_LIVE_CODEX_HARNESS_THINKING=low \
  OPENCLAW_LIVE_CODEX_HARNESS_COMPACTION_STRESS_TURNS=8 \
  OPENCLAW_LIVE_CODEX_HARNESS_LARGE_OUTPUT_BYTES=800000 \
  OPENCLAW_LIVE_CODEX_HARNESS_DEBUG=1 \
  node --import tsx scripts/test-live.mts --quiet src/gateway/gateway-codex-harness.live.test.ts

GPT-5.6 native Codex matrix:

OPENCLAW_LIVE_CODEX_HARNESS_AUTH=api-key \
  OPENCLAW_LIVE_CODEX_HARNESS_TARGETS='openai/gpt-5.6-sol=ultra,openai/gpt-5.6-terra=ultra,openai/gpt-5.6-luna=max' \
  pnpm test:docker:live-codex-harness

Live: OpenAI long context

• Objective: confirm exact-model embedded OpenClaw execution through a Gateway isolated per process, push past the long-context pricing threshold, see a first-class OpenAI Responses compaction entry, and verify opaque replay plus prefix pruning on the subsequent request.
• Test: src/gateway/gateway-openai-long-context.live.test.ts
• Enable: OPENCLAW_LIVE_OPENAI_LONG_CONTEXT=1
• Profiles: OPENCLAW_LIVE_OPENAI_LONG_CONTEXT_PROFILE=full picks the exact openai/gpt-5.6-sol with a 1050000 full window, 922000 safe active input, 128000 max output, and 700000 compaction threshold. reduced hits the same transport and persistence route with a smaller budget.
• Metrics: OPENCLAW_LIVE_OPENAI_LONG_CONTEXT_METRICS=1 reports phase timing and token counts. These values are informational, not pass/fail latency checks.
• Long output: OPENCLAW_LIVE_OPENAI_LONG_CONTEXT_OUTPUT=1 demands a deterministic reply between 4000 and 8000 output tokens.
• Optional raw read-tool stress: OPENCLAW_LIVE_OPENAI_LONG_CONTEXT_TOOL_OUTPUT=1. It stays out of the default recipe because the effective tool surface might use Code Mode instead of surfacing the raw read tool.

Full 922000 input-budget recipe:

OPENCLAW_LIVE_OPENAI_LONG_CONTEXT=1 \
  OPENCLAW_LIVE_OPENAI_LONG_CONTEXT_PROFILE=full \
  OPENCLAW_LIVE_OPENAI_LONG_CONTEXT_METRICS=1 \
  OPENCLAW_LIVE_OPENAI_LONG_CONTEXT_OUTPUT=1 \
  node --import tsx scripts/test-live.mts --quiet src/gateway/gateway-openai-long-context.live.test.ts

Reduced-budget recipe:

OPENCLAW_LIVE_OPENAI_LONG_CONTEXT=1 \
  OPENCLAW_LIVE_OPENAI_LONG_CONTEXT_PROFILE=reduced \
  OPENCLAW_LIVE_OPENAI_LONG_CONTEXT_METRICS=1 \
  OPENCLAW_LIVE_OPENAI_LONG_CONTEXT_OUTPUT=1 \
  node --import tsx scripts/test-live.mts --quiet src/gateway/gateway-openai-long-context.live.test.ts

Long-context hard oracles

The full embedded and native recipes serve as proof runs, not throughput tests. They fail unless these runtime contracts hold:

• Runtime and model identity match exactly: embedded OpenClaw or native Codex as requested, both on openai/gpt-5.6-sol.
• At least one provider request exceeds 272000 input tokens and every call reports priority service.
• Embedded OpenClaw receives and stores a first-class encrypted Responses compaction item, replays the exact opaque item on the next request, and trims the earlier input prefix. The encrypted content must never show up in display or diagnostics.
• Native Codex reports an effective window of 875900, grows past the 700000 total-scope threshold without a manual compact, and compacts automatically on the next turn.
• Each runtime produces a deterministic long response between 4000 and 8000 output tokens and keeps a durable marker through compaction and a Gateway restart.

Compaction duration, restart latency, turn latency, and total suite duration are reported as informational metrics only.

Warning

The full modes intentionally cross OpenAI's long-context pricing boundary and issue several large API calls. Above 272000 input tokens, the entire request is 2× input/cache and 1.5× output; Fast/Priority doubles that tier again. Use full mode only with explicit spend approval.

Fresh OpenAI API-key default:

OPENCLAW_LIVE_GATEWAY_OPENAI_API_DEFAULT=1 \
  OPENCLAW_LIVE_GATEWAY_PROVIDERS=openai \
  OPENCLAW_LIVE_GATEWAY_THINKING=off \
  pnpm test:live -- src/gateway/gateway-models.profiles.live.test.ts

This proof leaves OPENCLAW_LIVE_GATEWAY_MODELS unset, resolves the model via the fresh onboarding inference-selection seam, asserts openai/gpt-5.6-sol, and then runs a real gateway turn with that resolved model.

GPT-5.6 embedded OpenClaw matrix:

OPENCLAW_LIVE_GATEWAY_THINKING=ultra \
  OPENCLAW_LIVE_GATEWAY_PROVIDERS=openai \
  OPENCLAW_LIVE_GATEWAY_MODELS='openai/gpt-5.6-sol,openai/gpt-5.6-terra,openai/gpt-5.6-luna' \
  pnpm test:live -- src/gateway/gateway-models.profiles.live.test.ts

Docker notes:

• The Docker runner is located at scripts/test-live-codex-harness-docker.sh.
• It passes OPENAI_API_KEY, copies Codex CLI auth files when present, installs @openai/codex into a writable mounted npm prefix, stages the source tree, then runs only the Codex-harness live test.
• Docker turns on the image, MCP/tool, and Guardian probes by default. Set OPENCLAW_LIVE_CODEX_HARNESS_IMAGE_PROBE=0 or OPENCLAW_LIVE_CODEX_HARNESS_MCP_PROBE=0 or OPENCLAW_LIVE_CODEX_HARNESS_GUARDIAN_PROBE=0 when you need a narrower debug run.
• Docker applies the same explicit Codex runtime config, so legacy aliases or OpenClaw fallback cannot mask a Codex harness regression.
• Matrix targets run sequentially in one container. The Docker script scales its default 35-minute timeout by target count; any outer shell or CI timeout must allow the same total. Canonical CI keeps each GPT-5.6 target in a separate shard.

Recommended live recipes

Narrow, explicit allowlists are fastest and least flaky:

• Single model, direct (no gateway):

  • OPENCLAW_LIVE_MODELS="openai/gpt-5.6-luna" pnpm test:live src/agents/models.profiles.live.test.ts

• Small-model direct profile:

  • OPENCLAW_LIVE_MODELS=small pnpm test:live src/agents/models.profiles.live.test.ts

• Small-model gateway profile:

  • OPENCLAW_LIVE_GATEWAY_MODELS=small pnpm test:live src/gateway/gateway-models.profiles.live.test.ts

• Ollama Cloud API smoke:

  • OPENCLAW_LIVE_TEST=1 OPENCLAW_LIVE_OLLAMA=1 OPENCLAW_LIVE_OLLAMA_BASE_URL=https://ollama.com OPENCLAW_LIVE_OLLAMA_MODEL=glm-5.1:cloud OPENCLAW_LIVE_OLLAMA_WEB_SEARCH=0 pnpm test:live -- extensions/ollama/ollama.live.test.ts

• Single model, gateway smoke:

  • OPENCLAW_LIVE_GATEWAY_MODELS="openai/gpt-5.6-luna" pnpm test:live src/gateway/gateway-models.profiles.live.test.ts

• Tool calling across several providers:

  • OPENCLAW_LIVE_GATEWAY_MODELS="openai/gpt-5.6-luna,anthropic/claude-opus-4-6,google/gemini-3.5-flash,deepseek/deepseek-v4-flash,zai/glm-5.1,minimax/MiniMax-M3" pnpm test:live src/gateway/gateway-models.profiles.live.test.ts

• Z.AI Coding Plan GLM-5.3 direct smoke:

  • ZAI_CODING_LIVE_TEST=1 pnpm test:live src/agents/zai.live.test.ts

• Google focus:

  • Gemini (API key): OPENCLAW_LIVE_GATEWAY_MODELS="google/gemini-3.5-flash" pnpm test:live src/gateway/gateway-models.profiles.live.test.ts

• Google adaptive thinking smoke (qa manual from the private QA CLI - requires OPENCLAW_ENABLE_PRIVATE_QA_CLI=1 and a source checkout; see QA overview):

  • Gemini 3 dynamic default: OPENCLAW_ENABLE_PRIVATE_QA_CLI=1 pnpm openclaw qa manual --provider-mode live-frontier --model google/gemini-3.1-pro-preview --alt-model google/gemini-3.1-pro-preview --message '/think adaptive Reply exactly: GEMINI_ADAPTIVE_OK' --timeout-ms 180000
  • Gemini 2.5 dynamic budget: OPENCLAW_ENABLE_PRIVATE_QA_CLI=1 pnpm openclaw qa manual --provider-mode live-frontier --model google/gemini-2.5-flash --alt-model google/gemini-2.5-flash --message '/think adaptive Reply exactly: GEMINI25_ADAPTIVE_OK' --timeout-ms 180000

Notes:

• google/... uses the Gemini API (API key).
• google-gemini-cli/... uses the local Gemini CLI on your machine (separate auth + tooling quirks).
• google-antigravity/... is not a registered provider or supported setup path. Do not add it to live-test allowlists.
• Gemini API vs Gemini CLI:
  • API: OpenClaw calls Google's hosted Gemini API over HTTP (API key / profile auth); this is what most users mean by "Gemini".
  • CLI: OpenClaw shells out to a local gemini binary; it has its own auth and can behave differently (streaming/tool support/version skew).

Live: model matrix (what we cover)

Live is opt-in, so there is no fixed "CI model list." OPENCLAW_LIVE_MODELS=modern / OPENCLAW_LIVE_GATEWAY_MODELS=modern (and their all alias) run the curated priority list from HIGH_SIGNAL_LIVE_MODEL_PRIORITY in src/agents/live-model-filter.ts, in this priority order:

The hand-picked small-model set (OPENCLAW_LIVE_MODELS=small / OPENCLAW_LIVE_GATEWAY_MODELS=small), sourced from SMALL_LIVE_MODEL_PRIORITY:

Details regarding the modern list:

• Providers codex and codex-cli are left out of the default modern sweep, since they handle CLI-backend/ACP behavior, which gets its own tests above. By default, openai/gpt-5.6 goes through the Codex app-server harness; refer to Live: Codex app-server harness smoke.
• In the modern sweep, fireworks, google, openrouter, and xai run only their explicitly listed model ids, with no automatic expansion to all models from those providers.
• Make sure OPENCLAW_LIVE_GATEWAY_MODELS includes at least one vision-capable model, such as Claude, Gemini, or OpenAI-family variants, so the image probe gets exercised.

Execute a gateway smoke test with tools and image support across a selected cross-provider group:

OPENCLAW_LIVE_GATEWAY_MODELS="openai/gpt-5.6-luna,anthropic/claude-opus-4-6,google/gemini-3.1-pro-preview,google/gemini-3.5-flash,deepseek/deepseek-v4-flash,zai/glm-5.1,minimax/MiniMax-M3" pnpm test:live src/gateway/gateway-models.profiles.live.test.ts

Extra coverage beyond the curated lists is optional (useful to have; choose a model you have enabled that supports tools):

• Mistral: mistral/...
• Cerebras: cerebras/... (if you have access)
• LM Studio: lmstudio/... (local; tool calling depends on API mode)

Aggregators / alternate gateways

With the right keys enabled, you can also run tests through:

• OpenRouter: openrouter/... (hundreds of models; use openclaw models scan to locate candidates that support both tools and images)
• OpenCode: opencode/... for Zen and opencode-go/... for Go (authentication via OPENCODE_API_KEY / OPENCODE_ZEN_API_KEY)

Additional providers you could add to the live matrix (assuming you have the necessary credentials or configuration):

• Plugins shipped by the first-party provider: anthropic, cerebras, github-copilot, google, google-gemini-cli, google-vertex, groq, mistral, openai, openrouter, opencode, opencode-go, xai, zai
• Through models.providers (custom endpoints): minimax (cloud/API), plus any proxy compatible with OpenAI or Anthropic (LM Studio, vLLM, LiteLLM, etc.)

Tip

Avoid writing "all models" directly into docs. The definitive list comes from whatever discoverModels(...) outputs on your system, combined with the keys that are present.

Credentials (never commit)

Live tests locate credentials using the same mechanism as the CLI. What that means in practice:

• When the CLI functions, the live tests should pick up identical keys.

• A "no creds" message from a live test calls for the same troubleshooting steps you would apply to openclaw models list / model selection.

• Auth profiles scoped per agent: ~/.openclaw/agents/<agentId>/agent/auth-profiles.json (this is what "profile keys" refers to in the live tests)

• Configuration: ~/.openclaw/openclaw.json (or OPENCLAW_CONFIG_PATH)

• Legacy OAuth directory: ~/.openclaw/credentials/ (copied into the staged live home when it exists, but not treated as the primary profile-key store)

• For local live runs, the active config (with agents.*.workspace / agentDir overrides removed) and each agent's auth-profiles.json are copied, not the remainder of that agent's folder, so workspace/ and sandboxes/ data never ends up in the staged home, along with the legacy credentials/ directory and supported external CLI auth files/dirs (.claude.json, .claude/.credentials.json, .claude/settings*.json, .claude/backups, .codex/auth.json, .codex/config.toml, .gemini, .minimax) into a temporary test home.

To depend on environment keys, set them before running local tests, or use the Docker runners listed below with an explicit OPENCLAW_PROFILE_FILE.

Deepgram live (audio transcription)

• Test: extensions/deepgram/audio.live.test.ts
• Enable: DEEPGRAM_API_KEY=... DEEPGRAM_LIVE_TEST=1 pnpm test:live extensions/deepgram/audio.live.test.ts

BytePlus coding plan live

• Test: extensions/byteplus/live.test.ts
• Enable: BYTEPLUS_API_KEY=... BYTEPLUS_LIVE_TEST=1 pnpm test:live extensions/byteplus/live.test.ts
• Optional model override: BYTEPLUS_CODING_MODEL=ark-code-latest

ComfyUI workflow media live

• Test: extensions/comfy/comfy.live.test.ts
• Enable: OPENCLAW_LIVE_TEST=1 COMFY_LIVE_TEST=1 pnpm test:live -- extensions/comfy/comfy.live.test.ts
• Scope:
  • Covers the comfy image, video, and music_generate paths
  • Omits each capability unless plugins.entries.comfy.config.<capability> is set up
  • Helpful after modifying comfy workflow submission, polling, downloads, or plugin registration

Image generation live

• Test: test/image-generation.runtime.live.test.ts
• Command: pnpm test:live test/image-generation.runtime.live.test.ts
• Harness: pnpm test:live:media image
• Scope:
  • Lists all image-generation provider plugins that are currently registered
  • Checks provider env vars that were already exported before any probing
  • Defaults to live/env API keys over stored auth profiles, so outdated test keys in auth-profiles.json won't hide actual shell credentials
  • Omits providers lacking a valid auth/profile/model
  • Sends each configured provider through the shared image-generation runtime:
    • <provider>:generate
    • <provider>:edit if the provider indicates edit support
• Bundled providers currently included:
  • deepinfra
  • fal
  • google
  • minimax
  • openai
  • openrouter
  • vydra
  • xai
• Filtering options:
  • OPENCLAW_LIVE_IMAGE_GENERATION_PROVIDERS="openai,google,openrouter,xai"
  • OPENCLAW_LIVE_IMAGE_GENERATION_PROVIDERS="deepinfra"
  • OPENCLAW_LIVE_IMAGE_GENERATION_MODELS="openai/gpt-image-2,google/gemini-3.1-flash-image,openrouter/google/gemini-3.1-flash-image-preview,xai/grok-imagine-image"
  • OPENCLAW_LIVE_IMAGE_GENERATION_CASES="google:flash-generate,google:pro-edit,openrouter:generate,xai:default-generate,xai:default-edit"
• Auth-related options:
  • OPENCLAW_LIVE_REQUIRE_PROFILE_KEYS=1 to enforce profile-store auth, bypassing env-only overrides

For the shipped CLI path, add an infer smoke once the provider/runtime live test succeeds:

OPENCLAW_LIVE_TEST=1 OPENCLAW_LIVE_INFER_CLI_TEST=1 pnpm test:live -- test/image-generation.infer-cli.live.test.ts
openclaw infer image providers --json
openclaw infer image generate \
  --model google/gemini-3.1-flash-image \
  --prompt "Minimal flat test image: one blue square on a white background, no text." \
  --output ./openclaw-infer-image-smoke.png \
  --json

This exercises CLI argument parsing, config/default-agent resolution, bundled plugin activation, the shared image-generation runtime, and the live provider request. Plugin dependencies must be present before runtime load.

Music generation live

• Test: extensions/music-generation-providers.live.test.ts
• Enable: OPENCLAW_LIVE_TEST=1 pnpm test:live -- extensions/music-generation-providers.live.test.ts
• Harness: pnpm test:live:media music
• Scope:
  • Tests the shared bundled music-generation provider path
  • Now includes fal, google, minimax, and openrouter
  • Checks provider env vars that were already exported before any probing
  • Defaults to live/env API keys over stored auth profiles, so outdated test keys in auth-profiles.json won't hide actual shell credentials
  • Omits providers lacking a valid auth/profile/model
  • Runs both declared runtime modes when available:
    • generate with prompt-only input
    • edit when the provider declares capabilities.edit.enabled
  • comfy is handled in its own separate live file, not this shared sweep
• Filtering options:
  • OPENCLAW_LIVE_MUSIC_GENERATION_PROVIDERS="google,minimax"
  • OPENCLAW_LIVE_MUSIC_GENERATION_MODELS="google/lyria-3-clip-preview,minimax/music-2.6"
• Auth-related options:
  • OPENCLAW_LIVE_REQUIRE_PROFILE_KEYS=1 to enforce profile-store auth, bypassing env-only overrides

Video generation live

• Test: extensions/video-generation-providers.live.test.ts
• Enable: OPENCLAW_LIVE_TEST=1 pnpm test:live -- extensions/video-generation-providers.live.test.ts
• Harness: pnpm test:live:media video
• Scope:
  • Covers the shared bundled video-generation provider path across alibaba, byteplus, deepinfra, fal, google, minimax, openai, openrouter, pixverse, qwen, runway, together, vydra, xai
  • Uses the release-safe smoke path by default: one text-to-video request per provider, a one-second lobster prompt, and a per-provider operation cap sourced from OPENCLAW_LIVE_VIDEO_GENERATION_TIMEOUT_MS (defaulting to 180000)
  • FAL is excluded by default since provider-side queue latency can dominate release time; pass OPENCLAW_LIVE_VIDEO_GENERATION_PROVIDERS="fal" (or clear the skip list) to include it explicitly
  • Checks already-exported provider env vars before probing
  • Prefers live/env API keys over stored auth profiles by default, so stale test keys in auth-profiles.json do not hide real shell credentials
  • Omits providers lacking usable auth/profile/model
  • Executes only generate by default
  • Set OPENCLAW_LIVE_VIDEO_GENERATION_FULL_MODES=1 to also run declared transform modes when available:
    • imageToVideo when the provider declares capabilities.imageToVideo.enabled and the selected provider/model accepts buffer-backed local image input in the shared sweep
    • videoToVideo when the provider declares capabilities.videoToVideo.enabled and the selected provider/model accepts buffer-backed local video input in the shared sweep
  • Current declared-but-skipped imageToVideo provider in the shared sweep:
    • vydra (buffer-backed local image input is not supported in this lane)
  • Provider-specific Vydra coverage:
    • OPENCLAW_LIVE_TEST=1 OPENCLAW_LIVE_VYDRA_VIDEO=1 pnpm test:live -- extensions/vydra/vydra.live.test.ts
    • That file runs veo3 text-to-video plus a kling image-to-video lane that uses a remote image URL fixture by default (OPENCLAW_LIVE_VYDRA_KLING_IMAGE_URL to override).
  • Provider-specific xAI coverage:
    • OPENCLAW_LIVE_TEST=1 OPENCLAW_LIVE_XAI_VIDEO=1 pnpm test:live -- extensions/xai/xai.live.test.ts -t "classic Grok Imagine"
    • The classic case generates a square local PNG first frame, omits geometry, requests a one-second image-to-video clip, polls to completion, and verifies the downloaded buffer.
    • OPENCLAW_LIVE_TEST=1 OPENCLAW_LIVE_XAI_VIDEO=1 pnpm test:live -- extensions/xai/xai.live.test.ts -t "Grok Imagine Video 1.5"
    • The 1.5 case generates a local PNG first frame, requests a one-second 1080P image-to-video clip, polls to completion, and verifies the downloaded buffer.
  • Current videoToVideo live coverage:
    • runway only when the selected model resolves to gen4_aleph
  • Current declared-but-skipped videoToVideo providers in the shared sweep:
    • alibaba, google, openai, qwen, xai because those paths currently require remote http(s) reference URLs rather than buffer-backed local input
• Optional narrowing:
  • OPENCLAW_LIVE_VIDEO_GENERATION_PROVIDERS="deepinfra,google,openai,runway"
  • OPENCLAW_LIVE_VIDEO_GENERATION_MODELS="google/veo-3.1-fast-generate-preview,openai/sora-2,runway/gen4_aleph"
  • OPENCLAW_LIVE_VIDEO_GENERATION_SKIP_PROVIDERS="" to include every provider in the default sweep, including FAL
  • OPENCLAW_LIVE_VIDEO_GENERATION_TIMEOUT_MS=60000 to reduce each provider operation cap for an aggressive smoke run
• Optional auth behavior:
  • OPENCLAW_LIVE_REQUIRE_PROFILE_KEYS=1 to force profile-store auth and ignore env-only overrides

Media live harness

• Command: pnpm test:live:media
• Entrypoint: test/e2e/qa-lab/media/hosted-media-provider-live.ts, which invokes pnpm test:live -- <suite-test-file> for every suite that is selected, keeping heartbeat and quiet-mode settings aligned with other pnpm test:live executions.
• Purpose:
  • Executes the combined image, music, and video live suites using a single entrypoint native to the repository
  • Pulls in any absent provider environment variables automatically from ~/.profile
  • By default, restricts each suite to providers with currently valid authentication
• Flags:
  • --providers <csv> acts as a global provider filter; --image-providers / --music-providers / --video-providers limit a filter to a specific suite
  • --all-providers bypasses the authentication-based auto-filter
  • --allow-empty terminates with code 0 if filtering results in no providers that can run
  • --quiet / --no-quiet are forwarded directly to test:live
• Examples:
  • pnpm test:live:media
  • pnpm test:live:media image video --providers openai,google,minimax
  • pnpm test:live:media video --video-providers openai,runway --all-providers
  • pnpm test:live:media music --quiet

Related

• Testing - covers unit, integration, QA, and Docker suites