Live Network-Touching Tests for OpenClaw Gateway

For quick setup, QA runners, unit/integration suites, and Docker workflows, refer to Testing. This page documents live tests that interact with the network: model matrix, CLI backends, ACP, media providers, and credential handling.

Live tests vs your real gateway

Live suites and ad hoc smoke tests must never interfere with a gateway currently handling production traffic, whether yours or another operator's:

• Provide your own gateway: use the in-process gateway (Layer 2 below) or launch a dev instance with an isolated state directory (OPENCLAW_STATE_DIR=<scratch>) and an available port. Do not bind the default gateway port (18789) if a real gateway is already using it.
• Do not run openclaw gateway stop/restart (or launchctl/systemctl/tmux equivalents) on a service you did not start during this session, as that belongs to the operator's live instance. Obtain explicit permission first.
• Need realistic data? Copy the live state or database into your dev state directory and test against the copy. Running in-place migrations on a live gateway's state also requires explicit approval.

Live: local smoke commands

Export the necessary provider key into the process environment before running ad hoc live checks.

Safe media smoke test:

pnpm openclaw infer tts convert --local --json \
  --text "OpenClaw live smoke." \
  --output /tmp/openclaw-live-smoke.mp3

Safe voice-call readiness smoke test:

pnpm openclaw voicecall setup --json
pnpm openclaw voicecall smoke --to "+15555550123"

voicecall smoke acts as a dry run unless --yes is also provided; use --yes only when you plan to place an actual call. For Twilio, Telnyx, and Plivo, a successful readiness check requires a public webhook URL. Local or private loopback URLs are rejected because those providers cannot reach them.

Live: Android node capability sweep

• Test: src/gateway/android-node.capabilities.live.test.ts
• Script: pnpm android:test:integration
• Goal: execute every command currently advertised by a connected Android node and verify command contract behavior.
• Scope:
  • Preconditioned or manual setup (the suite does not install, run, or pair the app).
  • Command-by-command gateway node.invoke validation for the selected Android node.
• Required pre-setup:
  • Android app already connected and paired to the gateway.
  • App kept in the foreground.
  • Permissions and capture consent granted for capabilities you expect to pass.
• Optional target overrides:
  • OPENCLAW_ANDROID_NODE_ID or OPENCLAW_ANDROID_NODE_NAME.
  • OPENCLAW_ANDROID_GATEWAY_URL / OPENCLAW_ANDROID_GATEWAY_TOKEN / OPENCLAW_ANDROID_GATEWAY_PASSWORD.
• Full Android setup details: Android App

Live: model smoke (profile keys)

Live model tests are divided into two layers so failures remain isolated:

• "Direct model" indicates whether the provider or model can respond at all with the given key.
• "Gateway smoke" indicates whether the full gateway and agent pipeline works for that model, including sessions, history, tools, sandbox policy, and more.

The curated model lists below reside in src/agents/live-model-filter.ts and evolve over time. Treat the arrays there as authoritative, not this page.

MiniMax M3 uses minimax/MiniMax-M3 as its default provider or model reference.

Layer 1: Direct model completion (no gateway)

• Test: src/agents/models.profiles.live.test.ts
• Goal:
  • List discovered models
  • Use getApiKeyForModel to select models you have credentials for
  • Run a small completion per model, including targeted regressions where needed
• How to enable:
  • pnpm test:live (or OPENCLAW_LIVE_TEST=1 if invoking Vitest directly)
  • Set OPENCLAW_LIVE_MODELS=modern, small, or all (alias for modern) to actually execute this suite. Otherwise it skips, so pnpm test:live alone stays focused on gateway smoke.
• How to select models:
  • OPENCLAW_LIVE_MODELS=modern runs the curated high-signal priority list (see Live: model matrix)
  • OPENCLAW_LIVE_MODELS=small runs the curated small-model priority list
  • OPENCLAW_LIVE_MODELS=all is an alias for modern
  • Or OPENCLAW_LIVE_MODELS="openai/gpt-5.6-luna,anthropic/claude-opus-4-6,..." (comma-separated allowlist)
  • Local Ollama small-model runs default to http://127.0.0.1:11434; set OPENCLAW_LIVE_OLLAMA_BASE_URL only for LAN, custom, or Ollama Cloud endpoints.
  • Modern/all and small sweeps default to their curated-list length as a cap. Set OPENCLAW_LIVE_MAX_MODELS=0 for an exhaustive sweep of the selected profile, or a positive number for a smaller cap.
  • Exhaustive sweeps use OPENCLAW_LIVE_TEST_TIMEOUT_MS for the entire direct-model test timeout. Default: 60 minutes.
  • Direct-model probes run with 20-way parallelism by default. Set OPENCLAW_LIVE_MODEL_CONCURRENCY to override.
• How to select providers:
  • OPENCLAW_LIVE_PROVIDERS="google,google-antigravity,google-gemini-cli" (comma-separated allowlist)
• Where keys come from:
  • By default: profile store and environment fallbacks
  • Set OPENCLAW_LIVE_REQUIRE_PROFILE_KEYS=1 to enforce profile store only
• Why this exists:
  • Separates "provider API is broken or key is invalid" from "gateway agent pipeline is broken"
  • Contains small, isolated regressions, for example OpenAI Responses or Codex Responses reasoning replay and tool-call flows

Layer 2: Gateway + dev agent smoke (what "@openclaw" actually does)

• Test: src/gateway/gateway-models.profiles.live.test.ts
• Purpose:
  • Start an in-process gateway
  • Create or update an agent:dev:* session (model override applied per run)
  • Step through models with keys and verify:
    • A "meaningful" reply appears (no tools used)
    • A real tool call works (read probe)
    • Optional extra tool probes (exec plus read probe)
    • OpenAI regression scenarios (tool call only then follow up) continue to function
• Probe breakdown (to help you diagnose failures quickly):
  • read probe: the test places a nonce file in the workspace and instructs the agent to read it and return the nonce.
  • exec+read probe: the test directs the agent to exec a nonce into a temporary file, then read it back.
  • Image probe: the test attaches a generated PNG (containing "CAT" plus a random code) and expects the model to return cat <CODE>.
  • Implementation reference: src/gateway/gateway-models.profiles.live.test.ts and test/helpers/live-image-probe.ts.
• How to turn on:
  • pnpm test:live (or OPENCLAW_LIVE_TEST=1 when calling Vitest directly)
• How to choose models:
  • Default: the curated high signal (modern) priority list is used
  • OPENCLAW_LIVE_GATEWAY_MODELS=small runs the curated small model list through the full gateway and agent pipeline
  • OPENCLAW_LIVE_GATEWAY_MODELS=all is a shorthand for modern
  • Alternatively, set OPENCLAW_LIVE_GATEWAY_MODELS="provider/model" (or a comma separated list) to restrict the selection
  • The modern/all and small gateway sweeps cap at the length of their curated list by default; set OPENCLAW_LIVE_GATEWAY_MAX_MODELS=0 for a full selected sweep or a positive number for a smaller cap.
• How to pick providers (to avoid routing everything through OpenRouter):
  • OPENCLAW_LIVE_GATEWAY_PROVIDERS="google,google-antigravity,google-gemini-cli,openai,anthropic,zai,minimax" (comma separated allowlist)
• Tool and image probes are always active in this live test:
  • read probe plus exec+read probe (tool stress test)
  • The image probe runs when the model advertises support for image input
  • High level flow:
    • The test creates a small PNG with "CAT" and a random code (test/helpers/live-image-probe.ts)
    • It sends the image via agent attachments: [{ mimeType: "image/png", content: "<base64>" }]
    • The gateway converts attachments into images[] (src/gateway/server-methods/agent.ts plus src/gateway/chat-attachments.ts)
    • The embedded agent forwards a multimodal user message to the model
    • Assertion: the reply contains cat and the code (OCR tolerance allows minor errors)

Tip

To discover what you can test on your machine and see the exact provider/model IDs, run:

openclaw models list
openclaw models list --json

Live: CLI backend smoke (Claude, Gemini, or other local CLIs)

• Test: src/gateway/gateway-cli-backend.live.test.ts
• Purpose: verify the Gateway and agent pipeline using a local CLI backend, without altering your default configuration.
• Backend specific smoke defaults are stored within the owning plugin's cli-backend.ts definition.
• Enable:
  • pnpm test:live (or OPENCLAW_LIVE_TEST=1 when calling Vitest directly)
  • OPENCLAW_LIVE_CLI_BACKEND=1
• Defaults:
  • Default provider and model: claude-cli/claude-sonnet-4-6
  • Command, arguments, and image behavior are taken from the owning CLI backend plugin metadata.
• Optional overrides:
  • OPENCLAW_LIVE_CLI_BACKEND_MODEL="claude-cli/claude-sonnet-4-6"
  • OPENCLAW_LIVE_CLI_BACKEND_COMMAND="/full/path/to/claude"
  • OPENCLAW_LIVE_CLI_BACKEND_ARGS='["-p","--output-format","json"]'
  • OPENCLAW_LIVE_CLI_BACKEND_IMAGE_PROBE=1 to attach a real image (file paths are inserted into the prompt). Disabled by default in Docker recipes.
  • OPENCLAW_LIVE_CLI_BACKEND_IMAGE_ARG="--image" to supply image file paths as CLI arguments rather than injecting them into the prompt.
  • OPENCLAW_LIVE_CLI_BACKEND_IMAGE_MODE="repeat" (or "list") to control how image arguments are passed when IMAGE_ARG is set.
  • OPENCLAW_LIVE_CLI_BACKEND_RESUME_PROBE=1 to send a second turn and test the resume flow.
  • OPENCLAW_LIVE_CLI_BACKEND_MODEL_SWITCH_PROBE=1 to enable the Claude Sonnet to Opus same session continuity probe when the chosen model supports a switch target. Disabled by default, including in Docker recipes.
  • OPENCLAW_LIVE_CLI_BACKEND_MCP_PROBE=1 to enable the MCP or tool loopback probe. Disabled by default in Docker recipes.

Example:

  OPENCLAW_LIVE_CLI_BACKEND=1 \
  OPENCLAW_LIVE_CLI_BACKEND_MODEL="claude-cli/claude-sonnet-4-6" \
  pnpm test:live src/gateway/gateway-cli-backend.live.test.ts

Cheap Gemini MCP configuration smoke test:

OPENCLAW_LIVE_TEST=1 \
  pnpm test:live src/agents/cli-runner/bundle-mcp.gemini.live.test.ts

This does not ask Gemini to produce a response. It writes the same system settings that OpenClaw provides to Gemini, then runs gemini --debug mcp list to confirm that a saved transport: "streamable-http" server is normalized to Gemini's HTTP MCP format and can connect to a local streamable HTTP MCP server.

Docker recipe:

pnpm test:docker:live-cli-backend

Single provider Docker recipes:

pnpm test:docker:live-cli-backend:claude
pnpm test:docker:live-cli-backend:claude-subscription
pnpm test:docker:live-cli-backend:gemini

Notes:

• The Docker runner is located at scripts/test-live-cli-backend-docker.sh.
• Inside the repo Docker image, it executes the live CLI-backend smoke as the non‑root node user.
• It retrieves CLI smoke metadata from the owning plugin, then installs the correct Linux CLI package (@anthropic-ai/claude-code or @google/gemini-cli) into a cached writable prefix at OPENCLAW_DOCKER_CLI_TOOLS_DIR, which defaults to ~/.cache/openclaw/docker-cli-tools.
• codex-cli is deprecated as a bundled CLI backend; replace it with openai/* using the Codex app‑server runtime (refer to Live: Codex app-server harness smoke).
• pnpm test:docker:live-cli-backend:claude-subscription needs a portable Claude Code subscription OAuth, obtained either through ~/.claude/.credentials.json with claudeAiOauth.subscriptionType or via CLAUDE_CODE_OAUTH_TOKEN from claude setup-token. It first verifies direct claude -p inside Docker, then performs two Gateway CLI‑backend rounds without retaining Anthropic API‑key environment variables. This subscription path disables the Claude MCP/tool and image probes by default, because it draws on the signed‑in subscription’s usage quotas, and Anthropic may alter Claude Agent SDK / claude -p billing and rate‑limiting behaviour without an OpenClaw release.
• Claude and Gemini both support the same set of probes (text turn, image classification, MCP cron tool call, model‑switch continuity) via the flags listed above, but none of those probes are active by default. Enable them per flag as needed.

Live: APNs HTTP/2 proxy reachability

• Test: src/infra/push-apns-http2.live.test.ts
• Goal: route through a local HTTP CONNECT proxy to Apple’s sandbox APNs endpoint, transmit the APNs HTTP/2 validation request, and confirm that Apple’s actual 403 InvalidProviderToken response comes back through the proxy path.
• Enable:
  • OPENCLAW_LIVE_TEST=1 OPENCLAW_LIVE_APNS_REACHABILITY=1 pnpm test:live src/infra/push-apns-http2.live.test.ts
• Optional timeout:
  • OPENCLAW_LIVE_APNS_TIMEOUT_MS=30000

Live: ACP bind smoke (/acp spawn ... --bind here)

• Test: src/gateway/gateway-acp-bind.live.test.ts
• Goal: confirm the real ACP conversation‑bind flow with a live ACP agent:
  • send /acp spawn <agent> --bind here
  • bind a synthetic message‑channel conversation in place
  • send a normal follow‑up on that same conversation
  • check that the follow‑up appears in the bound ACP session transcript
• Enable:
  • pnpm test:live src/gateway/gateway-acp-bind.live.test.ts
  • OPENCLAW_LIVE_ACP_BIND=1
• Defaults:
  • ACP agents in Docker: claude,codex,gemini
  • ACP agent for direct pnpm test:live ...: claude
  • Synthetic channel: Slack DM‑style conversation context
  • ACP backend: acpx
• Overrides:
  • OPENCLAW_LIVE_ACP_BIND_AGENT=claude
  • OPENCLAW_LIVE_ACP_BIND_AGENT=codex
  • OPENCLAW_LIVE_ACP_BIND_AGENT=droid
  • OPENCLAW_LIVE_ACP_BIND_AGENT=gemini
  • OPENCLAW_LIVE_ACP_BIND_AGENT=opencode
  • OPENCLAW_LIVE_ACP_BIND_AGENTS=claude,codex,gemini
  • OPENCLAW_LIVE_ACP_BIND_AGENT_COMMAND='npx -y @agentclientprotocol/claude-agent-acp@<version>'
  • OPENCLAW_LIVE_ACP_BIND_CODEX_MODEL=gpt-5.6-luna
  • OPENCLAW_LIVE_ACP_BIND_OPENCODE_MODEL=opencode/kimi-k2.6
  • OPENCLAW_LIVE_ACP_BIND_IMAGE_PROBE=1 (or on/true/yes) to force the image probe on; any other value forces it off. It runs by default for every agent except opencode.
  • OPENCLAW_LIVE_ACP_BIND_REQUIRE_CRON=1
  • OPENCLAW_LIVE_ACP_BIND_PARENT_MODEL=openai/gpt-5.6-luna
• Notes:
  • This lane uses the gateway chat.send surface with admin‑only synthetic originating‑route fields, so tests can attach message‑channel context without simulating external delivery.
  • When OPENCLAW_LIVE_ACP_BIND_AGENT_COMMAND is not set, the test relies on the embedded acpx plugin’s built‑in agent registry for the chosen ACP harness agent.
  • Bound‑session cron MCP creation is best‑effort by default, because external ACP harnesses may cancel MCP calls after the bind/image proof has succeeded. Set OPENCLAW_LIVE_ACP_BIND_REQUIRE_CRON=1 to make that post‑bind cron probe strict.

Example:

OPENCLAW_LIVE_ACP_BIND=1 \
  OPENCLAW_LIVE_ACP_BIND_AGENT=claude \
  pnpm test:live src/gateway/gateway-acp-bind.live.test.ts

Docker recipe:

pnpm test:docker:live-acp-bind

Single-agent Docker recipes:

pnpm test:docker:live-acp-bind:claude
pnpm test:docker:live-acp-bind:codex
pnpm test:docker:live-acp-bind:droid
pnpm test:docker:live-acp-bind:gemini
pnpm test:docker:live-acp-bind:opencode

Docker notes:

• The Docker runner can be found at scripts/test-live-acp-bind-docker.sh.
• Its default behavior is to execute the ACP bind smoke test against the aggregate live CLI agents in a specific order: claude, codex, and finally gemini.
• To reduce the matrix, apply OPENCLAW_LIVE_ACP_BIND_AGENTS=claude, OPENCLAW_LIVE_ACP_BIND_AGENTS=codex, OPENCLAW_LIVE_ACP_BIND_AGENTS=droid, OPENCLAW_LIVE_ACP_BIND_AGENTS=gemini, or OPENCLAW_LIVE_ACP_BIND_AGENTS=opencode.
• It places the necessary CLI authentication material into the container, then installs the required live CLI (@anthropic-ai/claude-code, @openai/codex, Factory Droid through https://app.factory.ai/cli, @google/gemini-cli, or opencode-ai) if it is not already present. The ACP backend itself is the embedded acpx/runtime package from the official acpx plugin.
• For the Droid Docker variant, ~/.factory is staged for configuration, FACTORY_API_KEY is forwarded, and an API key is mandatory because local Factory OAuth or keyring authentication cannot be transferred into the container. It uses the built-in droid exec --output-format acp registry entry from ACPX.
• The OpenCode Docker variant operates as a strict single-agent regression lane. It creates a temporary OPENCODE_CONFIG_CONTENT default model from OPENCLAW_LIVE_ACP_BIND_OPENCODE_MODEL (the default is opencode/kimi-k2.6).
• Direct acpx CLI invocations are only a manual or workaround method for comparing behavior outside the Gateway. The Docker ACP bind smoke test validates the embedded acpx runtime backend of OpenClaw.

Live: Codex app-server harness smoke

• Purpose: confirm that the plugin-owned Codex harness operates correctly through the standard gateway agent process:
  • load the packaged codex plugin
  • pick an OpenAI model via /model <ref> --runtime codex
  • dispatch an initial gateway agent turn at the specified thinking level
  • issue a second turn to the same OpenClaw session and confirm the app-server thread can continue
  • execute /codex status and /codex models along the same gateway command route
  • optionally perform two Guardian-checked escalated shell tests: one harmless command that must be accepted and one simulated secret upload that must be refused so the agent requests confirmation
• Test identifier: src/gateway/gateway-codex-harness.live.test.ts
• Activation flag: OPENCLAW_LIVE_CODEX_HARNESS=1
• Harness default model: openai/gpt-5.6-luna
• Fresh OpenAI API key selection default: openai/gpt-5.6
• Default thinking level: low
• Model override: OPENCLAW_LIVE_CODEX_HARNESS_MODEL=openai/<model>
• Thinking override: OPENCLAW_LIVE_CODEX_HARNESS_THINKING=<level>
• Non-default model effort assertion: OPENCLAW_LIVE_CODEX_HARNESS_EXPECTED_EFFORT=<level>
• Matrix override: OPENCLAW_LIVE_CODEX_HARNESS_TARGETS=<model>=<thinking>,...
• Authentication mode: OPENCLAW_LIVE_CODEX_HARNESS_AUTH=codex-auth (default) relies on the copied Codex login; api-key uses OPENAI_API_KEY through the Codex app-server.
• Optional image probe: OPENCLAW_LIVE_CODEX_HARNESS_IMAGE_PROBE=1
• Optional MCP or tool probe: OPENCLAW_LIVE_CODEX_HARNESS_MCP_PROBE=1
• Optional Guardian probe: OPENCLAW_LIVE_CODEX_HARNESS_GUARDIAN_PROBE=1
• Optional resume stress: OPENCLAW_LIVE_CODEX_HARNESS_RESUME_STRESS=1 inserts four history turns, then shuts down and restarts the Gateway and Codex app-server three times while demanding the same native thread ID and conversation history. Adjust the bounded counts with OPENCLAW_LIVE_CODEX_HARNESS_RESUME_STRESS_HISTORY_TURNS (1-20) and OPENCLAW_LIVE_CODEX_HARNESS_RESUME_STRESS_RESTARTS (1-10).
• Optional fan-out stress: set OPENCLAW_LIVE_CODEX_HARNESS_SUBAGENT_PROBE=1 and OPENCLAW_LIVE_CODEX_HARNESS_SUBAGENT_COUNT (1-12). The harness launches every child at the same time, waits for all terminal runs to finish, and checks each distinct child response and native thread identity.
• Optional compaction stress: OPENCLAW_LIVE_CODEX_HARNESS_COMPACTION_STRESS=1 produces bounded native tool output, forces automatic compaction events, confirms the stored compaction count and hidden-marker recall, restarts the Gateway and physical Codex app-server, then repeats the output and compaction cycle. Tune the bounded work with OPENCLAW_LIVE_CODEX_HARNESS_COMPACTION_STRESS_TURNS (1-8) and OPENCLAW_LIVE_CODEX_HARNESS_LARGE_OUTPUT_BYTES (100000-800000).
• Full direct-API context: OPENCLAW_LIVE_CODEX_HARNESS_FULL_CONTEXT=1 applies the 922000 context and 700000 total compaction limits, sends dense bounded user turns, runs two explicit native compaction checkpoints per cycle, and proceeds with later turns after each checkpoint. It requires OPENCLAW_LIVE_CODEX_HARNESS_AUTH=api-key plus an absolute OPENCLAW_LIVE_CODEX_HARNESS_MODEL_CATALOG path. The catalog must expose the chosen model with max_context_window: 922000 so Codex does not restrict the override back to its normal catalog window. The standard reduced-threshold stress above keeps the stricter automatic-compaction and hidden-marker retention checks.
• Optional loop-relay opt-out probe: OPENCLAW_LIVE_CODEX_HARNESS_DISABLE_LOOP_RELAY=1
• The requested thinking preference may map to the closest effort that Codex advertises for that model. For example, Luna maps minimal to low.
• Known Codex catalog models derive that exact native effort automatically. Unknown model overrides must specify the expected mapped effort.
• The smoke forces provider or model agentRuntime.id: "codex" so a broken Codex harness cannot pass silently by falling back to OpenClaw.
• Auth: Codex app-server authentication from the local Codex subscription login, or OPENAI_API_KEY when OPENCLAW_LIVE_CODEX_HARNESS_AUTH=api-key. Docker can copy ~/.codex/auth.json and ~/.codex/config.toml for subscription runs.

Local recipe:

OPENCLAW_LIVE_CODEX_HARNESS=1 \
  OPENCLAW_LIVE_CODEX_HARNESS_IMAGE_PROBE=1 \
  OPENCLAW_LIVE_CODEX_HARNESS_MCP_PROBE=1 \
  OPENCLAW_LIVE_CODEX_HARNESS_GUARDIAN_PROBE=1 \
  OPENCLAW_LIVE_CODEX_HARNESS_MODEL=openai/gpt-5.6-luna \
  pnpm test:live -- src/gateway/gateway-codex-harness.live.test.ts

Docker recipe:

pnpm test:docker:live-codex-harness

Restart and history stress:

OPENCLAW_LIVE_CODEX_HARNESS_RESUME_STRESS=1 \
pnpm test:docker:live-codex-harness

Fan-out, large-output, compaction, and restart stress:

OPENCLAW_LIVE_CODEX_HARNESS_AUTH=api-key \
  OPENCLAW_LIVE_CODEX_HARNESS_SUBAGENT_PROBE=1 \
  OPENCLAW_LIVE_CODEX_HARNESS_SUBAGENT_COUNT=8 \
  OPENCLAW_LIVE_CODEX_HARNESS_RESUME_STRESS=1 \
  OPENCLAW_LIVE_CODEX_HARNESS_COMPACTION_STRESS=1 \
  pnpm test:docker:live-codex-harness

Full native Codex 922000 input-budget compaction stress:

OPENCLAW_LIVE_CODEX_HARNESS=1 \
  OPENCLAW_LIVE_CODEX_HARNESS_AUTH=api-key \
  OPENCLAW_LIVE_CODEX_HARNESS_FULL_CONTEXT=1 \
  OPENCLAW_LIVE_CODEX_HARNESS_MODEL_CATALOG=/absolute/path/to/models-api-1m.json \
  OPENCLAW_LIVE_CODEX_HARNESS_MODEL=openai/gpt-5.6-terra \
  OPENCLAW_LIVE_CODEX_HARNESS_THINKING=medium \
  OPENCLAW_LIVE_CODEX_HARNESS_COMPACTION_STRESS_TURNS=8 \
  OPENCLAW_LIVE_CODEX_HARNESS_LARGE_OUTPUT_BYTES=800000 \
  pnpm test:live -- src/gateway/gateway-codex-harness.live.test.ts

GPT-5.6 native Codex matrix:

OPENCLAW_LIVE_CODEX_HARNESS_AUTH=api-key \
  OPENCLAW_LIVE_CODEX_HARNESS_TARGETS='openai/gpt-5.6-sol=ultra,openai/gpt-5.6-terra=ultra,openai/gpt-5.6-luna=max' \
  pnpm test:docker:live-codex-harness

Live: OpenAI repeated compaction

• Purpose: exercise the embedded OpenClaw openai-responses agent loop through at least two real automatic compactions, then confirm a durable marker persists.
• Test identifier: src/agents/sessions/agent-session.openai-compaction.live.test.ts
• Activation flag: OPENCLAW_LIVE_OPENAI_COMPACTION=1
• Default model: gpt-5.6-luna
• Model override: OPENCLAW_LIVE_OPENAI_COMPACTION_MODEL=<model>
• The normal stress mode uses a reduced client context budget to reach the same real compaction path with bounded API spend.
• Full-context mode sets the client budget to 922000 and compaction reserve to 222000, so automatic compaction starts at 700000. It also requires an observed provider input count above the 272000 long-context pricing boundary.

Bounded live recipe:

OPENCLAW_LIVE_TEST=1 \
  OPENCLAW_LIVE_OPENAI_COMPACTION=1 \
  pnpm test:live -- src/agents/sessions/agent-session.openai-compaction.live.test.ts

Complete 922000 input budget recipe:

OPENCLAW_LIVE_TEST=1 \
  OPENCLAW_LIVE_OPENAI_COMPACTION=1 \
  OPENCLAW_LIVE_OPENAI_COMPACTION_FULL=1 \
  OPENCLAW_LIVE_OPENAI_COMPACTION_MODEL=gpt-5.6-terra \
  pnpm test:live -- src/agents/sessions/agent-session.openai-compaction.live.test.ts

Warning

Full mode deliberately crosses OpenAI's long context pricing boundary and can make several large API calls. Use it only with explicit spend approval.

Default for a fresh OpenAI API key:

OPENCLAW_LIVE_GATEWAY_OPENAI_API_DEFAULT=1 \
  OPENCLAW_LIVE_GATEWAY_PROVIDERS=openai \
  OPENCLAW_LIVE_GATEWAY_THINKING=off \
  pnpm test:live -- src/gateway/gateway-models.profiles.live.test.ts

This proof leaves OPENCLAW_LIVE_GATEWAY_MODELS unset, resolves the model through the fresh onboarding inference selection seam, asserts openai/gpt-5.6, and then runs a real gateway turn with that resolved model.

GPT-5.6 embedded OpenClaw matrix:

OPENCLAW_LIVE_GATEWAY_THINKING=ultra \
  OPENCLAW_LIVE_GATEWAY_PROVIDERS=openai \
  OPENCLAW_LIVE_GATEWAY_MODELS='openai/gpt-5.6-sol,openai/gpt-5.6-terra,openai/gpt-5.6-luna' \
  pnpm test:live -- src/gateway/gateway-models.profiles.live.test.ts

Docker notes:

• The Docker runner is located at scripts/test-live-codex-harness-docker.sh.
• It forwards OPENAI_API_KEY, copies Codex CLI auth files when present, installs @openai/codex into a writable mounted npm prefix, stages the source tree, then runs only the Codex harness live test.
• Docker activates the image, MCP/tool, and Guardian probes by default. Set OPENCLAW_LIVE_CODEX_HARNESS_IMAGE_PROBE=0 or OPENCLAW_LIVE_CODEX_HARNESS_MCP_PROBE=0 or OPENCLAW_LIVE_CODEX_HARNESS_GUARDIAN_PROBE=0 when you need a narrower debug run.
• Docker uses the same explicit Codex runtime config, so legacy aliases or OpenClaw fallback cannot hide a Codex harness regression.
• Matrix targets run sequentially in one container. The Docker script scales its default 35 minute timeout by target count; any outer shell or CI timeout must allow the same total. Canonical CI keeps each GPT-5.6 target in a separate shard.

Recommended live recipes

Narrow, explicit allowlists are fastest and least flaky:

• Single model, direct (no gateway):

  • OPENCLAW_LIVE_MODELS="openai/gpt-5.6-luna" pnpm test:live src/agents/models.profiles.live.test.ts

• Small model direct profile:

  • OPENCLAW_LIVE_MODELS=small pnpm test:live src/agents/models.profiles.live.test.ts

• Small model gateway profile:

  • OPENCLAW_LIVE_GATEWAY_MODELS=small pnpm test:live src/gateway/gateway-models.profiles.live.test.ts

• Ollama Cloud API smoke:

  • OPENCLAW_LIVE_TEST=1 OPENCLAW_LIVE_OLLAMA=1 OPENCLAW_LIVE_OLLAMA_BASE_URL=https://ollama.com OPENCLAW_LIVE_OLLAMA_MODEL=glm-5.1:cloud OPENCLAW_LIVE_OLLAMA_WEB_SEARCH=0 pnpm test:live -- extensions/ollama/ollama.live.test.ts

• Single model, gateway smoke:

  • OPENCLAW_LIVE_GATEWAY_MODELS="openai/gpt-5.6-luna" pnpm test:live src/gateway/gateway-models.profiles.live.test.ts

• Tool calling across several providers:

  • OPENCLAW_LIVE_GATEWAY_MODELS="openai/gpt-5.6-luna,anthropic/claude-opus-4-6,google/gemini-3.5-flash,deepseek/deepseek-v4-flash,zai/glm-5.1,minimax/MiniMax-M3" pnpm test:live src/gateway/gateway-models.profiles.live.test.ts

• Z.AI Coding Plan GLM-5.2 direct smoke:

  • ZAI_CODING_LIVE_TEST=1 pnpm test:live src/agents/zai.live.test.ts

• Google focus (Gemini API key + Antigravity):

  • Gemini (API key): OPENCLAW_LIVE_GATEWAY_MODELS="google/gemini-3.5-flash" pnpm test:live src/gateway/gateway-models.profiles.live.test.ts
  • Antigravity (OAuth): OPENCLAW_LIVE_GATEWAY_MODELS="google-antigravity/claude-opus-4-6-thinking,google-antigravity/gemini-3-pro-high" pnpm test:live src/gateway/gateway-models.profiles.live.test.ts

• Google adaptive thinking smoke (qa manual from the private QA CLI - requires OPENCLAW_ENABLE_PRIVATE_QA_CLI=1 and a source checkout; see QA overview):

  • Gemini 3 dynamic default: OPENCLAW_ENABLE_PRIVATE_QA_CLI=1 pnpm openclaw qa manual --provider-mode live-frontier --model google/gemini-3.1-pro-preview --alt-model google/gemini-3.1-pro-preview --message '/think adaptive Reply exactly: GEMINI_ADAPTIVE_OK' --timeout-ms 180000
  • Gemini 2.5 dynamic budget: OPENCLAW_ENABLE_PRIVATE_QA_CLI=1 pnpm openclaw qa manual --provider-mode live-frontier --model google/gemini-2.5-flash --alt-model google/gemini-2.5-flash --message '/think adaptive Reply exactly: GEMINI25_ADAPTIVE_OK' --timeout-ms 180000

Notes:

• google/... uses the Gemini API (API key).
• google-antigravity/... uses the Antigravity OAuth bridge (Cloud Code Assist style agent endpoint).
• google-gemini-cli/... uses the local Gemini CLI on your machine (separate auth + tooling quirks).
• Gemini API vs Gemini CLI:
  • API: OpenClaw calls Google's hosted Gemini API over HTTP (API key / profile auth); this is what most users mean by "Gemini".
  • CLI: OpenClaw shells out to a local gemini binary; it has its own auth and can behave differently (streaming/tool support/version skew).

Live: model matrix (what we cover)

Live is opt in, so there is no fixed "CI model list." OPENCLAW_LIVE_MODELS=modern / OPENCLAW_LIVE_GATEWAY_MODELS=modern (and their all alias) run the curated priority list from HIGH_SIGNAL_LIVE_MODEL_PRIORITY in src/agents/live-model-filter.ts, in this priority order:

The handpicked small-model set (OPENCLAW_LIVE_MODELS=small / OPENCLAW_LIVE_GATEWAY_MODELS=small), sourced from SMALL_LIVE_MODEL_PRIORITY:

Observations on the modern list:

• codex and codex-cli are left out of the default modern sweep because they cover CLI-backend/ACP behavior, which gets its own separate tests. openai/gpt-5.5 normally routes through the Codex app-server harness by default; refer to Live: Codex app-server harness smoke.
• In the modern sweep, fireworks, google, openrouter, and xai only execute their explicitly curated model identifiers, without automatic expansion to all models from those providers.
• Make sure OPENCLAW_LIVE_GATEWAY_MODELS includes at least one image-capable model, such as Claude, Gemini, or OpenAI-family vision variants, to exercise the image probe.

Run a gateway smoke test with tools and images across a curated cross-provider selection:

OPENCLAW_LIVE_GATEWAY_MODELS="openai/gpt-5.6-luna,anthropic/claude-opus-4-6,google/gemini-3.1-pro-preview,google/gemini-3.5-flash,google-antigravity/claude-opus-4-6-thinking,deepseek/deepseek-v4-flash,zai/glm-5.1,minimax/MiniMax-M3" pnpm test:live src/gateway/gateway-models.profiles.live.test.ts

Optional extra coverage outside the curated lists (beneficial but not required; pick a tools-capable model you have active):

• Mistral: mistral/...
• Cerebras: cerebras/... (provided you have access)
• LM Studio: lmstudio/... (local; tool support depends on the API mode)

Aggregators / alternate gateways

If your keys are active, you can also run tests through:

• OpenRouter: openrouter/... (hundreds of models; use openclaw models scan to identify candidates that support both tools and images)
• OpenCode: opencode/... for Zen and opencode-go/... for Go (authentication via OPENCODE_API_KEY / OPENCODE_ZEN_API_KEY)

Additional providers you can add to the live matrix (assuming you have credentials or configuration):

• Built-in: anthropic, cerebras, github-copilot, google, google-antigravity, google-gemini-cli, google-vertex, groq, mistral, openai, openrouter, opencode, opencode-go, xai, zai
• Through models.providers (custom endpoints): minimax (cloud/API), plus any proxy compatible with OpenAI or Anthropic, such as LM Studio, vLLM, or LiteLLM.

Tip

Avoid hardcoding "all models" in documentation. The definitive list is whatever discoverModels(...) returns on your machine, combined with whichever keys are available.

Credentials (never commit)

Live tests discover credentials using the same method as the CLI. Practical consequences:

• If the CLI functions correctly, live tests should locate the same keys.

• When a live test reports "no creds", troubleshoot the same way you would debug openclaw models list or model selection.

• Authentication profiles per agent: ~/.openclaw/agents/<agentId>/agent/auth-profiles.json (this is what "profile keys" refers to in live tests)

• Configuration: ~/.openclaw/openclaw.json (or OPENCLAW_CONFIG_PATH)

• Legacy OAuth directory: ~/.openclaw/credentials/ (copied into the staged live home when present, but not the primary profile key store)

• Local live runs copy the active configuration (with agents.*.workspace / agentDir overrides removed) and each agent's auth-profiles.json, not the rest of that agent's directory, so workspace/ and sandboxes/ data never reaches the staged home, plus the legacy credentials/ directory and supported external CLI authentication files and directories (.claude.json, .claude/.credentials.json, .claude/settings*.json, .claude/backups, .codex/auth.json, .codex/config.toml, .gemini, .minimax) into a temporary test home.

To depend on environment keys, export them before local tests or use the Docker runners listed below with an explicit OPENCLAW_PROFILE_FILE.

Deepgram live (audio transcription)

• Test: extensions/deepgram/audio.live.test.ts
• Enable: DEEPGRAM_API_KEY=... DEEPGRAM_LIVE_TEST=1 pnpm test:live extensions/deepgram/audio.live.test.ts

BytePlus coding plan live

• Test: extensions/byteplus/live.test.ts
• Enable: BYTEPLUS_API_KEY=... BYTEPLUS_LIVE_TEST=1 pnpm test:live extensions/byteplus/live.test.ts
• Optional model override: BYTEPLUS_CODING_MODEL=ark-code-latest

ComfyUI workflow media live

• Test: extensions/comfy/comfy.live.test.ts
• Enable: OPENCLAW_LIVE_TEST=1 COMFY_LIVE_TEST=1 pnpm test:live -- extensions/comfy/comfy.live.test.ts
• Scope:
  • Exercises the bundled comfy image, video, and music_generate paths
  • Skips each capability unless plugins.entries.comfy.config.<capability> is configured
  • Useful after modifying comfy workflow submission, polling, downloads, or plugin registration

Image generation live

• Test: test/image-generation.runtime.live.test.ts
• Command: pnpm test:live test/image-generation.runtime.live.test.ts
• Harness: pnpm test:live:media image
• Scope:
  • Enumerates every registered image generation provider plugin
  • Uses already exported provider environment variables before probing
  • Prefers live or environment API keys over stored authentication profiles by default, so stale test keys in auth-profiles.json do not hide real shell credentials
  • Skips providers without a usable authentication, profile, or model
  • Runs each configured provider through the shared image generation runtime:
    • <provider>:generate
    • <provider>:edit when the provider declares edit support
• Current bundled providers covered:
  • deepinfra
  • fal
  • google
  • minimax
  • openai
  • openrouter
  • vydra
  • xai
• Optional narrowing:
  • OPENCLAW_LIVE_IMAGE_GENERATION_PROVIDERS="openai,google,openrouter,xai"
  • OPENCLAW_LIVE_IMAGE_GENERATION_PROVIDERS="deepinfra"
  • OPENCLAW_LIVE_IMAGE_GENERATION_MODELS="openai/gpt-image-2,google/gemini-3.1-flash-image,openrouter/google/gemini-3.1-flash-image-preview,xai/grok-imagine-image"
  • OPENCLAW_LIVE_IMAGE_GENERATION_CASES="google:flash-generate,google:pro-edit,openrouter:generate,xai:default-generate,xai:default-edit"
• Optional authentication behavior:
  • OPENCLAW_LIVE_REQUIRE_PROFILE_KEYS=1 to force profile store authentication and ignore environment only overrides

For the shipped CLI path, add an infer smoke after the provider or runtime live test passes:

OPENCLAW_LIVE_TEST=1 OPENCLAW_LIVE_INFER_CLI_TEST=1 pnpm test:live -- test/image-generation.infer-cli.live.test.ts
openclaw infer image providers --json
openclaw infer image generate \
  --model google/gemini-3.1-flash-image \
  --prompt "Minimal flat test image: one blue square on a white background, no text." \
  --output ./openclaw-infer-image-smoke.png \
  --json

This covers CLI argument parsing, configuration and default agent resolution, bundled plugin activation, the shared image generation runtime, and the live provider request. Plugin dependencies must be present before runtime load.

Music generation live

• Test: extensions/music-generation-providers.live.test.ts
• Enable: OPENCLAW_LIVE_TEST=1 pnpm test:live -- extensions/music-generation-providers.live.test.ts
• Harness: pnpm test:live:media music
• Scope:
  • Validates the shared bundled music-generation provider path
  • Presently includes fal, google, minimax, and openrouter
  • Relies on provider environment variables already exported before any probing
  • By default, prefers live/env API keys over saved authentication profiles, preventing outdated test keys in auth-profiles.json from hiding real shell credentials
  • Omits providers lacking a usable auth/profile/model
  • When both declared runtime modes are present, executes each:
    • generate using prompt-only input
    • edit if the provider declares capabilities.edit.enabled
  • comfy uses its own dedicated live file instead of this shared sweep
• Optional narrowing:
  • OPENCLAW_LIVE_MUSIC_GENERATION_PROVIDERS="google,minimax"
  • OPENCLAW_LIVE_MUSIC_GENERATION_MODELS="google/lyria-3-clip-preview,minimax/music-2.6"
• Optional auth behavior:
  • OPENCLAW_LIVE_REQUIRE_PROFILE_KEYS=1 to enforce profile-store authentication, ignoring environment-only overrides

Video generation live

• Test: extensions/video-generation-providers.live.test.ts
• Enable: OPENCLAW_LIVE_TEST=1 pnpm test:live -- extensions/video-generation-providers.live.test.ts
• Harness: pnpm test:live:media video
• Scope:
  • Exercises the shared bundled video-generation provider path across alibaba, byteplus, deepinfra, fal, google, minimax, openai, openrouter, pixverse, qwen, runway, together, vydra, xai
  • Defaults to the release-safe smoke path: one text-to-video request per provider, a one-second lobster prompt, and a per-provider operation cap taken from OPENCLAW_LIVE_VIDEO_GENERATION_TIMEOUT_MS (180000 by default)
  • FAL is excluded by default because provider-side queue latency can dominate release time; pass OPENCLAW_LIVE_VIDEO_GENERATION_PROVIDERS="fal" (or clear the skip list) to run it explicitly
  • Uses already-exported provider environment variables before probing
  • Prefers live/env API keys over stored auth profiles by default, so stale test keys in auth-profiles.json do not mask real shell credentials
  • Skips providers with no usable auth, profile, or model
  • Runs only generate by default
  • Set OPENCLAW_LIVE_VIDEO_GENERATION_FULL_MODES=1 to also run declared transform modes when available:
    • imageToVideo when the provider declares capabilities.imageToVideo.enabled and the selected provider/model accepts buffer-backed local image input in the shared sweep
    • videoToVideo when the provider declares capabilities.videoToVideo.enabled and the selected provider/model accepts buffer-backed local video input in the shared sweep
  • Current declared-but-skipped imageToVideo provider in the shared sweep:
    • vydra (buffer-backed local image input is not supported in this lane)
  • Provider-specific Vydra coverage:
    • OPENCLAW_LIVE_TEST=1 OPENCLAW_LIVE_VYDRA_VIDEO=1 pnpm test:live -- extensions/vydra/vydra.live.test.ts
    • That file runs veo3 text-to-video plus a kling image-to-video lane that uses a remote image URL fixture by default (OPENCLAW_LIVE_VYDRA_KLING_IMAGE_URL to override).
  • Provider-specific xAI coverage:
    • OPENCLAW_LIVE_TEST=1 OPENCLAW_LIVE_XAI_VIDEO=1 pnpm test:live -- extensions/xai/xai.live.test.ts -t "classic Grok Imagine"
    • The classic case generates a square local PNG first frame, omits geometry, requests a one-second image-to-video clip, polls to completion, and verifies the downloaded buffer.
    • OPENCLAW_LIVE_TEST=1 OPENCLAW_LIVE_XAI_VIDEO=1 pnpm test:live -- extensions/xai/xai.live.test.ts -t "Grok Imagine Video 1.5"
    • The 1.5 case generates a local PNG first frame, requests a one-second 1080P image-to-video clip, polls to completion, and verifies the downloaded buffer.
  • Current videoToVideo live coverage:
    • runway only when the selected model resolves to gen4_aleph
  • Current declared-but-skipped videoToVideo providers in the shared sweep:
    • alibaba, google, openai, qwen, xai because those paths currently require remote http(s) reference URLs rather than buffer-backed local input
• Optional narrowing:
  • OPENCLAW_LIVE_VIDEO_GENERATION_PROVIDERS="deepinfra,google,openai,runway"
  • OPENCLAW_LIVE_VIDEO_GENERATION_MODELS="google/veo-3.1-fast-generate-preview,openai/sora-2,runway/gen4_aleph"
  • OPENCLAW_LIVE_VIDEO_GENERATION_SKIP_PROVIDERS="" to include every provider in the default sweep, including FAL
  • OPENCLAW_LIVE_VIDEO_GENERATION_TIMEOUT_MS=60000 to reduce each provider operation cap for an aggressive smoke run
• Optional auth behavior:
  • OPENCLAW_LIVE_REQUIRE_PROFILE_KEYS=1 to force profile-store auth and ignore env-only overrides

Media live harness

• Command: pnpm test:live:media
• Entrypoint: test/e2e/qa-lab/media/hosted-media-provider-live.ts executes pnpm test:live -- <suite-test-file> for each chosen suite, ensuring heartbeat and quiet-mode behavior align with other pnpm test:live runs.
• Purpose:
  • Launches the shared image, music, and video live suites using a single repository-native entrypoint
  • Automatically populates missing provider environment variables from ~/.profile
  • By default, restricts each suite to providers with currently valid authentication
• Flags:
  • --providers <csv> acts as a global provider filter; --image-providers, --music-providers, and --video-providers apply a filter to an individual suite
  • --all-providers bypasses the authentication-based automatic filter
  • --allow-empty causes 0 to terminate when no runnable providers remain after filtering
  • --quiet and --no-quiet are forwarded to test:live
• Examples:
  • pnpm test:live:media
  • pnpm test:live:media image video --providers openai,google,minimax
  • pnpm test:live:media video --video-providers openai,runway --all-providers
  • pnpm test:live:media music --quiet

Related

• Testing covers unit, integration, QA, and Docker suites