OpenClaw Testing: Unit, E2E, Live Suites and Docker Runners

OpenClaw ships three Vitest test suites (unit/integration, e2e, live) alongside Docker runners. This document explains what each suite covers, the commands tied to particular workflows, how live tests locate credentials, and the process for adding regression tests against real-world provider/model issues.

Note

The QA stack (qa-lab, qa-channel, live transport lanes) has its own documentation:

QA overview - covers architecture, command surface, scenario authoring, and the Matrix live lane.

Maturity scorecard - explains how release QA evidence informs stability and LTS decisions.

QA channel - details the synthetic transport plugin backing repo-driven scenarios.

Regular test suites and Docker/Parallels runners are the focus here. QA-specific runners below provides the exact qa commands and directs you back to the references above.

Quick start

Typical daily usage:

Full gate (required before pushing): pnpm build && pnpm check && pnpm check:test-types && pnpm test
Faster local full-suite execution on a machine with ample resources: pnpm test:max
Direct Vitest watch mode: pnpm test:watch
File-specific targeting also handles plugin/channel paths: pnpm test extensions/discord/src/monitor/message-handler.preflight.test.ts
When iterating on a single failure, start with targeted runs.
Docker-backed QA site: pnpm qa:lab:up
Linux VM-backed QA lane: pnpm openclaw qa suite --runner multipass --scenario channel-chat-baseline

For test modifications or added confidence:

Informational V8 coverage report: pnpm test:coverage
E2E suite: pnpm test:e2e

Test Temp Directories

Shared helpers from test/helpers/temp-dir.ts handle test-owned temporary directories, making ownership explicit and tying cleanup into the test lifecycle:

import { afterEach } from "vitest";
import { useAutoCleanupTempDirTracker } from "../helpers/temp-dir.js";

const tempDirs = useAutoCleanupTempDirTracker(afterEach);

it("uses a temp workspace", () => {
  const workspace = tempDirs.make("openclaw-example-");
  // use workspace
});

useAutoCleanupTempDirTracker(afterEach) deliberately omits a manual cleanup method, since Vitest manages cleanup after each test. Older lower-level helpers (makeTempDir, cleanupTempDirs, createTempDirTracker) remain available for tests not yet migrated; steer clear of new usage and avoid fresh bare fs.mkdtemp* calls unless a test explicitly validates raw temp-dir behavior. When a bare temp dir is truly necessary, include an auditable allow comment with justification:

// openclaw-temp-dir: allow verifies raw fs cleanup behavior
const workspace = fs.mkdtempSync(prefix);

node scripts/report-test-temp-creations.mjs flags new bare temp-dir creation and new manual shared-helper usage in added diff lines without interfering with existing cleanup patterns. It applies the same test-path classification as scripts/changed-lanes.mjs and excludes the shared helper implementation itself. check:changed executes this report for changed test paths as a warning-only CI signal (GitHub warning annotations, not failures).

Live and Docker/Parallels workflows

When troubleshooting real providers/models (requires real credentials):

Live suite (models + gateway tool/image probes): pnpm test:live
Quietly probe a single live file: pnpm test:live -- src/agents/models.profiles.live.test.ts
Runtime performance reporting: send OpenClaw Performance along with live_openai_candidate=true for an actual openai/gpt-5.6-luna agent turn, or deep_profile=true for Kova CPU/heap/trace artifacts. Daily scheduled runs post mock-provider, deep-profile, and GPT-5.6 Luna lane reports to openclaw/clawgrit-reports via a separate artifact-consuming publisher job; absent or invalid publisher credentials cause scheduled and profile=release runs to fail. Manual non-release dispatches retain the GitHub artifacts and treat report publication as optional. The mock-provider report additionally covers source-level gateway boot, memory, plugin-pressure, repeated fake-model hello-loop, and CLI startup metrics.
Docker live model sweep: pnpm test:docker:live-models
- Each chosen model executes a text turn plus a small file-read-style probe. Models whose metadata indicates image input also run a tiny image turn. Turn off the extra probes with OPENCLAW_LIVE_MODEL_FILE_PROBE=0 or OPENCLAW_LIVE_MODEL_IMAGE_PROBE=0 when isolating provider failures.
- CI coverage: daily OpenClaw Scheduled Live And E2E Checks and manual OpenClaw Release Checks both invoke the reusable live/E2E workflow with include_live_suites: true, which includes Docker live model matrix jobs sharded by provider.
- For targeted CI reruns, dispatch OpenClaw Live And E2E Checks (Reusable) with include_live_suites: true and live_models_only: true.
- Add new high-signal provider secrets to scripts/ci-hydrate-live-auth.sh plus .github/workflows/openclaw-live-and-e2e-checks-reusable.yml and its scheduled/release callers.
Native Codex bound-chat smoke: pnpm test:docker:live-codex-bind
- Executes a Docker live lane against the Codex app-server path, binds a synthetic Slack DM with /codex bind, runs /codex fast and /codex permissions, then confirms a plain reply and an image attachment flow through the native plugin binding instead of ACP.
Codex app-server harness smoke: pnpm test:docker:live-codex-harness
- Runs gateway agent turns through the plugin-owned Codex app-server harness, verifies /codex status and /codex models, and by default exercises image, cron MCP, sub-agent, and Guardian probes. Disable the sub-agent probe with OPENCLAW_LIVE_CODEX_HARNESS_SUBAGENT_PROBE=0 when isolating other failures. For a focused sub-agent check, disable the other probes: OPENCLAW_LIVE_CODEX_HARNESS_IMAGE_PROBE=0 OPENCLAW_LIVE_CODEX_HARNESS_MCP_PROBE=0 OPENCLAW_LIVE_CODEX_HARNESS_GUARDIAN_PROBE=0 OPENCLAW_LIVE_CODEX_HARNESS_SUBAGENT_PROBE=1 pnpm test:docker:live-codex-harness. This exits after the sub-agent probe unless OPENCLAW_LIVE_CODEX_HARNESS_SUBAGENT_ONLY=0 is set.
Codex on-demand install smoke: pnpm test:docker:codex-on-demand
- Installs the packaged OpenClaw tarball in Docker, runs OpenAI API-key onboarding, and verifies the Codex plugin plus @openai/codex dependency were downloaded into the managed npm project root on demand.
Codex npm-plugin live package smoke: pnpm test:docker:live-codex-npm-plugin
- Installs the candidate OpenClaw package and exact Codex plugin into Docker, then uses a real OpenAI key for CLI preflight and same-session turns.
- Its zero-retry medium-thinking follow-through turn must send progress, keep working through randomized workspace reads and an exact artifact write, then send completion. A progress-only terminal turn fails the lane.
Live plugin tool dependency smoke: pnpm test:docker:live-plugin-tool
- Packs a fixture plugin with a real slugify dependency, installs it through npm-pack:, verifies the dependency under the managed npm project root, then asks a live OpenAI model to call the plugin tool and return the hidden slug.
OpenClaw rescue command smoke: pnpm test:live:system-agent-rescue-channel
- Opt-in belt-and-suspenders check for the message-channel rescue command surface. Exercises /openclaw status, queues a persistent model change, replies /openclaw yes, and verifies the audit/config write path.
OpenClaw first-run Docker smoke: pnpm test:docker:system-agent-first-run
- Starts from an empty OpenClaw state dir and first proves the packaged openclaw setup CLI fails closed without inference. It then tests and activates fake Claude through the packaged activation module. Only afterward does a fuzzy packaged CLI request reach the planner and resolve to typed setup, followed by one-shot model, agent, Discord config, and SecretRef operations. It validates config and audit entries. This is supporting gate/operation evidence, not an interactive onboarding or OpenClaw agent/tool/approval proof. The same lane is exposed in QA Lab by pnpm openclaw qa suite --scenario system-agent-ring-zero-setup.
Moonshot/Kimi cost smoke: with MOONSHOT_API_KEY set, run openclaw models list --provider moonshot --json, then run an isolated openclaw agent --local --session-id live-kimi-cost --message 'Reply exactly: KIMI_LIVE_OK' --thinking off --json against moonshot/kimi-k2.6. Verify the JSON reports Moonshot/K2.6 and the assistant transcript stores normalized usage.cost.

Tip

When you only need one failing case, prefer narrowing live tests via the allowlist env vars described below.

QA-specific runners

These commands sit beside the main test suites when you need QA-lab realism.

CI runs QA Lab in dedicated workflows. Agentic parity is nested under QA-Lab - All Lanes and release validation, not a standalone PR workflow. Broad validation should use Full Release Validation with rerun_group=qa-parity for parity or rerun_group=qa-live for live QA. The direct OpenClaw Release Checks child alone may use rerun_group=qa as a manual aggregate of both groups. Stable/full, soak-enabled, and explicit qa-live release checks include the QA-live Matrix and Telegram lanes. Bounded beta-publish all without soak runs parity but defers those live lanes to postpublish-confidence. QA-Lab - All Lanes runs nightly on main and from manual dispatch with the mock parity lane, live Matrix lane, Convex-managed live Telegram lane, and Convex-managed live Discord lane as parallel jobs. Scheduled QA and selected release checks run the catalog-derived Matrix selection through the shared live adapter. Release transport checks use mock-openai/gpt-5.6-luna so they stay deterministic and avoid normal provider-plugin startup. These live transport gateways disable memory search; memory behavior stays covered by the QA parity suites.

Full release live media shards run on
ghcr.io/openclaw/openclaw-live-media-runner:ubuntu-24.04, which comes with
ffmpeg and ffprobe preinstalled. For Docker live model/backend shards, the shared
ghcr.io/openclaw/openclaw-live-test:<sha> image is built a single time per chosen
commit, and each shard pulls it using OPENCLAW_SKIP_DOCKER_BUILD=1 rather than rebuilding
it locally.

pnpm openclaw qa suite
- Executes repository-backed QA scenarios directly on the host machine.
- Produces qa-evidence.json, qa-suite-summary.json, and qa-suite-report.md artifacts at the top level for the chosen scenario set, which covers mixed flow, Vitest, and Playwright selections.
- When triggered via pnpm openclaw qa run --qa-profile <profile>, it includes the selected taxonomy profile scorecard in that same qa-evidence.json. smoke-ci generates minimal evidence (evidenceMode: "slim", without per-entry execution). release handles the curated release-readiness subset; all picks all active maturity categories and explicitly targets QA Profile Evidence workflow dispatches when a full scorecard artifact is required.
- Multiple selected scenarios run in parallel by default, each with its own isolated gateway worker. qa-channel sets concurrency to 4 by default, capped by the number of selected scenarios. Adjust the worker count with --concurrency <count>, or fall back to the older serial mode with --concurrency 1.
- Returns a non-zero exit code if any scenario fails. To get artifacts without that failing exit code, use --allow-failures.
- Supports provider modes live-frontier, mock-openai, and aimock. aimock launches a local AIMock-backed provider server for experimental fixture and protocol-mock coverage, leaving the scenario-aware mock-openai lane untouched.
pnpm openclaw qa coverage --match <query>
- Looks through scenario IDs, titles, surfaces, coverage IDs, docs refs, code refs, plugins, and provider requirements, then outputs the matching suite targets.
- Run this ahead of a QA Lab session when you know the affected behavior or file path but not the minimal scenario. It is advisory only, so still pick mock, live, Multipass, Matrix, or transport proof based on what is being changed.
pnpm test:plugins:kitchen-sink-live
- Puts the live OpenAI Kitchen Sink plugin gauntlet through QA Lab. Installs the external Kitchen Sink package, checks the plugin SDK surface inventory, probes /healthz and /readyz, records gateway CPU/RSS evidence, performs a live OpenAI turn, and inspects adversarial diagnostics. Live OpenAI auth is required, for example OPENAI_API_KEY. In hydrated Testbox sessions, the Testbox live-auth profile is sourced automatically when the openclaw-testbox-env helper exists.
pnpm test:gateway:cpu-scenarios
- Runs the gateway startup benchmark plus a small mock QA Lab scenario pack (channel-chat-baseline, memory-failure-fallback, gateway-restart-inflight-run) and writes a combined CPU observation summary under .artifacts/gateway-cpu-scenarios/.
- By default, only sustained hot CPU observations are flagged (--cpu-core-warn, default 0.9; --hot-wall-warn-ms, default 30000), so brief startup spikes are recorded as metrics without resembling the long gateway peg regression.
- Requires built dist artifacts; build first if the checkout lacks fresh runtime output.
pnpm openclaw qa suite --runner multipass
- Runs the same QA suite inside a disposable Multipass Linux VM, using the same scenario-selection and provider/model flags as qa suite.
- Live runs forward the QA auth inputs that make sense for the guest: env-based provider keys, the QA live provider config path, and CODEX_HOME when available.
- Output directories must remain under the repo root so the guest can write back through the mounted workspace.
- Produces the standard QA report and summary, plus Multipass logs under .artifacts/qa-e2e/....
pnpm qa:lab:up
- Launches the Docker-backed QA site for operator-style QA work.
pnpm test:docker:npm-onboard-channel-agent
- Creates an npm tarball from the current checkout, installs it globally in Docker, runs non-interactive OpenAI API-key onboarding, sets up Telegram by default, confirms the packaged plugin runtime loads without startup dependency repair, runs doctor, and executes one local agent turn against a mocked OpenAI endpoint.
- Use OPENCLAW_NPM_ONBOARD_CHANNEL=discord to run the same packaged-install lane with Discord.
pnpm test:docker:session-runtime-context
- Runs a deterministic built-app Docker smoke for embedded runtime context transcripts. Confirms hidden OpenClaw runtime context stays as a non-display custom message rather than leaking into the visible user turn, then seeds an affected broken session JSONL and checks that openclaw doctor --fix rewrites it to the active branch with a backup.
pnpm test:docker:npm-telegram-live
- Installs an OpenClaw package candidate in Docker, runs installed-package onboarding, configures Telegram through the installed CLI, then reuses the live Telegram QA lane with that installed package as the SUT Gateway.
- The trusted checkout owns the QA harness source, taxonomy, scenarios, dependencies, and private SDK build. The installed package remains the absolute CLI, Gateway, and bundled-plugin runtime under test, and its CLI writes the package candidate's persisted auth state.
- Defaults to OPENCLAW_NPM_TELEGRAM_PACKAGE_SPEC=openclaw@beta; set OPENCLAW_NPM_TELEGRAM_PACKAGE_TGZ=/path/to/openclaw-current.tgz or OPENCLAW_CURRENT_PACKAGE_TGZ to test a resolved local tarball instead of installing from the registry.
- Emits repeated RTT timing in qa-evidence.json by default with OPENCLAW_NPM_TELEGRAM_RTT_SAMPLES=20. Override OPENCLAW_NPM_TELEGRAM_RTT_SAMPLES, OPENCLAW_NPM_TELEGRAM_RTT_TIMEOUT_MS, or OPENCLAW_NPM_TELEGRAM_RTT_MAX_FAILURES to tune the run. OPENCLAW_NPM_TELEGRAM_RTT_CHECKS selects the Telegram QA scenario to sample; the supported RTT target is channel-canary. The package runner promotes that portable canary once to the first position, making canary+RTT the preflight before the remaining taxonomy-backed fail-fast release scenarios.
- Uses the same Telegram env credentials or Convex credential source as pnpm openclaw qa telegram. For CI/release automation, set OPENCLAW_NPM_TELEGRAM_CREDENTIAL_SOURCE=convex plus OPENCLAW_QA_CONVEX_SITE_URL and a role secret. If OPENCLAW_QA_CONVEX_SITE_URL and a Convex role secret are present in CI, the Docker wrapper selects Convex automatically.
- The wrapper validates Telegram or Convex credential env on the host before Docker build/install work. Set OPENCLAW_NPM_TELEGRAM_SKIP_CREDENTIAL_PREFLIGHT=1 solely for debugging scenarios that occur before credential setup.
- OPENCLAW_NPM_TELEGRAM_CREDENTIAL_ROLE=ci|maintainer replaces the shared OPENCLAW_QA_CREDENTIAL_ROLE for this particular lane. When Convex credentials are chosen and no role is defined, the wrapper applies ci in CI and maintainer outside CI.
- In GitHub Actions, this lane appears as the manual maintainer workflow NPM Telegram Beta E2E. Merges do not trigger it. The workflow relies on the qa-live-shared environment and Convex CI credential leases.
Additionally, GitHub Actions exposes Package Acceptance for side-run product validation against a single candidate package. It takes a Git ref, published npm spec, HTTPS tarball URL with SHA-256, trusted-URL policy, or tarball artifact from a different run (source=ref|npm|url|trusted-url|artifact), uploads the normalized openclaw-current.tgz as package-under-test, then executes the existing Docker E2E scheduler with smoke, package, product, full, or custom lane profiles. Configure telegram_mode=mock-openai or live-frontier to trigger the Telegram QA workflow against the same package-under-test artifact.
- Latest beta product proof:

gh workflow run package-acceptance.yml --ref main \
  -f source=npm \
  -f package_spec=openclaw@beta \
  -f suite_profile=product \
  -f telegram_mode=mock-openai

Proof for an exact tarball URL requires a digest and applies the public URL safety policy:

gh workflow run package-acceptance.yml --ref main \
  -f source=url \
  -f package_url=https://registry.npmjs.org/openclaw/-/openclaw-VERSION.tgz \
  -f package_sha256=<sha256> \
  -f suite_profile=package

Enterprise and private tarball mirrors adopt an explicit trusted-source policy:

gh workflow run package-acceptance.yml --ref main \
  -f source=trusted-url \
  -f trusted_source_id=enterprise-artifactory \
  -f package_url=https://packages.example.internal:8443/artifactory/openclaw/openclaw-VERSION.tgz \
  -f package_sha256=<sha256> \
  -f suite_profile=package

source=trusted-url pulls .github/package-trusted-sources.json from the trusted workflow ref and rejects URL credentials as well as any workflow-input private-network bypass. When the named policy specifies bearer auth, set the fixed OPENCLAW_TRUSTED_PACKAGE_TOKEN secret.

Artifact proof retrieves a tarball artifact from another Actions run:

gh workflow run package-acceptance.yml --ref main \
  -f source=artifact \
  -f artifact_run_id=<run-id> \
  -f artifact_name=<artifact-name> \
  -f suite_profile=smoke

pnpm test:docker:plugins
- Packages and installs the current OpenClaw build inside Docker, launches the Gateway with OpenAI configured, then activates bundled channel and plugins through config modifications.
- Confirms that setup discovery leaves unconfigured downloadable plugins missing, the first configured doctor repair explicitly installs every absent downloadable plugin, and a subsequent restart skips hidden dependency repair.
- Additionally installs a known older npm baseline, enables Telegram before running openclaw update --tag <candidate>, and checks that the candidate's post-update doctor removes legacy plugin dependency debris without any harness-side postinstall repair.
pnpm test:parallels:npm-update
- Executes the native packaged-install update smoke across Parallels guests. Each selected platform starts by installing the requested baseline package, then runs the installed openclaw update command within the same guest and confirms the installed version, update status, gateway readiness, and one local agent turn.
- While iterating on a single guest, use --platform macos, --platform windows, or --platform linux. For the summary artifact path and per-lane status, use --json.
- By default, the OpenAI lane relies on openai/gpt-5.6-luna for the live agent-turn proof. Pass --model <provider/model> or set OPENCLAW_PARALLELS_OPENAI_MODEL to verify a different OpenAI model.
- Enclose long local runs in a host timeout so Parallels transport stalls cannot eat up the remaining testing window:
```
timeout --foreground 150m pnpm test:parallels:npm-update -- --json
timeout --foreground 90m pnpm test:parallels:npm-update -- --platform windows --json
```
- Nested lane logs are written by the script under /tmp/openclaw-parallels-npm-update.*. Check windows-update.log, macos-update.log, or linux-update.log before concluding the outer wrapper is stuck.
- On a cold guest, Windows update can take 10 to 15 minutes in post-update doctor and package update work; that remains normal when the nested npm debug log keeps advancing.
- Avoid running this aggregate wrapper concurrently with individual Parallels macOS, Windows, or Linux smoke lanes. They share VM state and can conflict on snapshot restore, package serving, or guest gateway state.
- Because capability facades such as speech, image generation, and media understanding load through bundled runtime APIs, the post-update proof runs the normal bundled plugin surface even when the agent turn itself only verifies a simple text response.
pnpm openclaw qa aimock
- Brings up only the local AIMock provider server, meant for direct protocol smoke testing.
pnpm openclaw qa buzz
- Executes the Buzz live QA lane against an actual relay room, using dedicated driver and SUT identities.
- For local runs, --credential-file <path> is used with relayUrl, roomId, driverPrivateKey, and sutPrivateKey. Closed relays might additionally require driverAuthTag and sutAuthTag. Hosted relays demand wss://; ws:// is permitted solely for loopback development relays.
- By default, mock-openai is assumed, and canary plus mention-gating scenarios run through the genuine Buzz plugin path.
- --credential-source convex is supported with a pooled kind: "buzz" row. Both public keys must belong to the relay/room, and the SUT needs the Bot room role. Never employ a human owner or admin private key.
pnpm openclaw qa matrix
- Runs the Matrix live QA lane against a disposable Docker-backed Tuwunel homeserver. This works only from a source checkout; packaged installs omit qa-lab.
- For the full CLI, profile/scenario catalog, env vars, and artifact layout, see Matrix smoke lanes.
pnpm openclaw qa telegram
- Runs the Telegram live QA lane against a real private group, pulling the driver and SUT bot tokens from env.
- Needs OPENCLAW_QA_TELEGRAM_GROUP_ID, OPENCLAW_QA_TELEGRAM_DRIVER_BOT_TOKEN, and OPENCLAW_QA_TELEGRAM_SUT_BOT_TOKEN. The group id must be the numeric Telegram chat id.
- --credential-source convex enables shared pooled credentials. Default to env mode, or set OPENCLAW_QA_CREDENTIAL_SOURCE=convex to choose pooled leases.
- Defaults handle canary, mention gating, command addressing, /status, bot-to-bot mentioned replies, and core native command replies. mock-openai defaults also cover deterministic reply-chain and Telegram final-message streaming regressions. Use --list-scenarios for optional probes like session_status.
- A non-zero exit occurs when any scenario fails. For artifacts without a failing exit code, use --allow-failures.
- Two distinct bots in the same private group are required, and the SUT bot must expose a Telegram username.
- For stable bot-to-bot observation, turn on Bot-to-Bot Communication Mode in @BotFather for both bots and confirm the driver bot can observe group bot traffic.
- A Telegram QA report, summary, and qa-evidence.json are written under .artifacts/qa-e2e/.... Replying scenarios include RTT from the driver send request to the observed SUT reply.

Mantis Telegram Live wraps this lane for PR evidence. It runs the candidate ref with Convex-leased Telegram credentials, renders the redacted QA report/evidence bundle in a Crabbox desktop browser, records MP4 evidence, generates a motion-trimmed GIF, uploads the artifact bundle, and posts inline PR evidence through the Mantis GitHub App when pr_number is set. Maintainers can trigger it from the Actions UI via Mantis Scenario (scenario_id: telegram-live).

Mantis Telegram Desktop Proof is the agentic native Telegram Desktop before/after wrapper for PR visual proof. Start it from the Actions UI with freeform instructions, through Mantis Scenario (scenario_id: telegram-desktop-proof), or from a maintainer PR comment:

@openclaw-mantis
@openclaw-mantis verify the streamed reply stays visible while it arrives

ClawSweeper's mantis: telegram-visible-proof label starts this workflow automatically for branches in openclaw/openclaw. Fork PRs require the maintainer comment. Mantis reacts with 👀 when it accepts a comment, then posts the active workflow link in its evidence comment and replaces that same comment with the result. Any text after the mention is optional proof guidance. Manual requests stop before desktop setup and comment There was nothing visible to test in this PR at all. when the diff has no Telegram-visible behavior.

The Mantis agent reads the PR, decides what Telegram-visible behavior proves the change, runs the real-user Crabbox Telegram Desktop proof lane on baseline and candidate refs, iterates until the native GIFs are useful, writes a paired motionPreview manifest, and posts the same 2-column GIF table through the Mantis GitHub App when pr_number is set.

pnpm openclaw qa mantis telegram-desktop-builder
- Leases or reuses a Crabbox Linux desktop, installs native Telegram Desktop, configures OpenClaw with a leased Telegram SUT bot token, starts the gateway, and records screenshot/MP4 evidence from the visible VNC desktop.
- Defaults to --credential-source convex so workflows only need the Convex broker secret. Use --credential-source env with the same OPENCLAW_QA_TELEGRAM_* variables as pnpm openclaw qa telegram.
- Telegram Desktop still needs a user login/profile. The bot token configures OpenClaw only. Use --telegram-profile-archive-env <name> for a base64 .tgz profile archive, or use --keep-lease and log in manually through VNC once.
- Writes mantis-telegram-desktop-builder-report.md, mantis-telegram-desktop-builder-summary.json, telegram-desktop-builder.png, and telegram-desktop-builder.mp4 under the output directory.

Live transport lanes share one standard contract so new transports do not drift; the per-lane coverage matrix lives in QA overview - Live transport coverage. qa-channel is the broad synthetic suite and is not part of that matrix.

Shared Telegram credentials via Convex (v1)

When --credential-source convex (or OPENCLAW_QA_CREDENTIAL_SOURCE=convex) is turned on for live transport QA, the QA lab takes an exclusive lease from a Convex-backed pool, sends heartbeats for that lease while the lane is active, and gives the lease back during shutdown. The section's name was set before Buzz, Discord, Slack, and WhatsApp were supported; the lease contract remains the same across all types.

Convex project scaffold for reference: qa/convex-credential-broker/

Environment variables that are required:

OPENCLAW_QA_CONVEX_SITE_URL (such as https://your-deployment.convex.site)
One secret matching the chosen role:
- OPENCLAW_QA_CONVEX_SECRET_MAINTAINER for maintainer
- OPENCLAW_QA_CONVEX_SECRET_CI for ci
Picking the credential role:
- Through CLI: --credential-role maintainer|ci
- Via env default: OPENCLAW_QA_CREDENTIAL_ROLE (falls back to ci in CI, maintainer otherwise)

Environment variables that are optional:

OPENCLAW_QA_CREDENTIAL_LEASE_TTL_MS (standard 1200000)
OPENCLAW_QA_CREDENTIAL_HEARTBEAT_INTERVAL_MS (standard 30000)
OPENCLAW_QA_CREDENTIAL_ACQUIRE_TIMEOUT_MS (standard 90000)
OPENCLAW_QA_CREDENTIAL_HTTP_TIMEOUT_MS (standard 15000)
OPENCLAW_QA_CONVEX_ENDPOINT_PREFIX (standard /qa-credentials/v1)
OPENCLAW_QA_CREDENTIAL_OWNER_ID (trace id, not required)
OPENCLAW_QA_ALLOW_INSECURE_HTTP=1 permits loopback http:// Convex URLs for development confined to local machines.

In regular operation, OPENCLAW_QA_CONVEX_SITE_URL should rely on https://.

Admin commands for maintainers (adding, removing, or listing pools) specifically demand OPENCLAW_QA_CONVEX_SECRET_MAINTAINER.

CLI helpers intended for maintainers:

pnpm openclaw qa credentials doctor
pnpm openclaw qa credentials add --kind telegram --payload-file qa/telegram-credential.json
pnpm openclaw qa credentials list --kind telegram
pnpm openclaw qa credentials remove --credential-id <credential-id>

Before running live tests, invoke doctor to verify the Convex site URL, broker secrets, endpoint prefix, HTTP timeout, and whether admin and list endpoints are reachable, all without exposing secret values. For output that scripts and CI tools can parse, use --json.

Standard endpoint setup (OPENCLAW_QA_CONVEX_SITE_URL combined with /qa-credentials/v1). Requests are authenticated through an Authorization: Bearer <role secret> header; the bodies listed below omit that header:

POST /acquire
- Request: { kind, ownerId, actorRole, leaseTtlMs, heartbeatIntervalMs }
- Success: { status: "ok", credentialId, leaseToken, payload, leaseTtlMs?, heartbeatIntervalMs? }
- Exhausted or retryable: { status: "error", code: "POOL_EXHAUSTED" | "NO_CREDENTIAL_AVAILABLE", ... }
POST /payload-chunk
- Request: { kind, ownerId, actorRole, credentialId, leaseToken, index }
- Success: { status: "ok", index, data }
POST /heartbeat
- Request: { kind, ownerId, actorRole, credentialId, leaseToken, leaseTtlMs }
- Success: { status: "ok" } (or an empty 2xx)
POST /release
- Request: { kind, ownerId, actorRole, credentialId, leaseToken }
- Success: { status: "ok" } (or an empty 2xx)
POST /admin/add (restricted to maintainer secret)
- Request: { kind, actorId, payload, note?, status? }
- Success: { status: "ok", credential }
POST /admin/remove (restricted to maintainer secret)
- Request: { credentialId, actorId }
- Success: { status: "ok", changed, credential }
- Guard for active lease: { status: "error", code: "LEASE_ACTIVE", ... }
POST /admin/list (restricted to maintainer secret)
- Request: { kind?, status?, includePayload?, limit? }
- Success: { status: "ok", credentials, count }

Payload structure for the Telegram kind:

{ groupId: string, driverToken: string, sutToken: string }
groupId has to be a string holding a numeric Telegram chat id.
admin/add checks this format for kind: "telegram" and turns away payloads that are not properly formed.

Payload structure for the Telegram real-user kind:

{ groupId: string, sutToken: string, testerUserId: string, testerUsername: string, telegramApiId: string, telegramApiHash: string, tdlibDatabaseEncryptionKey: string, tdlibArchiveBase64: string, tdlibArchiveSha256: string, desktopTdataArchiveBase64: string, desktopTdataArchiveSha256: string }
groupId, testerUserId, and telegramApiId need to be strings of digits.
tdlibArchiveSha256 and desktopTdataArchiveSha256 have to be SHA-256 hex strings.
kind: "telegram-user" is set aside for the Mantis Telegram Desktop proof workflow. Generic QA Lab lanes are not allowed to take it.

Multi-channel payloads checked by the broker:

Buzz: { relayUrl: string, roomId: string, driverPrivateKey: string, sutPrivateKey: string, driverAuthTag?: string, sutAuthTag?: string }
Discord: { guildId: string, channelId: string, driverBotToken: string, sutBotToken: string, sutApplicationId: string, voiceChannelId?: string }
WhatsApp: { driverPhoneE164: string, sutPhoneE164: string, driverAuthArchiveBase64: string, sutAuthArchiveBase64: string, groupJid?: string }

Slack lanes may also draw from the pool, but Slack payload validation is handled in the Slack QA runner, not the broker. For Slack rows, use { channelId: string, driverBotToken: string, sutBotToken: string, sutAppToken: string }.

Adding a channel to QA

The architecture and scenario-helper names for new channel adapters are documented in QA overview - Adding a channel. The baseline requirement: build the transport runner on the shared qa-lab host seam, add an adapterFactory for shared scenarios, list qaRunners in the plugin manifest, mount as openclaw qa <runner>, and write scenarios under qa/scenarios/.

Test suites (what runs where)

View the suites as "stepping up realism" (and stepping up flakiness/cost).

Unit / integration (default)

Command: pnpm test
Config: untargeted runs rely on the vitest.full-*.config.ts shard set and can break multi-project shards into per-project configs for parallel scheduling
Files: core/unit inventories under src/**/*.test.ts, packages/**/*.test.ts, and test/**/*.test.ts; UI unit tests execute in the separate unit-ui shard
Scope:
- Pure unit tests
- In-process integration tests (gateway auth, routing, tooling, parsing, config)
- Deterministic regressions for known bugs
Expectations:
- Runs in CI
- No real keys required
- Should be fast and stable
- Resolver and public-surface loader tests must show broad api.js and runtime-api.js fallback behavior with generated tiny plugin fixtures, not real bundled plugin source APIs. Real plugin API loads belong in plugin-owned contract/integration suites.

Native dependency policy:

Default test installs skip optional native Discord opus builds. Discord voice uses bundled libopus-wasm, and @discordjs/opus stays disabled in allowBuilds so local tests and Testbox lanes do not compile the native addon.
Compare native opus performance in the libopus-wasm benchmark repo, not in default OpenClaw install/test loops. Do not set @discordjs/opus to true in the default allowBuilds; that makes unrelated install/test loops compile native code.

Projects, shards, and scoped lanes

Instead of one massive native root-project process, untargeted pnpm test executes thirteen smaller shard configurations (core-unit-fast, core-unit-src, core-unit-security, core-unit-ui, core-unit-support, core-support-boundary, core-tooling, core-contracts, core-bundled, core-runtime, agentic, auto-reply, extensions). Peak RSS on busy machines drops, and auto-reply/plugin work no longer starves unrelated suites.
Because a multi-shard watch loop is impractical, pnpm test --watch continues to rely on the native root vitest.config.ts project graph.
Explicit file and directory targets are routed through scoped lanes first by pnpm test, pnpm test:watch, and pnpm test:perf:imports, so pnpm test extensions/discord/src/monitor/message-handler.preflight.test.ts does not incur the full root project startup cost.
By default, pnpm test:changed expands changed git paths into cheap scoped lanes: direct test edits, sibling *.test.ts files, explicit source mappings, and local import-graph dependents. Config, setup, and package edits do not trigger broad test runs unless you explicitly invoke OPENCLAW_TEST_CHANGED_BROAD=1 pnpm test:changed.
For narrow work, pnpm check:changed serves as the standard smart local check gate. It categorizes the diff into core, core tests, extensions, extension tests, apps, docs, release metadata, live Docker tooling, and tooling, then executes the corresponding typecheck, lint, and guard commands. Vitest tests are not run; use pnpm test:changed or explicit pnpm test <target> for test evidence. Version-only bumps in release metadata run targeted version, config, and root-dependency checks, with a guard that rejects package changes outside the top-level version field.
Focused checks apply to live Docker ACP harness edits: shell syntax for the live Docker auth scripts and a live Docker scheduler dry-run. package.json changes are included only when the diff is confined to scripts["test:docker:live-*"]; dependency, export, version, and other package-surface edits still go through the broader guards.
Import-light unit tests from agents, commands, plugins, auto-reply helpers, plugin-sdk, and similar pure utility areas are directed to the unit-fast lane, which bypasses test/setup-openclaw-runtime.ts; stateful or runtime-heavy files remain on the existing lanes.
Selected plugin-sdk and commands helper source files also map changed-mode runs to explicit sibling tests in those light lanes, so helper edits avoid rerunning the full heavy suite for that directory.
auto-reply provides dedicated buckets for top-level core helpers, top-level reply.* integration tests, and the src/auto-reply/reply/** subtree. CI further splits the reply subtree into agent-runner, dispatch, and commands/state-routing shards, preventing one import-heavy bucket from owning the entire Node tail.
Normal PR/main CI deliberately skips the bundled plugin batch sweep and the release-only agentic-plugins shard. Full Release Validation dispatches the separate Plugin Prerelease child workflow for those plugin-heavy suites on release candidates.

Embedded runner coverage

When you change message-tool discovery inputs or compaction runtime context, maintain both levels of coverage.
Add focused helper regressions for pure routing and normalization boundaries.
Keep the embedded runner integration suites healthy: src/agents/embedded-agent-runner/compact.hooks.test.ts, src/agents/embedded-agent-runner/run.overflow-compaction.test.ts, and src/agents/embedded-agent-runner/run.overflow-compaction.loop.test.ts.
Those suites confirm that scoped ids and compaction behavior still flow through the real run.ts / compact.ts paths; helper-only tests cannot replace those integration paths.

Vitest pool and isolation defaults

The base Vitest config defaults to threads.
The shared Vitest config fixes isolate: false and applies the non-isolated runner across the root projects, e2e, and live configs.
The root UI lane retains its jsdom setup and optimizer, but also runs on the shared non-isolated runner.
Each pnpm test shard inherits the same threads + isolate: false defaults from the shared Vitest config.
scripts/run-vitest.mjs adds --no-maglev for Vitest child Node processes by default to cut V8 compile churn during large local runs. Set OPENCLAW_VITEST_ENABLE_MAGLEV=1 to compare against stock V8 behavior.
scripts/run-vitest.mjs stops explicit non-watch Vitest runs after 5 minutes with no stdout or stderr output. Set OPENCLAW_VITEST_NO_OUTPUT_TIMEOUT_MS=0 to disable the watchdog for an intentionally silent investigation.

Fast local iteration

pnpm changed:lanes reveals which architectural lanes a diff touches.
The pre-commit hook only handles formatting. It re-stages formatted files and skips lint, typecheck, and tests.
When you need the smart local check gate, run pnpm check:changed explicitly before handoff or push.
By default, pnpm test:changed sends work through cheap scoped lanes. Only use OPENCLAW_TEST_CHANGED_BROAD=1 pnpm test:changed when the agent determines that a harness, config, package, or contract edit truly needs broader Vitest coverage.
pnpm test:max and pnpm test:changed:max follow the same routing rules, but with a higher worker cap.
Local worker auto-scaling stays deliberately cautious and pulls back when the host load average is already elevated, so concurrent Vitest runs cause less damage by default.
The base Vitest config marks the projects/config files as forceRerunTriggers, keeping changed-mode reruns accurate when test wiring shifts.
The config leaves OPENCLAW_VITEST_FS_MODULE_CACHE active on supported hosts; set OPENCLAW_VITEST_FS_MODULE_CACHE_PATH=/abs/path for a single explicit cache location when profiling directly.

Perf debugging

pnpm test:perf:imports turns on Vitest import-duration reporting and import-breakdown output.
pnpm test:perf:imports:changed limits the same profiling view to files modified since origin/main.
Shard timing data lands in .artifacts/vitest-shard-timings.json. Whole-config runs key on the config path; include-pattern CI shards append the shard name so filtered shards get tracked separately.
If one hot test still burns most of its time in startup imports, keep heavy dependencies behind a narrow local *.runtime.ts seam and mock that seam directly, rather than deep-importing runtime helpers just to pass them through vi.mock(...).
pnpm test:perf:changed:bench -- --ref <git-ref> compares routed test:changed against the native root-project path for that committed diff and outputs wall time plus macOS max RSS.
pnpm test:perf:changed:bench -- --worktree benchmarks the current dirty tree by sending the changed file list through scripts/test-projects.mts and the root Vitest config.
pnpm test:perf:profile:main writes a main-thread CPU profile covering Vitest/Vite startup and transform overhead.
pnpm test:perf:profile:runner writes runner CPU+heap profiles for the unit suite with file parallelism off.

Stability (gateway)

Command: pnpm test:stability:gateway
Config: test/vitest/vitest.gateway.config.ts, test/vitest/vitest.logging.config.ts, and test/vitest/vitest.infra.config.ts, each capped at one worker
Scope:
- Launches a real loopback Gateway with diagnostics on by default
- Pushes synthetic gateway message, memory, and large-payload churn through the diagnostic event path
- Queries diagnostics.stability over the Gateway WS RPC
- Covers diagnostic stability bundle persistence helpers
- Verifies the recorder stays bounded, synthetic RSS samples remain under the pressure budget, and per-session queue depths drain back to zero
Expectations:
- CI-safe and keyless
- Narrow lane for stability-regression follow-up, not a replacement for the full Gateway suite

E2E (repo aggregate)

Command: pnpm test:e2e
Scope:
- Executes the gateway smoke E2E lane
- Executes the mocked Control UI browser E2E lane
Expectations:
- CI-safe and keyless
- Playwright Chromium must be installed

E2E (gateway smoke)

Command: pnpm test:e2e:gateway
Config: test/vitest/vitest.e2e.config.ts
Files: src/**/*.e2e.test.ts, test/**/*.e2e.test.ts, and bundled-plugin E2E tests under extensions/
Runtime defaults:
- Uses Vitest threads with isolate: false, matching the rest of the repo.
- Uses one worker by default to keep non-isolated gateway state deterministic.
- Runs in silent mode by default to cut console I/O overhead.
Useful overrides:
- OPENCLAW_E2E_WORKERS=<n> to opt into parallel workers (capped at 16).
- OPENCLAW_E2E_VERBOSE=1 to turn verbose console output back on.
Scope:
- Multi-instance gateway end-to-end behavior
- WebSocket/HTTP surfaces, node pairing, and heavier networking
Expectations:
- Runs in CI (when enabled in the pipeline)
- No real keys required
- More moving parts than unit tests (can be slower)

E2E (Control UI mocked browser)

Command: pnpm test:ui:e2e
Config: test/vitest/vitest.ui-e2e.config.ts
Files: ui/src/**/*.e2e.test.ts
Scope:
- Starts the Vite Control UI
- Drives a real Chromium page through Playwright
- Swaps the Gateway WebSocket for deterministic in-browser mocks
Expectations:
- Runs in CI as part of pnpm test:e2e
- No real Gateway, agents, or provider keys required
- Browser dependency must be present (pnpm --dir ui exec playwright install chromium)

E2E: OpenShell backend smoke

Command: pnpm test:e2e:openshell
File: extensions/openshell/src/backend.e2e.test.ts
Scope:
- Reuses an active local OpenShell gateway
- Builds a sandbox from a temporary local Dockerfile
- Exercises OpenClaw's OpenShell backend over real sandbox ssh-config + SSH exec
- Checks remote-canonical filesystem behavior through the sandbox fs bridge
Expectations:
- Opt-in only; not part of the default pnpm test:e2e run
- Needs a local openshell CLI plus a working Docker daemon
- Needs an active local OpenShell gateway and its config source
- Uses isolated HOME / XDG_CONFIG_HOME, then destroys the test sandbox
Useful overrides:
- OPENCLAW_E2E_OPENSHELL=1 to enable the test when running the broader e2e suite manually
- OPENCLAW_E2E_OPENSHELL_COMMAND=/path/to/openshell to point at a non-default CLI binary or wrapper script
- OPENCLAW_E2E_OPENSHELL_CONFIG_HOME=/path/to/config to expose the registered gateway config to the isolated test
- OPENCLAW_E2E_OPENSHELL_HOST_IP=172.18.0.1 to override the Docker gateway IP used by the host policy fixture

Live (real providers + real models)

Command: pnpm test:live
Config: test/vitest/vitest.live.config.ts
Files: src/**/*.live.test.ts, test/**/*.live.test.ts, plus bundled-plugin live tests located under extensions/
Default: enabled through pnpm test:live (which assigns OPENCLAW_LIVE_TEST=1)
Scope:
- "Can this provider/model actually function today with genuine credentials?"
- Detect provider format shifts, tool-calling oddities, authentication failures, and rate limit behavior
Expectations:
- Intentionally not CI-stable (real networks, provider policies, quotas, outages)
- Incurs costs / consumes rate limits
- Better to run targeted subsets rather than "everything"
Live runs rely on previously exported API keys and staged auth profiles.
By default, live runs still isolate HOME and duplicate config/auth data into a temporary test home, so unit fixtures cannot alter your actual ~/.openclaw.
Set OPENCLAW_LIVE_USE_REAL_HOME=1 only when you deliberately need live tests to access your real home directory.
pnpm test:live defaults to a quieter mode: it preserves [live] ... progress output and suppresses gateway bootstrap logs/Bonjour chatter. Set OPENCLAW_LIVE_TEST_QUIET=0 if you want the complete startup logs restored.
API key rotation (provider-specific): configure *_API_KEYS with comma/semicolon syntax or *_API_KEY_1, *_API_KEY_2 (for instance OPENAI_API_KEYS, ANTHROPIC_API_KEYS, GEMINI_API_KEYS) or per-live override via OPENCLAW_LIVE_*_KEY; tests retry upon rate limit responses.
Progress/heartbeat output:
- Live suites write progress lines to stderr, so lengthy provider calls remain visibly active even when Vitest console capture is quiet.
- test/vitest/vitest.live.config.ts turns off Vitest console interception, letting provider/gateway progress lines stream immediately during live runs.
- Adjust direct-model heartbeats with OPENCLAW_LIVE_HEARTBEAT_MS.
- Adjust gateway/probe heartbeats with OPENCLAW_LIVE_GATEWAY_HEARTBEAT_MS.

Which suite should I run?

Refer to this decision table:

Editing logic/tests: execute pnpm test (and pnpm test:coverage if you modified a substantial amount)
Touching gateway networking / WS protocol / pairing: include pnpm test:e2e
Debugging "my bot is down" / provider-specific failures / tool calling: run a focused pnpm test:live

Live (network-touching) tests

For the live model matrix, CLI backend smokes, ACP smokes, Codex app-server harness, and all media-provider live tests (Deepgram, BytePlus, ComfyUI, image, music, video, media harness), plus credential handling for live runs

see Testing live suites. For the dedicated update and plugin validation checklist, see Testing updates and plugins.

Docker runners (optional "works in Linux" checks)

These Docker runners divide into two categories:

Live-model runners: test:docker:live-models and test:docker:live-gateway execute only their corresponding profile-key live file within the repo's Docker image (src/agents/models.profiles.live.test.ts and src/gateway/gateway-models.profiles.live.test.ts), while your local config directory, workspace, and optional profile env file are mounted. The local entrypoints that match are test:live:models-profiles and test:live:gateway-profiles.
Docker live runners maintain practical limits where necessary: test:docker:live-models defaults to the curated supported high-signal set, and test:docker:live-gateway defaults to OPENCLAW_LIVE_GATEWAY_SMOKE=1, OPENCLAW_LIVE_GATEWAY_MAX_MODELS=8, OPENCLAW_LIVE_GATEWAY_STEP_TIMEOUT_MS=45000, and OPENCLAW_LIVE_GATEWAY_MODEL_TIMEOUT_MS=90000. Set OPENCLAW_LIVE_MAX_MODELS or the gateway env vars when you explicitly want a smaller cap or larger scan.
test:docker:all builds the live Docker image once via test:docker:live-build, packs OpenClaw once as an npm tarball through scripts/package-openclaw-for-docker.mjs, then builds or reuses two scripts/e2e/Dockerfile images. The bare image serves only as the Node/Git runner for install, update, and plugin-dependency lanes; those lanes mount the prebuilt tarball. The functional image installs the same tarball into /app for built-app functionality lanes. Docker lane definitions are in scripts/lib/docker-e2e-scenarios.mts; planner logic is in scripts/lib/docker-e2e-plan.mts; scripts/test-docker-all.mjs executes the chosen plan. The aggregate uses a weighted local scheduler: OPENCLAW_DOCKER_ALL_PARALLELISM controls process slots, while resource caps prevent heavy live, npm-install, and multi-service lanes from starting simultaneously. If a single lane exceeds the active caps, the scheduler can still start it when the pool is empty and then keeps it running alone until capacity becomes available. Defaults are 10 slots, OPENCLAW_DOCKER_ALL_LIVE_LIMIT=9, OPENCLAW_DOCKER_ALL_NPM_LIMIT=5, and OPENCLAW_DOCKER_ALL_SERVICE_LIMIT=7; adjust OPENCLAW_DOCKER_ALL_WEIGHT_LIMIT or OPENCLAW_DOCKER_ALL_DOCKER_LIMIT (and other OPENCLAW_DOCKER_ALL_<RESOURCE>_LIMIT overrides) only when the Docker host has extra headroom. The runner performs a Docker preflight by default, removes stale OpenClaw E2E containers, prints status every 30 seconds, stores successful lane timings in .artifacts/docker-tests/lane-timings.json, and uses those timings to start longer lanes first on later runs. Use OPENCLAW_DOCKER_ALL_DRY_RUN=1 to print the weighted lane manifest without building or running Docker, or node scripts/test-docker-all.mjs --plan-json to print the CI plan for selected lanes, package and image needs, and credentials.
Package Acceptance is the GitHub-native package gate for "does this installable tarball work as a product?" It resolves one candidate package from source=npm, source=ref, source=url, source=trusted-url, or source=artifact, uploads it as package-under-test, then runs the reusable Docker E2E lanes against that exact tarball instead of repacking the selected ref. Profiles are ordered by breadth: smoke, package, product, and full (plus custom for an explicit lane list). See Testing updates and plugins for the package, update, and plugin contract, published-upgrade survivor matrix, release defaults, and failure triage.
After tsdown completes, scripts/check-cli-bootstrap-imports.mts handles build and release verification. Starting from dist/entry.js and dist/cli/run-main.js, the guard traverses the static built graph and aborts if any external package (Commander, prompt UI, undici, logging, and other startup-heavy dependencies all qualify) is statically imported by that pre-dispatch bootstrap graph before command dispatch occurs. Additionally, it limits the bundled gateway run chunk to 70 KB and blocks static imports of recognized cold gateway paths (control-ui-assets, diagnostic-stability-bundle, onboard-helpers, process-respawn, restart-sentinel, server-close, server-reload-handlers) from that chunk. Separately, scripts/release-check.ts performs smoke tests on the packed CLI using --help, onboard --help, doctor --help, status --json --timeout 1, config schema, and models list --provider openai.
Legacy compatibility for Package Acceptance is limited to 2026.4.25 (with 2026.4.25-beta.* included). Up to that cutoff, the harness only permits gaps in shipped-package metadata: omitted private QA inventory entries, absent gateway install --wrapper, missing patch files in the tarball-derived git fixture, no persisted update.channel, legacy plugin install-record locations, missing marketplace install-record persistence, and config metadata migration during plugins update. For packages after 2026.4.25, these scenarios become strict failures.
Container smoke runners: test:docker:openwebui, test:docker:onboard, test:docker:npm-onboard-channel-agent, test:docker:release-user-journey, test:docker:release-typed-onboarding, test:docker:release-media-memory, test:docker:release-upgrade-user-journey, test:docker:release-plugin-marketplace, test:docker:skill-install, test:docker:update-channel-switch, test:docker:upgrade-survivor, test:docker:published-upgrade-survivor, test:docker:session-runtime-context, test:docker:agents-delete-shared-workspace, test:docker:gateway-network, test:docker:browser-cdp-snapshot, test:docker:mcp-channels, test:docker:agent-bundle-mcp-tools, test:docker:cron-mcp-cleanup, test:docker:plugins, test:docker:plugin-update, test:docker:plugin-lifecycle-matrix, and test:docker:config-reload launch one or more real containers and validate higher-level integration paths.
Docker/Bash E2E lanes that install the packed OpenClaw tarball via scripts/lib/openclaw-e2e-instance.sh set npm install to OPENCLAW_E2E_NPM_INSTALL_TIMEOUT (default 600s; use 0 to turn off the wrapper for debugging).

For live-model Docker runners, only the necessary CLI auth homes are bind-mounted (or all supported ones when the run isn't narrowed), then copied into the container home before execution so external-CLI OAuth can refresh tokens without altering the host auth store.

Direct model checks: pnpm test:docker:live-models (executed via scripts/test-live-models-docker.sh)
ACP bind smoke test: pnpm test:docker:live-acp-bind (run through scripts/test-live-acp-bind-docker.sh; defaults to Claude, Codex, and Gemini, with strict Droid/OpenCode checks enabled by pnpm test:docker:live-acp-bind:droid and pnpm test:docker:live-acp-bind:opencode)
CLI backend smoke test: pnpm test:docker:live-cli-backend (using scripts/test-live-cli-backend-docker.sh)
Codex app-server harness smoke test: pnpm test:docker:live-codex-harness (via scripts/test-live-codex-harness-docker.sh)
Gateway and dev agent: pnpm test:docker:live-gateway (through scripts/test-live-gateway-models-docker.sh)
Observability smoke tests: pnpm qa:otel:smoke, pnpm qa:prometheus:smoke, and pnpm qa:observability:smoke are private QA source-checkout lanes. They are deliberately excluded from package Docker release lanes because the npm tarball does not include QA Lab.
Open WebUI live smoke test: pnpm test:docker:openwebui (run with scripts/e2e/openwebui-docker.sh)
Onboarding wizard (TTY, full scaffolding): pnpm test:docker:onboard (executed via scripts/e2e/onboard-docker.sh)
Npm tarball onboarding/channel/agent smoke test: pnpm test:docker:npm-onboard-channel-agent globally installs the packed OpenClaw tarball in Docker, sets up OpenAI through env-ref onboarding and Telegram by default, runs doctor, and executes one mocked OpenAI agent turn. You can reuse a prebuilt tarball with OPENCLAW_CURRENT_PACKAGE_TGZ=/path/to/openclaw-*.tgz, avoid the host rebuild using OPENCLAW_NPM_ONBOARD_HOST_BUILD=0, or change channels via OPENCLAW_NPM_ONBOARD_CHANNEL=discord or OPENCLAW_NPM_ONBOARD_CHANNEL=slack.
Release user journey smoke: pnpm test:docker:release-user-journey performs a global install of the packed OpenClaw tarball within a clean Docker home, walks through onboarding, sets up a mocked OpenAI provider, executes a single agent turn, handles external plugin installation and removal, points ClickClack at a local fixture, checks messaging in both directions, restarts Gateway, and invokes doctor.
Release typed onboarding smoke: pnpm test:docker:release-typed-onboarding installs the packed tarball, guides openclaw onboard through a genuine TTY session, configures OpenAI as an env-ref provider, confirms no raw key is stored, and runs a mocked agent turn.
Release media/memory smoke: pnpm test:docker:release-media-memory installs the packed tarball, confirms image comprehension from a PNG attachment, validates OpenAI-compatible image generation output, checks memory search recall, and ensures recall persists across a Gateway restart.
Release upgrade user journey smoke: pnpm test:docker:release-upgrade-user-journey installs the newest published baseline older than the candidate tarball by default, configures provider/plugin/ClickClack state on the published package, upgrades to the candidate tarball, then repeats the core agent/plugin/channel journey. When no older published baseline is available, the candidate version is reused. Use OPENCLAW_RELEASE_UPGRADE_BASELINE_SPEC=openclaw@<version> to override the baseline.
Release plugin marketplace smoke: pnpm test:docker:release-plugin-marketplace installs from a local fixture marketplace, updates the installed plugin, removes it, and confirms the plugin CLI vanishes along with pruned install metadata.
Skill install smoke: pnpm test:docker:skill-install installs the packed OpenClaw tarball globally in Docker, disables uploaded archive installs in config, resolves the current live ClawHub skill slug from search, installs it with openclaw skills install, and verifies the installed skill plus .clawhub origin/lock metadata.
Update channel switch smoke: pnpm test:docker:update-channel-switch installs the packed OpenClaw tarball globally in Docker, switches from package stable to git dev, verifies the persisted channel and plugin post-update work, then switches back to package stable and checks update status.
Upgrade survivor smoke: pnpm test:docker:upgrade-survivor installs the packed OpenClaw tarball over a dirty old-user fixture with agents, channel config, plugin allowlists, stale plugin dependency state, and existing workspace/session files. It runs package update plus non-interactive doctor without live provider or channel keys, then starts a loopback Gateway and checks config/state preservation plus startup/status budgets.
Published upgrade survivor smoke: pnpm test:docker:published-upgrade-survivor installs openclaw@latest by default, seeds realistic existing-user files, configures that baseline with a baked command recipe, validates the resulting config, updates that published install to the candidate tarball, runs non-interactive doctor, writes .artifacts/upgrade-survivor/summary.json, then starts a loopback Gateway and checks configured intents, state preservation, startup, /healthz, /readyz, and RPC status budgets. Override one baseline with OPENCLAW_UPGRADE_SURVIVOR_BASELINE_SPEC, ask the aggregate scheduler to expand exact local baselines with OPENCLAW_UPGRADE_SURVIVOR_BASELINE_SPECS such as openclaw@2026.5.2 openclaw@2026.4.23 openclaw@2026.4.15, and expand issue-shaped fixtures with OPENCLAW_UPGRADE_SURVIVOR_SCENARIOS such as reported-issues; the reported-issues set includes configured-plugin-installs for automatic external OpenClaw plugin install repair. Package Acceptance exposes those as published_upgrade_survivor_baseline, published_upgrade_survivor_baselines, and published_upgrade_survivor_scenarios, resolves meta baseline tokens such as last-stable-4 or all-since-2026.4.23, and Full Release Validation expands the release-soak package gate to last-stable-4 2026.4.23 2026.5.2 2026.4.15 plus reported-issues.
Session runtime context smoke: pnpm test:docker:session-runtime-context verifies hidden runtime context transcript persistence plus doctor repair of affected duplicated prompt-rewrite branches.
Bun global install smoke: bash scripts/e2e/bun-global-install-smoke.sh packs the current tree, installs it with bun install -g in an isolated home, and verifies openclaw infer image providers --json returns bundled image providers instead of hanging. Reuse a prebuilt tarball with OPENCLAW_BUN_GLOBAL_SMOKE_PACKAGE_TGZ=/path/to/openclaw-*.tgz, skip the host build with OPENCLAW_BUN_GLOBAL_SMOKE_HOST_BUILD=0, or copy dist/ from a built Docker image with OPENCLAW_BUN_GLOBAL_SMOKE_DIST_IMAGE=openclaw-dockerfile-smoke:local.
Installer Docker smoke: bash scripts/test-install-sh-docker.sh shares one npm cache across its root, update, and direct-npm containers. Update smoke defaults to npm latest as the stable baseline before upgrading to the candidate tarball. Override with OPENCLAW_INSTALL_SMOKE_UPDATE_BASELINE=2026.4.22 locally, or with the Install Smoke workflow's update_baseline_version input on GitHub. Non-root installer checks keep an isolated npm cache so root-owned cache entries do not mask user-local install behavior. Set OPENCLAW_INSTALL_SMOKE_NPM_CACHE_DIR=/path/to/cache to reuse the root/update/direct-npm cache across local reruns.
Install Smoke CI skips the duplicate direct-npm global update with OPENCLAW_INSTALL_SMOKE_SKIP_NPM_GLOBAL=1; run the script locally without that env when direct npm install -g coverage is needed.
Agents delete shared workspace CLI smoke: pnpm test:docker:agents-delete-shared-workspace (script: scripts/e2e/agents-delete-shared-workspace-docker.sh) builds the root Dockerfile image by default, seeds two agents with one workspace in an isolated container home, runs agents delete --json, and verifies valid JSON plus retained workspace behavior. Reuse the install-smoke image with OPENCLAW_AGENTS_DELETE_SHARED_WORKSPACE_E2E_IMAGE=openclaw-dockerfile-smoke:local OPENCLAW_AGENTS_DELETE_SHARED_WORKSPACE_E2E_SKIP_BUILD=1.
Gateway networking and host lifecycle: pnpm test:docker:gateway-network (script: scripts/e2e/gateway-network-docker.sh) preserves the two-container LAN WebSocket auth/health smoke, then uses loopback Admin HTTP to prove prepare fencing, retained-control access, resume recovery, and a prepared same-container stop/start. The restart check must finish before the original lease expires, verifies that suspension state is process-local while persisted Gateway config and container identity survive, and emits machine-readable phase timing JSON.
Browser CDP snapshot smoke: pnpm test:docker:browser-cdp-snapshot (script: scripts/e2e/browser-cdp-snapshot-docker.sh) builds the source E2E image plus a Chromium layer, starts Chromium with raw CDP, runs browser doctor --deep, and verifies CDP role snapshots cover link URLs, cursor-promoted clickables, iframe refs, and frame metadata.
OpenAI Responses web_search minimal reasoning regression: pnpm test:docker:openai-web-search-minimal (script: scripts/e2e/openai-web-search-minimal-docker.sh) launches a mocked OpenAI server via Gateway, confirms web_search escalates reasoning.effort from minimal to low, then triggers the provider schema rejection and confirms the raw detail shows up in Gateway logs.
MCP channel bridge (seeded Gateway + stdio bridge + raw Claude notification-frame smoke): pnpm test:docker:mcp-channels (script: scripts/e2e/mcp-channels-docker.sh)
OpenClaw bundle MCP tools (real stdio MCP server + embedded OpenClaw profile allow/deny smoke): pnpm test:docker:agent-bundle-mcp-tools (script: scripts/e2e/agent-bundle-mcp-tools-docker.sh)
Cron/subagent MCP cleanup (real Gateway + stdio MCP child teardown after isolated cron and one-shot subagent runs): pnpm test:docker:cron-mcp-cleanup (script: scripts/e2e/cron-mcp-cleanup-docker.sh)
Plugins (install/update smoke for local path, file:, npm registry with hoisted dependencies, malformed npm package metadata, git moving refs, ClawHub kitchen-sink, marketplace updates, and Claude-bundle enable/inspect): pnpm test:docker:plugins (script: scripts/e2e/plugins-docker.sh) Set OPENCLAW_PLUGINS_E2E_CLAWHUB=0 to skip the ClawHub block, or override the default kitchen-sink package/runtime pair with OPENCLAW_PLUGINS_E2E_CLAWHUB_SPEC and OPENCLAW_PLUGINS_E2E_CLAWHUB_ID. Without OPENCLAW_CLAWHUB_URL/CLAWHUB_URL, the test relies on a hermetic local ClawHub fixture server.
Plugin update unchanged smoke: pnpm test:docker:plugin-update (script: scripts/e2e/plugin-update-unchanged-docker.sh)
Plugin lifecycle matrix smoke: pnpm test:docker:plugin-lifecycle-matrix installs the packed OpenClaw tarball in a bare container, installs an npm plugin, toggles enable/disable, upgrades and downgrades it through a local npm registry, deletes the installed code, then verifies uninstall still removes stale state while logging RSS/CPU metrics for each lifecycle phase.
Config reload metadata smoke: pnpm test:docker:config-reload (script: scripts/e2e/config-reload-source-docker.sh)
Plugins: pnpm test:docker:plugins covers install/update smoke for local path, file:, npm registry with hoisted dependencies, git moving refs, ClawHub fixtures, marketplace updates, and Claude-bundle enable/inspect. pnpm test:docker:plugin-update covers unchanged update behavior for installed plugins. pnpm test:docker:plugin-lifecycle-matrix covers resource-tracked npm plugin install, enable, disable, upgrade, downgrade, and missing-code uninstall.

To prebuild and reuse the shared functional image manually:

OPENCLAW_DOCKER_E2E_IMAGE=openclaw-docker-e2e-functional:local pnpm test:docker:e2e-build
OPENCLAW_DOCKER_E2E_IMAGE=openclaw-docker-e2e-functional:local OPENCLAW_SKIP_DOCKER_BUILD=1 pnpm test:docker:mcp-channels

Suite-specific image overrides such as OPENCLAW_GATEWAY_NETWORK_E2E_IMAGE still win when set. When OPENCLAW_SKIP_DOCKER_BUILD=1 points at a remote shared image, the scripts pull it if it is not already local. The QR and installer Docker tests keep their own Dockerfiles because they validate package/install behavior rather than the shared built-app runtime.

The live-model Docker runners also bind-mount the current checkout read-only and stage it into a temporary workdir inside the container. This keeps the runtime image slim while still running Vitest against your exact local source/config. The staging step skips large local-only caches and app build outputs such as .pnpm-store, .worktrees, __openclaw_vitest__, and app-local .build or Gradle output directories so Docker live runs do not spend minutes copying machine-specific artifacts. They also set OPENCLAW_SKIP_CHANNELS=1 so gateway live probes do not start real Telegram/Discord/etc. channel workers inside the container. test:docker:live-models still runs pnpm test:live, so pass through OPENCLAW_LIVE_GATEWAY_* as well when you need to narrow or exclude gateway live coverage from that Docker lane.

test:docker:openwebui is a higher-level compatibility smoke: it starts an OpenClaw gateway container with the OpenAI-compatible HTTP endpoints enabled, starts a pinned Open WebUI container against that gateway, signs in through Open WebUI, verifies /api/models exposes openclaw/default, then sends a real chat request through Open WebUI's /api/chat/completions proxy. Set OPENWEBUI_SMOKE_MODE=models for release-path CI checks that should stop after Open WebUI sign-in and model discovery, without waiting on a live model completion. The first run can be noticeably slower because Docker may need to pull the Open WebUI image and Open WebUI may need to finish its own cold-start setup. This lane expects a usable live model key, provided through the process environment, staged auth profiles, or an explicit OPENCLAW_PROFILE_FILE. Successful runs print a small JSON payload like { "ok": true, "model": "openclaw/default", ... }.

test:docker:mcp-channels is intentionally deterministic and does not need a real Telegram, Discord, or iMessage account. It boots a seeded Gateway container, starts a second container that spawns openclaw mcp serve, then verifies routed conversation discovery, transcript reads, attachment metadata, live event queue behavior, outbound send routing, and Claude-style channel + permission notifications over the real stdio MCP bridge. The notification check inspects the raw stdio MCP frames directly so the smoke validates what the bridge actually emits, not just what a specific client SDK happens to surface.

test:docker:agent-bundle-mcp-tools is deterministic and does not need a live model key. It builds the repo Docker image, starts a real stdio MCP probe server inside the container, materializes that server through the embedded OpenClaw bundle MCP runtime, executes the tool, then verifies coding and messaging keep bundle-mcp tools while minimal and tools.deny: ["bundle-mcp"] filter them.

test:docker:cron-mcp-cleanup is deterministic and does not need a live model key. It starts a seeded Gateway with a real stdio MCP probe server, runs an isolated cron turn and a sessions_spawn one-shot child turn, then verifies the MCP child process exits after each run.

Manual ACP plain-language thread smoke (not CI):

bun scripts/dev/discord-acp-plain-language-smoke.ts --channel <discord-channel-id> ...
Keep this script for regression/debug workflows. It may be needed again for ACP thread routing validation, so do not delete it.

Useful env vars:

OPENCLAW_CONFIG_DIR=... (default: ~/.openclaw) attached to /home/node/.openclaw
OPENCLAW_WORKSPACE_DIR=... (default: ~/.openclaw/workspace) attached to /home/node/.openclaw/workspace
OPENCLAW_PROFILE_FILE=... mounted and sourced ahead of test execution
OPENCLAW_DOCKER_PROFILE_ENV_ONLY=1 to confirm that only env vars sourced from OPENCLAW_PROFILE_FILE are present, using temporary config/workspace directories and no external CLI auth mounts
OPENCLAW_DOCKER_CLI_TOOLS_DIR=... (default: ~/.cache/openclaw/docker-cli-tools, unless the run already employs a CI/managed bind dir) attached to /home/node/.npm-global for cached CLI installs within Docker
External CLI auth dirs/files under $HOME are mounted read-only under /host-auth..., then transferred into /home/node/... before tests commence
- Default dirs (applied when the run is not restricted to specific providers): .factory, .gemini, .minimax
- Default files: ~/.codex/auth.json, ~/.codex/config.toml, .claude.json, ~/.claude/.credentials.json, ~/.claude/settings.json, ~/.claude/settings.local.json
- Narrowed provider runs attach only the required dirs/files deduced from OPENCLAW_LIVE_PROVIDERS / OPENCLAW_LIVE_GATEWAY_PROVIDERS
- Manually override with OPENCLAW_DOCKER_AUTH_DIRS=all, OPENCLAW_DOCKER_AUTH_DIRS=none, or a comma-separated list like OPENCLAW_DOCKER_AUTH_DIRS=.claude,.codex
OPENCLAW_LIVE_GATEWAY_MODELS=... / OPENCLAW_LIVE_MODELS=... to limit the run
OPENCLAW_LIVE_GATEWAY_PROVIDERS=... / OPENCLAW_LIVE_PROVIDERS=... to filter providers inside the container
OPENCLAW_SKIP_DOCKER_BUILD=1 to reuse an existing openclaw:local-live image for reruns that skip rebuilding
OPENCLAW_LIVE_REQUIRE_PROFILE_KEYS=1 to guarantee creds originate from the profile store (not env)
OPENCLAW_OPENWEBUI_MODEL=... to pick the model the gateway exposes for the Open WebUI smoke
OPENCLAW_OPENWEBUI_PROMPT=... to override the nonce-check prompt used by the Open WebUI smoke
OPENWEBUI_IMAGE=... to override the pinned Open WebUI image tag

Docs sanity

After editing docs, execute checks with pnpm check:docs. When in-page heading checks are also required, run full Mintlify anchor validation via pnpm docs:check-links:anchors.

Offline regression (CI-safe)

These are "real pipeline" regressions without actual providers:

Gateway tool calling (mock OpenAI, real gateway + agent loop): src/gateway/gateway.test.ts (case: "runs a mock OpenAI tool call end-to-end via gateway agent loop")
Gateway wizard (WS wizard.start/wizard.next, writes config + auth enforced): src/gateway/gateway.test.ts (case: "runs wizard over ws and writes auth token config")

Agent reliability evals (skills)

A handful of CI-safe tests already exist that resemble "agent reliability evals":

Mock tool-calling through the real gateway + agent loop (src/gateway/gateway.test.ts).
End-to-end wizard flows that verify session wiring and config effects (src/gateway/gateway.test.ts).

For skills, the following remains absent (see Skills):

Decisioning: when skills appear in the prompt, does the agent choose the correct skill (or skip irrelevant ones)?
Compliance: does the agent read SKILL.md before use and adhere to required steps/args?
Workflow contracts: multi-turn scenarios that verify tool order, session history carryover, and sandbox boundaries.

Future evals should prioritize determinism first:

A scenario runner using mock providers to assert tool calls + order, skill file reads, and session wiring.
A compact suite of skill-focused scenarios (use vs avoid, gating, prompt injection).
Optional live evals (opt-in, env-gated) only after the CI-safe suite is established.

Contract tests (plugin and channel shape)

Contract tests confirm that every registered plugin and channel conforms to its interface contract. They traverse all discovered plugins and execute a series of shape and behavior assertions. The default pnpm test unit lane intentionally omits these shared seam and smoke files; run the contract commands explicitly when modifying shared channel or provider surfaces.

Commands

All contracts: pnpm test:contracts
Channel contracts only: pnpm test:contracts:channels
Provider contracts only: pnpm test:contracts:plugins

Channel contracts

Found in src/channels/plugins/contracts/*.contract.test.ts. Current top-level categories:

channel-catalog - metadata for bundled or registry channel catalog entries
plugin (registry-backed, sharded) - the fundamental plugin registration structure
surfaces-only (registry-backed, sharded) - per-surface shape validations covering actions, setup, status, outbound, messaging, threading, directory, and gateway
session-binding (registry-backed) - how session binding behaves
outbound-payload - message payload layout and its normalization
group-policy (fallback) - default group policy application per channel
threading (registry-backed, sharded) - thread id management
directory (registry-backed, sharded) - the directory or roster API
registry and plugins-core.* - internals for the channel plugin registry, its loader, and config-write authorization

The inbound dispatch-capture and outbound-payload harness helpers that these suites rely on are made available internally via src/plugin-sdk/channel-contract-testing.ts (excluded from npm, not a public SDK subpath); no standalone inbound.contract.test.ts file exists in this directory.

Provider contracts

Found in src/plugins/contracts/*.contract.test.ts. The current categories are:

shape - plugin manifest, API, and runtime export structure
plugin-registration (+ parallel) - manifest registration scenarios
package-manifest - what a package manifest must satisfy
loader - plugin loader setup and teardown behavior
registry - plugin contract registry contents and lookups
providers - shared behavior across bundled providers, plus web-search providers
auth-choice - auth choice metadata and setup behavior
provider-catalog-deprecation - metadata for deprecated provider catalogs
wizard.choice-resolution, wizard.model-picker, wizard.setup-options - provider setup wizard contracts
embedding-provider, memory-embedding-provider, web-fetch-provider, tts - capability-specific provider contracts
session-actions, session-attachments, session-entry-projection - plugin-owned session state contracts
scheduled-turns - plugin scheduled turn metadata and timestamp bounds
host-hooks, run-context-lifecycle, runtime-import-side-effects, runtime-seams - plugin host and runtime lifecycle plus import-boundary contracts
extension-runtime-dependencies - runtime dependency placement for extensions

When to run

When plugin-sdk exports or subpaths change
When a channel or provider plugin is added or modified
When plugin registration or discovery is refactored

Contract tests execute in CI and do not need real API keys.

Adding regressions (guidance)

When addressing a provider or model issue found in live:

Add a CI-safe regression when feasible (mock or stub provider, or capture the exact request-shape transformation)
If the issue is inherently live-only (rate limits, auth policies), keep the live test narrow and opt-in through env vars
Aim for the smallest layer that catches the bug:
- provider request conversion or replay bug -> direct models test
- gateway session, history, or tool pipeline bug -> gateway live smoke or CI-safe gateway mock test
SecretRef traversal guardrail:
- src/secrets/exec-secret-ref-id-parity.test.ts derives one sampled target per SecretRef class from registry metadata (listSecretTargetRegistryEntries()), then verifies that traversal-segment exec ids are rejected.
- When adding a new includeInPlan SecretRef target family in src/secrets/target-registry-data.ts, update classifyTargetClass in that test. The test deliberately fails on unclassified target ids, so new classes cannot be silently skipped.