Running Tests Locally and Remotely: Vitest, Force, Coverage

• Full testing kit (suites, live, Docker): Testing
• Update and plugin package validation: Testing updates and plugins

Agent default

Agent sessions execute a handful of focused tests and low-cost static checks locally only when the source is trusted and the current dependency install is present. Running untrusted repository tooling on your machine is never allowed. Larger suites, modified gates with typecheck/lint fan-out, builds, Docker, package lanes, E2E, live proof, and cross-platform validation are dispatched remotely through Crabbox. Heavy proof from trusted maintainers defaults to Blacksmith Testbox. The configured Testbox workflow injects credentials, so untrusted contributor or fork code must rely on secretless fork CI or sanitized direct AWS Crabbox instead.

Avoid pre-warming for anticipated tasks. Fetch the backend lazily when the first heavy command is ready, reuse the returned tbx_... id for subsequent heavy commands, sync the current checkout on every run, and terminate it before handoff.

After the first successful reuse, the wrapper records the lease's base, dependency, and Testbox workflow fingerprint under .crabbox/testbox-leases/. Source-only edits continue reusing the warmed box. A changed merge base, lockfile, package-manager input, wrapper, or Testbox workflow fails closed and demands a fresh lease. Every run still syncs the current checkout. OPENCLAW_TESTBOX_ALLOW_STALE=1 is reserved for intentional diagnostics, not release proof.

The local test commands listed below serve human workflows and bounded agent proof. Remote-provider unavailability must be reported; it does not authorize silently running a broad local gate.

For untrusted heavy proof, warm lazily with --provider aws. Every run must set CRABBOX_ENV_ALLOW=CI, pass --provider aws --no-hydrate, and use a fresh temporary remote HOME before installing dependencies or running tests. Use a newly warmed lease dedicated to that untrusted source; never reuse a trusted or previously hydrated lease. Launch an installed trusted Crabbox binary from a clean trusted main checkout and fetch only the remote PR with --fresh-pr; never execute the untrusted checkout's wrapper or config locally. Unset CRABBOX_AWS_INSTANCE_PROFILE and fail closed unless resolved aws.instanceProfile is empty. Before any install/test, use trusted absolute-path tools to require an IMDSv2 token, prove the IAM credentials endpoint returns 404, and verify remote git rev-parse HEAD equals the full reviewed PR head SHA. Bind the lease to that SHA and stop/rewarm when the head changes. Upload trusted scripts/crabbox-untrusted-bootstrap.sh from clean main alongside --fresh-pr; it installs pinned Node/pnpm, verifies the SHA and package-manager pin, isolates HOME, installs dependencies, then executes the requested test. If the broker cannot prove no role or no remote PR exists, use secretless fork CI. Do not use hydrate-github, --no-sync, or a credential-hydrated Testbox workflow. Unset all CRABBOX_TAILSCALE* overrides, force --network public --tailscale=false, clear exit-node/LAN flags, and require crabbox inspect to report public networking with no Tailscale state before uploading any script.

Routine local order

1. pnpm test:changed for changed-scope Vitest proof.
2. pnpm test <path-or-filter> for one file, directory, or explicit target.
3. pnpm test only when you intentionally need the full local Vitest suite.

In a Codex worktree or linked/sparse checkout, agents avoid direct local pnpm test* / pnpm check* / pnpm crabbox:run:

• Bounded focused proof with ready dependencies: node scripts/run-vitest.mjs <path-or-filter>.
• Classify-first changed check: node scripts/check-changed.mjs; docs-only, no-change, and small metadata plans stay local when dependencies are ready, while heavy or dependency-missing plans delegate to Testbox.
• Explicit kept-lease broad proof: node scripts/crabbox-wrapper.mjs run --provider blacksmith-testbox ... -- env OPENCLAW_CHECK_CHANGED_REMOTE_CHILD=1 OPENCLAW_CHANGED_LANES_RAW_SYNC=1 corepack pnpm check:changed so pnpm runs inside Testbox.
• The wrapper's final exitCode and timing JSON are the command result. A delegated Blacksmith GitHub Actions run may show cancelled after a successful SSH command because the Testbox is stopped from outside the keepalive action; check the wrapper summary and command output before treating that as a failure.
• OPENCLAW_HEAVY_CHECK_LOCK_SCOPE=worktree <local-heavy-check command>: keeps heavy-check serialization inside the current worktree instead of the Git common dir for commands such as pnpm check:changed and targeted pnpm test .... Use it only on high-capacity local hosts when you intentionally run independent checks across linked worktrees.

Core commands

Test wrapper runs end with a short [test] passed|failed|skipped ... in ... summary; Vitest's own duration line stays the per-shard detail.

Shared test state and process helpers

• src/test-utils/openclaw-test-state.ts: pull this in from Vitest whenever a test demands its own isolated HOME, OPENCLAW_STATE_DIR, OPENCLAW_CONFIG_PATH, config fixture, workspace, agent dir, or auth-profile store.
• pnpm test:env-mutations:report: a non-blocking report of tests and harnesses that write directly to HOME, OPENCLAW_STATE_DIR, OPENCLAW_CONFIG_PATH, OPENCLAW_WORKSPACE_DIR, or adjacent env keys. Use it to spot candidates for moving to the shared test-state helper.
• test/helpers/openclaw-test-instance.ts: process-level E2E tests that need a live Gateway, CLI env, log capture, and teardown all handled in a single spot.
• Docker and Bash E2E lanes that load scripts/lib/docker-e2e-image.sh can hand docker_e2e_test_state_shell_b64 <label> <scenario> to the container and extract it via scripts/lib/openclaw-e2e-instance.sh; multi-home scripts can supply docker_e2e_test_state_function_b64 and invoke openclaw_test_state_create <label> <scenario> in every flow. node --import tsx scripts/lib/openclaw-test-state.mts -- create --label <name> --scenario <name> --env-file <path> --json emits a sourceable host env file (the -- placed before create stops newer Node runtimes from reading --env-file as a Node flag). Lanes that spin up a Gateway can source scripts/lib/openclaw-e2e-instance.sh for entrypoint resolution, mock OpenAI startup, foreground or background launch, readiness checks, state env export, log collection, and process shutdown.

Control UI, TUI, and extension lanes

• Control UI mocked E2E: pnpm test:ui:e2e executes the Vitest plus Playwright lane that boots the Vite Control UI and drives a real Chromium page against a mocked Gateway WebSocket. Tests are kept in ui/src/**/*.e2e.test.ts; shared mocks and controls sit in ui/src/test-helpers/control-ui-e2e.ts. pnpm test:e2e covers this lane. Agent runs default to Testbox and Crabbox, including targeted proof; reach for node scripts/run-vitest.mjs run --config test/vitest/vitest.ui-e2e.config.ts --configLoader runner ui/src/e2e/chat-flow.messaging.e2e.test.ts only when you need an explicit local fallback.
• TUI PTY tests: node scripts/run-vitest.mjs run --config test/vitest/vitest.tui-pty.config.ts runs the quick fake-backend PTY lane. OPENCLAW_TUI_PTY_INCLUDE_LOCAL=1 or pnpm tui:pty:test:watch --mode local triggers the slower tui --local smoke, which stubs only the external model endpoint. CI also sets OPENCLAW_TUI_PTY_USE_BUILT_CLI=1 after dist/ is built; that flag is only for when exact-head built artifacts already exist. Assert stable visible text or fixture calls, never raw ANSI snapshots.
• pnpm test:extensions and pnpm test extensions cover every extension and plugin shard. Heavy channel plugins, the browser plugin, and OpenAI each get their own shard; the remaining plugin groups stay batched. pnpm test extensions/<id> runs a single bundled plugin lane.
• Source files that have a sibling test resolve to that sibling first, then fall back to wider directory globs. Helper changes under src/channels/plugins/contracts/test-helpers, src/plugin-sdk/test-helpers, and src/plugins/contracts rely on a local import graph to run importing tests instead of sweeping every shard when the dependency path is exact.
• Contract directory targets fan out to their contract lanes: pnpm test src/channels/plugins/contracts runs the four channel contract configs and pnpm test src/plugins/contracts runs the plugin contracts config, because the generic channels and plugins projects leave out contracts/**.
• auto-reply is broken into three dedicated configs (core, top-level, reply) so the reply harness does not crowd out the lighter top-level status, token, and helper tests.
• Selected plugin-sdk and commands test files go through dedicated light lanes that keep only test/setup.ts, leaving runtime-heavy cases on their current lanes.
• The base Vitest config defaults to pool: "threads" and isolate: false, with the shared non-isolated runner enabled across all repo configs.
• pnpm test:channels runs vitest.channels.config.ts.

Gateway and E2E

• The untargeted pnpm test full suite already covers gateway tests; to execute them in isolation, use pnpm test:gateway.
• pnpm test:e2e: repo E2E aggregate = pnpm test:e2e:gateway && pnpm test:ui:e2e.
• pnpm test:e2e:gateway: gateway end-to-end smoke tests (multi-instance WS/HTTP/node pairing). Defaults to threads + isolate: false with one worker in vitest.e2e.config.ts; opt into parallelism with OPENCLAW_E2E_WORKERS=<n> (capped at 16), and enable verbose logs with OPENCLAW_E2E_VERBOSE=1.
• pnpm test:live: provider live tests (Claude/Minimax/DeepSeek/z.ai/etc, gated by *.live.test.ts). Requires API keys and LIVE=1 (or OPENCLAW_LIVE_TEST=1) to unskip; verbose output with OPENCLAW_LIVE_TEST_QUIET=0.

Full Docker suite (pnpm test:docker:all)

The shared live-test image gets built, OpenClaw is packed once as an npm tarball, a bare Node/Git runner image plus a functional image that installs that tarball into /app are built or reused, and Docker smoke lanes run through a weighted scheduler. scripts/package-openclaw-for-docker.mjs serves as the stable local/CI package packer entrypoint and checks the tarball plus dist/postinstall-inventory.json before Docker consumes it.

• Bare image (OPENCLAW_DOCKER_E2E_BARE_IMAGE): installer/update/plugin-dependency lanes; mounts the prebuilt tarball instead of copied repo sources.
• Functional image (OPENCLAW_DOCKER_E2E_FUNCTIONAL_IMAGE): normal built-app functionality lanes.
• Lane definitions: scripts/lib/docker-e2e-scenarios.mts. Planner: scripts/lib/docker-e2e-plan.mts. Executor: scripts/test-docker-all.mjs.
• node scripts/test-docker-all.mjs --plan-json emits the scheduler-owned CI plan (lanes, image kinds, package/live-image needs, state scenarios, credential checks) without building or running Docker.

Scheduling knobs (env vars, defaults in parentheses):

Env var pattern for resource caps is OPENCLAW_DOCKER_ALL_<RESOURCE>_LIMIT (resource name uppercased, non-alphanumerics collapsed to _).

Other runner behaviors: Docker is preflighted by default, stale OpenClaw E2E containers get cleaned up, provider CLI tool caches are shared across compatible lanes, and new pooled lanes stop being scheduled after the first failure unless OPENCLAW_DOCKER_ALL_FAIL_FAST=0 is set. On a low-parallelism host, a lane that exceeds the effective weight/resource cap can still launch from an empty pool and run solo until capacity frees up. Per-lane logs, summary.json, failures.json, and phase timings are stored under .artifacts/docker-tests/<run-id>/; use pnpm test:docker:timings <summary.json> to check slow lanes and pnpm test:docker:rerun <run-id|summary.json|failures.json> to output cheap targeted rerun commands.

Notable Docker lanes

Sandbox compatibility lanes

Local PR gate

For local PR land/gate checks, run:

• pnpm check:changed
• pnpm check
• pnpm check:test-types
• pnpm build
• pnpm test
• pnpm check:docs

When pnpm test behaves erratically on a busy machine, run it a second time before you classify it as a regression, then narrow things down with pnpm test <path/to/test>. On hosts where memory is tight:

• OPENCLAW_VITEST_MAX_WORKERS=1 pnpm test
• OPENCLAW_VITEST_FS_MODULE_CACHE_PATH=/tmp/openclaw-vitest-cache pnpm test:changed

Test performance tooling

• pnpm test:perf:imports: turns on Vitest import-duration and import-breakdown reporting, yet keeps scoped lane routing for explicit file or directory targets. The same profiling is limited to files modified since origin/main via pnpm test:perf:imports:changed.
• For an identical committed git diff, pnpm test:perf:changed:bench -- --ref <git-ref> compares the routed changed-mode path against the native root-project run; pnpm test:perf:changed:bench -- --worktree benchmarks the current worktree changes without requiring a commit first.
• A CPU profile for the Vitest main thread is written by pnpm test:perf:profile:main (.artifacts/vitest-main-profile); CPU and heap profiles for the unit runner come from pnpm test:perf:profile:runner (.artifacts/vitest-runner-profile).
• pnpm test:perf:groups --full-suite --allow-failures --output .artifacts/test-perf/baseline-before.json: executes every full-suite Vitest leaf config one after another and records grouped duration data along with per-config JSON and log artifacts. Full-suite reports separate files by default, so module graphs and GC pauses left over from earlier files do not get attributed to later assertions; pass -- --no-isolate only when you deliberately want to profile shared-worker buildup. Before attempting slow-test fixes, the Test Performance Agent relies on this as its baseline. After a performance-oriented change, pnpm test:perf:groups:compare .artifacts/test-perf/baseline-before.json .artifacts/test-perf/after-agent.json compares the grouped reports.
• Timing data for the local environment gets updated in .artifacts/vitest-shard-timings.json by full, extension, and include-pattern shard runs; later whole-config runs then use those timings to distribute slow and fast shards. Include-pattern CI shards append the shard name to the timing key, keeping filtered shard timings visible without overwriting whole-config timing data. Set OPENCLAW_TEST_PROJECTS_TIMINGS=0 to disregard the local timing artifact.

Benchmarks

Model latency (scripts/bench-model.ts)

pnpm tsx scripts/bench-model.ts --runs 10

Optional env: MINIMAX_API_KEY, MINIMAX_BASE_URL, MINIMAX_MODEL, ANTHROPIC_API_KEY. Default prompt: "Reply with a single word: ok. No punctuation or extra text."

CLI startup (scripts/bench-cli-startup.ts)

pnpm test:startup:bench
pnpm test:startup:bench:smoke
pnpm test:startup:bench:save
pnpm test:startup:bench:update
pnpm test:startup:bench:check
pnpm tsx scripts/bench-cli-startup.ts --runs 12
pnpm tsx scripts/bench-cli-startup.ts --preset real --case status --case gatewayStatus --runs 3
pnpm tsx scripts/bench-cli-startup.ts --entry openclaw.mjs --entry-secondary dist/entry.js --preset all

Presets:

• startup: --version, --help, health, health --json, status --json, status
• real: health, status, status --json, sessions, sessions --json, tasks --json, tasks list --json, tasks audit --json, agents list --json, gateway status, gateway status --json, gateway health --json, config get gateway.port
• all: both presets combined

Output includes sampleCount, avg, p50, p95, min/max, exit-code/signal distribution, and max RSS per command. --cpu-prof-dir / --heap-prof-dir write V8 profiles per run.

Saved output: pnpm test:startup:bench:smoke produces .artifacts/cli-startup-bench-smoke.json; pnpm test:startup:bench:save produces .artifacts/cli-startup-bench-all.json (runs=5 warmup=1). A checked-in fixture, test/fixtures/cli-startup-bench.json, gets refreshed by pnpm test:startup:bench:update and validated against pnpm test:startup:bench:check.

Gateway startup (scripts/bench-gateway-startup.ts)

The default points to the built CLI entry at dist/entry.js; make sure pnpm build runs beforehand. To benchmark the source runner instead, pass --entry scripts/run-node.mjs and store those numbers apart from the built-entry baselines.

pnpm test:startup:gateway -- --runs 5 --warmup 1
pnpm test:startup:gateway -- --case skipChannels --case fiftyPlugins --runs 5
node --import tsx scripts/bench-gateway-startup.ts --case default --runs 5 --output .artifacts/gateway-startup.json

Case ids: default, skipChannels (channel startup skipped), oneInternalHook, allInternalHooks, fiftyPlugins (50 manifest plugins), fiftyStartupLazyPlugins (50 startup-lazy manifest plugins).

The output captures first process output, /healthz, /readyz, HTTP listen log time, Gateway ready log time, CPU time, CPU core ratio, max RSS, heap, startup trace metrics, event-loop delay, and plugin lookup-table detail metrics. In the child Gateway environment, the script assigns OPENCLAW_GATEWAY_STARTUP_TRACE=1.

/healthz signals liveness, meaning the HTTP server can respond. /readyz indicates usable readiness, which settles after startup plugin sidecars, channels, and ready-critical post-attach work finish. Startup hooks fire asynchronously and fall outside the readiness guarantee. The ready log time comes from the Gateway's internal clock, handy for process-side attribution, but it does not replace the external /readyz probe.

For change comparisons, rely on JSON output or --output. Turn to --cpu-prof-dir only after trace output points to import, compile, or CPU-bound work that phase timings alone cannot clarify.

Gateway restart (scripts/bench-gateway-restart.ts)

Restricted to macOS and Linux, since SIGUSR1 handles in-process restarts and fails right away on Windows. The built-entry default and --entry scripts/run-node.mjs override match the gateway startup described above.

pnpm test:restart:gateway -- --case skipChannels --runs 1 --restarts 5
pnpm test:restart:gateway -- --case default --runs 3 --restarts 3 --warmup 1

Case ids: skipChannels, skipChannelsAcpxProbe (ACPX startup probe on), skipChannelsNoAcpxProbe (probe off), default, fiftyPlugins.

Output includes next /healthz, next /readyz, downtime, restart ready timing, CPU, RSS, startup trace metrics for the replacement process, and restart trace metrics for signal handling, active-work drain, close phases, next start, ready timing, and memory snapshots. The script sets OPENCLAW_GATEWAY_STARTUP_TRACE=1 and OPENCLAW_GATEWAY_RESTART_TRACE=1.

Run this benchmark when a change affects restart signaling, close handlers, startup-after-restart, sidecar shutdown, service handoff, or readiness after restart. Begin with skipChannels to separate Gateway mechanics from channel startup; only after the narrow case explains the restart path should you use default or plugin-heavy cases. Trace metrics serve as attribution hints, not final judgments. Judge a restart change using multiple samples, the matching owner span, /healthz//readyz behavior, and the user-visible restart contract.

Onboarding E2E (Docker)

Optional, and only needed for containerized onboarding smoke tests. A clean cold-start flow in a Linux container:

scripts/e2e/onboard-docker.sh

Through a pseudo-tty, it drives the interactive wizard, checks config/workspace/session state, then launches the gateway and executes openclaw health.

QR import smoke (Docker)

Confirms the maintained QR runtime helper loads under the supported Docker Node runtimes, with Node 24 as default and Node 22 compatible:

pnpm test:docker:qr

Related

• Testing
• Testing live
• Testing updates and plugins