Running Tests Locally with Vitest and Using Force/Coverage Modes

• Full test suite (suites, live, Docker): Testing
• Update and plugin package validation: Testing updates and plugins

Agent default

Agent sessions execute a limited set of targeted tests and inexpensive static checks on the local machine only when the source is trusted and an existing dependency installation is available. Never run untrusted repository tooling on your local system. Larger test suites, modified gates involving typecheck or lint fan-out, builds, Docker, package pipelines, end-to-end tests, live verification, and cross-platform validation are handled remotely through Crabbox. Trusted maintainer heavy verification defaults to Blacksmith Testbox. The configured Testbox workflow injects credentials, so code from untrusted contributors or forks must use a secretless fork CI or a sanitized direct AWS Crabbox instead.

Do not pre-warm for anticipated work. Obtain the backend lazily when the first heavy command is ready, reuse the returned tbx_... id for subsequent heavy commands, synchronize the current checkout on every execution, and terminate it before handoff.

After the first successful reuse, the wrapper records the lease's base, dependency, and Testbox workflow fingerprint under .crabbox/testbox-leases/. Source-only edits keep reusing the warmed box. A changed merge base, lockfile, package-manager input, wrapper, or Testbox workflow fails closed and demands a fresh lease. Every run still synchronizes the current checkout. OPENCLAW_TESTBOX_ALLOW_STALE=1 is intended only for deliberate diagnostics, not release verification.

The local test commands listed below are for human workflows and bounded agent verification. Remote-provider unavailability must be reported; it does not grant permission to silently run a broad local gate.

For untrusted heavy verification, lazily warm with --provider aws. Every run must set CRABBOX_ENV_ALLOW=CI, pass --provider aws --no-hydrate, and use a fresh temporary remote HOME before installing dependencies or running tests. Use a newly warmed lease dedicated to that untrusted source; never reuse a trusted or previously hydrated lease. Launch an installed trusted Crabbox binary from a clean trusted main checkout and fetch only the remote PR with --fresh-pr; never execute the untrusted checkout's wrapper or config locally. Unset CRABBOX_AWS_INSTANCE_PROFILE and fail closed unless resolved aws.instanceProfile is empty. Before any install or test, use trusted absolute-path tools to require an IMDSv2 token, prove the IAM credentials endpoint returns 404, and verify remote git rev-parse HEAD equals the full reviewed PR head SHA. Bind the lease to that SHA and stop or rewarm when the head changes. Upload trusted scripts/crabbox-untrusted-bootstrap.sh from clean main alongside --fresh-pr; it installs pinned Node and pnpm, verifies the SHA and package-manager pin, isolates HOME, installs dependencies, then executes the requested test. If the broker cannot prove no role or no remote PR exists, use secretless fork CI. Do not use hydrate-github, --no-sync, or a credential-hydrated Testbox workflow. Unset all CRABBOX_TAILSCALE* overrides, force --network public --tailscale=false, clear exit-node and LAN flags, and require crabbox inspect to report public networking with no Tailscale state before uploading any script.

Routine local order

1. pnpm test:changed for changed-scope Vitest verification.
2. pnpm test <path-or-filter> for a single file, directory, or explicit target.
3. pnpm test only when you deliberately need the full local Vitest suite.

In a Codex worktree or linked or sparse checkout, agents avoid direct local pnpm test* / pnpm check* / pnpm crabbox:run:

• Bounded focused verification with ready dependencies: node scripts/run-vitest.mjs <path-or-filter>.
• Classify-first changed check: node scripts/check-changed.mjs; docs-only, no-change, and small metadata plans stay local when dependencies are ready, while heavy or dependency-missing plans delegate to Testbox.
• Explicit kept-lease broad verification: node scripts/crabbox-wrapper.mjs run --provider blacksmith-testbox ... -- env OPENCLAW_CHECK_CHANGED_REMOTE_CHILD=1 OPENCLAW_CHANGED_LANES_RAW_SYNC=1 corepack pnpm check:changed so pnpm runs inside Testbox.
• The wrapper's final exitCode and timing JSON are the command result. A delegated Blacksmith GitHub Actions run may show cancelled after a successful SSH command because the Testbox is stopped from outside the keepalive action; check the wrapper summary and command output before treating that as a failure.
• OPENCLAW_HEAVY_CHECK_LOCK_SCOPE=worktree <local-heavy-check command>: keeps heavy-check serialization inside the current worktree instead of the Git common dir for commands such as pnpm check:changed and targeted pnpm test .... Use it only on high-capacity local hosts when you deliberately run independent checks across linked worktrees.

Core commands

Test wrapper runs conclude with a brief [test] passed|failed|skipped ... in ... summary; Vitest's own duration line remains the per-shard detail.

Shared test state and process helpers

• src/test-utils/openclaw-test-state.ts: import from Vitest when a test requires a separate HOME, OPENCLAW_STATE_DIR, OPENCLAW_CONFIG_PATH, config fixture, workspace, agent directory, or auth profile store.
• pnpm test:env-mutations:report: a non blocking report for tests or harnesses that directly modify HOME, OPENCLAW_STATE_DIR, OPENCLAW_CONFIG_PATH, OPENCLAW_WORKSPACE_DIR, or related environment keys. Use it to locate candidates for migration to the shared test state helper.
• test/helpers/openclaw-test-instance.ts: process level E2E tests that need a running Gateway, CLI environment, log capture, and cleanup consolidated in a single location.
• Docker or Bash E2E lanes sourcing scripts/lib/docker-e2e-image.sh can hand docker_e2e_test_state_shell_b64 <label> <scenario> into the container and decode it with scripts/lib/openclaw-e2e-instance.sh; multi home scripts can supply docker_e2e_test_state_function_b64 and invoke openclaw_test_state_create <label> <scenario> in each flow. node scripts/lib/openclaw-test-state.mjs -- create --label <name> --scenario <name> --env-file <path> --json produces a sourceable host environment file (the -- before create prevents newer Node runtimes from interpreting --env-file as a Node flag). Lanes that start a Gateway can source scripts/lib/openclaw-e2e-instance.sh for entrypoint resolution, mock OpenAI startup, foreground or background launch, readiness probes, state environment export, log dumps, and process cleanup.

Control UI, TUI, and extension lanes

• Control UI mocked E2E: pnpm test:ui:e2e executes the Vitest and Playwright lane that launches the Vite Control UI and drives a real Chromium page against a mocked Gateway WebSocket. Tests reside in ui/src/**/*.e2e.test.ts; shared mocks and controls live in ui/src/test-helpers/control-ui-e2e.ts. pnpm test:e2e includes this lane. Agent runs default to Testbox or Crabbox, including targeted proof; use node scripts/run-vitest.mjs run --config test/vitest/vitest.ui-e2e.config.ts --configLoader runner ui/src/ui/e2e/chat-flow.e2e.test.ts only for an explicit local fallback.
• TUI PTY tests: node scripts/run-vitest.mjs run --config test/vitest/vitest.tui-pty.config.ts runs the fast fake backend PTY lane. OPENCLAW_TUI_PTY_INCLUDE_LOCAL=1 or pnpm tui:pty:test:watch --mode local runs the slower tui --local smoke, which mocks only the external model endpoint. Assert stable visible text or fixture calls, not raw ANSI snapshots.
• pnpm test:extensions and pnpm test extensions execute all extension and plugin shards. Heavy channel plugins, the browser plugin, and OpenAI run as dedicated shards; other plugin groups stay batched. pnpm test extensions/<id> runs one bundled plugin lane.
• Source files with sibling tests map to that sibling before falling back to wider directory globs. Helper edits under src/channels/plugins/contracts/test-helpers, src/plugin-sdk/test-helpers, and src/plugins/contracts use a local import graph to run importing tests instead of broad running every shard when the dependency path is precise.
• Contract directory targets fan out to their contract lanes: pnpm test src/channels/plugins/contracts runs the four channel contract configs and pnpm test src/plugins/contracts runs the plugin contracts config, since the generic channels or plugins projects exclude contracts/**.
• auto-reply splits into three dedicated configs (core, top-level, reply) so the reply harness does not dominate the lighter top level status, token, and helper tests.
• Selected plugin-sdk and commands test files route through dedicated light lanes that keep only test/setup.ts, leaving runtime heavy cases on their existing lanes.
• Base Vitest config defaults to pool: "threads" and isolate: false, with the shared non isolated runner enabled across repo configs.
• pnpm test:channels runs vitest.channels.config.ts.

Gateway and E2E

• Gateway integration is enabled only when you opt in using OPENCLAW_TEST_INCLUDE_GATEWAY=1 pnpm test or pnpm test:gateway.
• pnpm test:e2e: repo-level end-to-end aggregate is set to pnpm test:e2e:gateway && pnpm test:ui:e2e.
• pnpm test:e2e:gateway: end-to-end smoke tests for the gateway (multi-instance WS/HTTP/node pairings). By default it uses threads plus isolate: false with adaptive workers inside vitest.e2e.config.ts; adjust with OPENCLAW_E2E_WORKERS=<n>, enable detailed logging with OPENCLAW_E2E_VERBOSE=1.
• pnpm test:live: live tests for providers (Claude, Minimax, DeepSeek, z.ai, and others, controlled by *.live.test.ts). API keys and either LIVE=1 or OPENCLAW_LIVE_TEST=1 are needed to bypass skipping; turn on verbose output with OPENCLAW_LIVE_TEST_QUIET=0.

Full Docker suite (pnpm test:docker:all)

Constructs the shared live-test image, packages OpenClaw once as an npm tarball, builds or reuses a minimal Node/Git runner image along with a functional image that installs that tarball into /app, then executes Docker smoke lanes via a weighted scheduler. scripts/package-openclaw-for-docker.mjs acts as the single packaging tool for local and CI environments, validating both the tarball and dist/postinstall-inventory.json before Docker uses them.

• Minimal image (OPENCLAW_DOCKER_E2E_BARE_IMAGE): handles installer, update, and plugin-dependency lanes; it mounts the prebuilt tarball instead of copying source from the repo.
• Functional image (OPENCLAW_DOCKER_E2E_FUNCTIONAL_IMAGE): runs standard built-app functionality lanes.
• Lane definitions: scripts/lib/docker-e2e-scenarios.mjs. Planner: scripts/lib/docker-e2e-plan.mjs. Executor: scripts/test-docker-all.mjs.
• node scripts/test-docker-all.mjs --plan-json outputs the scheduler-owned CI plan (lanes, image types, package and live-image requirements, state scenarios, credential checks) without building or launching Docker.

Scheduling controls (environment variables, defaults listed in parentheses):

The environment variable pattern for resource caps follows OPENCLAW_DOCKER_ALL_<RESOURCE>_LIMIT (the resource name in uppercase, with non-alphanumeric characters replaced by _).

By default, the runner performs a Docker preflight, removes stale OpenClaw E2E containers, reuses provider CLI tool caches across compatible lanes, and halts scheduling of new pooled lanes after the first error unless OPENCLAW_DOCKER_ALL_FAIL_FAST=0 is configured. On a host with limited parallelism, a lane that exceeds the effective weight or resource cap can still start from an empty pool and run alone until it frees resources. Per-lane logs, summary.json, failures.json, and phase timings are stored under .artifacts/docker-tests/<run-id>/; use pnpm test:docker:timings <summary.json> to examine slow lanes and pnpm test:docker:rerun <run-id|summary.json|failures.json> to generate inexpensive rerun commands.

Notable Docker lanes

Local PR gate

For local PR land or gate checks, execute:

• pnpm check:changed
• pnpm check
• pnpm check:test-types
• pnpm build
• pnpm test
• pnpm check:docs

If pnpm test becomes flaky on a loaded host, rerun it once before treating it as a regression, then isolate it with pnpm test <path/to/test>. On memory-constrained hosts:

• OPENCLAW_VITEST_MAX_WORKERS=1 pnpm test
• OPENCLAW_VITEST_FS_MODULE_CACHE_PATH=/tmp/openclaw-vitest-cache pnpm test:changed

Test performance tooling

• pnpm test:perf:imports: activates Vitest import duration and import breakdown reporting, while keeping scoped lane routing for explicit file or directory targets. pnpm test:perf:imports:changed applies the same profiling only to files changed after origin/main.
• pnpm test:perf:changed:bench -- --ref <git-ref> measures the routed changed mode path against the native root project run for the same committed git diff; pnpm test:perf:changed:bench -- --worktree measures the current worktree change set without an initial commit.
• pnpm test:perf:profile:main generates a CPU profile for the Vitest main thread (.artifacts/vitest-main-profile); pnpm test:perf:profile:runner generates CPU and heap profiles for the unit runner (.artifacts/vitest-runner-profile).
• pnpm test:perf:groups --full-suite --allow-failures --output .artifacts/test-perf/baseline-before.json: executes each full suite Vitest leaf configuration in sequence and outputs grouped duration data along with per config JSON and log artifacts. Full suite reports isolate files by default so retained module graphs and GC pauses from earlier files are not attributed to later assertions; pass -- --no-isolate only when intentionally profiling shared worker accumulation. The Test Performance Agent uses this as its baseline before attempting slow test fixes. pnpm test:perf:groups:compare .artifacts/test-perf/baseline-before.json .artifacts/test-perf/after-agent.json compares grouped reports after a performance focused change.
• Full, extension, and include pattern shard runs update local timing data in .artifacts/vitest-shard-timings.json; later whole config runs use those timings to balance slow and fast shards. Include pattern CI shards append the shard name to the timing key, which keeps filtered shard timings visible without replacing whole config timing data. Set OPENCLAW_TEST_PROJECTS_TIMINGS=0 to ignore the local timing artifact.

Benchmarks

Model latency (scripts/bench-model.ts)

pnpm tsx scripts/bench-model.ts --runs 10

Optional env: MINIMAX_API_KEY, MINIMAX_BASE_URL, MINIMAX_MODEL, ANTHROPIC_API_KEY. Default prompt: "Reply with a single word: ok. No punctuation or extra text."

CLI startup (scripts/bench-cli-startup.ts)

pnpm test:startup:bench
pnpm test:startup:bench:smoke
pnpm test:startup:bench:save
pnpm test:startup:bench:update
pnpm test:startup:bench:check
pnpm tsx scripts/bench-cli-startup.ts --runs 12
pnpm tsx scripts/bench-cli-startup.ts --preset real --case status --case gatewayStatus --runs 3
pnpm tsx scripts/bench-cli-startup.ts --entry openclaw.mjs --entry-secondary dist/entry.js --preset all

Presets:

• startup: --version, --help, health, health --json, status --json, status
• real: health, status, status --json, sessions, sessions --json, tasks --json, tasks list --json, tasks audit --json, agents list --json, gateway status, gateway status --json, gateway health --json, config get gateway.port
• all: both presets combined

Output includes sampleCount, avg, p50, p95, min/max, exit code and signal distribution, and max RSS per command. --cpu-prof-dir / --heap-prof-dir write V8 profiles per run.

Saved output: pnpm test:startup:bench:smoke writes .artifacts/cli-startup-bench-smoke.json; pnpm test:startup:bench:save writes .artifacts/cli-startup-bench-all.json (runs=5 warmup=1). Checked in fixture: test/fixtures/cli-startup-bench.json, refreshed by pnpm test:startup:bench:update, compared by pnpm test:startup:bench:check.

Gateway startup (scripts/bench-gateway-startup.ts)

Defaults to the built CLI entry at dist/entry.js; you must run pnpm build first. To measure the source runner instead, pass --entry scripts/run-node.mjs and keep those results separate from built-entry baselines.

pnpm test:startup:gateway -- --runs 5 --warmup 1
pnpm test:startup:gateway -- --case skipChannels --case fiftyPlugins --runs 5
node --import tsx scripts/bench-gateway-startup.ts --case default --runs 5 --output .artifacts/gateway-startup.json

Case ids: default, skipChannels (channel startup skipped), oneInternalHook, allInternalHooks, fiftyPlugins (50 manifest plugins), fiftyStartupLazyPlugins (50 startup-lazy manifest plugins).

The output includes first process output, /healthz, /readyz, HTTP listen log time, Gateway ready log time, CPU time, CPU core ratio, max RSS, heap, startup trace metrics, event-loop delay, and plugin lookup-table detail metrics. The script sets OPENCLAW_GATEWAY_STARTUP_TRACE=1 in the child Gateway environment.

/healthz represents liveness (whether the HTTP server can respond). /readyz indicates usable readiness (startup plugin sidecars, channels, and ready-critical post-attach work have settled). Startup hooks dispatch asynchronously and are not covered by the readiness guarantee. Ready log time is the Gateway's internal timestamp, useful for process-side attribution but not a replacement for the external /readyz probe.

When comparing changes, use JSON output or --output. Use --cpu-prof-dir only after trace output points at import, compile, or CPU-bound work that phase timings alone cannot explain.

Gateway restart (scripts/bench-gateway-restart.ts)

Supported on macOS and Linux only (uses SIGUSR1 for in-process restarts; fails immediately on Windows). Uses the same built-entry default and --entry scripts/run-node.mjs override as gateway startup above.

pnpm test:restart:gateway -- --case skipChannels --runs 1 --restarts 5
pnpm test:restart:gateway -- --case default --runs 3 --restarts 3 --warmup 1

Case ids: skipChannels, skipChannelsAcpxProbe (ACPX startup probe on), skipChannelsNoAcpxProbe (probe off), default, fiftyPlugins.

Output includes next /healthz, next /readyz, downtime, restart ready timing, CPU, RSS, startup trace metrics for the replacement process, and restart trace metrics for signal handling, active-work drain, close phases, next start, ready timing, and memory snapshots. The script sets OPENCLAW_GATEWAY_STARTUP_TRACE=1 and OPENCLAW_GATEWAY_RESTART_TRACE=1.

Use this benchmark when a change touches restart signaling, close handlers, startup-after-restart, sidecar shutdown, service handoff, or readiness after restart. Start with skipChannels to isolate Gateway mechanics from channel startup; use default or plugin-heavy cases only after the narrow case explains the restart path. Trace metrics are attribution hints, not verdicts. Judge a restart change from multiple samples, the matching owner span, /healthz//readyz behavior, and the user-visible restart contract.

Onboarding E2E (Docker)

Optional; only needed for containerized onboarding smoke tests. Full cold-start flow in a clean Linux container:

scripts/e2e/onboard-docker.sh

Drives the interactive wizard via a pseudo-tty, verifies config/workspace/session files, then starts the gateway and runs openclaw health.

QR import smoke (Docker)

Ensures the maintained QR runtime helper loads under the supported Docker Node runtimes (Node 24 default, Node 22 compatible):

pnpm test:docker:qr

Related

• Testing
• Testing live
• Testing updates and plugins