QA Overview: End-to-End Automation with OpenClaw

The private QA stack exercises OpenClaw in a realistic, channel-shaped way that a unit test cannot.

Pieces:

• extensions/qa-channel: synthetic message channel with DM, channel, thread, reaction, edit, and delete surfaces.
• extensions/qa-lab: debugger UI, QA bus, scenario runners, and live transport adapters for observing the transcript, injecting inbound messages, and exporting a Markdown report.
• qa/: repo-backed seed assets for the kickoff task and baseline QA scenarios.
• Mantis: before/after live verification for bugs that need real transports, browser screenshots, VM state, and PR evidence.

Command surface

Every QA flow runs under pnpm openclaw qa <subcommand>. Many have pnpm qa:* script aliases; both forms work.

Profile-backed qa run

Profile-backed qa run pulls membership data from taxonomy.yaml, then routes the resolved scenarios onward via qa suite. Instead of creating separate lanes, --surface and --category narrow down the chosen profile. The resulting qa-evidence.json carries a profile scorecard summary, including counts for selected categories and IDs for missing coverage; the individual evidence entries stay the definitive reference for tests, coverage roles, and outcomes. Taxonomy feature coverage IDs act as exact proof targets rather than aliases: primary scenario coverage satisfies matching IDs, while secondary coverage remains advisory. Every coverage ID is precisely taxonomy-surface.feature, drawn from the short surface ID in taxonomy.yaml. A scenario's separate surface field serves as an execution/reporting label (such as channel or runtime-tool); it does not determine taxonomy ownership. An explicit profile coverage ID picks every eligible primary owner for that ID, with deduplication by scenario. Neither scenario file order nor taxonomy order influences membership or execution sequencing.

scenario.execution.channels functions as an OR eligibility list: a channel-specific runner can execute the scenario on any one listed channel. Profile-backed execution broadens that same list across all channels the selected driver supports, and the profile run succeeds only when every expanded channel execution succeeds. This rule applies uniformly across all taxonomy profiles.

Slim evidence skips per-entry execution and assigns evidenceMode: "slim"; smoke-ci is the default for slim, and --evidence-mode full brings back full entries:

pnpm openclaw qa run \
  --qa-profile smoke-ci \
  --category channels.conversation-routing-and-delivery \
  --provider-mode mock-openai \
  --output-dir .artifacts/qa-e2e/smoke-ci-profile-dispatch

For deterministic profile proof with mock model providers and Crabline local provider servers, use smoke-ci. For Stable/LTS proof against live channels, use release. Use all only for explicit full-taxonomy evidence runs; it picks every active maturity category and can be dispatched through the QA Profile Evidence GitHub Actions workflow with qa_profile=all. When a command also requires an OpenClaw root profile, place the root profile before the QA command:

pnpm openclaw --profile work qa run --qa-profile smoke-ci

Operator flow

The current QA operator flow uses a two-pane QA site:

• Left: Gateway dashboard (Control UI) with the agent.
• Right: QA Lab, showing the Slack-ish transcript and scenario plan.

Run it with:

pnpm qa:lab:up

That builds the QA site, starts the Docker-backed gateway lane, and exposes the QA Lab page where an operator or automation loop can give the agent a QA mission, observe real channel behavior, and record what worked, failed, or stayed blocked.

The Runner's Scenarios panel can launch flow, Playwright, Vitest, and script catalog entries together. Profile uses the taxonomy-owned membership plan; checking scenarios creates an explicit override, while Profile in the Scenarios panel returns to server-resolved profile membership.

Config also exposes Provider lane, primary and alternate models, Execution channel, Channel driver, Evidence mode, Runtime pair, and Runtime-pair lane (core, extended, or soak). Provider/model, runtime, and channel-driver choices remain independent: for example, Real frontier providers can use the Crabline channel driver, and Synthetic (mock) can use Real channels. The server resolves taxonomy membership, provider/model eligibility, declared execution.channel, runtime-pair-lane membership, and supported execution kinds before launch. The Run panel shows the selected execution kinds plus explicit exclusions or errors. Unknown, empty explicit, profile-incompatible, or lane-incompatible selections fail closed instead of being replaced by a default suite.

For faster QA Lab UI iteration without rebuilding the Docker image each time, start the stack with a bind-mounted QA Lab bundle:

pnpm openclaw qa docker-build-image
pnpm qa:lab:build
pnpm qa:lab:up:fast
pnpm qa:lab:watch

qa:lab:up:fast keeps the Docker services on a prebuilt image and bind-mounts extensions/qa-lab/web/dist into the qa-lab container. qa:lab:watch rebuilds that bundle on change, and the browser auto-reloads when the QA Lab asset hash changes.

Observability smokes

Note

Observability QA stays source-checkout only. The npm tarball intentionally omits QA Lab (and qa-channel), so package Docker release lanes do not run qa commands. Run these from a built source checkout when changing diagnostics instrumentation.

qa:otel:smoke starts a local OTLP/HTTP receiver, runs a minimal QA-channel agent turn, then asserts traces, metrics, and logs are exported. It decodes the exported protobuf trace spans and checks the release-critical shape: openclaw.run, openclaw.harness.run, a latest GenAI semantic-convention model-call span, openclaw.context.assembled, and openclaw.message.delivery must all be present. The smoke forces OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental, so the model-call span must use the {gen_ai.operation.name} {gen_ai.request.model} name; model calls must not export StreamAbandoned on successful turns; raw diagnostic IDs and openclaw.content.* attributes must stay out of the trace. The scenario prompt asks the model to reply with a fixed marker and to withhold a fixed secret string; the raw OTLP payloads must not contain either, or the QA session key derived from the scenario id. It writes otel-smoke-summary.json next to the QA suite artifacts.

qa:prometheus:smoke confirms that unauthenticated scrape attempts are turned away, and afterwards verifies that the authenticated scrape carries release-critical metric families, free of prompt text, response text, raw diagnostic identifiers, auth tokens, or local paths.

Matrix live lane

For a transport-real Matrix lane that needs no model-provider credentials, go with the deterministic mock OpenAI provider:

pnpm openclaw qa matrix --provider-mode mock-openai

For the live-frontier provider lane, pass in OpenAI-compatible credentials explicitly:

OPENCLAW_LIVE_OPENAI_KEY="${OPENAI_API_KEY}" \
  pnpm openclaw qa matrix --provider-mode live-frontier

Plain pnpm openclaw qa matrix executes every flow scenario that explicitly marks itself as Matrix-eligible through execution.channel or execution.channels, and it keeps going even when scenarios fail. For a quicker feedback loop, use --fail-fast, or run --scenario <id> to target a specific subset, including portable scenarios without any channel restriction.

Matrix live implementations are located under extensions/qa-lab/src/live-transports/matrix/scenarios/.

The adapter spins up a disposable Tuwunel homeserver in Docker (default image ghcr.io/matrix-construct/tuwunel:v1.8.2, pinned to its multi-architecture OCI index digest; server name matrix-qa.test, port 28008), sets up temporary driver, SUT, and observer users, seeds the required rooms, and logs the redacted request/response boundary. After that, it runs the actual Matrix plugin inside a child QA gateway scoped to that transport (no qa-channel) and cleans up the environment.

The v1.8.2 GHCR index resolves to sha256:6f950bb139411a7964781e986321e395e045e4a6a52240a4dda9d23d04075f78. docker buildx imagetools inspect ghcr.io/matrix-construct/tuwunel:v1.8.2 reports manifests for linux/arm64, linux/amd64, linux/amd64/v2, and linux/amd64/v3.

Common options:

Matrix QA does not lease shared Matrix credentials: the adapter creates disposable users locally, so it does not accept --credential-source or --credential-role. Override the homeserver image with OPENCLAW_QA_MATRIX_TUWUNEL_IMAGE; adjust negative no-reply assertions with OPENCLAW_QA_MATRIX_NO_REPLY_WINDOW_MS (default 8000, clamped to the active scenario timeout). The single-shot command normally forces a clean exit after artifacts flush because Matrix crypto native handles can outlive cleanup; set OPENCLAW_QA_MATRIX_DISABLE_FORCE_EXIT=1 only for a direct test harness that needs the command to return instead.

Each run writes the normal QA Lab artifacts under the selected output directory: qa-suite-report.md, qa-suite-summary.json, and qa-evidence.json. If cleanup fails, run the printed docker compose ... down --remove-orphans recovery command. On slow runners, increase the no-reply window; on fast CI, a smaller window can shorten negative assertions.

The catalog covers transport behavior that unit tests cannot prove end to end: mention gating, allow-bot policies, allowlists, top-level and threaded replies, DM routing, reaction handling, inbound edit suppression, restart replay dedupe, homeserver interruption recovery, approval metadata delivery, media handling, and Matrix E2EE bootstrap/recovery/verification flows. The E2EE CLI scenarios also drive openclaw matrix encryption setup and verification commands through the same disposable homeserver before checking gateway replies.

CI uses the same command surface in .github/workflows/qa-live-transports-convex.yml. Scheduled, release, and manual runs fan the catalog-derived selection across five deterministic shards so membership stays scenario-owned while each job remains within its timeout.

Discord Mantis scenarios

Discord also has Mantis-only opt-in scenarios for bug reproduction. Use --scenario discord-status-reactions-tool-only for the explicit status reaction timeline, or --scenario discord-thread-reply-filepath-attachment to create a real Discord thread and verify that message.thread-reply preserves a filePath attachment. These scenarios stay out of the default live Discord lane because they are before/after repro probes rather than broad smoke coverage. The thread-attachment Mantis workflow can also add a logged-in Discord Web witness video when MANTIS_DISCORD_VIEWER_CHROME_PROFILE_DIR or MANTIS_DISCORD_VIEWER_CHROME_PROFILE_TGZ_B64 is configured in the QA environment. That viewer profile is only for visual capture; the pass/fail decision still comes from the Discord REST oracle.

For the other transport-real smoke lanes:

pnpm openclaw qa buzz
pnpm openclaw qa discord
pnpm openclaw qa slack
pnpm openclaw qa telegram
pnpm openclaw qa whatsapp

They target a pre-existing real channel with two bots or accounts (driver + SUT). Required env vars, scenario lists, output artifacts, and the Convex credential pool for those five transports are documented in Buzz, Discord, Slack, Telegram, and WhatsApp QA reference below.

Mantis Slack desktop and visual-task runners

For a full Slack desktop VM run with VNC rescue, run:

pnpm openclaw qa mantis slack-desktop-smoke \
  --gateway-setup \
  --scenario slack-canary \
  --keep-lease

Running that command reserves a Crabbox desktop or browser machine, boots the Slack live lane within the VM, launches Slack Web in the VNC browser, grabs the desktop, and returns slack-qa/, slack-desktop-smoke.png, and slack-desktop-smoke.mp4 (if video capture is possible) to the Mantis artifact directory. Because Crabbox desktop and browser leases come preloaded with the capture utilities and browser or native-build helper packages, the scenario should only install fallback tools on older leases. Mantis records overall and per-phase timings in mantis-slack-desktop-smoke-report.md, so slow executions reveal whether time went to lease warmup, credential retrieval, remote setup, or artifact copying. After manually logging into Slack Web via VNC, reuse --lease-id <cbx_...>; reused leases also keep Crabbox's pnpm store cache warm. The default --hydrate-mode source checks out the source and performs install and build steps inside the VM. Only use --hydrate-mode prehydrated when the reused remote workspace already contains node_modules and a compiled dist/; this mode skips the costly install and build phase and fails closed if the workspace is not prepared. With --gateway-setup, Mantis keeps a persistent OpenClaw Slack gateway active in the VM on port 38973; otherwise, the command runs the standard bot-to-bot Slack QA lane and terminates after artifact capture.

To demonstrate the native Slack approval UI with desktop evidence, execute the Mantis approval checkpoint mode:

pnpm openclaw qa mantis slack-desktop-smoke \
  --approval-checkpoints \
  --credential-source convex \
  --credential-role maintainer

This mode cannot be combined with --gateway-setup. It runs the Slack approval scenarios, rejects any non-approval scenario IDs, pauses at each pending and resolved approval state, converts the observed Slack API message into approval-checkpoints/<scenario>-pending.png and approval-checkpoints/<scenario>-resolved.png, and then fails if any checkpoint, message evidence, acknowledgment, or rendered screenshot is absent or empty. Fresh CI leases may still display Slack sign-in in slack-desktop-smoke.png; the approval checkpoint images serve as the visual proof for this lane.

By default, the checkpoint run keeps the two standard Slack approval scenarios. To capture either opt-in Codex approval route, pick it explicitly with --scenario slack-codex-approval-exec-native or --scenario slack-codex-approval-plugin-native; Mantis accepts both and produces the same pending and resolved screenshot pair. The runner extends its checkpoint and remote-command timeouts for each chosen Codex route so the full approval, agent completion, and resolved-update flow can complete.

The operator checklist, GitHub workflow dispatch command, evidence-comment contract, hydrate-mode decision table, timing interpretation, and failure handling steps are documented in Mantis Slack Desktop Runbook.

For an agent or CV style desktop task, run:

pnpm openclaw qa mantis visual-task \
  --browser-url https://example.net \
  --expect-text "Example Domain" \
  --vision-model openai/gpt-5.6-luna

visual-task leases or reuses a Crabbox desktop or browser machine, launches crabbox record --while, operates the visible browser through a nested visual-driver, captures visual-task.png, runs openclaw infer image describe against the screenshot when --vision-mode image-describe is selected, and outputs visual-task.mp4, mantis-visual-task-summary.json, mantis-visual-task-driver-result.json, and mantis-visual-task-report.md. When --expect-text is set, the vision prompt requests a structured JSON verdict (visible, evidence, reason) and only succeeds when the model reports visible: true with evidence citing the expected text; a visible: false response that only quotes the target text still fails the assertion. Use --vision-mode metadata for a no-model smoke test that verifies the desktop, browser, screenshot, and video plumbing without invoking an image-understanding provider. Recording is a mandatory artifact for visual-task; if Crabbox captures no non-empty visual-task.mp4, the task fails even when the visual driver passed. On failure, Mantis holds the lease for VNC unless the task had already succeeded and --keep-lease was not set.

Credential pool health check

Before using pooled live credentials, run:

pnpm openclaw qa credentials doctor

The doctor checks the Convex broker environment (OPENCLAW_QA_CONVEX_SITE_URL, OPENCLAW_QA_CONVEX_ENDPOINT_PREFIX), validates endpoint settings, reports only set or missing status for OPENCLAW_QA_CONVEX_SECRET_CI and OPENCLAW_QA_CONVEX_SECRET_MAINTAINER, and checks admin and list reachability when the maintainer secret is available.

Canonical scenario coverage

The root taxonomy.yaml defines semantic coverage IDs. Scenario YAML files under qa/scenarios/ map each scenario to those IDs and hold execution metadata; execution.channel or execution.channels specifies channel requirements. Taxonomy profiles pick coverage IDs or entire categories, and the catalog resolves their primary scenario owners. Transport runners apply channel and provider eligibility to that result instead of keeping scenario-ID allowlists. The channel driver is a swappable run-level implementation choice.

Static qa coverage output reports the taxonomy-to-scenario mapping. Actual proof comes from qa-evidence.json, which logs the executed scenario, coverage IDs, channel, driver actually used, and result. Channel and driver are report dimensions, not extra coverage-ID vocabularies or scenario eligibility axes.

For a disposable Linux VM lane without bringing Docker into the QA path, run:

pnpm openclaw qa suite --runner multipass --scenario channel-chat-baseline

This boots a fresh Multipass guest, installs dependencies, builds OpenClaw inside the guest, runs qa suite, then copies the normal QA report and summary back into .artifacts/qa-e2e/... on the host. It reuses the same scenario-selection behavior as qa suite on the host.

Host and Multipass suite runs execute multiple selected scenarios in parallel with isolated gateway workers by default. qa-channel defaults to concurrency 4, capped by the selected scenario count. Use --concurrency <count> to tune the worker count, or --concurrency 1 for serial execution. Use qa run --qa-profile personal-agent --provider-mode mock-openai for the personal assistant benchmark, or --qa-profile observability for the source checkout telemetry checks. CI uses the same profile resolver for smoke-ci; none of these selectors maintains a second scenario-ID list.

The command exits non-zero when any scenario fails. Use --allow-failures when you want artifacts without a failing exit code.

Live runs forward the supported QA auth inputs that are practical for the guest: env-based provider keys, the QA live provider config path, and CODEX_HOME when present. Keep --output-dir under the repo root so the guest can write back through the mounted workspace.

Buzz, Discord, Slack, Telegram, and WhatsApp QA reference

The Matrix adapter uses the disposable Docker-backed lane documented above. Buzz, Discord, Slack, Telegram, and WhatsApp run against pre-existing real transports, so their reference lives here.

Shared CLI flags

These lanes register through the shared QA runner CLI contract. Transport plugins may own the registration while QA Lab remains the suite host. They accept the same flags:

Each lane exits non-zero on any failed scenario. --allow-failures writes artifacts without setting a failing exit code. Telegram also accepts --list-scenarios to print available scenario ids and exit; the other lanes do not expose that flag.

Buzz QA

pnpm openclaw qa buzz \
  --credential-file /secure/path/buzz-qa-credentials.json

Targets one real Buzz room with two dedicated Nostr identities. The driver publishes inbound room events; the SUT identity is configured in the child OpenClaw Gateway and its outbound events are observed from the relay. The default mock-openai provider proves the real Buzz transport without requiring a model-provider credential.

Local runs use --credential-file <path> with a private JSON file containing relayUrl, roomId, driverPrivateKey, and sutPrivateKey. Closed relays may also need driverAuthTag and sutAuthTag. Relative paths resolve from --repo-root. Hosted relays must use wss://; plaintext ws:// is accepted only for loopback development relays.

Both identities must be members of the dedicated room, and the SUT public key must have the Bot role. A hosted closed relay may also require both public keys to be enrolled as relay members. Use dedicated QA identities only; never use a human owner or admin private key. Keep all private keys and authorization values out of logs, command lines, artifacts, screenshots, and source control.

The default scenarios are:

• channel-canary
• channel-mention-gating

Each execution stores qa-suite-report.md, qa-suite-summary.json, and qa-evidence.json in the chosen output folder. While the report shows the actual Buzz relay path, credential values are withheld.

Telegram QA

pnpm openclaw qa telegram

A single real private Telegram group is exercised by two separate bots, one acting as driver and the other as SUT. A Telegram username is required for the SUT bot. For optimal bot-to-bot observation, both bots should have Bot-to-Bot Communication Mode turned on in @BotFather.

When --credential-source env is active, these environment variables are mandatory:

• OPENCLAW_QA_TELEGRAM_GROUP_ID - numeric chat id, provided as a string.
• OPENCLAW_QA_TELEGRAM_DRIVER_BOT_TOKEN
• OPENCLAW_QA_TELEGRAM_SUT_BOT_TOKEN

The release profile picks taxonomy-owned Telegram scenarios that specify the channel, employ the flow execution kind, and align with the requested provider and model lane. Explicit --scenario values refine that same selection rather than circumventing its restrictions. To display the current selection along with regression references, use pnpm openclaw qa telegram --list-scenarios --provider-mode mock-openai. Passing --model enforces the same model constraint on both listing and execution.

telegram-startup-getme-live acts as a catalog script generator, not a live-adapter flow. Execute it via qa suite --scenario telegram-startup-getme-live; the specialized qa telegram command and --list-scenarios deliberately exclude it.

Artifacts produced:

• qa-suite-report.md
• qa-suite-summary.json
• qa-evidence.json - evidence records for the live transport checks, containing profile, coverage, provider, channel, artifacts, result, and RTT fields.

Package Telegram runs adhere to the same Telegram credential contract. The standard package Telegram live lane includes repeated RTT measurement as part of its normal operation; the RTT distribution is incorporated into qa-evidence.json under result.timing for the chosen RTT check.

OPENCLAW_QA_CREDENTIAL_SOURCE=convex \
pnpm test:docker:npm-telegram-live

When OPENCLAW_QA_CREDENTIAL_SOURCE=convex is configured, the package live wrapper obtains a kind: "telegram" credential, injects the leased group/driver/SUT bot environment into the installed-package run, maintains the lease with heartbeats, and frees it upon shutdown. By default, the package wrapper performs 20 RTT checks of channel-canary, applies a 30s RTT timeout, and uses Convex role maintainer outside CI when Convex is selected. Adjust OPENCLAW_NPM_TELEGRAM_RTT_SAMPLES, OPENCLAW_NPM_TELEGRAM_RTT_TIMEOUT_MS, or OPENCLAW_NPM_TELEGRAM_RTT_MAX_FAILURES to modify RTT measurement without needing a separate RTT command or a Telegram-specific summary format.

Discord QA

pnpm openclaw qa discord

One real private Discord guild channel is targeted with two bots: a driver bot managed by the harness and a SUT bot launched by the child OpenClaw gateway via the bundled Discord plugin. The checks confirm channel mention handling, that the SUT bot has registered the native /help command with Discord, and opt-in Mantis evidence scenarios.

When --credential-source env is set, these environment variables are required:

• OPENCLAW_QA_DISCORD_GUILD_ID
• OPENCLAW_QA_DISCORD_CHANNEL_ID
• OPENCLAW_QA_DISCORD_DRIVER_BOT_TOKEN
• OPENCLAW_QA_DISCORD_SUT_BOT_TOKEN
• OPENCLAW_QA_DISCORD_SUT_APPLICATION_ID - must equal the SUT bot user id returned by Discord, otherwise the lane fails immediately.

Optional settings:

• OPENCLAW_QA_DISCORD_VOICE_CHANNEL_ID designates the voice/stage channel for discord-voice-autojoin; if omitted, the scenario chooses the first visible voice/stage channel available to the SUT bot.

Discord YAML module scenarios (qa/scenarios/channels/discord-*.yaml):

• discord-canary
• discord-mention-gating
• discord-native-help-command-registration
• discord-progress-draft-lifecycle - executes a deterministic tool turn, verifies the final answer lacks any synthesized activity receipt, confirms the working draft is removed after a successful final, and confirms an error final keeps its draft visible as diagnostic context.
• discord-voice-autojoin - opt-in voice scenario. Runs independently, activates channels.discord.voice.autoJoin, and verifies the SUT bot's current Discord voice state matches the target voice/stage channel. Convex Discord credentials may include optional voiceChannelId; otherwise the runner adapter locates the first visible voice/stage channel in the guild.
• discord-transcripts-voice-authorization - opt-in live-model scenario. A real driver-bot message first demonstrates that a sender excluded from the target voice channel receives a visible transcript-tool denial without a join. That same sender is then allowlisted and must start, stop, and leave live capture. The scenario writes redacted JSON evidence and removes its known Discord messages during cleanup.
• discord-status-reactions-tool-only - opt-in Mantis scenario. Runs independently because it shifts the SUT to always-on, tool-only guild replies with messages.statusReactions.enabled=true, then records a REST reaction timeline plus HTML/PNG visual artifacts. Mantis before/after reports also keep scenario-provided MP4 artifacts as baseline.mp4 and candidate.mp4.
• discord-thread-reply-filepath-attachment - opt-in Mantis scenario; refer to Discord Mantis scenarios.

Run the Discord voice auto-join scenario explicitly:

pnpm openclaw qa discord \
  --scenario discord-voice-autojoin \
  --provider-mode mock-openai

Run the transcript authorization scenario with a Convex lease:

pnpm openclaw qa discord \
  --scenario discord-transcripts-voice-authorization \
  --provider-mode live-frontier \
  --credential-source convex \
  --credential-role maintainer

Run the Mantis status-reaction scenario explicitly:

pnpm openclaw qa discord \
  --scenario discord-status-reactions-tool-only \
  --provider-mode live-frontier \
  --model openai/gpt-5.6-luna \
  --alt-model openai/gpt-5.6-luna \
  --fast

Output artifacts:

• qa-suite-report.md
• qa-suite-summary.json
• qa-evidence.json - evidence entries for the live transport checks.
• discord-qa-reaction-timelines.json and discord-status-reactions-tool-only-timeline.png when the status-reaction scenario runs.

Slack QA

pnpm openclaw qa slack

A single real private Slack channel is targeted, with two separate bots in play: one driver bot under harness control, and a SUT bot launched by the child OpenClaw gateway through the bundled Slack plugin.

Required env when --credential-source env:

• OPENCLAW_QA_SLACK_CHANNEL_ID
• OPENCLAW_QA_SLACK_DRIVER_BOT_TOKEN
• OPENCLAW_QA_SLACK_SUT_BOT_TOKEN
• OPENCLAW_QA_SLACK_SUT_APP_TOKEN

Optional:

• OPENCLAW_QA_SLACK_APPROVAL_CHECKPOINT_DIR enables visual approval checkpoints for Mantis. The adapter writes <scenario>.pending.json and <scenario>.resolved.json, then waits for matching .ack.json files.
• OPENCLAW_QA_SLACK_APPROVAL_CHECKPOINT_TIMEOUT_MS overrides the checkpoint acknowledgement timeout. The default is 120000.

Canonical YAML scenarios exposed through the Slack live adapter:

• thread-follow-up
• thread-isolation

Slack YAML module scenarios (qa/scenarios/channels/slack-*.yaml):

• slack-canary
• slack-mention-gating
• slack-mpim-app-mention-dedupe - opens a real C-prefixed group DM, verifies exactly one SUT reply after message/app-mention twin delivery, confirms a native threaded follow-up can recall that bot reply, then closes the MPIM.
• slack-allowlist-block
• slack-channel-disabled-warning - opt-in real-Slack probe that confirms a configured disabled channel emits a structured warning without replying.
• slack-top-level-reply-shape
• slack-restart-resume
• slack-progress-commentary-true, slack-progress-commentary-false, slack-progress-commentary-omitted, and slack-progress-commentary-verbose-dedupe - opt-in real-Slack probes for independent commentary/tool-progress controls, the omitted-key legacy default, and single-delivery behavior when durable verbose progress is on.
• slack-reaction-glyph-native - opt-in live message-tool reaction scenario. Instructs the agent to pass the exact ✅ glyph and confirms Slack stored white_check_mark for the SUT bot on the target message.
• slack-chart-presentation-native - opt-in portable chart scenario that verifies the native data_visualization block and exact accessible text.
• slack-table-presentation-native - opt-in portable table scenario that verifies the native data_table block, exact rows, and accessible text.
• slack-table-invalid-blocks-fallback - opt-in direct-transport scenario that sends a structurally readable over-limit raw table with 101 data rows plus its header through the production Slack send path, proves Slack itself returns invalid_blocks, and verifies the stored formatting-disabled fallback is complete and has no native data block. Scenario details keep only safe error-code, count, and boolean evidence.
• slack-approval-exec-native - opt-in native Slack exec approval scenario. Requests an exec approval through the gateway, verifies the Slack message has native approval buttons, resolves it, and verifies the resolved Slack update.
• slack-approval-plugin-native - opt-in native Slack plugin approval scenario. Enables exec and plugin approval forwarding together so plugin events are not suppressed by exec approval routing, then verifies the same pending/resolved native Slack UI path.
• slack-codex-approval-exec-native - opt-in Codex Guardian command approval scenario. Enables the Codex plugin in Guardian mode, routes a Slack-originated Gateway agent turn through the Codex app-server harness, waits for the native Slack plugin approval prompt for codex, resolves it, and verifies the Codex turn finishes with the expected command-output and assistant markers.
• slack-codex-approval-plugin-native - opt-in Codex Guardian file approval scenario. Uses an outside-workspace apply_patch instruction so Codex emits the app-server file-change approval route, then verifies the same native Slack pending/resolved approval path, final assistant marker, and exact file contents before cleanup.

The Codex approval scenarios require an openai/* or codex/* --model, the normal live model credentials, and Codex auth or API-key auth accepted by the Codex plugin. The scenario details include the Codex app-server method, selected Codex model key, final Codex turn status, and operation-marker verification alongside the redacted Slack approval metadata.

Output artifacts:

• qa-suite-report.md
• qa-suite-summary.json
• qa-evidence.json - evidence entries for the live transport checks.
• approval-checkpoints/ - only when Mantis sets OPENCLAW_QA_SLACK_APPROVAL_CHECKPOINT_DIR; contains checkpoint JSON, acknowledgement JSON, and pending/resolved screenshots.

Setting up the Slack workspace

The lane needs two distinct Slack apps in one workspace, plus a channel both bots are members of.

• channelId - the Cxxxxxxxxxx id of a channel where both bots are members. Pick a channel used only for testing; the lane sends a post on every execution.
• driverBotToken - bot token (xoxb-...) belonging to the Driver app.
• sutBotToken - bot token (xoxb-...) for the SUT app, which must be a distinct Slack app from the driver so its bot user id differs.
• sutAppToken - app-level token (xapp-...) from the SUT app with connections:write, enabling Socket Mode so the SUT app can receive events.

Choose a Slack workspace dedicated to QA rather than reusing one from production.

The SUT manifest shown below deliberately restricts the bundled Slack plugin's production install (extensions/slack/src/setup-shared.ts:12) to only the permissions and events exercised by the live Slack QA suite. For the setup users encounter in production, refer to Slack channel quick setup; the QA Driver/SUT combination is kept separate on purpose because the lane requires two different bot user ids within a single workspace.

1. Create the Driver app

Navigate to api.slack.com/apps → Create New App → From a manifest → select the QA workspace, paste the manifest below, then click Install to Workspace:

{
  "display_information": {
    "name": "OpenClaw QA Driver",
    "description": "Test driver bot for OpenClaw QA Slack live lane"
  },
  "features": {
    "bot_user": {
      "display_name": "OpenClaw QA Driver",
      "always_online": true
    }
  },
  "oauth_config": {
    "scopes": {
      "bot": ["chat:write", "channels:history", "groups:history", "users:read"]
    }
  },
  "settings": {
    "socket_mode_enabled": false
  }
}

Grab the Bot User OAuth Token (xoxb-...) - that value goes into driverBotToken. The driver only needs to send messages and identify itself; no events or Socket Mode are required.

2. Create the SUT app

Repeat Create New App → From a manifest in the same workspace. This QA app uses a trimmed version of the bundled Slack plugin's production manifest (extensions/slack/src/setup-shared.ts:12): reaction scopes and events are left out because the live Slack QA suite does not yet test reaction handling.

{
  "display_information": {
    "name": "OpenClaw QA SUT",
    "description": "OpenClaw QA SUT connector for OpenClaw"
  },
  "features": {
    "bot_user": {
      "display_name": "OpenClaw QA SUT",
      "always_online": true
    },
    "app_home": {
      "home_tab_enabled": true,
      "messages_tab_enabled": true,
      "messages_tab_read_only_enabled": false
    }
  },
  "oauth_config": {
    "scopes": {
      "bot": [
        "app_mentions:read",
        "assistant:write",
        "channels:history",
        "channels:read",
        "chat:write",
        "commands",
        "emoji:read",
        "files:read",
        "files:write",
        "groups:history",
        "groups:read",
        "im:history",
        "im:read",
        "im:write",
        "mpim:history",
        "mpim:read",
        "mpim:write",
        "pins:read",
        "pins:write",
        "usergroups:read",
        "users:read"
      ]
    }
  },
  "settings": {
    "socket_mode_enabled": true,
    "event_subscriptions": {
      "bot_events": [
        "app_home_opened",
        "app_mention",
        "channel_rename",
        "member_joined_channel",
        "member_left_channel",
        "message.channels",
        "message.groups",
        "message.im",
        "message.mpim",
        "pin_added",
        "pin_removed"
      ]
    }
  }
}

Once Slack creates the app, complete two steps on its settings page:

• Install to Workspace → copy the Bot User OAuth Token → that value becomes sutBotToken.
• Basic Information → App-Level Tokens → Generate Token and Scopes → add scope connections:write → save → copy the xapp-... value → that becomes sutAppToken.

Confirm the two bots have distinct user ids by invoking auth.test with each token. The runtime tells driver and SUT apart by user id; using the same app for both causes mention-gating to fail right away.

3. Create the channel

In the QA workspace, make a channel (e.g. #openclaw-qa) and invite both bots from within it:

/invite @OpenClaw QA Driver
/invite @OpenClaw QA SUT

Copy the Cxxxxxxxxxx id from channel info → About → Channel ID - that becomes channelId. A public channel is fine; for a private channel, both apps already hold groups:history, so the harness's history reads still work.

4. Register the credentials

Two approaches exist. Use environment variables for local debugging on one machine (set the four OPENCLAW_QA_SLACK_* variables and pass --credential-source env), or populate the shared Convex pool so CI and other maintainers can lease them.

For the Convex pool, put the four fields into a JSON file:

{
  "channelId": "Cxxxxxxxxxx",
  "driverBotToken": "xoxb-...",
  "sutBotToken": "xoxb-...",
  "sutAppToken": "xapp-..."
}

With OPENCLAW_QA_CONVEX_SITE_URL and OPENCLAW_QA_CONVEX_SECRET_MAINTAINER exported in your shell, register and verify:

pnpm openclaw qa credentials add \
  --kind slack \
  --payload-file slack-creds.json \
  --note "QA Slack pool seed"

pnpm openclaw qa credentials list --kind slack --status all --json

Expect count: 1, status: "active", and no lease field.

5. Verify end to end

Run the lane locally to confirm both bots can communicate through the broker:

pnpm openclaw qa slack \
  --credential-source convex \
  --credential-role maintainer \
  --output-dir .artifacts/qa-e2e/slack-local

A successful run finishes in well under 30 seconds, and qa-suite-report.md lists both slack-canary and slack-mention-gating at status pass. If the lane stalls for about 90 seconds and ends with Convex credential pool exhausted for kind "slack", either the pool is empty or every row is leased - qa credentials list --kind slack --status all --json tells you which.

WhatsApp QA

pnpm openclaw qa whatsapp

Targets two dedicated WhatsApp Web accounts: a driver account managed by the harness and a SUT account launched by the child OpenClaw gateway via the bundled WhatsApp plugin.

Required env when --credential-source env:

• OPENCLAW_QA_WHATSAPP_DRIVER_PHONE_E164
• OPENCLAW_QA_WHATSAPP_SUT_PHONE_E164
• OPENCLAW_QA_WHATSAPP_DRIVER_AUTH_ARCHIVE_BASE64
• OPENCLAW_QA_WHATSAPP_SUT_AUTH_ARCHIVE_BASE64

Optional:

• OPENCLAW_QA_WHATSAPP_GROUP_JID enables group scenarios such as whatsapp-mention-gating, whatsapp-group-pending-history-context, whatsapp-broadcast-group-fanout, whatsapp-group-activation-always, whatsapp-group-reply-to-bot-triggers, group action/media/poll scenarios, and whatsapp-group-allowlist-block.

WhatsApp YAML scenarios (qa/scenarios/channels/whatsapp-*.yaml):

• Baseline and group gating: whatsapp-canary, whatsapp-pairing-block, whatsapp-mention-gating, whatsapp-group-pending-history-context, whatsapp-group-activation-always, whatsapp-group-reply-to-bot-triggers, whatsapp-top-level-reply-shape, whatsapp-restart-resume, whatsapp-group-allowlist-block.
• Native commands: whatsapp-help-command, whatsapp-status-command, whatsapp-commands-command, whatsapp-tools-compact-command, whatsapp-whoami-command, whatsapp-context-command, whatsapp-native-new-command.
• Reply and final-output behavior: whatsapp-tool-only-usage-footer, whatsapp-reply-to-message, whatsapp-group-reply-to-message, whatsapp-reply-to-mode-batched, whatsapp-reply-context-isolation, whatsapp-reply-delivery-shape, whatsapp-stream-final-message-accounting.
• User-path message actions: whatsapp-agent-message-action-react begins with a genuine driver DM, enables the model to invoke the message tool, and tracks the native WhatsApp response. whatsapp-agent-message-action-upload-file adopts the same approach for message(action=upload-file) and tracks native WhatsApp media. whatsapp-group-agent-message-action-react and whatsapp-group-agent-message-action-upload-file demonstrate the same user-facing actions within a real WhatsApp group.
• Group fanout: whatsapp-broadcast-group-fanout begins with a single WhatsApp group message that mentions the agent and checks for distinct visible replies from main and qa-second.
• Group activation: whatsapp-group-activation-always switches a live group session to /activation always, confirms that a group message without a mention rouses the agent, then reverts to /activation mention. whatsapp-group-reply-to-bot-triggers plants a bot reply, sends a native quoted reply to it without an explicit mention, and confirms the agent wakes from that reply context.
• Inbound media and structured messages: whatsapp-inbound-image-caption, whatsapp-audio-preflight, whatsapp-inbound-structured-messages, whatsapp-group-audio-gating, whatsapp-inbound-reaction-no-trigger. These push real WhatsApp image, audio, document, location, contact, sticker, and reaction events through the driver.
• Direct Gateway contract probes: whatsapp-outbound-media-matrix, whatsapp-outbound-document-preserves-filename, whatsapp-outbound-poll, whatsapp-outbound-send-serialization, whatsapp-group-outbound-media, whatsapp-group-outbound-poll, whatsapp-message-actions, whatsapp-reply-context-isolation, whatsapp-reply-delivery-shape. These intentionally skip model prompting and verify deterministic Gateway/channel send, poll, and message.action contracts.
• Access-control coverage: whatsapp-access-control-dm-open, whatsapp-access-control-dm-disabled, whatsapp-access-control-group-open, whatsapp-access-control-group-disabled, whatsapp-group-allowlist-block.
• Native approvals: whatsapp-approval-exec-deny-native, whatsapp-approval-exec-native, whatsapp-approval-exec-reaction-native, whatsapp-approval-exec-group-reaction-native, whatsapp-approval-plugin-native.
• Status reactions: whatsapp-status-reactions, whatsapp-status-reaction-lifecycle.

The taxonomy profile and lane constraints dictate the default settings for WhatsApp. mock-openai executes eligible scenarios in a deterministic manner through the actual WhatsApp transport, mocking only model output; live-frontier filters out scenarios whose provider or model contract necessitates the mock lane.

Structured live events (text, media, location, reaction, and poll) are observed by the WhatsApp QA driver, which can also proactively send media, polls, contacts, locations, and stickers. Rather than accessing private WhatsApp runtime files, QA Lab imports this driver via the @openclaw/whatsapp/api.js package surface. For group observations, fromJid serves as the group JID while participantJid and fromPhoneE164 identify the participant sender. By default, message content is redacted. Direct Gateway probes for polls, upload-file, media, group poll, group media, and reply-shape are transport/API contract checks; they do not constitute proof that a user prompt led the agent to select the same action. Evidence of user-path actions comes from scenarios like whatsapp-agent-message-action-react and whatsapp-group-agent-message-action-react, where the driver transmits a standard WhatsApp message and QA Lab observes the resulting native WhatsApp artifact. Each WhatsApp scenario includes its posture (user-path, direct-gateway, or native-approval) so that evidence is not misread as a stronger contract than it actually demonstrates.

Output artifacts:

• qa-suite-report.md
• qa-suite-summary.json
• qa-evidence.json - evidence entries for the live transport checks.

Convex credential pool

Rather than reading the env vars above, Buzz, Discord, Slack, Telegram, and WhatsApp lanes can lease credentials from a shared Convex pool. Pass --credential-source convex (or set OPENCLAW_QA_CREDENTIAL_SOURCE=convex); QA Lab obtains an exclusive lease, heartbeats it for the run's duration, and releases it upon shutdown. Pool kinds include "buzz", "discord", "slack", "telegram", and "whatsapp".

Payload shapes validated by the broker on admin/add:

• Buzz (kind: "buzz"): { relayUrl: string, roomId: string, driverPrivateKey: string, sutPrivateKey: string, driverAuthTag?: string, sutAuthTag?: string } - relayUrl must use wss://, with ws:// permitted only for loopback relays; roomId must be a channel UUID, and the identities must be distinct.
• Discord (kind: "discord"): { guildId: string, channelId: string, driverBotToken: string, sutBotToken: string, sutApplicationId: string, voiceChannelId?: string }.
• Telegram (kind: "telegram"): { groupId: string, driverToken: string, sutToken: string } - groupId must be a numeric chat-id string.
• Telegram real user (kind: "telegram-user"): { groupId: string, sutToken: string, testerUserId: string, testerUsername: string, telegramApiId: string, telegramApiHash: string, tdlibDatabaseEncryptionKey: string, tdlibArchiveBase64: string, tdlibArchiveSha256: string, desktopTdataArchiveBase64: string, desktopTdataArchiveSha256: string } - Mantis Telegram Desktop proof only. Generic QA Lab lanes must not acquire this kind.
• WhatsApp (kind: "whatsapp"): { driverPhoneE164: string, sutPhoneE164: string, driverAuthArchiveBase64: string, sutAuthArchiveBase64: string, groupJid?: string } - phone numbers must be distinct E.164 strings.

The Mantis Telegram Desktop proof workflow holds one exclusive Convex telegram-user lease for both the TDLib CLI driver and Telegram Desktop witness, then releases it after publishing proof.

When a PR needs a deterministic visual diff, Mantis can use the same mock model reply on main and on the PR head while the Telegram formatter or delivery layer changes. Capture defaults are tuned for PR comments: standard Crabbox class, 24fps desktop recording, 24fps motion GIF, and 1920px preview width. Before/after comments should publish a clean bundle that contains only the intended GIFs.

Slack lanes can also use the pool. Slack payload shape checks currently live in the Slack QA runner rather than the broker; use { channelId: string, driverBotToken: string, sutBotToken: string, sutAppToken: string }, with a Slack channel id like Cxxxxxxxxxx. See Setting up the Slack workspace for app and scope provisioning.

Operational env vars and the Convex broker endpoint contract live in Testing → Shared Telegram credentials via Convex (the section name predates the multi-channel pool; the lease semantics are shared across kinds).

Repo-backed seeds

Seed assets live in qa/:

• qa/scenarios/index.yaml
• qa/scenarios/<theme>/*.yaml

Identity-sensitive channel changes rely on the isolated channel-participant-identity-inspection QA Channel flow. It spins up a real ephemeral Gateway and a mock provider, then checks admitted runs using the same openclaw audit --run ... --explain JSON and human surfaces that operators rely on. The flow also handles lifecycle-owned restart and verifies the row count for rejected pre-run ingress.

These are deliberately committed to git so both humans and the agent can see the QA plan.

qa-lab remains a generic YAML scenario runner. Each scenario YAML file acts as the single source of truth for one test run and should specify:

• top-level title
• scenario metadata
• optional category, capability, lane, and risk metadata in scenario
• docs and code references in scenario
• optional plugin requirements in scenario
• optional gateway config patch in scenario
• executable top-level flow for flow scenarios, or scenario.execution.kind / scenario.execution.path for Vitest and Playwright scenarios

The reusable runtime surface behind flow stays generic and cross-cutting. For instance, YAML scenarios can mix transport-side helpers with browser-side helpers that drive the embedded Control UI through the Gateway browser.request seam, no special-case runner required.

Scenario files should be organized by product capability, not by source tree folder. Keep scenario IDs stable when files move; use docsRefs and codeRefs for implementation traceability.

The baseline list should remain broad enough to cover:

• DM and channel chat
• thread behavior
• message action lifecycle
• cron callbacks
• memory recall
• model switching
• subagent handoff
• repo-reading and docs-reading
• one small build task such as Lobster Invaders

Provider mock lanes

qa suite provides two local provider mock lanes:

• mock-openai is the scenario-aware OpenClaw mock. It stays the default deterministic mock lane for repo-backed QA and parity gates.
• aimock launches an AIMock-backed provider server for experimental protocol, fixture, record/replay, and chaos coverage. It is additive and does not replace the mock-openai scenario dispatcher.

Provider-lane implementation lives under extensions/qa-lab/src/providers/. Each provider owns its defaults, local server startup, gateway model config, auth-profile staging needs, and live/mock capability flags. Shared suite and gateway code routes through the provider registry instead of branching on provider names.

Transport adapters

qa-lab owns a generic transport seam for YAML QA scenarios. qa-channel is the synthetic default. crabline starts local provider-shaped servers and runs OpenClaw's normal channel plugins against them. live is reserved for real provider credentials and external channels.

At the architecture level, the split is:

• qa-lab owns generic scenario execution, worker concurrency, artifact writing, and reporting.
• The transport adapter owns gateway config, readiness, inbound and outbound observation, transport actions, and normalized transport state.
• YAML scenario files under qa/scenarios/ define the test run; qa-lab provides the reusable runtime surface that executes them.

Adding a channel

Adding a channel to the YAML QA system requires the channel implementation plus a scenario pack that exercises the channel contract. For smoke CI coverage, add the matching Crabline local provider server and expose it through the crabline driver.

Do not add a new top-level QA command root when the shared qa-lab host can own the flow.

qa-lab owns the shared host mechanics:

• the openclaw qa command root
• suite startup and teardown
• worker concurrency
• artifact writing
• report generation
• scenario execution
• compatibility aliases for older qa-channel scenarios

Runner plugins own the transport contract:

• how openclaw qa <runner> is mounted beneath the shared qa root
• how the gateway is configured for that transport
• how readiness is checked
• how inbound events are injected
• how outbound messages are observed
• how transcripts and normalized transport state are exposed
• how transport-backed actions are executed
• how transport-specific reset or cleanup is handled

The minimum adoption bar for a new channel:

1. Keep qa-lab as the owner of the shared qa root.
2. Implement the transport runner on the shared qa-lab host seam.
3. Keep transport-specific mechanics inside the runner plugin or channel harness.
4. Mount the runner as openclaw qa <runner> instead of registering a competing root command. Runner plugins should declare qaRunners in openclaw.plugin.json and export a matching qaRunnerCliRegistrations array from a lightweight qa-runner-api.ts surface. Installed plugins using the shipped runtime-api.ts contract remain supported through 2026-10-01 while authors migrate. Keep runner execution behind lazy entrypoints. An optional adapterFactory exposes the transport to shared scenarios without changing the command's existing scenario catalog. Same-channel partitions are serial unless the factory declares that every instance owns isolated credentials or disposable servers, Gateway state, and artifact paths. Module-backed flow scenarios additionally require adapterFactory.supportsModuleFlows: true; those factories must return adapters that implement prepareFlow.
5. Author or adapt YAML scenarios under the themed qa/scenarios/ directories.
6. Use the generic scenario helpers for new scenarios.
7. Keep existing compatibility aliases working unless the repo is doing an intentional migration.

The decision rule is strict:

• If behavior can be expressed once in qa-lab, put it in qa-lab.
• If behavior depends on one channel transport, keep it in that runner plugin or plugin harness.
• If a scenario needs a new capability that more than one channel can use, add a generic helper instead of a channel-specific branch in suite.ts.
• If a behavior is only meaningful for one transport, keep the scenario transport-specific and make that explicit in the scenario contract.

Scenario helper names

Preferred generic helpers for new scenarios:

• waitForTransportReady
• waitForChannelReady
• injectInboundMessage
• injectOutboundMessage
• waitForOutboundMessage
• waitForNoTransportOutbound
• getTransportSnapshot
• readTransportMessage
• readTransportTranscript
• formatTransportTranscript
• resetTransport

For older setups, compatibility aliases stay in place: waitForQaChannelReady, waitForNoOutbound, formatConversationTranscript, and resetBus. But when writing fresh scenarios, stick with the generic names. Outbound checks should rely on the canonical waitForOutboundMessage rather than introducing transport-specific or channel-specific outbound wait aliases.

Reporting

From the observed bus timeline, qa-lab produces a Markdown protocol report. That report is meant to clarify:

• What succeeded
• What failed
• What remained blocked
• Which follow-up scenarios are likely worth adding

To see the full inventory of available scenarios, which helps when estimating follow-up effort or connecting a new transport, run pnpm openclaw qa coverage (append --json for output that machines can parse). When you need focused proof for a specific behavior or file path, use pnpm openclaw qa coverage --match <query>. The match report scans scenario metadata, documentation references, code references, coverage IDs, plugins, and provider requirements, then prints the qa suite --scenario ... targets that match.

Each qa suite execution writes top-level qa-evidence.json, qa-suite-summary.json, and qa-suite-report.md artifacts for the chosen scenario set. Scenarios declaring execution.kind: vitest or execution.kind: playwright execute the corresponding test path and also produce per-scenario logs. For scenarios that declare execution.kind: script, the evidence producer runs at execution.path through node --import tsx (with ${outputDir} and ${scenarioId} expanded in execution.args); that producer writes its own qa-evidence.json, whose entries get pulled into the suite output and whose artifact paths resolve relative to the producer qa-evidence.json. When qa suite is reached via qa run --qa-profile, the same qa-evidence.json also carries the profile scorecard summary for the selected taxonomy categories.

Treat coverage output as a discovery tool, not a replacement for gates; the chosen scenario still demands the correct provider mode, live transport, Multipass, Testbox, or release lane for the behavior under test. For scorecard context, refer to Maturity scorecard.

For character and style evaluation, execute the same scenario across several live model refs and generate a judged Markdown report:

pnpm openclaw qa character-eval \
  --model openai/gpt-5.6-luna,thinking=medium,fast \
  --model openai/gpt-5.2,thinking=xhigh \
  --model openai/gpt-5,thinking=xhigh \
  --model anthropic/claude-opus-4-8,thinking=high \
  --model anthropic/claude-sonnet-4-6,thinking=high \
  --model zai/glm-5.1,thinking=high \
  --model moonshot/kimi-k2.5,thinking=high \
  --model google/gemini-3.1-pro-preview,thinking=high \
  --judge-model openai/gpt-5.6-sol,thinking=xhigh,fast \
  --judge-model anthropic/claude-opus-4-8,thinking=high \
  --blind-judge-models \
  --concurrency 16 \
  --judge-concurrency 16

This command launches local QA gateway child processes, not Docker. Character eval scenarios should establish the persona through SOUL.md, then proceed with ordinary user turns like chat, workspace help, and small file tasks. The candidate model must not learn that it is under evaluation. The command keeps every full transcript, captures basic run stats, and then asks the judge models in fast mode with xhigh reasoning where supported to rank runs by naturalness, vibe, and humor. Use --blind-judge-models when comparing providers: the judge prompt still receives every transcript and run status, but candidate refs are swapped for neutral labels such as candidate-01; after parsing, the report maps rankings back to the real refs.

Candidate runs default to high thinking, with medium for GPT-5.6 Luna and xhigh for older OpenAI eval refs that support it. Override a specific candidate inline with --model provider/model,thinking=<level>; inline options also accommodate fast, no-fast, and fast=<bool>. --thinking <level> continues to set a global fallback, and the older --model-thinking <provider/model=level> form remains for compatibility. OpenAI candidate refs default to fast mode so priority processing applies where the provider supports it. Pass --fast only when you want fast mode forced on for every candidate model. Candidate and judge durations are logged in the report for benchmark analysis, but judge prompts explicitly instruct not to rank by speed. Both candidate and judge model runs default to concurrency 16. Lower --concurrency or --judge-concurrency when provider limits or local gateway pressure make a run too noisy.

If no candidate --model is supplied, the character evaluation falls back to openai/gpt-5.6-luna, openai/gpt-5.2, openai/gpt-5, anthropic/claude-opus-4-8, anthropic/claude-sonnet-4-6, zai/glm-5.1, moonshot/kimi-k2.5, and google/gemini-3.1-pro-preview. In the absence of a --judge-model, the default judges are openai/gpt-5.6-sol,thinking=xhigh,fast and anthropic/claude-opus-4-8,thinking=high.

Related docs

• Maturity scorecard
• Personal agent benchmark pack
• QA Channel
• Testing
• Dashboard