Mantis: Visual CI Evidence for OpenClaw Behavior

Mantis generates visual CI evidence and a PR comment for OpenClaw behavior.
Live transport scenarios compare a known-bad baseline with a candidate ref;
focused browser lanes may instead prove one candidate against a deterministic
mocked transport. Discord shipped first with real bot auth, guild channels,
reactions, threads, and a browser witness. Slack, Telegram, and focused Control
UI chat lanes exist too; WhatsApp and Matrix are unimplemented.

Ownership

• OpenClaw (extensions/qa-lab/src/mantis/*): scenario runtime, pnpm openclaw qa mantis <command> CLI, evidence schema.
• QA Lab (extensions/qa-lab/src/live-transports/*): live transport harness, driver/SUT bots, report/evidence writers.
• Crabbox (openclaw/crabbox): warmed Linux machines, leases, VNC, crabbox media preview.
• GitHub Actions (.github/workflows/mantis-*.yml): remote entrypoints, artifact retention.
• ClawSweeper: parses maintainer PR commands, dispatches workflows, posts the final PR comment.

CLI commands

All commands are pnpm openclaw qa mantis <command>, defined in
extensions/qa-lab/src/mantis/cli.ts. Requires OPENCLAW_ENABLE_PRIVATE_QA_CLI=1
at build/run time (bundled workflows set OPENCLAW_BUILD_PRIVATE_QA=1 and
OPENCLAW_ENABLE_PRIVATE_QA_CLI=1 before building).

Every command accepts --repo-root <path> and --output-dir <path>; Crabbox
commands also accept --crabbox-bin, --provider, --machine-class/--class,
--lease-id, --idle-timeout, --ttl, and --keep-lease. Local CLI defaults
for provider/class are hetzner/beast unless noted otherwise; CI workflows
usually override both.

discord-smoke

pnpm openclaw qa mantis discord-smoke \
  --output-dir .artifacts/qa-e2e/mantis/discord-smoke

Calls the Discord REST API (https://discord.com/api/v10) to fetch the bot
user, the guild, the guild's channels, and the target channel, asserts the
channel belongs to the guild, then (unless --skip-post) posts a message and
adds a 👀 reaction. Writes mantis-discord-smoke-summary.json and
mantis-discord-smoke-report.md.

Token resolution order: --token-file value, then OPENCLAW_QA_DISCORD_MANTIS_BOT_TOKEN
(override with --token-env), then a file named by OPENCLAW_QA_DISCORD_MANTIS_BOT_TOKEN_FILE
(override with --token-file-env). Guild/channel ids come from
OPENCLAW_QA_DISCORD_GUILD_ID / OPENCLAW_QA_DISCORD_CHANNEL_ID (override with
--guild-id / --channel-id) and must be 17-20 digit Discord snowflakes. Set
OPENCLAW_QA_REDACT_PUBLIC_METADATA=1 to replace bot/guild/channel/message ids
and names with <redacted> in the published summary and report.

run

pnpm openclaw qa mantis run \
  --transport discord \
  --scenario discord-status-reactions-tool-only \
  --baseline origin/main \
  --candidate HEAD \
  --output-dir .artifacts/qa-e2e/mantis/local-discord-status-reactions

--transport currently only accepts discord. --scenario is one of two
built-in ids, each with its own default baseline ref and expected before/after
labels (extensions/qa-lab/src/mantis/run.runtime.ts):

--candidate is set to HEAD by default. Additional flags include --credential-source (standard convex), --credential-role (standard ci), --provider-mode (standard live-frontier), --fast (standard on), --skip-install, and --skip-build.

For both the baseline and the candidate, the runner creates separate git worktree checkouts under <output-dir>/worktrees/, executes pnpm install/pnpm build in each (unless those steps are skipped), and then applies pnpm openclaw qa discord --scenario <id> --model openai/gpt-5.4 --alt-model openai/gpt-5.4 --allow-failures to every worktree. Each lane outputs discord-qa-reaction-timelines.json along with a <scenario-id>-timeline.html/.png pair; the runner transfers this evidence to baseline//candidate/, writes comparison.json, mantis-report.md, and mantis-evidence.json into the output directory, and returns a nonzero exit code when the comparison fails (baseline fail and candidate pass).

In the second Discord scenario (discord-thread-reply-filepath-attachment), a parent message is posted with the driver bot, a real thread gets created, the SUT's message.thread-reply action is invoked with a repo-local filePath, and the thread is polled for the reply and the attachment filename. The expected attachment name is mantis-thread-report.md.

desktop-browser-smoke

pnpm openclaw qa mantis desktop-browser-smoke \
  --output-dir .artifacts/qa-e2e/mantis/desktop-browser

A Crabbox desktop is leased or reused, a browser launches inside the VNC session targeting --browser-url (standard https://openclaw.ai) or a rendered --html-file, a wait occurs, a screenshot is taken with scrot, an MP4 is optionally recorded with ffmpeg, and desktop-browser-smoke.png / .mp4 / remote-metadata.json are rsynced back to --output-dir.

Flags:

• --lease-id <cbx_...> reuses a warmed desktop rather than creating a fresh one.
• --browser-profile-dir <remote-path> reuses a remote Chrome user-data-dir, allowing a persistent desktop to remain logged in across runs (applied for a long-lived Discord Web viewer profile).
• --browser-profile-archive-env <name> restores a base64 .tgz Chrome profile archive from that env var prior to launch (standard OPENCLAW_MANTIS_BROWSER_PROFILE_TGZ_B64); this is used for logged-in witnesses such as Discord Web.
• --video-duration <seconds> sets the MP4 capture duration (standard 10s).
• --keep-lease (or OPENCLAW_MANTIS_KEEP_VM=1) keeps a lease created by this run open for VNC inspection; failed runs that created a lease also retain it by default.

For Discord Web evidence, Mantis relies on a dedicated viewer account rather than a bot token. The Discord REST oracle (through qa discord) stays authoritative; when OPENCLAW_QA_DISCORD_CAPTURE_UI_METADATA=1 is configured, the scenario additionally writes a Discord Web URL artifact, and OPENCLAW_QA_DISCORD_KEEP_THREADS=1 keeps the thread open long enough for the browser to access it.

The GitHub workflow leans on a persistent viewer profile through MANTIS_DISCORD_VIEWER_CHROME_PROFILE_DIR (complete profile archives can exceed GitHub's secret size cap); for minimal or bootstrap profiles it can fall back to restoring a base64 .tgz sourced from MANTIS_DISCORD_VIEWER_CHROME_PROFILE_TGZ_B64. When neither source is set up, the workflow still emits the deterministic baseline/candidate screenshots and logs while noting that the logged-in witness was not used.

slack-desktop-smoke

pnpm openclaw qa mantis slack-desktop-smoke \
  --output-dir .artifacts/qa-e2e/mantis/slack-desktop \
  --gateway-setup \
  --scenario slack-canary \
  --keep-lease

It leases or reuses a Crabbox desktop, syncs the checkout into the VM, executes pnpm openclaw qa slack within it, launches Slack Web in the VNC browser, grabs the desktop image, and pulls back both the Slack QA artifacts (slack-qa/) and the VNC screenshot/video locally. Among all Mantis shapes, this one is unique: the SUT gateway and the browser share the same VM.

When --gateway-setup is present, the command sets up a persistent disposable OpenClaw home at $HOME/.openclaw-mantis/slack-openclaw inside the VM, adjusts Slack Socket Mode configuration for the designated channel, starts openclaw gateway run --dev --allow-unconfigured --port 38973, and keeps Chrome active in the VNC session; dropping --gateway-setup switches to the standard bot-to-bot Slack QA path.

Required env for --credential-source env (local default is env; role default is maintainer):

• OPENCLAW_QA_SLACK_CHANNEL_ID
• OPENCLAW_QA_SLACK_DRIVER_BOT_TOKEN
• OPENCLAW_QA_SLACK_SUT_BOT_TOKEN
• OPENCLAW_QA_SLACK_SUT_APP_TOKEN
• OPENCLAW_LIVE_OPENAI_KEY for the remote model lane (if only OPENAI_API_KEY is present locally, Mantis duplicates it to OPENCLAW_LIVE_OPENAI_KEY before calling Crabbox)

With --credential-source convex, Mantis pulls the Slack SUT credential from the shared pool prior to VM creation and passes channel id, app token, and bot token into the VM as OPENCLAW_MANTIS_SLACK_* env vars, so GitHub workflows require only the Convex broker secret, never raw Slack tokens.

Additional flags: --slack-url <url> targets a specific URL (otherwise Mantis computes https://app.slack.com/client/<team>/<channel> from auth.test); --slack-channel-id <id> defines the gateway allowlist channel; OPENCLAW_MANTIS_SLACK_BROWSER_PROFILE_DIR governs the persistent Chrome profile within the VM (default $HOME/.config/openclaw-mantis/slack-chrome-profile); --approval-checkpoints executes the native Slack approval scenarios (slack-approval-exec-native, slack-approval-plugin-native) and outputs pending/resolved checkpoint screenshots instead of gateway setup (cannot be combined with --gateway-setup); --hydrate-mode source|prehydrated, --provider-mode, --model, --alt-model, and --fast forward to the Slack live lane.

Checkpoint screenshots for approvals come from the Slack API message the scenario saw, not the live Slack UI; slack-desktop-smoke.png merely demonstrates Slack Web itself when the lease's browser profile was already authenticated.

Telegram Desktop recorder

The Telegram Desktop recorder acts as a standalone operator tool, called directly via pnpm qa:telegram-desktop-recorder. It never touches OpenClaw. Its standard recording commands send no Telegram messages. The optional actions command operates solely on the measured Telegram window through bounded click, key, type, and sleep actions; those actions can send as the signed-in Telegram user. The operator owns the turn and those side effects: start the SUT, send via a real Telegram user, then instruct the recorder on which message to display. The operator also provides --user-driver, the command the recorder shells out to for TDLib operations it cannot perform itself (confirm-qr, terminate-session). Any driver exposing those two verbs works, including this repo's scripts/e2e/telegram-user-driver.py.

The Mantis Telegram Desktop Proof workflow invokes the recorder with its local Docker provider. Its OpenClaw SUT stays isolated in the lane-attested container boundary while Telegram Desktop runs in the prebaked local image.

Start recording. --session names the run-scoped session handle: when it already points at a healthy authorized desktop, start reuses it and only starts a fresh capture in the new output directory; otherwise it provisions and QR-authorizes a desktop first.

pnpm qa:telegram-desktop-recorder start \
  --session .artifacts/qa-e2e/desktop-recorder.json \
  --output-dir .artifacts/qa-e2e/telegram-desktop \
  --chat -1001234567890 \
  --user-driver "python3 /path/to/telegram-user-driver.py" \
  --json

Open a recorded group post with view --session <recorder.json> --message-id <id>. Use screenshot --session <recorder.json> for a still image. Run stop --session <recorder.json> --crop telegram-window to copy the recording and logs and create motion GIFs; the authorized desktop remains alive for the next start, so repeated captures skip provisioning and QR login. When the run is done, run teardown --session <recorder.json> to end the Telegram Desktop authorization and release the Crabbox lease; the box stays inspectable over WebVNC until then.

The recorder defaults to Crabbox's local Docker desktop path. Build the pinned image once, then run start without coordinator access:

bash scripts/mantis/build-telegram-desktop-image.sh

--provider aws points at a Crabbox catalog-only Telegram variant image (--image-sdk telegram-desktop=7.0.9) so the standard desktop image never includes the client. Publishing that variant requires Crabbox coordinator admin (crabbox image create / image promote) and is not yet part of this repository; until it is published, use the local Docker image above. Either image must include an executable Telegram Desktop at /opt/Telegram/Telegram, a readable desktop version marker, wmctrl, xdotool, scrot, ffmpeg, zbarimg, and xdpyinfo, plus a reachable DISPLAY=:99. Crabbox rejects the lease when no matching image is promoted, and recorder startup fails when the contract is incomplete. It does not install packages or download replacements.

telegram-desktop-builder

pnpm openclaw qa mantis telegram-desktop-builder \
  --credential-source convex \
  --credential-role maintainer \
  --keep-lease

Leases or reuses a Crabbox desktop, installs native Linux Telegram Desktop, optionally restores a user-session archive, configures OpenClaw with the leased Telegram SUT bot token, starts openclaw gateway run --dev --allow-unconfigured --port 38974, posts a driver-bot readiness message to the leased private group, then captures a screenshot and MP4. A bot token only configures OpenClaw; it never logs Telegram Desktop in. The desktop viewer is a separate Telegram user session restored from --telegram-profile-archive-env <name> or logged in manually through VNC and kept alive with --keep-lease.

Flags: --lease-id <cbx_...> reruns against a VM already logged in to Telegram Desktop; --telegram-profile-archive-env <name> restores a base64 .tgz profile archive before launch; --telegram-profile-dir <remote-path> sets the remote profile directory (default $HOME/.local/share/TelegramDesktop); --no-gateway-setup installs and opens Telegram Desktop only; --credential-source/--credential-role default to convex/maintainer.

Evidence manifest

Every scenario that publishes to a PR writes mantis-evidence.json next to its report:

{
  "schemaVersion": 1,
  "id": "discord-status-reactions",
  "title": "Mantis Discord Status Reactions QA",
  "summary": "Human-readable top summary for the PR comment.",
  "scenario": "discord-status-reactions-tool-only",
  "comparison": {
    "baseline": { "sha": "...", "status": "fail", "expected": "queued-only" },
    "candidate": { "sha": "...", "status": "pass", "expected": "queued -> thinking -> done" },
    "pass": true
  },
  "artifacts": [
    {
      "kind": "timeline",
      "lane": "baseline",
      "label": "Baseline queued-only",
      "path": "baseline/timeline.png",
      "targetPath": "baseline.png",
      "alt": "Baseline Discord timeline",
      "width": 420
    }
  ]
}

Artifact path is relative to the manifest's directory; targetPath is relative to the configured R2/S3 artifact prefix. scripts/mantis/publish-pr-evidence.mjs rejects path traversal and skips entries with "required": false when the file is missing.

Artifact kinds: timeline (deterministic before/after screenshot), desktopScreenshot (VNC/browser screenshot), motionPreview (inline animated GIF from the recording), motionClip (motion-trimmed MP4), fullVideo (full recording), metadata (JSON/log sidecar), report (Markdown report).

A run's on-disk artifact layout:

.artifacts/qa-e2e/mantis/<run-id>/
  mantis-report.md
  mantis-evidence.json
  baseline/
  candidate/
  comparison.json

Screenshots are evidence, not secrets, but still need redaction discipline: private channel names, usernames, or message content may appear. Set OPENCLAW_QA_REDACT_PUBLIC_METADATA=1 for public artifact uploads; it is enabled by default in the Discord/Slack/Telegram GitHub workflows.

GitHub automation

scripts/mantis/publish-pr-evidence.mjs is the reusable publisher. Workflows call it with the manifest, target PR, artifact target root, comment marker, artifact URL, run URL, and request source. It uploads declared artifacts to the Mantis R2 bucket, builds a summary-first PR comment with inline images/previews and linked videos, then updates the existing marker comment or creates a new one. Required env:

• MANTIS_ARTIFACT_R2_ACCESS_KEY_ID
• MANTIS_ARTIFACT_R2_SECRET_ACCESS_KEY
• MANTIS_ARTIFACT_R2_BUCKET (workflows set openclaw-crabbox-artifacts)
• MANTIS_ARTIFACT_R2_ENDPOINT
• MANTIS_ARTIFACT_R2_REGION (workflows set auto)
• MANTIS_ARTIFACT_R2_PUBLIC_BASE_URL (workflows set https://artifacts.openclaw.ai)

Comments post through the Mantis GitHub App (MANTIS_GITHUB_APP_ID / MANTIS_GITHUB_APP_PRIVATE_KEY), not github-actions[bot], using a hidden marker comment as the upsert key.

Both Mantis Discord Status Reactions and Mantis Telegram Live accept baseline_ref/candidate_ref and check that the resolved SHA is either an ancestor of origin/main, a release tag (v*), or the head of an open PR before proceeding with secret-bearing credentials.

Comment triggers, from a PR with write/maintain/admin access:

@openclaw-mantis
@openclaw-mantis verify the streamed reply stays visible while it arrives

Mantis Telegram Desktop Proof reacts to a PR comment only when the commenter holds write, maintain, or admin access. A simple mention kicks off the desktop proof; any leftover text becomes optional proof guidance. Mantis responds with 👀 when it accepts the request, then includes the active run link in its evidence comment and swaps that same comment for the result. ClawSweeper's mantis: telegram-visible-proof label starts the proof automatically for branches in openclaw/openclaw; fork PRs still need an explicit maintainer comment. Manual runs first review the diff and halt before desktop setup when no Telegram-visible behavior is present to test.

The other scenario workflows stay accessible through manual Actions dispatch.

ClawSweeper can also trigger a scenario directly:

@clawsweeper mantis discord discord-status-reactions-tool-only

Machines and secrets

Local CLI Crabbox defaults are --provider hetzner --class beast; adjust with --provider, --class/--machine-class, or OPENCLAW_MANTIS_CRABBOX_PROVIDER / OPENCLAW_MANTIS_CRABBOX_CLASS. GitHub workflows commonly override both (for instance --class standard, and the Slack workflow's aws/hetzner provider choice input). If a provider is too slow or unavailable, add it behind the same Crabbox interface instead of hardcoding a fallback.

VM baseline: Linux with a desktop-capable Chrome/Chromium, CDP access, VNC/ noVNC, Node 22.22.3+, 24.15+, or 25.9+ and pnpm, an OpenClaw checkout, and outbound access to the target transport, GitHub, model providers, and the credential broker.

Credential and environment names used across Mantis commands and workflows:

• OPENCLAW_QA_DISCORD_MANTIS_BOT_TOKEN
• OPENCLAW_QA_DISCORD_GUILD_ID
• OPENCLAW_QA_DISCORD_CHANNEL_ID
• Running qa mantis run --credential-source env locally also depends on OPENCLAW_QA_DISCORD_DRIVER_BOT_TOKEN, OPENCLAW_QA_DISCORD_SUT_BOT_TOKEN, and OPENCLAW_QA_DISCORD_SUT_APPLICATION_ID. In GitHub workflows, --credential-source convex plus the broker credentials listed below are typically used rather than direct Discord bot tokens.
• OPENCLAW_QA_REDACT_PUBLIC_METADATA=1 is meant for publishing public artifacts
• OPENCLAW_QA_CONVEX_SITE_URL, OPENCLAW_QA_CONVEX_SECRET_CI
• OPENAI_API_KEY (alternatively, the Telegram Desktop proof-specific OPENCLAW_MANTIS_AGENT_OPENAI_API_KEY)
• CRABBOX_COORDINATOR / CRABBOX_COORDINATOR_TOKEN (workflows additionally support OPENCLAW_QA_MANTIS_CRABBOX_COORDINATOR / _TOKEN as a substitute, converting them to the standard names before Crabbox is invoked)
• CRABBOX_ACCESS_CLIENT_ID, CRABBOX_ACCESS_CLIENT_SECRET
• MANTIS_GITHUB_APP_ID, MANTIS_GITHUB_APP_PRIVATE_KEY

The Mantis runner must avoid outputting Discord/Slack/Telegram bot tokens, provider API keys, browser cookies, auth profile contents, VNC passwords, or raw credential payloads under any circumstances. Should a token appear in an issue, PR, chat, or log, rotate it once the replacement secret is saved.

Run outcomes

Transport scenarios before and after are distinguished by these outcomes, so a flaky environment is not mistaken for a product regression:

• Bug reproduced: the baseline failed exactly as the scenario predicted.
• Harness failure: the environment setup, credentials, transport API, browser, or provider broke before the oracle could be evaluated.

A candidate-only browser proof reports whether the candidate passed the mocked Gateway and visible UI assertions; it makes no claim about baseline reproduction.

Adding a scenario

Live transport scenarios are defined in TypeScript for each transport (see MANTIS_SCENARIO_CONFIGS within extensions/qa-lab/src/mantis/run.runtime.ts for the Discord before/after structure), not as a separate declarative file format. Each scenario requires: an id and title, transport, required credentials, baseline ref policy, candidate ref policy, OpenClaw config patch, setup/stimulus steps, expected baseline and candidate oracle, visual capture targets, timeout budget, and cleanup steps.

A focused candidate-only browser proof may rely on a dedicated deterministic E2E test and workflow. Keep its scope explicit, verify the candidate ref prior to execution, isolate secret-backed publishing, and emit the same evidence manifest contract.

Prefer small, typed oracles over vision checks: Discord reaction state or message references, Slack thread ts/reaction API state, email message ids and headers. Use browser screenshots when UI is the only reliable observable, and keep vision checks additive to a platform-API oracle where one exists.

Following Discord, Slack, and Telegram, the same runner shape extends to WhatsApp (QR login, re-identification, delivery, media, reactions) and Matrix (encrypted rooms, thread/reply relations, restart resume); neither is implemented yet.

Open questions

• Which Discord bot serves as the driver versus the SUT when the existing Mantis bot is reused?
• For how long should GitHub retain Mantis artifacts for PRs?
• When should ClawSweeper automatically suggest a Mantis scenario instead of waiting for a maintainer command?
• Should screenshots be redacted or cropped before upload for public PRs?