CI Pipeline: Job Graph, Scope Gates, and Release Umbrellas

OpenClaw CI activates when code is pushed to main (excluding Markdown and docs/** paths at the trigger level), for every pull request that is not in draft mode, and through manual dispatch. Canonical pushes to main use single-flight execution: the CI concurrency group allows one full integration cycle to finish while GitHub queues only the most recent pending push. New merges replace that queued run instead of aborting work that has already registered a Blacksmith matrix. Pull requests still cancel outdated heads, and manual dispatches operate in separate groups. preflight analyzes the diff and disables expensive lanes when only unrelated areas have changed. Manual workflow_dispatch dispatches deliberately bypass smart scoping and run the complete graph for release candidates and broad validation. Android lanes require opt-in via include_android (or the release_gate input). Release-only plugin coverage resides in the separate Plugin Prerelease workflow and triggers only from Full Release Validation or an explicit manual dispatch.

Pipeline overview

Standalone Periphery workflows enforce zero dead-code findings for the iOS and macOS apps. The shared OpenClawKit workflow scans both consumers in parallel and reports a declaration only when Periphery emits the same Swift USR from both builds. Its generated OpenClawProtocol/GatewayModels.swift schema contract is retained as generator-owned code rather than treated as app-local dead code.

Fail-fast order

1. preflight determines which lanes are present. The docs-scope and changed-scope logic are steps contained within this job rather than separate jobs. Canonical main launches right away, but its concurrency group permits only one complete run and merges later pushes into a single newest pending run. Main pushes relevant to nodes also serialize the exclusive dependency-disk writer and its size maintenance here before downstream jobs can mount the key; Blacksmith may present a fresh commit only to a later workflow run, so same-run consumers retain the local fallback verified by the marker.

2. security-fast, check-*, check-additional-*, check-docs, and skills-python fail rapidly without waiting for the heavier artifact and platform matrix jobs.

3. build-artifacts and the locale checks run concurrently with the fast Linux lanes. PRs for the control UI and native app source exclude generated locale snapshots and resources; their serialized refresh workflows repair and auto-merge isolated generated PRs in the background. Source CI still blocks outdated source inventories and unsafe localization calls. Generated PRs, manual CI, and release preparation enforce full translated and platform-generated parity. Canonical release/YYYY.M.PATCH branches may include release-prep locale repairs along with other generated release output.

4. Heavier platform and runtime lanes branch out after that: checks-fast-core, checks-fast-contracts-plugins-*, checks-fast-contracts-channels-*, checks-node-*, checks-windows, macos-node, macos-swift, ios-build, and android.

5. openclaw/ci-gate waits for every selected lane. Preflight and security must succeed; downstream jobs may be skipped only when the manifest did not select them. A failed or canceled selected lane causes the aggregate to fail.

The merge coordinator may reuse an authenticated successful openclaw/ci-gate for the same pull-request head for up to 24 hours. This prevents rewriting a contributor branch after unrelated main changes. The reusable result does not replace the separate strict, App-owned test-merge check against the current main. A later pending or failed rerun does not erase an earlier successful result for that unchanged head during the freshness window.

The default-branch ruleset requires the GitHub Actions-owned openclaw/ci-gate check. Repository maintainers and admins have an audited break-glass bypass intended only for signed direct fast-forward landings; the organization ruleset still blocks deletion and non-fast-forward updates. Normal pull-request merges should continue to use the gate rather than bypass failed CI. The separate strict App-owned test-merge check still binds the head to the current main.

GitHub may mark superseded pull-request jobs as cancelled when a newer head lands. Treat that as CI noise unless the newest run for the same PR is also failing. Canonical main runs are not canceled after admission; when merge traffic arrives, GitHub replaces only the older pending run with the newest tip. Matrix jobs use fail-fast: false, and build-artifacts reports embedded channel, core-support-boundary, and gateway-watch failures directly instead of queuing tiny verifier jobs. The automatic CI concurrency key is versioned (CI-v7-*) so a GitHub-side zombie in an old queue group cannot indefinitely block newer main runs. Manual full-suite runs use CI-manual-v1-* and do not cancel in-progress runs. The plugin-list startup-memory guard keeps a 350 MiB ceiling on self-hosted Blacksmith Linux and allows 425 MiB on GitHub-hosted Linux, whose RSS baseline is higher for the same built CLI.

Use pnpm ci:timings, pnpm ci:timings:recent, or node scripts/ci-run-timings.mjs <run-id> to summarize wall time, queue time, slowest jobs, failures, and the pnpm-store-warmup fanout barrier from GitHub Actions. The in-workflow ci-timings-summary job exists in ci.yml but is currently disabled (if: false); run the timing helper locally instead. For build timing, check the build-artifacts job's Build dist step: pnpm build:ci-artifacts prints [build-all] phase timings: and includes ui:build; the job also uploads the startup-memory artifact.

PR context and evidence

External contributor PRs run a PR context and evidence gate from .github/workflows/real-behavior-proof.yml. The workflow checks out the trusted workflow revision (github.workflow_sha) and evaluates the PR body only; it does not execute code from the contributor branch.

The gate applies to PR authors who are not repository owners, members, collaborators, or bots. It passes when the PR body contains authored What Problem This Solves and Evidence sections. Evidence can be a focused test, CI result, screenshot, recording, terminal output, live observation, redacted log, or artifact link. The body provides intent and useful validation; reviewers inspect the code, tests, and CI to assess correctness.

When the check fails, update the PR body instead of pushing another code commit.

Scope and routing

Scope logic lives in scripts/ci-changed-scope.mjs and is covered by unit tests in src/scripts/ci-changed-scope.test.ts. Manual dispatch skips changed-scope detection and makes the preflight manifest act as if every scoped area changed.

Separate iOS and macOS Periphery workflows enforce a zero-findings dead-code policy. Each runs only when a non-draft pull request touches its native scan scope, or when manually dispatched.

• CI workflow edits validate the Node CI graph, workflow linting, and the Windows lane (ci.yml runs it), but they do not independently force iOS, Android, or macOS native builds; those platform lanes remain scoped to platform source modifications.
• Workflow Sanity executes actionlint and zizmor across all workflow YAML files, the composite-action interpolation guard, and the conflict-marker guard. The PR-scoped security-fast job additionally runs zizmor over modified workflow files so workflow security findings fail early in the main CI graph.
• Docs on main pushes are validated by the standalone Docs workflow using the same ClawHub docs mirror that CI employs, so mixed code and docs pushes do not also enqueue the CI check-docs shard. Pull requests and manual CI still run check-docs from CI when documentation has changed.
• TUI PTY operates within the checks-node-core-runtime-tui-pty Linux Node shard for TUI modifications. The shard runs test/vitest/vitest.tui-pty.config.ts with OPENCLAW_TUI_PTY_INCLUDE_LOCAL=1, covering both the deterministic TuiBackend fixture lane and the slower tui --local smoke that mocks only the external model endpoint.
• CI routing-only edits, the small set of core-test fixtures the fast task runs directly, and narrow plugin contract helper edits follow a fast Node-only manifest path: preflight, security-fast, and only the fast lanes the change touches, a single checks-fast-core CI-routing task, the two plugin contract shards, or both. That path skips build artifacts, Node 22 compatibility, channel contracts, full core shards, bundled-plugin shards, and additional guard matrices.
• Windows Node checks are confined to Windows-specific process and path wrappers, npm/pnpm/UI runner helpers, package manager configuration, and the CI workflow surfaces that execute that lane; unrelated source, plugin, install-smoke, and test-only changes remain on the Linux Node lanes.

The slowest Node test families are split or balanced so each job stays small without over-reserving runners:

• Plugin contracts and channel contracts each run as two weighted Blacksmith-backed shards with the standard GitHub runner fallback.
• Core unit fast and support lanes run separately; core runtime infra splits into process, shared, hooks, secrets, and three cron domain shards.
• Auto-reply runs as balanced workers, with the reply subtree divided into agent-runner, commands, dispatch, session, and state-routing shards.
• Agentic gateway/server (control-plane) configs split across chat, auth, model, HTTP/plugin, runtime, and startup lanes instead of waiting on built artifacts.
• Normal CI packs only isolated infra include-pattern shards into deterministic bundles of at most 64 test files, reducing the Node matrix without merging non-isolated command/cron, stateful agents-core, or gateway/server suites. Heavy fixed suites stay on 8 vCPU while the bundled and lower-weight lanes use 4 vCPU.
• Pull requests on the canonical repository reuse the changed-test resolver against the synthetic merged-tree diff. Precise changes run one targeted Node job; each selected test file gets its own process so stateful suite isolation remains intact. The planner combines sibling tests with import-graph dependents and falls back to the existing 14-job compact full-suite plan for workspace package, package/lockfile, shared harness, split-config, renamed, or deleted changes, public extension-contract changes, tests with special shard setup, partially resolved or empty targets, oversized path or target plans, and planner errors. Targeted plans always retain the full built-artifact boundary gate because its repository scanners cannot be derived from imports. main pushes run the same full compact suite: pending intermediate push events can be coalesced, so the newest surviving run must validate the complete integration tree rather than only its final single-push diff. Manual dispatches and release gates retain the full named per-shard matrix.
• The full Node matrix admits the consistently slow serial tooling, auto-reply command shards, and broad core-fast cache writer first. This keeps the 28-job cap while preventing critical-path work and the next run's transform seed from slipping into a later wave.
• Broad browser, QA, media, and miscellaneous plugin tests use their dedicated Vitest configs instead of the shared plugin catch-all. Include-pattern shards record timing entries using the CI shard name, so .artifacts/vitest-shard-timings.json can distinguish a whole config from a filtered shard.
• Linux Node shard jobs persist Vitest's experimental filesystem module cache through the upstream Actions cache API, which Blacksmith transparently accelerates on its runners. Every CI shard is restore-only and unpacks the protected seed into its own runner-local root; the shard wrapper then gives concurrent Vitest processes separate live subdirectories. Only the non-cancelling daily or explicitly dispatched warmer saves a new immutable archive, so pull requests cannot publish transforms or mint per-PR cache families. A transform-input fingerprint clears incompatible lockfile, package, tsconfig, and Vitest-config generations. The protected writer scans and prunes its restored cache to 75% after it exceeds 2 GiB. Vitest hashes module id, source content, environment, and resolved transform config, so ordinary partial source changes keep unchanged entries warm while changed modules miss safely. Coarse restore prefixes bridge workflow runs; normal Actions cache LRU and inactivity eviction bound old immutable archives.
• Trusted Linux Node jobs also bind the pnpm store and node_modules from one protected dependency disk per supported Node line. Package manifests, install settings, runner platform, and the exact Node patch stay out of the disk key; an exact runtime and install-input fingerprint decides whether a job reuses the tree or reinstalls and refreshes the same disk. Manifests are canonicalized before hashing. The audited direct root hooks retain only pnpm's install lifecycle scripts, so formatting and ordinary test/build script edits keep the warm dependency tree; unaudited lifecycle-hook drift fails closed until its source inputs join the fingerprint contract. Dependency, package-manager, hook-source, and lockfile changes always invalidate the snapshot. A matching fingerprint is necessary but not sufficient: setup also checks the importer archive and manifest checksums, then verifies registry-backed lockfile dependencies retained by postinstall against the package manifests Node resolves from their importers. Missing or stale importer content falls back to a fresh install instead of serving the root hoist. A pull request whose read-only snapshot is unusable detaches the workspace bind and installs into runner-local storage, avoiding slow writes to a clone it cannot publish. Sticky cold installs disable pnpm's inner fetch retries and make up to three bounded full-install attempts from the progressively warmed store; a timeout remains a failure. After a content-validated restore or frozen-lockfile install, setup disables pnpm's redundant pre-run dependency check: the repository intentionally prunes plugin-local node_modules, which pnpm otherwise treats as stale and repairs through unsafe concurrent implicit installs during shard fanout. Canonical main preflight is the sole writer and measures the store on every refresh, running pnpm store prune only after retired package versions push it above 8 GiB. Blacksmith snapshot publication is asynchronous even after a writer job completes, so the first run after a fresh key or fingerprint can remain cold; later content-validated exact-marker restores are the rollout proof. Required CI jobs and pull requests get disposable clones, so dependency changes do not create new disks, competing snapshots, or a cache lock that can cancel builds.
• Node shard and build-artifact jobs also restore Node's portable on-disk compile cache through immutable Actions caches. Independent test and build namespaces prevent their writers from replacing each other's archives: the scheduled test warmer owns the protected test seed, while build-artifacts may publish at most one protected build archive per UTC day from trusted main pushes. PR and ordinary test jobs only read protected snapshots, so feature-branch bytecode never enters the shared seed and PR traffic creates no cache archives. This reuses V8 bytecode for Node-loaded orchestration, build tooling, and external dependencies across different checkout paths, including when only part of the source graph changes. Vitest child processes disable an inherited compile cache because coverage can be enabled inside dynamic configs and V8 coverage can lose source-position precision when scripts are deserialized from bytecode.
• The build-artifact job also persists content-fingerprinted build-all step outputs. CI's self-built plugin SDK declarations hash the complete repository-owned TypeScript/JSON source graph, exclude installed and generated directories, and restore both flat declarations and package bridges after tsdown clears dist. Documentation, workflow, plugin, and other changes outside that graph can reuse the declaration snapshot; source changes rebuild it before the export gate runs.
• Full declaration builds split tsdown into AI, workspace-package, and unified groups. Each group caches declarations only, then still rebuilds runtime JavaScript before restoring those declarations. Core or plugin changes therefore invalidate only the large unified graph, while workspace-package changes conservatively invalidate every dependent declaration group. Public full builds generally use an immutable Actions cache; coarse restore keys seed partial changes, per-group content fingerprints reject stale data, and GitHub's cache quota evicts old generations. The weekly Node 22 lane instead publishes a 14-day artifact after successful main runs and restores only artifacts whose immutable producer identity resolves to that workflow on main, avoiding quota churn without allowing PR code to write a shared cache. Private-QA declarations are never persisted in Actions caches because cache namespaces are not confidentiality boundaries.
• check-additional-* stripes the supplemental boundary guard list (scripts/run-additional-boundary-checks.mjs) into one prompt-heavy shard (check-additional-boundaries-a, which includes the Codex prompt snapshot drift check) and one combined shard for the remaining stripes (check-additional-boundaries-bcd), each running independent guards concurrently and printing per-check timings. Package-boundary compile/canary work stays together, and runtime topology architecture runs separately from the gateway watch coverage embedded in build-artifacts.
• On the 32-vCPU self-hosted build runner, Gateway watch, channel tests, and the core support-boundary shard start together inside build-artifacts after dist/ and dist-runtime/ are already built. GitHub-hosted fallback runs keep Gateway watch serial so low-core contention cannot consume its readiness deadline.

Once admitted, standard Linux CI can run up to 28 concurrent Node test jobs, with 12 available for the smaller fast/check lanes. Windows and Android are limited to two concurrent jobs each because those runner pools are smaller. Compact whole-configuration batches have a 120-minute batch timeout, and include-pattern groups share the same capped job budget.

Android CI executes both testPlayDebugUnitTest and testThirdPartyDebugUnitTest before building the Play debug APK. The third-party flavor does not have a separate source set or manifest. Its unit-test lane still compiles the flavor using the SMS and call-log BuildConfig flags, but avoids creating a duplicate debug APK packaging job on every Android-relevant push. Each active Gradle task has one protected sticky disk. PR jobs use disposable clones, while protected runs refresh content-addressed Gradle entries in place.

Blacksmith sticky-disk keys are intentionally scoped to supported runtime or task dimensions, never to PR number, commit, run, branch, or dependency hash. Runtime transform and compile caches use the Actions cache instead of sticky disks because immutable archives provide verifiable restore and save results and avoid failures from mutable snapshot promotion. After a sticky key-version migration, add only the exact obsolete key, architecture, and region identities to .github/retired-sticky-disks.json, dispatch Sticky Disk Cleanup from main with the same dimensions and confirmation, verify deletion, and then remove those entries. The workflow routes ARM identities to an ARM runner, rejects runner-region mismatches, uses Blacksmith's exact-key deletion action, and never deletes Docker builder caches or wildcard prefixes. Actions cache archives use standard LRU and inactivity eviction.

The check-dependencies shard runs production Knip checks for dependencies, unused files, and unused exports. The unused-file guard fails when a PR adds a new unreviewed unused file or leaves a stale allowlist entry, while preserving intentional dynamic plugin, generated, build, live-test, and package bridge surfaces that Knip cannot resolve statically. The unused-export guard excludes test-support files and fails on every unused production export. Intentional dynamic consumers must be modeled in config/knip.config.ts. Historical targets run the export guard when they provide it and otherwise keep their older dead-code fallback.

ClawSweeper activity forwarding

.github/workflows/clawsweeper-dispatch.yml is the target-side bridge from OpenClaw repository activity into ClawSweeper. It does not check out or execute untrusted pull request code. The workflow creates a GitHub App token from CLAWSWEEPER_APP_PRIVATE_KEY, then dispatches compact repository_dispatch payloads to openclaw/clawsweeper.

The workflow has four lanes:

• clawsweeper_item for exact issue and pull request review requests;
• clawsweeper_comment for explicit ClawSweeper commands in issue comments;
• clawsweeper_commit_review for commit-level review requests on main pushes;
• github_activity for general GitHub activity that the ClawSweeper agent may inspect.

The github_activity lane forwards only normalized metadata: event type, action, actor, repository, item number, URL, title, state, and short excerpts for comments or reviews when present. It intentionally does not forward the full webhook body. The receiving workflow in openclaw/clawsweeper is .github/workflows/github-activity.yml, which posts the normalized event to the OpenClaw Gateway hook for the ClawSweeper agent.

General activity is observation, not delivery by default. The ClawSweeper agent receives the Discord target in its prompt and should post to #clawsweeper only when the event is surprising, actionable, risky, or operationally useful. Routine opens, edits, bot churn, duplicate webhook noise, and normal review traffic should result in NO_REPLY.

Treat GitHub titles, comments, bodies, review text, branch names, and commit messages as untrusted data throughout this path. They are input for summarization and triage, not instructions for the workflow or agent runtime.

Manual dispatches

Manual CI dispatches run the same job graph as normal CI but force every non-Android scoped lane on: Linux Node shards, bundled-plugin shards, plugin and channel contract shards, Node 22 compatibility, check-*, check-additional-*, built-artifact smoke checks, docs checks, Python skills, Windows, macOS, iOS build, and Control UI and native app i18n. Automatic source PRs verify native extraction inventory and Android and Apple localization safety without requiring translated or platform-generated output in the same PR. The serialized Native App Locale Refresh workflow rebuilds those artifacts in one isolated PR and enables exact-head auto-merge after required checks pass. Full native parity remains blocking for generated-artifact PRs, manual CI, Full Release Validation, and release prep. Control UI locale parity remains advisory on automatic PR and main runs and blocking on manual and release CI. Standalone manual CI dispatches run Android only with include_android=true (the release_gate input also forces Android). The full release umbrella enables Android by passing include_android=true. Plugin prerelease static checks, the release-only agentic-plugins shard, the full extension batch sweep, and plugin prerelease Docker lanes are excluded from CI. The Docker prerelease suite runs only when Full Release Validation dispatches the separate Plugin Prerelease workflow with the release-validation gate enabled.

PR max-lines checks derive the baseline from the checked-out synthetic merge tree and verify its head parent against the event head. Manual runs use a unique concurrency group so a release-candidate full suite is not cancelled by another push or PR run on the same ref. The optional target_ref input lets a trusted caller run that graph against a branch, tag, or full commit SHA while using the workflow file from the selected dispatch ref. The max-lines baseline is compared with the target's merge base against the default-branch head resolved for that run. The release_gate input is an exact-SHA maintainer fallback for capacity-stalled PR CI. It requires target_ref to be a full commit SHA that matches the dispatched branch head and pull_request_number to identify the open PR whose merge tree is validated.

gh workflow run ci.yml --ref release/YYYY.M.PATCH
gh workflow run ci.yml --ref main -f target_ref=<branch-or-sha> -f include_android=true
gh workflow run full-release-validation.yml --ref main -f ref=<branch-or-sha>

Gateway extended-stable runs npm preflight, Full Release Validation, and plugin npm release from extended-stable/YYYY.M.33. Core publish consumes those three run IDs plus the validation attempt. release-ci/* evidence is invalid because publish binds every run to the canonical branch and release SHA. The tag publishes Gateway images and only the extended-stable* aliases. The path skips the regular orchestrator and its ClawHub, native-app, GitHub Release, website, and private dist-tag surfaces. See Monthly Gateway extended-stable publication for commands and recovery.

Runners

Runner registration budget

According to OpenClaw's current GitHub runner-registration bucket, 10,000 self-hosted runner registrations are allowed every 5 minutes in ghx api rate_limit. Before each tuning pass, verify actions_runner_registration because GitHub may adjust this bucket. All Blacksmith runner registrations within the openclaw organization share this limit, so deploying another Blacksmith installation does not create a separate bucket.

For burst control, treat Blacksmith labels as the constrained resource. Jobs that only route, notify, summarize, select shards, or execute brief CodeQL scans should remain on GitHub-hosted runners unless they have demonstrated Blacksmith-specific requirements. Any new Blacksmith matrix, larger max-parallel, or high-frequency workflow must document its worst-case registration count and keep the organization-level target below roughly 60% of the live bucket. Given the current 10,000-registration bucket, this translates to a 6,000-registration operating target, leaving buffer for concurrent repositories, retries, and burst overlap.

The changed-target PR plan reduces the common Node test burst from 14 Blacksmith registrations to one. Broad-risk PRs retain the 14-registration compact fallback, so the worst case does not increase.

Canonical-repo CI continues using Blacksmith as the default runner path for normal push and pull-request runs. workflow_dispatch and non-canonical repository runs rely on GitHub-hosted runners, but normal canonical runs do not currently check Blacksmith queue health or automatically fall back to GitHub-hosted labels when Blacksmith is unavailable.

Surface ratchets

Two shrink-only budgets protect the configuration surface. Both fail CI on growth until the budget file is consciously updated in the same PR, and both require a ratchet-down when cleanup reduces the actual count.

• config/env-var-count-budget.txt limits the number of distinct OPENCLAW_* names in production source under src/, packages/, and extensions/ (tests and QA Lab excluded). This is checked by node scripts/check-env-var-count.mjs. When removing env vars, lower the number in the same PR. Adding one is a config-surface decision, so justify it in the PR body.
• docs/.generated/config-baseline.counts.json limits the per-kind (core/channel/plugin) openclaw.json schema entry counts. This is checked by pnpm config:docs:check; regenerate with pnpm config:docs:gen after any schema change.

Local equivalents

pnpm changed:lanes                            # inspect the local changed-lane classifier for origin/main...HEAD
pnpm check:changed                            # smart local check gate: changed formatting/typecheck/lint/guards by boundary lane
pnpm check                                    # fast local gate: prod tsgo + sharded lint + parallel fast guards
pnpm check:test-types
pnpm check:timed                              # same gate with per-stage timings
pnpm build:strict-smoke
pnpm check:architecture
pnpm test:gateway:watch-regression
OPENCLAW_TUI_PTY_INCLUDE_LOCAL=1 node scripts/run-vitest.mjs run --config test/vitest/vitest.tui-pty.config.ts
pnpm test                                     # vitest tests
pnpm test:changed                             # cheap smart changed Vitest targets
pnpm test:ui                                  # Control UI unit/browser suite
pnpm ui:i18n:check                            # generated Control UI locale parity (release gate)
pnpm native:i18n:baseline                     # update source-owned native extraction inventory
pnpm native:i18n:verify                       # source inventory + Android/Apple localization safety
pnpm native:i18n:check                        # strict translated/platform-generated parity (release gate)
pnpm test:channels
pnpm test:contracts:channels
pnpm check:docs                               # docs format + lint + broken links
pnpm build                                    # build dist when CI artifact/smoke checks matter
pnpm ios:build                                # generate and build the iOS app project
pnpm ci:timings                               # summarize the latest origin/main push CI run
pnpm ci:timings:recent                        # compare recent successful main CI runs
node scripts/ci-run-timings.mjs <run-id>      # summarize wall time, queue time, and slowest jobs
node scripts/ci-run-timings.mjs --latest-main # ignore issue/comment noise and choose origin/main push CI
node scripts/ci-run-timings.mjs --recent 10   # compare recent successful main CI runs
pnpm test:perf:groups --full-suite --allow-failures --output .artifacts/test-perf/baseline-before.json
pnpm test:perf:groups:compare .artifacts/test-perf/baseline-before.json .artifacts/test-perf/after-agent.json
pnpm test:startup:memory
pnpm test:extensions:memory -- --json .artifacts/openclaw-performance/source/mock-provider/extension-memory.json
pnpm perf:kova:summary --report .artifacts/kova/reports/mock-provider/report.json --output .artifacts/kova/summary.md

OpenClaw Performance

OpenClaw Performance is the product/runtime performance workflow. It runs daily on main and supports manual dispatch:

gh workflow run openclaw-performance.yml --ref main -f profile=diagnostic -f repeat=3
gh workflow run openclaw-performance.yml --ref main -f profile=smoke -f repeat=1 -f deep_profile=true -f live_openai_candidate=true
gh workflow run openclaw-performance.yml --ref main -f target_ref=v2026.5.2 -f profile=diagnostic -f repeat=3

Manual dispatch typically benchmarks the workflow ref. Set target_ref to benchmark a release tag or another branch with the current workflow implementation. Published report paths and latest pointers are keyed by the tested ref, and each index.md records the tested ref/SHA, workflow ref/SHA, Kova ref, profile, lane auth mode, model, repeat count, and scenario filters.

The workflow installs OCM from a pinned release and Kova from openclaw/Kova at the pinned kova_ref input, then executes three lanes:

• mock-provider: Run Kova diagnostic scenarios against a local build runtime using deterministic fake OpenAI-compatible authentication.
• mock-deep-profile: Profile CPU, heap, and trace for startup, gateway, and agent-turn hotspots. Executes on a schedule or on dispatch with deep_profile=true.
• live-openai-candidate: Perform a real OpenAI openai/gpt-5.6-luna agent turn, skipped when OPENAI_API_KEY is unavailable. Runs on a schedule or on dispatch with live_openai_candidate=true.

After the Kova pass, the mock-provider lane also runs OpenClaw-native source probes: gateway boot timing and memory across default, skipped-channel, internal-hook, and fifty-plugin startup cases; bundled plugin import RSS, repeated mock-OpenAI channel-chat-baseline hello loops, CLI startup commands against the booted gateway, and the SQLite state smoke performance probe. When the previous published mock-provider source report exists for the tested ref, the source summary compares current RSS and heap values against that baseline and flags large RSS increases as watch. The source probe Markdown summary resides at source/index.md in the report bundle, with raw JSON alongside it.

Every lane uploads its full GitHub artifact, which includes CPU, heap, trace, and compressed diagnostic bundles. A separate publisher job downloads and validates those artifacts, then creates a short-lived ClawSweeper GitHub App token scoped only to openclaw/clawgrit-reports contents and passes it solely to the Git push step. It commits report.json, report.md, index.md, source-probe artifacts, and bundle metadata and checksums under openclaw-performance/<tested-ref>/<run-id>-<attempt>/<lane>/; the complete diagnostic archive remains in the linked Actions artifact. The publisher rejects any report file exceeding 50 MB before attempting a push. The current tested-ref pointer is openclaw-performance/<tested-ref>/latest-<lane>.json. Scheduled runs and profile=release dispatches fail if app-token creation or report publication fails. Manual non-release dispatches keep publication advisory and retain the GitHub artifacts when authentication or publishing fails. The previous source baseline is fetched anonymously from the public reports repository, so a successful baseline fetch does not confirm publisher authentication.

Full Release Validation

Full Release Validation is the manual umbrella workflow for "run everything before release." It accepts a branch, tag, or full commit SHA, dispatches the manual CI workflow with that target (including Android), dispatches Plugin Prerelease for release-only plugin, package, static, and Docker proof, dispatches OpenClaw Performance against the target SHA, and dispatches OpenClaw Release Checks for install smoke, package acceptance, cross-OS package checks, QA Lab parity, Matrix, Telegram, and gated Discord, WhatsApp, and Slack lanes (advisory maturity scorecard rendering is opt-in via run_maturity_scorecard). Stable and full profiles always include exhaustive live, E2E, and Docker release-path soak coverage; the beta profile can opt in with run_release_soak=true. The canonical package Telegram E2E runs inside Package Acceptance, so a full candidate does not start a duplicate live poller. After publishing, pass release_package_spec to reuse the shipped npm package across release checks, Package Acceptance, Docker, cross-OS, and Telegram without rebuilding. Use npm_telegram_package_spec only for a focused published-package Telegram rerun. The Codex plugin live package lane uses the same selected state by default: published release_package_spec=openclaw@<tag> derives codex_plugin_spec=npm:@openclaw/codex@<tag>, while SHA and artifact runs pack extensions/codex from the selected ref. Set codex_plugin_spec explicitly for custom plugin sources such as npm:, npm-pack:, or git: specs. Its live agent proof sends visible progress, continues through randomized workspace reads and an exact artifact write, then sends completion.

See Full release validation for the stage matrix, exact workflow job names, profile differences, artifacts, and focused rerun handles.

OpenClaw Release Publish is the manual mutating release workflow. Dispatch regular beta and stable publishes from trusted main after the release tag exists and after the OpenClaw npm preflight has succeeded (the preflight runs pnpm plugins:sync:check among its checks). The tag still selects the exact release commit, including a commit on release/YYYY.M.PATCH; Tideclaw alpha publishes keep using their matching alpha branch. It requires the saved preflight_run_id and a successful full_release_validation_run_id and its exact full_release_validation_run_attempt, dispatches Plugin NPM Release for all publishable plugin packages, dispatches Plugin ClawHub Release for the same release SHA, and only then dispatches OpenClaw NPM Release. Stable publish also requires an exact windows_node_tag; the workflow verifies the Windows source release and compares its x64 and ARM64 installers with the candidate-approved windows_node_installer_digests input before any publish child, then promotes and verifies those same pinned installer digests plus the exact companion asset and checksum contract before publishing the GitHub release draft. Focused plugin-only repairs use plugin_publish_scope=selected with a nonempty package list. Plugin-only all-publishable runs require the same immutable npm preflight and Full Release Validation evidence as a core publish.

gh workflow run openclaw-release-publish.yml \
  --ref main \
  -f tag=vYYYY.M.PATCH-beta.N \
  -f preflight_run_id=<successful-openclaw-npm-preflight-run-id> \
  -f full_release_validation_run_id=<successful-full-release-validation-run-id> \
  -f full_release_validation_run_attempt=<successful-full-release-validation-run-attempt> \
  -f npm_dist_tag=beta

For pinned commit proof on a fast-moving branch, use the helper instead of gh workflow run ... --ref main -f ref=<sha>:

pnpm ci:full-release --sha <full-sha>

GitHub workflow dispatch refs must be branches or tags, not raw commit SHAs. The helper pushes a temporary release-ci/<sha>-... branch at a trusted main workflow SHA, passes the requested target SHA through the workflow ref input, reuses strict exact-target evidence when available, verifies every child workflow headSha matches the trusted workflow SHA, and deletes the temporary branch when the run completes. Pass -f reuse_evidence=false to force fresh validation. The umbrella verifier also fails if any child workflow ran at a different workflow SHA.

release_profile determines the live/provider breadth used in release checks. Manual release workflows default to stable; only use full when you deliberately want the full advisory provider/media matrix. Stable and full release checks always run the exhaustive live/E2E and Docker release-path soak; the beta profile can opt in with run_release_soak=true.

• beta preserves the fastest OpenAI/core release-critical lanes.
• stable adds the stable provider/backend set.
• full runs the broad advisory provider/media matrix.

The umbrella records the dispatched child run IDs, and the final Verify full validation job rechecks current child run conclusions and appends slowest-job tables for each child run. If a child workflow is rerun and turns green, rerun only the parent verifier job to refresh the umbrella result and timing summary.

For recovery, both Full Release Validation and OpenClaw Release Checks accept rerun_group. Use all for a release candidate, ci for only the normal full CI child, plugin-prerelease for only the plugin prerelease child, performance for only the OpenClaw Performance child, release-checks for every release child, or a narrower group: install-smoke, cross-os, live-e2e, package, qa, qa-parity, qa-live, or npm-telegram on the umbrella. This keeps a failed release box rerun bounded after a focused fix. For one failed cross-OS lane, combine rerun_group=cross-os with cross_os_suite_filter, for example windows/packaged-upgrade; long cross-OS commands emit heartbeat lines and packaged-upgrade summaries include per-phase timings. Selected Matrix and Telegram QA lanes block normal release validation, as does the core runtime-pair tool coverage gate. QA parity, runtime parity, and the gated Discord, WhatsApp, and Slack live lanes are advisory.

OpenClaw Release Checks uses the trusted workflow ref to resolve the selected ref once into a release-package-under-test tarball, then passes that artifact to cross-OS checks and Package Acceptance, plus the live/E2E release-path Docker workflow when soak coverage runs. That keeps the package bytes consistent across release boxes and avoids repacking the same candidate in multiple child jobs. For the Codex npm-plugin live lane, release checks either pass a matching published plugin spec derived from release_package_spec, pass the operator-supplied codex_plugin_spec, or leave the input blank so the Docker script packs the selected checkout's Codex plugin.

Duplicate Full Release Validation runs for ref=main and rerun_group=all supersede the older umbrella. The parent monitor cancels any child workflow it has already dispatched when the parent is cancelled, so newer main validation does not sit behind a stale two-hour release-check run. Release branch/tag validation and focused rerun groups keep cancel-in-progress: false.

Live and E2E shards

The release live/E2E child keeps broad native pnpm test:live coverage, but it runs it as named shards through scripts/test-live-shard.mjs instead of one serial job:

• native-live-src-agents and native-live-src-agents-zai-coding
• native-live-src-gateway-core
• provider-filtered native-live-src-gateway-profiles jobs
• native-live-src-gateway-backends
• native-live-src-infra
• native-live-test
• native-live-extensions-a-k
• native-live-extensions-l-n
• native-live-extensions-moonshot
• native-live-extensions-openai
• native-live-extensions-o-z-other
• native-live-extensions-xai
• split media audio/video shards and provider-filtered music shards

That keeps the same file coverage while making slow live provider failures easier to rerun and diagnose. The aggregate native-live-src-gateway, native-live-extensions-o-z, native-live-extensions-media, and native-live-extensions-media-music shard names remain valid for manual one-shot reruns.

The native live media shards run in ghcr.io/openclaw/openclaw-live-media-runner:ubuntu-24.04, built by the Live Media Runner Image workflow. That image preinstalls ffmpeg and ffprobe; media jobs only verify the binaries before setup. Keep Docker-backed live suites on normal Blacksmith runners, container jobs are the wrong place to launch nested Docker tests.

Docker backed live model and backend shards each rely on a single shared ghcr.io/openclaw/openclaw-live-test:<sha>-<extensions> image per selected commit. The live release pipeline builds and pushes that image once; afterward, the Docker live model, provider sharded gateway, CLI backend, ACP bind, and Codex harness shards all run with OPENCLAW_SKIP_DOCKER_BUILD=1. Gateway Docker shards have explicit script level timeout caps set below the workflow job timeout, so a stuck container or a failed cleanup path terminates quickly rather than eating up the entire release check budget. If those shards rebuild the full source Docker target on their own, the release run is misconfigured and will waste wall clock time on duplicate image builds.

Package Acceptance

Use Package Acceptance when the question is "does this installable OpenClaw package work as a product?" This differs from normal CI: normal CI validates the source tree, whereas package acceptance validates a single tarball through the same Docker E2E harness that users run after install or update.

Jobs

1. resolve_package checks out workflow_ref, resolves one package candidate, writes .artifacts/docker-e2e-package/openclaw-current.tgz, writes .artifacts/docker-e2e-package/package-candidate.json, uploads both as the package-under-test artifact, and prints the source, workflow ref, package ref, version, SHA-256, and profile in the GitHub step summary.
2. package_integrity downloads the package-under-test artifact and enforces the public package tarball contract with scripts/check-openclaw-package-tarball.mjs.
3. docker_acceptance calls openclaw-live-and-e2e-checks-reusable.yml with the resolved package source SHA (falling back to workflow_ref) and package_artifact_name=package-under-test. The reusable workflow downloads that artifact, validates the tarball inventory, prepares package digest Docker images when needed, and runs the selected Docker lanes against that package instead of packing the workflow checkout. When a profile selects multiple targeted docker_lanes, the reusable workflow prepares the package and shared images once, then fans those lanes out as parallel targeted Docker jobs with unique artifacts.
4. package_telegram optionally calls NPM Telegram Beta E2E. It runs when telegram_mode is not none and installs the same package-under-test artifact when Package Acceptance resolved one; standalone Telegram dispatch can still install a published npm spec.
5. summary fails the workflow if package resolution, integrity, Docker acceptance, or the optional Telegram lane failed. The advisory input downgrades acceptance failures to warnings for advisory callers.

Candidate sources

• source=npm accepts only openclaw@extended-stable, openclaw@beta, openclaw@latest, or an exact OpenClaw release version such as openclaw@2026.4.27-beta.2. Use this for published extended stable, prerelease, or stable acceptance.
• source=ref packs a trusted package_ref branch, tag, or full commit SHA. The resolver fetches OpenClaw branches or tags, verifies the selected commit is reachable from repository branch history or a release tag, installs dependencies in a detached worktree, and packs it with scripts/package-openclaw-for-docker.mjs.
• source=url downloads a public HTTPS .tgz; package_sha256 is required. This path rejects URL credentials, non default HTTPS ports, private, internal, or special use hostnames or resolved IPs, and redirects outside the same public safety policy.
• source=trusted-url downloads an HTTPS .tgz from a named trusted source policy in .github/package-trusted-sources.json; package_sha256 and trusted_source_id are required. Use this only for maintainer owned enterprise mirrors or private package repositories that need configured hosts, ports, path prefixes, redirect hosts, or private network resolution. If the policy declares bearer auth, the workflow uses the fixed OPENCLAW_TRUSTED_PACKAGE_TOKEN secret; URL embedded credentials are still rejected.
• source=artifact downloads one .tgz from artifact_run_id and artifact_name; package_sha256 is optional but should be supplied for externally shared artifacts.

Keep workflow_ref and package_ref separate. workflow_ref is the trusted workflow or harness code that runs the test. package_ref is the source commit that gets packed when source=ref. This allows the current test harness to validate older trusted source commits without running old workflow logic.

Suite profiles

• smoke, npm-onboard-channel-agent, gateway-network, config-reload
• package, npm-onboard-channel-agent, doctor-switch, update-channel-switch, skill-install, update-corrupt-plugin, upgrade-survivor, published-upgrade-survivor, root-managed-vps-upgrade, update-restart-auth, plugins-offline, plugin-update
• product, the package set with live plugins coverage in place of plugins-offline, plus mcp-channels, cron-mcp-cleanup, openai-web-search-minimal, openwebui
• full, full Docker release-path chunks with OpenWebUI
• custom, exact docker_lanes; mandatory when suite_profile=custom

Offline plugin coverage is used by the package profile, so published-package validation does not depend on live ClawHub availability. The optional Telegram lane reuses the package-under-test artifact within NPM Telegram Beta E2E, while the published npm spec path remains for standalone dispatches.

Refer to Testing updates and plugins for the dedicated policy on update and plugin testing, including local commands, Docker lanes, Package Acceptance inputs, release defaults, and failure triage.

Package Acceptance is invoked by release checks with source=artifact, the prepared release package artifact, suite_profile=custom, docker_lanes='doctor-switch update-channel-switch skill-install update-corrupt-plugin upgrade-survivor published-upgrade-survivor root-managed-vps-upgrade update-restart-auth plugins-offline plugin-update plugin-binding-command-escape', and telegram_mode=mock-openai. This ensures that package migration, updates, live ClawHub skill installation, stale-plugin-dependency cleanup, configured-plugin install repair, offline plugins, plugin updates, and Telegram proof all operate on the same resolved package tarball. To run the same matrix against the shipped npm package without a rebuild after publishing a beta, set release_package_spec on Full Release Validation or OpenClaw Release Checks; set package_acceptance_package_spec only when Package Acceptance requires a different package than the rest of release validation. Cross-OS release checks still cover OS-specific onboarding, installer, and platform behavior; package and update product validation should begin with Package Acceptance.

One published package baseline per run is validated by the published-upgrade-survivor Docker lane in the blocking release path. Within Package Acceptance, the resolved package-under-test tarball is always the candidate, and published_upgrade_survivor_baseline selects the fallback published baseline (defaulting to openclaw@latest); rerun commands for failed lanes preserve that baseline. When Full Release Validation uses run_release_soak=true or release_profile=full, it sets published_upgrade_survivor_baselines='last-stable-4 2026.4.23 2026.5.2 2026.4.15' and published_upgrade_survivor_scenarios=reported-issues to expand across the four most recent stable npm releases, pinned plugin-compatibility boundary releases, and issue-shaped fixtures for Feishu config, preserved bootstrap/persona files, configured OpenClaw plugin installs, tilde log paths, and stale legacy plugin dependency roots. Survivor selections for multi-baseline published-upgrade are sharded by baseline into separate targeted Docker runner jobs. The separate Update Migration workflow uses the update-migration Docker lane with all-since-2026.4.23 baselines and plugin-deps-cleanup scenarios when the goal is exhaustive published update cleanup rather than normal Full Release CI breadth. Local aggregate runs can pass exact package specs with OPENCLAW_UPGRADE_SURVIVOR_BASELINE_SPECS, keep a single lane with OPENCLAW_UPGRADE_SURVIVOR_BASELINE_SPEC such as openclaw@2026.4.15, or set OPENCLAW_UPGRADE_SURVIVOR_SCENARIOS for the scenario matrix. A baked openclaw config set command recipe configures the baseline in the published lane, records recipe steps in summary.json, and probes /healthz, /readyz, along with RPC status after Gateway start. The Windows packaged and installer fresh lanes also verify that an installed package can import a browser-control override from a raw absolute Windows path. When set, the OpenAI cross-OS agent-turn smoke defaults to OPENCLAW_CROSS_OS_OPENAI_MODEL; otherwise it uses openai/gpt-5.6-luna, so the install and gateway proof employs the lower-cost GPT-5.6 test tier.

Legacy compatibility windows

Package Acceptance maintains bounded backward compatibility windows for packages already published. The compatibility path is available to packages through 2026.4.25, which includes 2026.4.25-beta.*:

• known private QA entries in dist/postinstall-inventory.json can reference files omitted from the tarball;
• doctor-switch may bypass the gateway install --wrapper persistence subcase when the package does not expose that flag;
• update-channel-switch may remove missing pnpm patchedDependencies from the tarball-derived fake git fixture and log absent persisted update.channel;
• plugin smokes may read legacy install-record paths or tolerate missing marketplace install-record persistence;
• plugin-update may permit config metadata migration while requiring the install record and no-reinstall behavior to remain unchanged.

The published 2026.4.26 package may also issue warnings for local build metadata stamp files that were already distributed, and packages through 2026.5.20 may warn instead of failing when npm-shrinkwrap.json is absent. Newer packages must adhere to the modern contracts; the same conditions produce failures instead of warnings or skips.

Examples

# Validate the current beta package with product-level coverage.
gh workflow run package-acceptance.yml \
  --ref main \
  -f workflow_ref=main \
  -f source=npm \
  -f package_spec=openclaw@beta \
  -f suite_profile=product \
  -f telegram_mode=mock-openai

# Validate the published extended-stable package with package coverage.
gh workflow run package-acceptance.yml \
  --ref main \
  -f workflow_ref=main \
  -f source=npm \
  -f package_spec=openclaw@extended-stable \
  -f suite_profile=package \
  -f telegram_mode=mock-openai

# Pack and validate a release branch with the current harness.
gh workflow run package-acceptance.yml \
  --ref main \
  -f workflow_ref=main \
  -f source=ref \
  -f package_ref=release/YYYY.M.PATCH \
  -f suite_profile=package \
  -f telegram_mode=mock-openai

# Validate a tarball URL. SHA-256 is mandatory for source=url.
gh workflow run package-acceptance.yml \
  --ref main \
  -f workflow_ref=main \
  -f source=url \
  -f package_url=https://example.com/openclaw-current.tgz \
  -f package_sha256=<64-char-sha256> \
  -f suite_profile=smoke

# Validate a tarball from a named trusted private mirror policy.
gh workflow run package-acceptance.yml \
  --ref main \
  -f workflow_ref=main \
  -f source=trusted-url \
  -f trusted_source_id=enterprise-artifactory \
  -f package_url=https://packages.example.internal:8443/artifactory/openclaw/openclaw-current.tgz \
  -f package_sha256=<64-char-sha256> \
  -f suite_profile=smoke

# Reuse a tarball uploaded by another Actions run.
gh workflow run package-acceptance.yml \
  --ref main \
  -f workflow_ref=main \
  -f source=artifact \
  -f artifact_run_id=<run-id> \
  -f artifact_name=package-under-test \
  -f suite_profile=custom \
  -f docker_lanes='install-e2e plugin-update'

When troubleshooting a failed package acceptance run, begin with the resolve_package summary to verify the package source, version, and SHA-256. Next, examine the docker_acceptance child run and its Docker artifacts: .artifacts/docker-tests/**/summary.json, failures.json, lane logs, phase timings, and rerun commands. Prefer rerunning the specific failed package profile or exact Docker lanes rather than rerunning the entire release validation.

Install smoke

The Install Smoke workflow no longer executes on pull requests or main pushes. Its nightly or manual wrapper and release validation both invoke the read-only install-smoke-reusable.yml core, and every run follows the full install-smoke path on GitHub-hosted runners:

• The root Dockerfile smoke image is built once per target SHA, tied to the workflow revision and producer attempt in an immutable artifact, then loaded by the CLI smoke, agents delete shared-workspace CLI smoke, container gateway-network E2E, and bundled matrix plugin build-arg smoke. The plugin smoke validates runtime dependency install mirroring and confirms the plugin loads without entry-escape diagnostics.
• QR package install and the installer or update Docker smokes (including Rocky Linux installer lanes and an update lane against a configurable update_baseline_version npm baseline) run as separate jobs so installer work does not wait behind the root image smokes.

The slow Bun global install image-provider smoke is separately gated by run_bun_global_install_smoke. It runs on the nightly schedule, is enabled by default for workflow calls from release checks, and manual Install Smoke dispatches can opt into it. Normal PR CI still runs the fast Bun launcher regression lane for Node-relevant changes. QR and installer Docker tests maintain their own install-focused Dockerfiles.

Local Docker E2E

pnpm test:docker:all prebuilds a single shared live-test image, packs OpenClaw once as an npm tarball, and builds two shared scripts/e2e/Dockerfile images:

• a bare Node or Git runner for installer, update, and plugin-dependency lanes;
• a functional image that installs the same tarball into /app for normal functionality lanes.

Docker lane definitions reside in scripts/lib/docker-e2e-scenarios.mjs, planner logic lives in scripts/lib/docker-e2e-plan.mjs, and the runner only executes the selected plan. The scheduler selects the image per lane using OPENCLAW_DOCKER_E2E_BARE_IMAGE and OPENCLAW_DOCKER_E2E_FUNCTIONAL_IMAGE, then runs lanes with OPENCLAW_SKIP_DOCKER_BUILD=1.

Tunables

A lane exceeding its effective cap can still start from an empty pool, then runs alone until it releases capacity. The local aggregate preflights Docker, removes stale OpenClaw E2E containers, emits active-lane status, persists lane timings for longest-first ordering, and stops scheduling new pooled lanes after the first failure by default.

Reusable live/E2E workflow

The reusable live or E2E workflow asks scripts/test-docker-all.mjs --plan-json which package, image kind, live image, lane, and credential coverage is required. scripts/docker-e2e.mjs then converts that plan into GitHub outputs and summaries. It either packs OpenClaw through scripts/package-openclaw-for-docker.mjs, downloads a current-run package artifact, or downloads a package artifact from package_artifact_run_id, then validates the tarball inventory. The default no-push-artifact path builds package-digest-tagged bare or functional images through Blacksmith's Docker layer cache, packs the exact image bytes into an immutable workflow artifact, and has each consumer verify and load that artifact. existing-only instead requires explicit docker_e2e_bare_image or docker_e2e_functional_image GHCR refs and never builds or pushes. Those registry pulls use a bounded 180-second per-attempt timeout so a stuck stream retries quickly instead of consuming most of the CI critical path. After successful scheduled validation, openclaw-scheduled-live-checks.yml passes the immutable tested-image manifest to the separate package-write publisher; read-only release and prerelease callers never traverse that writer.

Release-path chunks

Release Docker coverage uses smaller chunked jobs with OPENCLAW_SKIP_DOCKER_BUILD=1, so each chunk validates and loads only the artifact-backed image type it requires (or pulls it under explicit existing-only reuse) and runs multiple lanes through the same weighted scheduler:

• OPENCLAW_DOCKER_ALL_PROFILE=release-path
• OPENCLAW_DOCKER_ALL_CHUNK=core | package-update-openai | package-update-anthropic | package-update-core | plugins-runtime-plugins | plugins-runtime-services | plugins-runtime-install-a..h | openwebui

Current release Docker chunks are core, package-update-openai, package-update-anthropic, package-update-core, plugins-runtime-plugins, plugins-runtime-services, plugins-runtime-install-a through plugins-runtime-install-h, and openwebui. package-update-openai contains the live Codex plugin package lane, which installs the candidate OpenClaw package, installs the Codex plugin from codex_plugin_spec or a same-ref tarball with explicit Codex CLI install approval, runs Codex CLI preflight and same-session agent turns, then runs a zero-retry medium-thinking turn that sends progress, reads randomized workspace inputs, writes their exact artifact, and sends completion. plugins-runtime-core, plugins-runtime, and plugins-integrations remain aggregate plugin/runtime aliases. The install-e2e lane alias stays the aggregate manual rerun alias for both provider installer lanes.

OpenWebUI runs as a standalone openwebui chunk on a dedicated large-disk Blacksmith runner whenever stable or full release-path coverage requests it, even when the reusable workflow routes supported jobs to GitHub-hosted runners. Isolating the external image pull prevents the large image from competing with the shared package and plugin images in plugins-runtime-services; legacy aggregate plugin/runtime chunks still include OpenWebUI for compatible manual reruns. Bundled-channel update lanes retry once for transient npm network failures.

Each chunk uploads .artifacts/docker-tests/ with lane logs, timings, summary.json, failures.json, phase timings, scheduler plan JSON, slow-lane tables, and per-lane rerun commands. The workflow docker_lanes input runs selected lanes against images prepared for that run instead of the chunk jobs, which keeps failed-lane debugging bounded to one targeted Docker job; if a selected lane is a live Docker lane, the targeted job builds the live-test image locally for that rerun. The rerun helper validates the failure artifact's exact selected target SHA and manual dispatch repacks that ref, because the internal reusable-workflow package tuple is not part of the workflow_dispatch schema. Generated commands include prepared image inputs and shared_image_policy=existing-only only when those inputs are GHCR-backed; runner-local artifact tags are omitted so a fresh runner rebuilds them. An explicit target override drops recovered GHCR image refs unless the artifact proves they match the override. Artifact-generated workflow-definition refs are also omitted because full-release temporary branches are deleted; dispatch uses the repository default branch unless the operator explicitly overrides it.

pnpm test:docker:rerun <run-id>      # download Docker artifacts and print combined/per-lane targeted rerun commands
pnpm test:docker:timings <summary>   # slow-lane and phase critical-path summaries

The scheduled live/E2E workflow runs the full release-path Docker suite daily and, after it succeeds, invokes the explicit publisher for the exact tested image artifacts.

Plugin Prerelease

Plugin Prerelease is more expensive product/package coverage, so it is a separate workflow dispatched by Full Release Validation or by an explicit operator. Normal pull requests, main pushes, and standalone manual CI dispatches keep that suite off. It balances bundled plugin tests across eight extension workers; those extension shard jobs run up to two plugin config groups at a time with one Vitest worker per group and a larger Node heap so import-heavy plugin batches do not create extra CI jobs. The release-only Docker prerelease path (enabled by the full_release_validation input) batches targeted Docker lanes in groups of four to avoid reserving dozens of runners for one-to-three-minute jobs. The workflow also uploads an informational plugin-inspector-advisory artifact from @openclaw/plugin-inspector; inspector findings are triage input and do not change the blocking Plugin Prerelease gate.

QA Lab

QA Lab has dedicated CI lanes outside the main smart-scoped workflow. Agentic parity is nested under the broad QA and release harnesses, not a standalone PR workflow. Use Full Release Validation with rerun_group=qa-parity when parity should ride with a broad validation run.

• The QA-Lab - All Lanes workflow runs nightly on main and on manual dispatch; it fans out mock parity plus live Matrix, Telegram, Discord, WhatsApp, and Slack jobs. Live jobs use the qa-live-shared environment; Telegram, Discord, WhatsApp, and Slack use Convex leases, while Matrix provisions disposable local credentials.

Release checks run Matrix and Telegram live transport lanes with the deterministic mock provider and mock-qualified models (mock-openai/gpt-5.6-luna and mock-openai/gpt-5.6-luna-alt) so the channel contract is isolated from live model latency and normal provider-plugin startup. The live transport gateway disables memory search because QA parity covers memory behavior separately; provider connectivity is covered by the separate live model, native provider, and Docker provider suites.

Scheduled and release Matrix gates use the shared QA Lab suite host and live adapter with the release scenarios. The CLI default and manual workflow input remain all; manual all dispatches fan out the transport, media, e2ee-smoke, e2ee-deep, and e2ee-cli profiles so the 93-scenario proof stays within per-job timeouts. Focused manual dispatches select fast, release, or transport in one job.

OpenClaw Release Checks also runs the release-critical QA Lab lanes before release approval; its QA parity gate runs the candidate and baseline packs as parallel lane jobs, then downloads both artifacts into a small report job for the final parity comparison.

For normal PRs, follow scoped CI/check evidence instead of treating parity as a required status.

CodeQL

The CodeQL workflow is intentionally a narrow first-pass security scanner, not the full repository sweep. Daily, manual, main push, and non-draft pull request guard runs scan Actions workflow code plus the highest-risk JavaScript/TypeScript surfaces with high-confidence security queries filtered to high/critical security-severity.

The pull request guard remains minimal, activating only for modifications under .github/actions, .github/codeql, .github/workflows, packages, scripts, src, or process-owning bundled plugin runtime paths, and executes the same high-confidence security matrix used by the scheduled workflow. CodeQL analysis for Android and macOS is excluded from default PR checks.

Security categories

Platform-specific security shards

• CodeQL Android Critical Security, scheduled Android security shard. Builds the Android app manually for CodeQL on the smallest Blacksmith Linux runner accepted by workflow sanity. Uploads under /codeql-critical-security/android.
• CodeQL macOS Critical Security, weekly/manual macOS security shard. Builds the macOS app manually for CodeQL on Blacksmith macOS, filters dependency build results out of uploaded SARIF, and uploads under /codeql-critical-security/macos. Kept outside daily defaults because macOS build dominates runtime even when clean.

Critical Quality categories

CodeQL Critical Quality serves as the corresponding non-security shard. It executes only error-severity, non-security JavaScript and TypeScript quality queries across limited high-value surfaces on GitHub-hosted Linux runners, preventing quality scans from consuming Blacksmith runner-registration budget. Its pull request guard is deliberately smaller than the scheduled profile: non-draft PRs run only the matching shards for the surfaces they touch, from thirteen PR-routable shards, agent-runtime-boundary, channel-runtime-boundary, config-boundary, core-auth-secrets, gateway-runtime-boundary, mcp-process-runtime-boundary, memory-runtime-boundary, network-runtime-boundary, plugin-boundary, plugin-sdk-package-contract, plugin-sdk-reply-runtime, provider-runtime-boundary, and session-diagnostics-boundary. ui-control-plane and web-media-runtime-boundary are excluded from PR runs. CodeQL configuration and quality workflow changes trigger the full PR shard set (the network runtime shard activates based on its own CodeQL config files and network-owning source paths).

Manual dispatch accepts:

profile=all|agent-runtime-boundary|config-boundary|core-auth-secrets|channel-runtime-boundary|gateway-runtime-boundary|memory-runtime-boundary|mcp-process-runtime-boundary|network-runtime-boundary|plugin-boundary|plugin-sdk-package-contract|plugin-sdk-reply-runtime|provider-runtime-boundary|session-diagnostics-boundary

These narrow profiles serve as teaching and iteration hooks for running a single quality shard in isolation.

Quality remains separate from security so that quality results can be scheduled, measured, disabled, or expanded without masking security signals. Swift, Python, and bundled-plugin CodeQL expansion should be reintroduced as scoped or sharded follow-up work only after the narrow profiles achieve stable runtime and signal.

Maintenance workflows

Docs Agent

The Docs Agent workflow is an event-driven Codex maintenance lane that keeps existing documentation aligned with recently merged changes. It has no fixed schedule: a successful non-bot push CI run on main can trigger it, and manual dispatch can execute it directly. Workflow-run invocations are skipped when main has progressed or when another non-skipped Docs Agent run was created within the last hour. When it runs, it reviews the commit range from the previous non-skipped Docs Agent source SHA to the current main, so a single hourly run can cover all main changes accumulated since the last documentation pass.

Test Performance Agent

The Test Performance Agent workflow functions as an event-triggered Codex maintenance pipeline for slower tests. It lacks a fixed schedule: a successful non-bot push CI run on main may activate it, though it is skipped if another workflow-run invocation has already executed or is currently executing on that UTC day. Manual dispatch bypasses this daily activity restriction. This pipeline builds a complete grouped Vitest performance report, restricts Codex to making only small coverage-preserving test performance fixes rather than broad refactors, then re-executes the full-suite report and rejects any changes that lower the passing baseline test count. The grouped report captures per-configuration wall time and peak RSS on both Linux and macOS, so the before-and-after comparison reveals test memory differences alongside duration differences. If the baseline contains failing tests, Codex may fix only obvious failures, and the after-agent full-suite report must pass before any commit occurs. When main advances before the bot push is applied, the pipeline rebases the validated patch, re-runs pnpm check:changed, and retries the push; conflicting stale patches are skipped. It uses GitHub-hosted Ubuntu so the Codex action can maintain the same drop-sudo safety posture as the docs agent.

Duplicate PRs After Merge

The Duplicate PRs After Merge workflow is a manual maintainer pipeline for cleaning up duplicates after a merge. It defaults to dry-run mode and closes only explicitly listed PRs when apply=true. Before making changes on GitHub, it confirms the landed PR is merged and that each duplicate either shares a referenced issue or has overlapping changed hunks.

gh workflow run duplicate-after-merge.yml \
  -f landed_pr=70532 \
  -f duplicate_prs='70530,70592' \
  -f apply=true

Local check gates and changed routing

Config baseline count ratchet

pnpm config:docs:check rejects undocumented growth in configuration surfaces and corrupt or outdated count snapshots. When an intentional product change adds schema paths after review, run pnpm config:docs:gen, examine the core/channel/plugin count differences and generated SHA-256 files, and commit the deliberate baseline increase along with the schema, help, labels, migration, and tests. Do not manually edit the counts file to bypass the ratchet.

Configuration authors must also assign tiers to new leaves for Settings. Add advanced: false or advanced: true at the leaf, or place the key under an ancestor whose tier all descendants should inherit. Unclassified roots fail the schema quality test with copy-paste stubs; paths without an ancestor are treated as advanced by default. The curated common-leaf snapshot makes intentional tier changes visible during review.

Local changed-lane logic is located in scripts/changed-lanes.mjs and executed by scripts/check-changed.mjs. That local check gate enforces stricter architecture boundaries compared to the broader CI platform scope:

• core production changes run core prod and core test typecheck plus core lint/guards;
• core test-only changes run only core test typecheck plus core lint;
• extension production changes run extension prod and extension test typecheck plus extension lint;
• extension test-only changes run extension test typecheck plus extension lint;
• public Plugin SDK or plugin-contract changes expand to extension typecheck because extensions depend on those core contracts (Vitest extension sweeps remain explicit test work);
• release metadata-only version bumps run targeted version/config/root-dependency checks;
• unknown root/config changes fall back safely to all check lanes.

Local changed-test routing lives in scripts/test-projects.test-support.mjs and is intentionally less expensive than check:changed: direct test edits run themselves, source edits prefer explicit mappings, then sibling tests and import-graph dependents. Shared group-room delivery configuration is one of the explicit mappings: changes to the group visible-reply config, source reply delivery mode, or the message-tool system prompt route through the core reply tests plus Discord and Slack delivery regressions so a shared default change fails before the first PR push. Use OPENCLAW_TEST_CHANGED_BROAD=1 pnpm test:changed only when the change is harness-wide enough that the cheap mapped set is not a reliable proxy.

Testbox validation

Crabbox is the repository-owned remote-box wrapper for maintainer Linux verification. Agent sessions keep one or a few focused tests and cheap static checks local only for trusted source when the existing dependency install is ready. They use Crabbox for larger suites and computationally intensive work, including builds, typechecks, lint fan-out, Docker, package lanes, E2E, live proof, and CI parity. Trusted maintainer heavy proof defaults to blacksmith-testbox, and .crabbox.yaml now defaults to it. Its configured workflow hydrates provider and agent credentials, so untrusted contributor or fork code must use secretless fork CI or sanitized direct AWS Crabbox instead. Sanitized AWS runs set CRABBOX_ENV_ALLOW=CI, pass --no-hydrate, and use a fresh temporary remote HOME; this prevents the repo OPENCLAW_* allowlist and existing auth profiles from reaching untrusted code. They use a newly warmed lease dedicated to that untrusted source, never a trusted or previously hydrated lease. Launch an installed trusted Crabbox binary from a clean trusted main checkout and fetch only the remote PR with --fresh-pr; never execute the untrusted checkout's wrapper or config locally. Unset CRABBOX_AWS_INSTANCE_PROFILE and fail closed unless resolved aws.instanceProfile is empty. Before any install or test, use trusted absolute-path tools to require an IMDSv2 token, prove the IAM credentials endpoint returns 404, and compare remote git rev-parse HEAD to the full reviewed PR head SHA. Bind the lease to that SHA and stop or rewarm on head change. Upload trusted scripts/crabbox-untrusted-bootstrap.sh from clean main alongside --fresh-pr; it installs pinned Node/pnpm, verifies the SHA and package-manager pin, isolates HOME, installs dependencies, then executes the requested test. Unset all CRABBOX_TAILSCALE* overrides, force --network public --tailscale=false, clear exit-node/LAN flags, and require crabbox inspect to report public networking with no Tailscale state before uploading any script. Owned AWS/Hetzner capacity also remains the fallback for Blacksmith outages, quota issues, or explicit owned-capacity testing.

Agents do not pre-warm for anticipated work. Acquire a Testbox lazily when the first heavy command is ready, reuse the returned tbx_... id for later heavy commands, sync the current checkout on every run, and stop it before handoff.

Crabbox-backed Blacksmith runs warm, claim, sync, run, report, and clean up one-shot Testboxes. The built-in sync sanity check fails fast when git status --short on the synced box shows at least 200 tracked deletions, which catches disappearing root files such as pnpm-lock.yaml. For intentional large-deletion PRs, set CRABBOX_ALLOW_MASS_DELETIONS=1 for the remote command.

Crabbox also terminates a local Blacksmith CLI invocation that stays in the sync phase for more than five minutes without post-sync output. Set CRABBOX_BLACKSMITH_SYNC_TIMEOUT_MS=0 to disable that guard, or use a larger millisecond value for unusually large local diffs.

Before a first run, check the wrapper from the repo root:

pnpm crabbox:run -- --help | sed -n '1,120p'

The repo wrapper refuses a stale Crabbox binary that does not advertise the selected provider, and Blacksmith-backed runs require Crabbox 0.22.0 or newer so the wrapper gets the current Testbox sync, queue, and cleanup behavior. In Codex worktrees or linked/sparse checkouts, avoid the local pnpm crabbox:run script because pnpm may reconcile dependencies before Crabbox starts; invoke the node wrapper directly instead:

node scripts/crabbox-wrapper.mjs run --provider blacksmith-testbox --timing-json --shell -- "pnpm test <path-or-filter>"

When using the sibling checkout, rebuild the ignored local binary before timing or proof work:

version="$(git -C ../crabbox describe --tags --always --dirty | sed 's/^v//')" \
  && go build -C ../crabbox -trimpath -ldflags "-s -w -X github.com/openclaw/crabbox/internal/cli.version=${version}" -o bin/crabbox ./cmd/crabbox

The blacksmith: block in .crabbox.yaml already pins the org, workflow, job, and ref defaults, so the explicit flags below are optional. Changed gate:

pnpm crabbox:run -- --provider blacksmith-testbox \
  --blacksmith-org openclaw \
  --blacksmith-workflow .github/workflows/ci-check-testbox.yml \
  --blacksmith-job check \
  --blacksmith-ref main \
  --idle-timeout 90m \
  --ttl 240m \
  --timing-json \
  --shell -- \
  "corepack pnpm check:changed"

Focused test rerun on Testbox when local dependencies are unavailable or the target fans out:

pnpm crabbox:run -- --provider blacksmith-testbox \
  --idle-timeout 90m \
  --ttl 240m \
  --timing-json \
  --shell -- \
  "corepack pnpm test <path-or-filter>"

Full suite:

pnpm crabbox:run -- --provider blacksmith-testbox \
  --idle-timeout 90m \
  --ttl 240m \
  --timing-json \
  --shell -- \
  "corepack pnpm test"

Read the final JSON summary. The useful fields are provider, leaseId, syncDelegated, exitCode, commandMs, and totalMs. For delegated Blacksmith Testbox runs, the Crabbox wrapper exit code and JSON summary are the command result. The linked GitHub Actions run owns hydration and keepalive; it can finish as cancelled when the Testbox is stopped externally after the SSH command has already returned. Treat that as a cleanup or status artifact unless the wrapper exitCode is non-zero or the command output shows a failed test. One-shot Blacksmith-backed Crabbox runs should stop the Testbox automatically; if a run is interrupted or cleanup is unclear, inspect live boxes and stop only the boxes you created.

blacksmith testbox list --all
blacksmith testbox status --id <tbx_id>
blacksmith testbox stop --id <tbx_id>

Only enable reuse when you deliberately need to run several commands on a single hydrated box.

node scripts/crabbox-wrapper.mjs run --provider blacksmith-testbox --id <tbx_id> --timing-json --shell -- "corepack pnpm test <path-or-filter>"
pnpm crabbox:stop -- <tbx_id>

Reuse the lease, not stale source code. Omit --no-sync so every run uploads the current checkout; include it only when you intentionally want to rerun an unchanged, already synced tree. Code from untrusted contributors or forks must use CRABBOX_ENV_ALLOW=CI, --provider aws --no-hydrate, and a fresh temporary remote HOME for each command; install dependencies inside that sanitized command before testing begins. Reuse only a newly warmed lease that is dedicated to the same untrusted source; never reuse a trusted or previously hydrated lease. Never run the untrusted checkout's wrapper or configuration locally: instead, launch the installed trusted Crabbox binary from clean trusted main and pass --fresh-pr on every run. Keep CRABBOX_AWS_INSTANCE_PROFILE unset, reject any non-empty resolved instance profile, require a trusted remote IMDS no-role proof, and verify the reviewed head SHA before installing or testing. Bind the lease to that SHA; stop and rewarm whenever the head changes. If no remote PR exists, use secretless fork CI. Never select hydrate-github or the credential-hydrated Blacksmith workflow for untrusted source.

When Crabbox is the broken layer but Blacksmith itself functions, use direct Blacksmith only for diagnostic purposes such as list, status, and cleanup. Fix the Crabbox path before treating a direct Blacksmith run as maintainer proof.

If blacksmith testbox list --all and blacksmith testbox status work but new warmups sit queued without an IP or Actions run URL after a few minutes, treat this as pressure from the Blacksmith provider, queue, billing, or org-level limits. Stop the queued IDs you created, avoid starting more Testboxes, and move the proof to the owned Crabbox capacity path below while someone checks the Blacksmith dashboard, billing, and org limits.

Escalate to owned Crabbox capacity only when Blacksmith is down, quota-limited, missing the needed environment, or owned capacity is the explicit objective.

CRABBOX_CAPACITY_REGIONS=eu-west-1,eu-west-2,eu-central-1,us-east-1,us-west-2 \
  pnpm crabbox:warmup -- --provider aws --class standard --market on-demand --idle-timeout 90m
pnpm crabbox:hydrate -- --provider aws --id <cbx_id-or-slug>
pnpm crabbox:run -- --provider aws --id <cbx_id-or-slug> --timing-json --shell -- "pnpm check:changed"
pnpm crabbox:stop -- --provider aws <cbx_id-or-slug>

Under AWS pressure, avoid class=beast unless the task genuinely requires 48xlarge-class CPU. A beast request starts at 192 vCPUs and is the easiest way to hit regional EC2 Spot or On-Demand Standard quota. The repo-owned .crabbox.yaml defaults to class: standard, on-demand market, and capacity.hints: true so brokered AWS leases print selected region or market, quota pressure, Spot fallback, and high-pressure class warnings. Use fast for heavier broad checks, large only after standard and fast are insufficient, and beast only for exceptional CPU-bound lanes such as full-suite or all-plugin Docker matrices, explicit release or blocker validation, or high-core performance profiling. Do not use beast for pnpm check:changed, focused tests, docs-only work, ordinary lint or typecheck, small E2E reproductions, or Blacksmith outage triage. Use --market on-demand for capacity diagnosis so Spot market churn is not mixed into the signal.

.crabbox.yaml owns provider, sync, and GitHub Actions hydration defaults. Crabbox sync never transfers .git, so the hydrated Actions checkout keeps its own remote Git metadata instead of syncing maintainer-local remotes and object stores, and the repo configuration additionally excludes local runtime and build artifacts (such as .artifacts and test reports) that should never be transferred. .github/workflows/crabbox-hydrate.yml owns checkout, Node or pnpm setup, origin/main fetch, and the non-secret environment handoff for owned-cloud crabbox run --id <cbx_id> commands.

Related

• Install overview
• Development channels