Testing Updates and Plugins in OpenClaw

Checklist for update and plugin validation: prove the installable package can update real user state, repair stale legacy state through doctor, and still install, load, update, and uninstall plugins from every supported source.

For the broader test runner map, see Testing. For live provider keys and network-touching suites, see Testing live.

What we protect

• A package tarball is complete, has a valid dist/postinstall-inventory.json, and does not depend on unpacked repo files.
• A user can move from an older published package to the candidate package without losing config, agents, sessions, workspaces, plugin allowlists, or channel config.
• openclaw doctor --fix --non-interactive owns legacy cleanup and repair paths. Startup should not grow hidden compatibility migrations for stale plugin state.
• Plugin installs work from local directories, git repos, npm packages, and the ClawHub registry path.
• Plugin npm dependencies install in one managed npm project per plugin, get scanned before trust, and get removed through npm uninstall during plugin uninstall so hoisted dependencies do not linger.
• Plugin update is a no-op when nothing changed: install records, resolved source, installed dependency layout, and enabled state stay intact.

Local proof during development

Start narrow:

pnpm changed:lanes --json
pnpm check:changed
pnpm test:changed

For plugin install, uninstall, dependency, or package-inventory changes, also run the focused tests that cover the edited seam:

pnpm test src/plugins/uninstall.test.ts src/infra/package-dist-inventory.test.ts test/scripts/package-acceptance-workflow.test.ts

Before any package Docker lane consumes a tarball, prove the package artifact:

pnpm release:check

release:check runs generated config/docs and plugin checks (config schema, config docs baseline, plugin SDK exports and surface budget, plugin versions/inventory), writes the package dist inventory, runs npm pack --dry-run, rejects forbidden packed files, installs the tarball into a temp prefix, runs postinstall, and smokes bundled channel entrypoints.

For a Plugin SDK change, compare the exact commits separately:

base_sha=$(git merge-base origin/main HEAD)
head_sha=$(git rev-parse HEAD)
pnpm plugin-sdk:api:diff -- --base "$base_sha" --head "$head_sha"

Release npm preflight uses the same readable diff against the prior published dist-tag and prints the 8-character acknowledgement digest required when that release changes the Plugin SDK API.

Docker lanes

The Docker lanes are the product-level proof. They install or update a real package inside Linux containers and assert behavior through CLI commands, Gateway startup, HTTP probes, RPC status, and filesystem state.

Use focused lanes while iterating:

pnpm test:docker:plugins
pnpm test:docker:plugin-lifecycle-matrix
pnpm test:docker:plugin-update
pnpm test:docker:upgrade-survivor
pnpm test:docker:published-upgrade-survivor
pnpm test:docker:update-restart-auth
pnpm test:docker:update-migration

Important lanes:

• test:docker:plugins covers plugin install smoke, local folder installs, local folder update skip behavior, local folders with preinstalled dependencies, file: package installs, git installs with CLI execution, git moving-ref updates, npm registry installs with hoisted transitive dependencies, npm update no-ops, malformed npm package metadata rejection, local ClawHub fixture installs and update no-ops, marketplace update behavior, and Claude-bundle enable/inspect. Set OPENCLAW_PLUGINS_E2E_CLAWHUB=0 to keep the ClawHub block hermetic/offline.
• test:docker:plugin-lifecycle-matrix installs the candidate package in a bare container, runs an npm plugin through install, inspect, disable, enable, explicit upgrade, explicit downgrade, and uninstall after deleting the plugin code. It logs RSS and CPU metrics per phase.
• test:docker:plugin-update validates that an unchanged installed plugin does not reinstall or lose install metadata during openclaw plugins update.
• test:docker:upgrade-survivor installs the candidate tarball over a dirty old-user fixture, runs package update plus non-interactive doctor, then starts a loopback Gateway and checks state preservation.
• test:docker:published-upgrade-survivor first installs a published baseline, configures it through a baked openclaw config set recipe, updates it to the candidate tarball, runs doctor, checks legacy cleanup, starts the Gateway, and probes /healthz, /readyz, and RPC status.
• test:docker:update-restart-auth installs the candidate package, starts a managed token-auth Gateway, unsets caller gateway auth env for openclaw update --yes --json, and requires the candidate update command to restart the Gateway before the normal probes.
• test:docker:update-migration is the cleanup-heavy published-update lane. It starts from a configured Discord/Telegram-style user state, runs baseline doctor so configured plugin dependencies have a chance to materialize, seeds legacy plugin dependency debris for a configured packaged plugin, updates to the candidate tarball, and requires post-update doctor to remove the legacy dependency roots.

Useful published-upgrade survivor variants:

OPENCLAW_UPGRADE_SURVIVOR_BASELINE_SPEC=openclaw@2026.4.23 \
OPENCLAW_UPGRADE_SURVIVOR_SCENARIO=versioned-runtime-deps \
pnpm test:docker:published-upgrade-survivor

OPENCLAW_UPGRADE_SURVIVOR_BASELINE_SPEC=openclaw@latest \
OPENCLAW_UPGRADE_SURVIVOR_SCENARIO=bootstrap-persona \
pnpm test:docker:published-upgrade-survivor

OPENCLAW_UPGRADE_SURVIVOR_BASELINE_SPEC=openclaw@2026.7.1-2 \
OPENCLAW_UPGRADE_SURVIVOR_SCENARIO=sqlite-volume \
pnpm test:docker:published-upgrade-survivor

Available scenarios: base, acpx-openclaw-tools-bridge, feishu-channel, bootstrap-persona, channel-post-core-restore, plugin-deps-cleanup, configured-plugin-installs, stale-source-plugin-shadow, tilde-log-path, meeting-transcripts-sqlite, versioned-runtime-deps, cron-scheduled-authority, and sqlite-volume. In aggregate runs, OPENCLAW_UPGRADE_SURVIVOR_SCENARIOS=reported-issues expands the release-soak fixtures but excludes the expensive sqlite-volume scenario. Use OPENCLAW_UPGRADE_SURVIVOR_SCENARIOS=far-reaching to include it.

The sqlite-volume scenario combines configured Matrix, Discord, and Telegram plugin/channel state with 4,800 sessions, 23,890 transcript events, and 2,200 cron crawl jobs by default. It verifies exact JSONL-to-SQLite and cron migration, legacy archival, database integrity, a second idempotent Doctor run, and Gateway startup. Scale it with OPENCLAW_UPGRADE_SURVIVOR_VOLUME_SESSIONS, OPENCLAW_UPGRADE_SURVIVOR_VOLUME_EVENTS_PER_SESSION, and OPENCLAW_UPGRADE_SURVIVOR_VOLUME_CRON_JOBS. The default Doctor budgets are 120 seconds for migration and 60 seconds for the idempotent pass; override them with OPENCLAW_UPGRADE_SURVIVOR_VOLUME_MIGRATION_BUDGET_SECONDS and OPENCLAW_UPGRADE_SURVIVOR_VOLUME_IDEMPOTENCE_BUDGET_SECONDS on slower hosts.

Full update migration is intentionally separate from Full Release CI. Use the manual Update Migration workflow when the release question is "can every published stable release from 2026.4.23 onward update to this candidate and clean up plugin dependency debris?":

gh workflow run update-migration.yml \
  --ref main \
  -f workflow_ref=main \
  -f package_ref=main \
  -f baselines=all-since-2026.4.23 \
  -f scenarios=plugin-deps-cleanup

Package Acceptance

Package Acceptance is the GitHub-native package gate. It resolves one candidate package into a package-under-test tarball, records version and SHA-256, then runs reusable Docker E2E lanes against that exact tarball. The workflow harness ref is separate from the package source ref, so current test logic can validate older trusted releases.

Candidate sources:

• source=npm: check openclaw@extended-stable, openclaw@beta, openclaw@latest, or a specific published version.
• source=ref: bundle a trusted branch, tag, or commit using the currently selected harness.
• source=url: verify a public HTTPS tarball that requires package_sha256. This route blocks URL credentials, non-standard HTTPS ports, private or internal hostnames, DNS/IP resolutions, special-use IP ranges, and unsafe redirects.
• source=trusted-url: verify an HTTPS tarball with mandatory package_sha256 and trusted_source_id following the maintainer-owned rules in .github/package-trusted-sources.json. Choose this for enterprise or private mirrors rather than relaxing source=url via an input-level allow-private flag. If policy enables bearer auth, it relies on the static OPENCLAW_TRUSTED_PACKAGE_TOKEN token.
• source=artifact: reuse a tarball submitted by a different Actions run.

Full Release Validation defaults to source=artifact, which is derived from the resolved release SHA. To prove post-publish state, supply package_acceptance_package_spec=openclaw@YYYY.M.PATCH so the identical upgrade matrix points at the deployed npm package instead.

Release checks invoke Package Acceptance with the package, update, restart, and plugin set:

doctor-switch update-channel-switch skill-install update-corrupt-plugin upgrade-survivor published-upgrade-survivor root-managed-vps-upgrade update-restart-auth plugins-offline plugin-update plugin-binding-command-escape

When release soak is active (mandatory for release_profile=stable and full), these extra flags are passed:

published_upgrade_survivor_baselines=last-stable-4 2026.4.23 2026.5.2 2026.4.15
published_upgrade_survivor_scenarios=reported-issues
telegram_mode=mock-openai

This approach keeps package migration, update channel switching, corrupt managed-plugin handling, stale plugin dependency cleanup, offline plugin coverage, plugin update behavior, and Telegram package QA on the same resolved artifact, without forcing the default release package gate to scan every published release.

last-stable-4 maps to the four most recent stable OpenClaw npm releases. Release package acceptance fixes 2026.4.23 as the initial plugin-update compatibility cutoff, 2026.5.2 as a plugin-architecture transition cutoff, and 2026.4.15 as an older 2026.4.1x published-update baseline; the resolver removes pins already present in the latest four. For complete published update migration testing, rely on all-since-2026.4.23 in the separate Update Migration workflow instead of Full Release CI. release-history stays available for manual broader sampling when the legacy pre-date anchor is also desired.

When several published-upgrade survivor baselines are chosen, the reusable Docker workflow splits each baseline into its own dedicated runner job. Every baseline shard still executes the selected scenario set, but logs and artifacts remain per-baseline, and wall time is capped by the slowest shard rather than a single large serial job.

Run a package profile manually to validate a candidate before release:

gh workflow run package-acceptance.yml \
  --ref main \
  -f workflow_ref=main \
  -f source=npm \
  -f package_spec=openclaw@beta \
  -f suite_profile=package \
  -f published_upgrade_survivor_baselines="last-stable-4 2026.4.23 2026.5.2 2026.4.15" \
  -f published_upgrade_survivor_scenarios=reported-issues \
  -f telegram_mode=mock-openai

For a published extended-stable canary, assign package_spec=openclaw@extended-stable. Package Acceptance turns that selector into a precise tarball before the Docker lanes start.

Pick suite_profile=product when the release question involves MCP channels, cron or subagent cleanup, OpenAI web search, or OpenWebUI. Choose suite_profile=full only when complete Docker release-path coverage is necessary.

Release default

For release candidates, the standard proof stack is:

1. pnpm check:changed and pnpm test:changed to catch source-level regressions.
2. pnpm release:check to confirm package artifact integrity.
3. Package Acceptance package profile or the release-check custom package lanes for install, update, restart, and plugin contracts.
4. Cross-OS release checks for OS-specific installer, onboarding, and platform behavior.
5. Live suites only when the modified area touches provider or hosted-service behavior.

On maintainer machines, broad gates and Docker or package product proof should run in Testbox unless local proof is explicitly needed.

Legacy compatibility

Compatibility leniency stays narrow and time limited:

• Packages up to 2026.4.25, including 2026.4.25-beta.*, may accept already-released package metadata gaps in Package Acceptance.
• The published 2026.4.26 package may emit warnings for local build metadata stamp files already shipped.
• Later packages must meet modern contracts. The same gaps fail instead of warning or skipping.

Do not add new startup migrations for these older shapes. Add or extend a doctor repair, then verify it with upgrade-survivor, published-upgrade-survivor, or update-restart-auth when the update command controls the restart.

Adding coverage

When modifying update or plugin behavior, add coverage at the lowest layer that can fail for the correct reason:

• Pure path or metadata logic: unit test next to the source.
• Package inventory or packed-file behavior: package-dist-inventory or tarball checker test.
• CLI install or update behavior: Docker lane assertion or fixture.
• Published-release migration behavior: published-upgrade-survivor scenario.
• Update-owned restart behavior: update-restart-auth.
• Registry or package source behavior: test:docker:plugins fixture or ClawHub fixture server.
• Dependency layout or cleanup behavior: assert both runtime execution and the filesystem boundary. npm dependencies may be hoisted inside the plugin's managed npm project, so tests should confirm that project is scanned or cleaned instead of assuming only the plugin package-local node_modules tree.

Keep new Docker fixtures hermetic by default. Use local fixture registries and fake packages unless the test specifically targets live registry behavior.

Failure triage

Begin with the artifact identity:

• Package Acceptance resolve_package summary: source, version, SHA-256, and artifact name.
• Docker artifacts: .artifacts/docker-tests/**/summary.json, failures.json, lane logs, and rerun commands.
• Upgrade survivor summary: .artifacts/upgrade-survivor/summary.json, including baseline version, candidate version, scenario, phase timings, and config recipe coverage.

Prefer rerunning the failed exact lane with the same package artifact over rerunning the entire release umbrella.