Cloud Workers Plan for Ephemeral Agent Sessions

Status

Revision 3 of the proposal, not yet implemented. Direction was finalized in July 2026; revision 2 incorporated findings from adversarial review (dedicated worker protocol, placement and environment state machines, git-aware inbound synchronization, one-way v1 handoff, and controlled egress security language). Revision 3 finalizes the sync ownership model (the worker authors commits, the gateway adopts them and publishes), introduces a no-git plain sync mode, fixes worker execution to run fully within the box, moves internet policy to provision time, and returns agent dispatch to milestone 3.

Problem

On a single machine, OpenClaw agent sessions run their loop, tools, and inference within the gateway process. Compute is limited by that machine, long tasks occupy it, and parallel work competes for it. Hosted products such as Cursor cloud agents, Claude Code on the web, and Codex cloud solve this with ephemeral per-task cloud sandboxes, but they require vendor infrastructure and vendor trust.

Operators who already have spare machines or can lease them cheaply have no way to say: run this session over there, show it in my sidebar like any other session, and discard the machine afterward.

Goals

• Execute a full agent session (loop plus tools) on an ephemeral remote machine called a cloud worker, while the session appears and streams in the Control UI exactly like a local session.
• No standing credentials on the worker (no provider authentication, no forge tokens) and no direct network egress; the box only needs a reachable sshd.
• Provision, sync, run, collect, and destroy are fully automated and provider-pluggable (first provider: Crabbox-style lease CLIs).
• Dispatch running work from the gateway to a worker at a turn boundary without losing transcript, session identity, or provider cache affinity when request bytes remain equivalent; pull results back safely.
• Both humans via the UI and agents via the tool can dispatch work to a cloud worker.
• Support sessions lasting days; lifetime is governed by policy, not a hard-coded limit.

Non-goals (v1)

• No external coding harnesses (Claude Code, Codex CLI) on workers. Worker sessions run only OpenClaw's embedded runner. Harness support is a v2 opt-in because harnesses perform their own inference with their own credentials.
• No best-of-N or parallel attempt fan-out.
• No VPN or tailnet dependency. Transport is SSH only.
• No new sandbox runtime. The worker machine serves as the isolation boundary; in-box OS sandboxing can be added later.
• No symmetric live migration in v1: dispatch goes from local to worker; worker to local requires a stopped session plus completed workspace reconciliation. Live two-way handoff builds on the same barrier machinery later.
• No JSON side-state on the gateway; environment, placement, cursor, and grant state live in SQLite.

Prior art (what we copy, what we invert)

• Cursor cloud agents: the agent loop runs in their cloud; the VM is a tool-execution target; an append-only conversation store is streamed to all clients; snapshot-after-install provides warm starts; self-hosted workers are outbound-only worker processes. We adopt the "conversation source of truth stays on the orchestrator" and streaming model; we invert loop placement (see decision below).
• Codex cloud: a two-phase runtime with a networked setup phase followed by an offline agent phase with secrets stripped; container-state cache enables fast follow-ups. We adopt the phase split as our egress posture and the cache idea for v2 warm images.
• Claude Code on the web: per-session VM; credential-isolating git proxy where real tokens never enter the sandbox and push is restricted to the session branch; filesystem snapshot after setup; teleport handoff equals a pushed branch plus replayed history. We adopt credential isolation and the handoff framing, but outbound sync uses rsync from the gateway so dirty working trees work and no forge token exists anywhere near the box.
• Copilot coding agent: default-deny egress with a package-registry allowlist. Our steady-state default is stronger with no direct egress at all because inference and websearch arrive through the SSH tunnel. See Security for why this is "controlled egress" rather than "zero egress".

Architecture decision: loop on the worker, inference through the gateway

Three placements were evaluated:

1. Loop stays on the gateway, worker executes tools (Cursor model). Safest failure domain because transcript, inference, approvals, and restart recovery all remain local, and a reviewer-preferred first milestone. Rejected as the product architecture: OpenClaw's non-exec tools are in-process filesystem operations, so every file read, edit, or grep becomes a network round trip or requires a large tool-surface refactor into coarse workspace RPCs; runtime behavior is chatty and latency-bound. We reuse its spirit where it is already built, such as exec offload to nodes, but do not build the tool-remoting layer.
2. Loop and inference both on the worker. Simplest failure domain, but model credentials including OAuth profiles must ship to throwaway machines, the gateway loses policy, routing, and audit control, and migration switches the provider-calling identity, invalidating provider caches.
3. Loop and tools on the worker, model calls proxied through the gateway. Chosen. One round trip per model turn instead of per tool call; tools run next to the code; the gateway remains the single owner of auth profiles, provider routing, and policy; the worker holds no secrets.

The cost of option 3 is a synchronous gateway dependency during each model turn, so its durability rules are part of the decision, not an afterthought:

• Gateway loss mid-turn fails the active provider call. The turn is marked failed and is retried as a new turn after reconnect; there is no transparent replay of an in-flight provider stream because of double-billing and double-tool-call risk.
• Every worker-to-gateway operation carries durable identity (see Worker protocol) so reconnects resume or fetch cached terminal results instead of dangling.
• The gateway is a capacity-managed component: concurrent-worker limits, flow control, and load shedding are in scope for v1 (see Capacity).

Because the gateway both stores the transcript and originates all provider traffic, the session is location-independent: moving the loop between gateway and worker changes nothing on the provider side and nothing in the UI data path. That is what makes dispatch and pull-back cheap.

Components

1. Environment state machine + provider contract

environments.* in the gateway protocol is currently a status-only projection. The durable core is a SQLite-owned environment record and state machine, designed before RPC shapes:

requested → provisioning → bootstrapping → ready → (attached|idle) → draining → destroying → destroyed | failed | orphaned

• Provisioning is crash-safe: the intent row is persisted before the provider call, with a deterministic operation id, so a gateway restart can adopt an in-flight lease instead of double-provisioning or orphaning a paid machine.
• Restart reconciliation and an orphan sweeper comparing provider inspect against local records are v1 requirements, not hardening.

Provider contract (plugin-implemented; no provider names or policy in core):

type WorkerProvider = {
  id: string;
  provision(profile: WorkerProfile, opId: string): Promise<WorkerLease>; // → ssh host/port/user/key material
  inspect(lease: { leaseId: string; profile: WorkerProfile }): Promise<LeaseStatus>; // adopt/health/orphan sweep
  renew?(leaseId: string): Promise<void>; // long-lived sessions vs provider TTLs
  destroy(lease: { leaseId: string; profile: WorkerProfile }): Promise<void>; // idempotent, returns only on proof of teardown
};

RPCs: environments.create, environments.destroy, extended environments.list/status (provider, lease id, state, age, idle time, attached sessions). First providers: a Crabbox-shape lease CLI wrapper (product path) and a static-SSH-host provider marked development-only. A worker on a shared host can read unrelated host data, so static hosts are for feature development, not the default posture.

2. Worker bootstrap: install OpenClaw on the box

No bespoke worker artifact, and no dependence on npm availability:

• Canonical install for all modes: a gateway-produced, content-hashed worker bundle (the gateway's own build output packed as a tarball), pushed over SSH and installed on the box. This covers dev builds and unreleased commits by construction.
• npm i -g openclaw@<exact gateway version> is an optimization when the gateway runs a released version; never latest.
• Bootstrap is idempotent; a warm lease with a matching bundle hash skips install. Raw machines may need a networked toolchain phase for the Node runtime, which is part of the setup phase and closed afterward.
• Handshake verifies worker build hash, protocol feature set, and runtime compatibility. The existing gateway version and protocol checks are insufficient for this because SSH-tunneled nodes are exempted from exact-version rejection, so worker admission performs its own exact-build check.

Worker mode (openclaw worker) acts as an entry point rather than a fork: it handles connections alongside an embedded agent runner, with session persistence and model calls routed through gateway RPCs. It must not activate any gateway surfaces: no channels, no automatic plugin startup beyond the session toolset, a disposable state directory, and no local authentication profiles.

3. Transport: everything over SSH

Connectivity is managed by the gateway; the worker only needs sshd:

• The gateway opens an SSH connection to the worker (credentials come from the provider lease, the host key is pinned from provisioning output, without StrictHostKeyChecking=no) and sets up a reverse tunnel that forwards a local worker socket to the gateway's WebSocket endpoint.
• Control and model traffic, along with workspace transfers, use separate SSH connections with the same pinned trust material, so rsync cannot cause head-of-line blocking for token streams.
• The environment runtime on the gateway handles tunnel lifecycle, including keepalive and reconnection with backoff. A tunnel interruption remains invisible at the session level: durable protocol state (described below) allows the worker to reattach and continue.

4. Worker protocol (dedicated; not the node protocol)

Adversarial review of the current node seams eliminated plain reuse as an option: pending node invocations are process-local promises that terminate when the connection drops, node idempotency keys are parsed but not deduplicated, and critically, a connected node can emit standard node events, including agent-run requests. Therefore, "node kind plus capability ceiling" does not function as an ingress security boundary. Workers receive an authenticated worker role with a restricted, versioned RPC and event allowlist; worker connections cannot reach any legacy node event handler.

For identity and credentials, provisioning generates a short-lived worker credential tied to the environment ID, worker key, bundle hash, the single allowed session, the permitted RPC set, and an expiration time. SSH-verified pairing still applies (the box was provisioned and the key is held), but authorization comes from the minted credential rather than the declared node surface.

Durable operation semantics (the structure borrows from the existing ACP runtime and its event ledger, which provides stable handles, per-session serialization, and durable (session, seq) replay):

• Every operation is scoped within (sessionId, lifecycleRevision, runId, ownerEpoch, streamKind, seq).
• Ownership epochs fence off stale workers: a replacement worker advances the epoch, and late results from the old epoch are deterministically rejected.
• At-least-once delivery uses persisted ACK cursors and cached terminal results in SQLite; deduplication is deterministic. No exactly-once guarantees exist.
• Explicit frames handle cancel, close, resume, and terminal results; credit and window-based flow control governs streams.
• Protocol feature negotiation operates independently of the general node protocol version.

5. Session backend RPCs

Two distinct contracts exist: the current codebase separates durable transcript mutations (owned by the session manager, using a JSONL tree with parent and leaf state) from process-local live events (streaming deltas, tool lifecycle, approvals). The worker protocol must maintain this separation:

• For durable transcript commits, the worker submits semantic append batches using runEpoch along with base-leaf compare-and-swap; the gateway session manager generates entry IDs and parent IDs. The worker can never provide trusted transcript rows, entry IDs, parent IDs, or foreign session IDs.
• For replayable live events, a typed event union includes worker sequence numbers, gateway ACKs, bounded retention, and late-event fencing. This feeds the existing agent-event fanout so that chat views, tool rows, and unread or status logic behave identically to local sessions.

For the inference proxy, the event vocabulary from the existing runtime proxy stream client (src/agents/runtime/proxy.ts) is reused, but the trust boundary shifts. The worker only sends session and run identity, an approved model reference, context, and constrained generation options. The gateway resolves provider, endpoint, auth, headers, routing, and cost policy from its own catalog. A worker-supplied model object, such as an attacker-controlled baseUrl, is rejected. Request-size limits, cancellation, audit, and terminal-result replay all apply. Gateway-resident tools like websearch execute on the gateway and return results over the same channel.

6. Workspace sync

The sync anchor is a gateway-local workspace with exclusive placement ownership. For git workspaces, this is a dedicated managed worktree (existing managed-worktree metadata, including branch, base, and snapshot ownership, forms the foundation). For non-git workspaces, it is a gateway-owned target directory. The user's live checkout is never used. Exclusive ownership while the session is placed remotely makes inbound sync conflict-free by design.

Ownership is split between commit and publish:

• The worker-side agent creates commits normally in its copy. git commit is a local operation without credentials; author identity is projected from gateway configuration. These commits remain inert objects until the gateway adopts them.
• The gateway handles everything requiring trust: verifying that inbound commits build on the recorded base, fast-forwarding the local worktree, pushing, creating PRs, and optionally signing or re-signing, all with gateway-local credentials. The worker never holds git or forge credentials and never contacts a remote.

Two sync modes exist, selected based on whether the workspace is a git repository:

• Git mode. For outbound sync, rsync transfers the worktree (including uncommitted and eligible untracked files; crabbox-style include/exclude rules apply, and .worktreeinclude is respected) over the tunnel's SSH identity, recorded as an immutable base manifest with content hashes and base commit. For inbound sync, new commits arrive as a git bundle or temporary ref against the recorded base; untracked artifacts return via an explicit manifest with size, type, and symlink-containment checks. Adoption verifies base ancestry and stops on divergence, so nothing silently overwrites either side. Deletes, renames, submodules, and symlink escapes are handled by manifest rules, not rsync heuristics.
• Plain mode, for non-git scenarios like building a project from scratch on the box. Outbound sync uses the same rsync and base manifest approach. Inbound sync uses a manifest-diffed mirror back into the gateway-owned target directory with delete propagation. This is safe for the same reason git mode is: exclusive ownership means no concurrent local edits exist to conflict with, and the base manifest still detects unexpected local drift and stops rather than overwriting.

Checkpointing protects long-running sessions from lease loss. Periodic inbound checkpoints occur, using session-branch commits in git mode and manifest snapshots in plain mode. The cadence is defined by profile policy, with a turn-based default.

7. Placement state machine, sessions, and UI

Runtime placement is a SQLite-owned state machine keyed to the session, not a pair of loose row fields:

local → requested → provisioning → syncing → starting → active(worker) → draining → reconciling → local | reclaimed | failed

It stores the environment ID, transition generation, active owner epoch, workspace base manifest, worker bundle hash, and last ACK cursors. Turn admission atomically claims placement before either loop starts a turn, so a local message admitted against a stale snapshot can never race a worker turn. Exactly one loop owns the session at any time.

For the UI:

• A worker session appears as a standard session row with additional placement metadata. It resides in the normal store, lists via sessions.list, and streams through existing subscriptions. The sidebar and chat require no new data path, only presentation changes: a worker badge and placement or environment status (provisioning / syncing / running / idle / reconciling / reclaimed).
• For creation UX, the session target bar in the sessions sidebar redesign gains a cloud worker destination alongside gateway and node. This requires a configured provider profile; the feature remains invisible until configured.
• For agent dispatch, a session tool allows an agent to hand work to a cloud worker, similar to how a human would dispatch work, creating a worker-backed sub-session like a subagent. This ships in the same milestone as human dispatch, gated by the same opt-in provider configuration. Recursion is structurally bounded: worker sessions cannot dispatch other workers in v1. Spend control uses per-environment accounting and audit rather than quota machinery.

Dispatch and handoff

v1 is deliberately asymmetric:

• Local to worker dispatch: pass the migration barrier described below, provision or reuse a worker, sync, flip placement, then the next turn executes remotely.
• Worker to local pull-back: stop the session by draining the worker using the same barrier, complete inbound reconciliation, then flip placement to local. This is not a live migration.
• Symmetric live handoff, which moves an actively working session in both directions without stopping, reuses the same barrier and reconciliation machinery and ships after fault-injection tests prove the barrier works.

The migration barrier is necessary because the turn boundary alone is insufficient: approvals, background processes, and released-lock transcript merges can straddle it.

1. Prevent new turn acceptance (placement claim).
2. Abort or drain currently active runs.
3. Withdraw pending execution approvals and grant permissions.
4. Drain transcript side-writes and live event acknowledgments.
5. Shut down worker child processes.
6. Isolate the previous owner by incrementing the owner epoch.
7. Reconcile the workspace (inbound, conflict-aware).
8. Install the new owner.

Cache affinity: since provider calls always come from the gateway regardless of placement, cache locality holds when the serialized provider request remains unchanged, same tool ordering, system instructions, provider wrappers, and cache metadata (all handled on the gateway side). This is a verifiable property, not a guess: byte-level equivalence checks between local and worker placement for each supported provider transport are required by the milestone that introduces the worker loop.

Security model

To be exact: the worker lacks direct outbound network access and holds no standing provider or forge credentials. It is not "zero egress", inference and gateway-executed tools count as controlled egress channels (a prompt-injected worker can still embed workspace data into model context or websearch queries). Consequently:

• Controlled-egress tracking: per-environment auditing and operator-visible accounting on the inference proxy and gateway tools. Rate and byte limits exist as protocol-level flow control (capacity), not as spend-quota infrastructure.
• Worker communication to the gateway is limited to the closed worker-protocol allowlist; transcript writes are structurally constrained (gateway-generated identifiers, single bound session).
• Worker execution has full permissions inside the box. The box is disposable and has no credentials, so per-command approval adds friction without securing anything; the guarded boundary is inbound reconciliation and audit. Execution never passes through the gateway node-approval path.
• Internet access is a provision-time provider choice: the environment profile decides at box creation (firewall, security group, no-egress network), optionally with a networked setup phase that the provider closes before the agent phase. The core does not offer a runtime network toggle.
• Box hygiene during provisioning: cloud metadata endpoint blocked or confirmed absent, no instance profile, no inherited SSH agent, no Docker socket, clean environment and home directory. SSH host keys pinned from provisioning output.
• Approvals and policies for anything on the gateway side (push, PR, provider calls) continue to function on the gateway.

Blast radius of a compromised worker session: the synced workspace copy plus what the audited proxy channels permit, no credentials, no direct network access, no gateway surface beyond the allowlist.

Capacity

The gateway relays every prompt and token stream for N workers, so v1 defines a capacity model rather than discovering it in production: concurrent-worker limits per gateway, per-stream credit windows (the current event stream queue is unbounded and the node socket buffer ceiling force-closes slow consumers, both unsuitable without changes), bounded disk spooling for bursts, and load shedding with visible backpressure states in the UI. Workspace transfer stays on its own SSH channel.

Lifecycle

• Idle auto-stop and TTL are provider-profile policy, not fixed constants. Defaults are generous with explicit keep-alive; days-long work is first-class (provider renew exists for lease-based backends); a session with an active turn or recent activity is never reclaimed.
• On worker death or reclaim: placement moves to reclaimed, the session row remains, the next message provisions a fresh worker and re-syncs from the last checkpoint. Conversation is never lost (gateway-side store); workspace changes since the last checkpoint are lost and the UI indicates this.
• Warm-lease reuse from day one (providers that support it); image snapshot after bootstrap is the v2 fast-start path.

Configuration surface

Minimal and opt-in: a provider profile block (provider id, credentials or CLI reference, sync rules, lifetime policy, budgets, optional setup phase) plus per-session placement selection. No new environment variables. Unconfigured installs see nothing.

Milestones

Implementation lands as small, independently mergeable PRs; each milestone below is a PR series, not one change.

1. Foundations: environment state machine + provider contract + crabbox-shape provider (static-SSH as dev harness), worker bundle bootstrap + admission handshake, SSH tunnel + host-key pinning, managed-worktree snapshot + outbound sync (git + plain modes). Orphan sweep + restart adoption.
2. Worker protocol + worker loop: authenticated worker role, durable ops/epochs/ACK cursors, transcript commit + live event contracts, inference proxy with gateway-resolved models, flow control. One provider, human dispatch of new sessions only, no handoff. Fault-injection tests (tunnel partition, gateway restart, worker death) gate exit.
3. Dispatch + pull-back + agent dispatch: migration barrier, placement state machine wired to UI target bar, inbound reconciliation + checkpoints, per-environment audit, capacity limits, agent dispatch tool (worker sessions cannot recurse). Prompt-cache byte-equivalence tests.
4. Symmetric live handoff, after milestone-3 fault-injection proof.

Later: ACP harnesses on workers as per-environment credential-hydration opt-in; snapshot/warm-image fast start; fan-out (N leases, same prompt); in-box OS sandboxing; richer artifact capture via the artifacts schema.

Open questions

• Plugin and skill availability on workers: repo-carried skills sync with the workspace for free; gateway-configured agent skills and plugins need an explicit sync or exclusion decision (tool/plugin manifest is part of the admission handshake either way).
• Checkpoint cadence default: turn-based versus time-based for very chatty sessions.
• How environment profiles interact with multi-agent routing (per-agent default profiles versus per-session selection only).