Computer Use: Desktop Control via computer.act

Computer use enables the gateway agent to observe and operate a compatible desktop system that is paired with it. Eligibility depends on advertised capabilities: the connected node must declare support for both computer.act and screen.snapshot. The node's descriptor spells out which v2 action, target, observation, and delivery families it supports, so the built-in computer tool only presents operations that the provider can genuinely carry out. Coordinate actions are tied to a reference frame supplied by the node; providers with the right capabilities can also target windows and elements, ask for background delivery, and produce structured results or refusal details. A model with vision capabilities drives the interface through the built-in computer agent tool.

A single uniform command, computer.act, is issued by the agent; it has no say in how the node executes it. On macOS, the node-local provider is chosen under Settings → General → Capabilities: Peekaboo, the default, keeps the existing in-process coordinate-action route, whereas CUA relies on a driver daemon embedded in OpenClaw.app. The app launches that daemon itself, which lets it inherit OpenClaw's Accessibility and Screen Recording permissions, and the app-owned node worker connects via a private socket. For Windows and Linux, the optional and experimental cua-computer plugin can be used, and it invokes the packaged CUA Driver SDK directly.

Provider selection does not fall back on a per-action basis. Switching providers shuts down the active execution surface, bumps the provider generation, and re-advertises the node commands. As a result, a CUA failure surfaces as an unavailable outcome rather than quietly replaying the action through Peekaboo.

Requirements

• A paired, connected node that advertises both computer.act and screen.snapshot, with screen.snapshot yielding displayFrameId.
• macOS fulfiller: the app setting Allow Computer Control must be enabled. It starts enabled by default; an explicit off selection remains off.
• macOS fulfiller: pick Peekaboo (default) or CUA. CUA is only available when the pinned driver is included in the signed app bundle; development builds lacking that artifact display driver not bundled.
• macOS fulfiller: OpenClaw must be granted Accessibility and Screen Recording. The native Peekaboo route additionally needs Event Posting access for its CoreGraphics input primitives.
• Windows/Linux fulfiller: the bundled cua-computer plugin is enabled on Windows x64/ARM64 or glibc-based Linux x64/ARM64. Its package ships the pinned CUA Driver SDK 0.19.3 runtime; no cua-driver executable, daemon, or MCP server is set up.
• The pairing update that includes computer.act is approved on the gateway.
• A vision-capable agent model.
• A tool policy that exposes computer. The default coding profile does not. Add computer to tools.alsoAllow; sandboxed agents also require it in tools.sandbox.tools.alsoAllow.

The computer agent tool

The built-in computer tool handles one action per call. Coordinates are non-negative integer pixels taken from the latest screenshot; the node translates them into display points. Coordinate actions must repeat the screenshot result's frameId, and an explicit screenIndex has to match that frame. OpenClaw also carries a display identity issued by the node from the screenshot into the action, so a display reconnect or geometry change fails closed rather than silently retargeting the same index. These checks reject guessed tokens and tokens originating from another delivered frame or display. A token does not guarantee freshness: apps can alter pixels on the same display after capture, so grab a new screenshot whenever the scene might have changed.

• Reads: screenshot.
• Pointer: left_click, right_click, middle_click, double_click, triple_click, mouse_move, left_click_drag (with startCoordinate), left_mouse_down, left_mouse_up.
• Scroll: scroll with scrollDirection (up|down|left|right) and scrollAmount (wheel ticks).
• Keyboard: type (text), key (combo such as cmd+shift+t or Return), hold_key (text combo held for duration seconds).
• Pacing: wait (duration seconds).

Providers that support the v2 window/element family can additionally offer list_apps, list_windows, get_accessibility_tree, get_cursor_position, get_window_state, launch_app, kill_app, bring_to_front, set_value, zoom, escalate_scope, and invoke_menu. The provider descriptor is the source of truth; actions that are unavailable are left out rather than faked through another provider.

The CUA provider also makes available the v2 browser family: get_browser_state, browser_prepare, browser_navigate, browser_click, browser_type, browser_dialog, browser_set_input_files, browser_download, and browser_pointer. To attach a discovered native browser window, call get_browser_state, then work with the returned opaque browserRef, pageRef, observation, and element references. These references are scoped to a single Computer Use run and driver generation; page-element observations become invalid after navigation, and a driver restart invalidates the entire browser reference set.

Additionally, CUA exposes get_recording_state, start_recording, stop_recording, and replay_trajectory. Recording and browser file operations rely on opaque openclaw:computer-resource handles. The node handles creation and validation of the underlying files and directories; agent actions never accept native paths, output roots, or helper executable paths. Handles are tied to one Computer Use execution and cannot be reused across another.

Modifier keys travel via the text field on click and scroll actions (shift, ctrl, alt, cmd). After an input action, the tool returns a fresh screenshot for the model to observe the result. If multiple computer-capable nodes are connected, explicitly pass node.

Screenshots are model-only: they are never auto-delivered to the chat channel. Treat all on-screen content as untrusted input; the tool warns the model not to follow on-screen instructions that conflict with the user's request.

CUA Driver provider

macOS app-owned daemon

The signed macOS app bundles the universal cua-driver 0.19.3 executable and offers CUA in the Computer Control provider picker. OpenClaw creates a private, owner-only socket directory under Application Support and starts cua-driver serve --embedded as a direct app child. It does not launch through the Gateway, the TypeScript worker, open(1), or NSWorkspace; those paths would break macOS's TCC responsibility chain and create a second permission identity.

The app waits until the private socket accepts connections before advertising CUA readiness. Its TypeScript node worker starts only the unprivileged MCP proxy against that socket and maps the same typed computer.act v2 actions used on other platforms. Permission changes restart the daemon, and provider changes, disabling Computer Control, app shutdown, or an unexpected child exit remove the advertised CUA commands until a fresh generation is ready.

Trust model

The Gateway is the authorization chokepoint; the driver is a dumb effector. OpenClaw deliberately leaves the daemon unceilinged and authorizes computer use above it through tool exposure, the dangerous-command allowlist, device and command pairing approval, node-local provider enablement, and OS permissions. This is the same authorization boundary used by the shipped Peekaboo fulfiller.

CUA Driver 0.19.3 fixes its permission mode and bounded manifest when the runtime starts. Exact PID/window grants, and any application-wide window grants, must be declared in that launch-approved manifest; an ask entry is a hard denial for unattended dispatch. OpenClaw instead drives applications, windows, and elements discovered while the agent is running. Bounded mode therefore cannot express this provider model without duplicating Gateway policy or preauthorizing broad application classes, so the app starts its managed daemon in unrestricted mode with approvals bypassed.

The computer.act node-invoke policy classifies exact arguments before transport dispatch. Forced app termination, browser navigation, browser downloads, browser file inputs, recording start, trajectory replay, and desktop-scope escalation are separate high-risk families; ordinary observation and input remain distinct. Classification does not add a per-action prompt or weaken the command-level gates: every action still requires the same exposed tool, armed command, approved pairing, enabled node provider, and OS permissions.

The managed endpoint is not part of the model contract. The CUA plugin registers no model tool, CLI command, service, or raw node-MCP descriptor, and its action schema accepts neither helper binaries, sockets, native sessions, driver arguments, nor provider tool names. On macOS only the app-owned worker receives the endpoint, while node shell execution is routed through the app host without that worker-only value. These boundaries prevent an OpenClaw model action from selecting an alternate route to the managed daemon.

CUA creates the Unix socket with mode 0600, and OpenClaw places it in a random owner-only 0700 directory. This excludes remote clients and other local users. It does not authenticate or sandbox processes running as the same logged-in user: those processes are inside this boundary and may be able to discover and use same-user resources. Unrestricted CUA mode does not contain a compromised user account. Stronger same-user isolation would require inherited connected IPC or an OS-enforced process boundary.

Loopback is also reachability, not identity: any process on the machine can connect to 127.0.0.1. A Gateway client therefore does not receive operator.write merely because it arrived over loopback. It must authenticate and pass the Gateway's device pairing and scope approval; without a separately trusted local or shared credential, another already-authorized device must approve the requested operator scope. The driver and its socket never make that decision.

The CUA descriptor advertises window, element, and browser targets; background and foreground delivery; image, accessibility, and browser observations; and recording. Peekaboo remains the default in this release and does not advertise recording.

Browser profiles

browser_prepare can launch a separate driver-owned Chromium process with a new ephemeral profile or a named isolated profile. It never modifies, copies, terminates, or attaches to the selected browser's existing profile. Existing-profile/CDP attachment remains unavailable because it requires the driver's protected embedding-host consent and revocation path; Gateway approval and computer.act arming do not substitute for that local consent.

Browser targets, pages, page elements, and dialogs are opaque capabilities. Retake browser state after navigation, reconnect, or a stale-reference refusal. The adapter never returns provider-native CDP target IDs, tab IDs, or page refs to the model.

Maintainer live-proof rig

The repository includes a development rig that preserves the real vertical path: agent-facing computer tool, Gateway node.invoke, paired node, and the selected node-local provider. It is deliberately isolated from the operator app and Gateway. The macOS path uses the signed app node; the Linux path uses the opt-in cua-computer plugin in a real X11 session.

macOS

Build a signed app from a clean, committed checkout, choose a fresh profile and non-default loopback port, and prepare the two config views:

scratch="$(mktemp -d /tmp/openclaw-cu-live.XXXXXX)"
scripts/dev/computer-use-macos-live-rig.sh prepare \
  cu-live-proof 29431 "$PWD/dist/OpenClaw.app" "$scratch" peekaboo

Run the emitted gateway and app commands in separate terminals. The split config is intentional: the externally launched daemon reads a scratch config with gateway.mode: "local", while the app profile reads gateway.mode: "remote", direct transport, and the daemon's loopback URL. If the app reads local mode, its Port Guardian owns the route instead of joining the external daemon. The rig keeps its validated launch fields in non-executable rig.json; later commands reject unknown fields or paths that do not match the scratch/profile layout. It also seeds a dedicated node identity, completed onboarding, unpaused state, Computer Control, and the checkout path used to start the debug node worker. There is no separate node-mode toggle.

In a third terminal, rerun the emitted nodes command until the paired entry is connected and advertises computer.act plus a computerUse descriptor. No operator-device approval step is involved: the loopback gateway silently pairs the rig's CLI identity on its first connect, and the proof runner is admitted as a local backend client without pairing at all. The rig keeps those two identities in separate state directories (cli-state and agent-state) because a paired operator device is pinned to the scopes of its first connect, and a CLI pairing would otherwise cap the proof client below operator.write.

If the node's command surface has not yet been approved, pull .pending[0].requestId from the nodes output and execute scripts/dev/computer-use-macos-live-rig.sh approve "$scratch" <request-id>.

Position a harmless editable fixture window behind a different frontmost app, then run the vertical:

scripts/dev/computer-use-macos-live-rig.sh proof \
  "$scratch" peekaboo "Computer Use Fixture" "background proof" "Editor"

The proof runner first requires the sole connected computer node to advertise the requested provider, then executes screenshot, list_windows, get_window_state, background element click and type, and re-observes the window. It saves the structured result and target-window before/after images under the scratch directory and fails unless the provider matches, the target started non-frontmost, the frontmost app and cursor stayed unchanged, target content changed, and the final effect was confirmed or a structured refusal. Restart the isolated app with the other provider and rerun the same proof. Do not use port 18789, the default profile, or /Applications/OpenClaw.app for this rig.

Linux X11 through Crabbox

Run Linux proof on a Crabbox Linux host, not on a macOS container. A direct AWS Crabbox lease with Xvfb is sufficient because Xvfb is a real X11 server; a local container on macOS is not remote Linux desktop proof. Install the X11 fixture prerequisites on the disposable host, then start an isolated session:

sudo apt-get update
sudo apt-get install -y at-spi2-core dbus-x11 gir1.2-gtk-3.0 jq openbox python3-gi x11-utils xdotool xvfb

dbus-run-session -- bash
export DISPLAY=:99 XDG_SESSION_TYPE=x11 NO_AT_BRIDGE=0
Xvfb "$DISPLAY" -screen 0 1280x800x24 -nolisten tcp &
openbox >/tmp/openclaw-cu-openbox.log 2>&1 &

scratch="$(mktemp -d /tmp/openclaw-cu-live.XXXXXX)"
scripts/dev/computer-use-macos-live-rig.sh prepare-linux \
  cu-linux-live-proof 29431 "$scratch"

Run the emitted gateway, node, and fixture commands in separate panes that inherit the same DISPLAY and DBUS_SESSION_BUS_ADDRESS. The isolated configs share one scratch-only random Gateway token, and the gateway silently approves loopback node-device pairing. The node command surface remains an explicit approval: run the emitted nodes command after the node finishes reconnecting, read .pending[0].requestId, and pass it to scripts/dev/computer-use-macos-live-rig.sh approve "$scratch" <request-id>. Rerun nodes until exactly one connected node advertises provider.id: "cua-computer".

Execute the same proof runner against the non-frontmost GTK fixture:

scripts/dev/computer-use-macos-live-rig.sh proof \
  "$scratch" cua "OpenClaw CUA X11 Target" "W3-LINUX CONFIRMED"

The result and window-before.png / window-after.png stay under the scratch directory. A confirmed mutation must preserve the sentinel as the active X11 window and leave the pointer unchanged. An upstream background_unavailable or background_occluded result is valid refusal evidence only when it remains structured and no foreground retry is attempted. The rig rejects native Wayland even when DISPLAY is also present for XWayland; switch to X11 instead of claiming Wayland coverage.

Windows and Linux (experimental, direct SDK)

The bundled cua-computer plugin provides an experimental fulfiller for Windows and Linux node hosts. It is disabled by default on those platforms (macOS enables it by default for the CUA fulfiller above) and uses the pinned CUA Driver SDK 0.19.3 contract directly:

1. Enable the plugin:

   openclaw plugins enable cua-computer
   

2. Verify the node-local SDK package before starting the node:

   openclaw doctor --lint --only cua-computer/driver-artifacts
   

   OpenClaw checks the SDK package version, the selected OS/CPU package version, regular-file identity, and the pinned SHA-256 digest of the native library and Node runtime. A clean check prints no findings. If it reports a COMPUTER_DRIVER_* error, reinstall or update OpenClaw on this node host and run the check again. Do not download a standalone cua-driver executable or add one to PATH; Windows and Linux use the npm-installed in-process SDK.

3. Start openclaw node run from the interactive desktop session. The plugin repeats the artifact verification at startup before it imports native code, creates its configured SDK runtime lazily, then creates separate fixed window- and desktop-scoped trusted sessions for each provider execution. escalate_scope reads the already-desktop session state, so the window identity remains immutable. Completion, cancellation, Gateway disconnect, provider switching, local Stop, and command-host shutdown all close that exact execution, finalize or discard its recording resources, close both sessions, and shut down its runtime.

4. Approve the pairing update that includes computer.act. Desktop computer.act is a built-in platform default, so plugin enablement plus that approval is the whole grant; no gateway.nodes.commands.allow entry is required. An operator who wants the command off can deny it:

   {
     gateway: {
       nodes: { commands: { deny: ["computer.act"] } },
     },
   }
   

   A denied command is withheld from the node's advertised surface together with its computer capability, and the Gateway logs which commands it withheld.

This fulfiller currently controls only the primary display. hold_key, left_mouse_down, and left_mouse_up are unavailable because the CUA Driver SDK has no desktop-scope held-input contract. Modifier-held clicks, scrolling, and dragging are rejected because the typed desktop methods do not accept modifiers. The key action accepts named keys, letters, and modifier combos (for example cmd+c or Return); digit and punctuation keys are rejected because the driver drops their layout-dependent shift state, so send that text through the type action instead. Cancellation is passed to the SDK for each node invocation.

The plugin calls CuaDriver.createConfigured, never bare create(). Its authorization ceiling, fixed window/desktop session identities, TTLs, and scopes are owned by OpenClaw; model-facing screen.snapshot and computer.act inputs cannot select a session or widen the window identity. Because the driver reports no stable display identity, frame authorization binds to the trusted session generation plus live primary-display geometry. A new session invalidates outstanding frames, but a same-geometry primary-display substitution inside one session cannot be detected; prefer a stable single-display session for this fulfiller.

On Windows and Linux this is a hard replacement of the former 0.10 daemon/MCP integration: OpenClaw does not spawn a CUA process or proxy an MCP client. macOS deliberately uses the app-owned embedded daemon described above so the driver remains in OpenClaw.app's TCC responsibility chain. Neither path falls back to another provider for an individual action.

The accepted driver record lives with the cua-computer package and supplies both the npm native-file digests and the macOS archive digest. Updating OpenClaw updates that record and the SDK packages together. There is no independent Windows/Linux driver updater or rollback directory because there is no separate driver installation on those hosts; roll back by installing the previous known-good OpenClaw package, then rerun the focused doctor check before restarting the node.

Troubleshooting

The cua-computer fulfiller surfaces typed error codes in the tool result and node logs. Common ones:

The computer.act node command

computer.act serves as the sole node command the tool uses for input routing (node.invoke with command: "computer.act"). Its characteristics:

• Locally enabled: the node advertises it only while Computer Control is active. The gateway can approve that advertised surface once at pairing.
• Capability-based: the tool demands a connected node to advertise both computer.act and screen.snapshot. The bundled macOS app and the opt-in experimental cua-computer plugin satisfy the same command pair.

Reads utilize screen.snapshot; no alternate capture path exists. See Camera and screen nodes for the shared capture command.

Authorization

1. Activate the platform fulfiller: on macOS, Settings → General → Capabilities → Allow Computer Control starts enabled, then select Peekaboo or CUA and grant Accessibility and Screen Recording under Settings → Permissions; on Windows/Linux, follow the experimental cua-computer setup above.

2. Approve the pairing update on the gateway (a new command forces re-pairing).

3. Expose the tool to the vision-capable agent. For the default coding profile:

   {
     tools: {
       alsoAllow: ["computer"],
       // Sandboxed agents need this second gate too:
       sandbox: { tools: { alsoAllow: ["computer"] } },
     },
   }
   

With node-local control enabled and the pairing update approved, computer.act remains persistently available as long as the node keeps advertising it. No lease, expiry, or arm/disarm command exists. Disabling Computer Control locally removes the advertised command, and the node rechecks the toggle at invocation time.

On macOS, default-on means a paired gateway can drive pointer and keyboard input as soon as the required macOS grants are in place. No per-action confirmation occurs. Turn off Allow Computer Control before pairing, or at any later point, to halt advertising and accepting computer.act.

gateway.nodes.commands.deny stays an explicit global revocation and always takes precedence; a denied computer.act is withheld from the node's advertised surface along with its computer capability, and the Gateway logs what it withheld. No fulfiller requires a gateway.nodes.commands.allow entry: computer.act is a built-in desktop platform default, so node-local enablement plus pairing approval constitutes the entire grant on every platform and for either fulfiller. An authenticated operator with operator.write can invoke an enabled, paired command through node.invoke; no per-action admin check exists.

Safety

• All layers (tool policy, gateway command policy, pairing, node-app setting, and platform permissions) must align. On macOS that includes Allow Computer Control, Accessibility, and Screen Recording; the native Peekaboo path also demands Event Posting. Actions execute while those durable controls stay enabled; no per-action confirmation occurs.
• The macOS fulfiller posts text one grapheme at a time, so cancellation, disconnect, pause, disable, or endpoint replacement halts it before the next grapheme. The experimental CUA Driver fulfiller passes node cancellation to the SDK for each call.
• CUA recording, replay, browser upload, and browser download paths are node-owned. The model receives only opaque execution-scoped resource handles; traversal, absolute paths, symlink escapes, and helper selection are blocked before driver dispatch.
• Screenshots are model-only and never auto-sent to chat (issue #44759).
• Treat screen content as untrusted; it can carry prompt injection.

macOS permission troubleshooting

The Computer Control status in Settings → General → Capabilities checks Accessibility, Event Posting, and Screen Recording separately. Screen capture can function while input remains denied because macOS stores those grants in separate TCC buckets.

If the status says Accessibility grant may be stale, OpenClaw may already appear enabled under System Settings → Privacy & Security → Accessibility even though macOS rejects it. This occurs when the Accessibility entry is pinned to an older app build. Select OpenClaw in that list, remove it with −, then re-add /Applications/OpenClaw.app. Quit and reopen OpenClaw after changing the grant because macOS can cache Accessibility trust for the lifetime of the process.

Relationship to other desktop-control paths

This is the agent-driven path. See Peekaboo bridge for how it relates to the PeekabooBridge host, Codex Computer Use, and the direct cua-driver MCP.