Debugging Tools: Watch Mode, Raw Model Streams, and Tracing Reasoning Leakage
This page covers runtime debug overrides, session trace output, and plugin lifecycle tracing for debugging streaming output and gateway iteration. It is intended for developers troubleshooting Neura Market's streaming and plugin systems.
Read this when
- You need to inspect raw model output for reasoning leakage
- You want to run the Gateway in watch mode while iterating
- You need a repeatable debugging workflow
Helpers for debugging streaming output, gateway iteration, and startup profiling.
Runtime debug overrides
/debug applies runtime-only configuration overrides (held in memory, not written to disk). This feature is off by default; activate it with commands.debug: true.
/debug show
/debug set channels.whatsapp.responsePrefix="[openclaw]"
/debug unset channels.whatsapp.responsePrefix
/debug reset
/debug reset removes every override and restores the on-disk configuration.
Session trace output
/trace outputs plugin-specific trace and debug lines for a single session without turning on full verbose mode. Use it for plugin diagnostics like Active Memory debug summaries; rely on /verbose for standard status or tool output.
/trace
/trace on
/trace off
Plugin lifecycle trace
Set OPENCLAW_PLUGIN_LIFECYCLE_TRACE=1 to get a phase-by-phase breakdown of plugin metadata, discovery, registry, runtime mirror, config mutation, and refresh operations. Output goes to stderr so JSON command results stay parseable.
Plugin load failures include their stack trace while this trace is active.
OPENCLAW_PLUGIN_LIFECYCLE_TRACE=1 openclaw plugins install tokenjuice --force
[plugins:lifecycle] phase="config read" ms=6.83 status=ok command="install"
[plugins:lifecycle] phase="slot selection" ms=94.31 status=ok command="install" pluginId="tokenjuice"
[plugins:lifecycle] phase="registry refresh" ms=51.56 status=ok command="install" reason="source-changed"
Reach for this before using a CPU profiler. From a source checkout, measure the built runtime with node dist/entry.js ... after pnpm build; pnpm openclaw ... also captures source-runner overhead.
For synchronous module-load timings, use the shared diagnostics surface instead of a separate plugin-only environment switch:
OPENCLAW_DIAGNOSTICS=plugin.load-profile openclaw plugins list
CLI startup and command profiling
Checked-in startup benchmarks:
pnpm test:startup:bench:smoke
pnpm tsx scripts/bench-cli-startup.ts --preset real --case status --runs 3
pnpm tsx scripts/bench-cli-startup.ts --preset real --cpu-prof-dir .artifacts/cli-cpu
For a one-off profile through the normal source runner, set OPENCLAW_RUN_NODE_CPU_PROF_DIR:
OPENCLAW_RUN_NODE_CPU_PROF_DIR=.artifacts/cli-cpu pnpm openclaw status
The source runner adds Node CPU profile flags and writes a .cpuprofile for the command. Use this before adding temporary instrumentation to command code.
For startup stalls that appear to be synchronous filesystem or module-loader work, add Node's sync I/O trace flag through the source runner:
OPENCLAW_TRACE_SYNC_IO=1 pnpm openclaw gateway --force
pnpm gateway:watch keeps this flag off by default for the watched Gateway child; set OPENCLAW_TRACE_SYNC_IO=1 when you want sync I/O trace output in watch mode as well.
Gateway watch mode
pnpm gateway:watch
By default this starts or restarts a tmux session named openclaw-gateway-watch-<profile> (for example openclaw-gateway-watch-main), with a port suffix like openclaw-gateway-watch-dev-19001 added only when OPENCLAW_GATEWAY_PORT differs from the default port 18789. It auto-attaches from interactive terminals; non-interactive shells, CI, and agent exec calls stay detached and print attach instructions instead:
tmux attach -t openclaw-gateway-watch-main
# Read recent output without attaching
tmux capture-pane -ep -t openclaw-gateway-watch-main -S -200
The pane uses tmux remain-on-exit, so startup failures remain available for attach or capture rather than deleting the session. Re-running pnpm gateway:watch respawns that pane.
The tmux pane runs the raw watcher:
node scripts/watch-node.mjs gateway --force
Before watching the configured or default port, the tmux wrapper stops the active profile's installed Gateway service. This hands the port to the source watcher without launchd, systemd, or Scheduled Task respawning and replacing it. The service stays installed; restore it after the watch session with:
pnpm openclaw gateway start
When an explicit --port or OPENCLAW_GATEWAY_PORT differs from the installed service's effective port, the wrapper leaves the service running so both Gateways can run side by side.
Foreground mode without tmux:
pnpm gateway:watch:raw
# or
OPENCLAW_GATEWAY_WATCH_TMUX=0 pnpm gateway:watch
Raw mode does not manage the installed service. Run pnpm openclaw gateway stop first when it uses the same port.
Keep tmux management but disable auto-attach:
OPENCLAW_GATEWAY_WATCH_ATTACH=0 pnpm gateway:watch
Profile watched Gateway CPU time when debugging startup or runtime hotspots:
pnpm gateway:watch --benchmark
The watch wrapper consumes --benchmark before invoking the Gateway and writes one V8 .cpuprofile per Gateway child exit under .artifacts/gateway-watch-profiles/. Stop or restart the watched gateway to flush the current profile, then open it with Chrome DevTools or Speedscope:
npx speedscope .artifacts/gateway-watch-profiles/*.cpuprofile
--benchmark-dir <path>: write profiles to a different location.--benchmark-no-force: skip the default--forceport cleanup and fail fast if the Gateway port is already in use.
Benchmark mode suppresses sync-I/O trace spam by default. Set OPENCLAW_TRACE_SYNC_IO=1 with --benchmark to get both CPU profiles and sync-I/O stack traces; in benchmark mode those trace blocks go to gateway-watch-output.log under the benchmark directory (filtered from the terminal pane), while normal Gateway logs stay visible.
The tmux wrapper carries common non-secret runtime selectors into the pane, including OPENCLAW_PROFILE, OPENCLAW_CONFIG_PATH, OPENCLAW_STATE_DIR, OPENCLAW_GATEWAY_PORT, and OPENCLAW_SKIP_CHANNELS. Put provider credentials in your normal profile or config, or use raw foreground mode for one-off ephemeral secrets.
If the watched Gateway exits during startup, the watcher runs openclaw doctor --fix --non-interactive once and restarts the Gateway child. Set OPENCLAW_GATEWAY_WATCH_AUTO_DOCTOR=0 to see the original startup failure without the dev-only repair pass.
By default, the managed tmux pane displays colored Gateway logs. To disable ANSI output, set FORCE_COLOR=0 when launching pnpm gateway:watch.
The watcher restarts whenever build-relevant files change under src/, extension source files, extension package.json and openclaw.plugin.json metadata, tsconfig.json, package.json, and tsdown.config.ts. Changes to extension metadata restart the gateway without triggering a rebuild; source and config changes still require rebuilding dist first.
Pass gateway CLI flags after gateway:watch, and they are forwarded on each restart. Repeating the same watch command respawns the named tmux pane; the raw watcher uses a single-watcher lock, so duplicate watcher parents are replaced rather than accumulated.
Dev profile + dev gateway (--dev)
Two separate --dev flags:
- Global
--dev(profile): isolates state under~/.openclaw-devand sets the default gateway port to19001(derived ports adjust accordingly). gateway --dev: instructs the Gateway to automatically create a default configuration and workspace when they are absent (and skip bootstrap).
Recommended workflow (dev profile combined with dev bootstrap):
pnpm gateway:dev
OPENCLAW_PROFILE=dev openclaw tui
If there is no global install, run the CLI through pnpm openclaw ....
What this accomplishes:
-
Profile isolation (global
--dev)OPENCLAW_PROFILE=devOPENCLAW_STATE_DIR=~/.openclaw-devOPENCLAW_CONFIG_PATH=~/.openclaw-dev/openclaw.jsonOPENCLAW_GATEWAY_PORT=19001(browser and canvas ports shift correspondingly)
-
Dev bootstrap (
gateway --dev)- Creates a minimal configuration if one does not exist (
gateway.mode=local, bind to loopback). - Assigns
agents.defaults.workspaceto the development workspace andagents.defaults.skipBootstrap=true. - Populates the workspace files if they are missing:
AGENTS.md,SOUL.md,TOOLS.md,IDENTITY.md,USER.md. - Default identity: C3-PO (protocol droid).
pnpm gateway:devalso configuresOPENCLAW_SKIP_CHANNELS=1to skip channel providers.
- Creates a minimal configuration if one does not exist (
Development Gateways ignore ambient channel environment triggers by default, so credentials inherited from your shell will not connect the development instance to live channel services. Explicit channels.<id> configuration still functions. Pass --dev-ambient-channels together with --dev to re-enable ambient channel auto-configuration for that specific run.
Reset workflow (starting fresh):
pnpm gateway:dev:reset
Note
--devis a global profile flag and is consumed by certain runners. To spell it out explicitly, use the environment variable form:OPENCLAW_PROFILE=dev openclaw gateway --dev --reset
--reset removes configuration, credentials, sessions, and the development workspace (moved to trash, not permanently deleted), then recreates the default development setup.
Tip
If a non-development gateway is already active (via launchd or systemd), stop it first:
openclaw gateway stop
Raw stream logging
OpenClaw can log the raw assistant stream before any filtering or formatting occurs. This is the most effective way to determine whether reasoning arrives as plain text deltas or as separate thinking blocks.
Enable it from the command line:
pnpm gateway:watch --raw-stream
Optional path override:
pnpm gateway:watch --raw-stream --raw-stream-path ~/.openclaw/logs/raw-stream.jsonl
Equivalent environment variables:
OPENCLAW_RAW_STREAM=1
OPENCLAW_RAW_STREAM_PATH=~/.openclaw/logs/raw-stream.jsonl
Default file location: ~/.openclaw/logs/raw-stream.jsonl
Safety notes
- Raw stream logs may include full prompts, tool output, and user data.
- Keep logs local and delete them after debugging is complete.
- If you share logs, remove secrets and personally identifiable information first.
Debugging in VSCode
Source maps are necessary because the build process hashes generated filenames. The included launch.json targets the Gateway service:
- Rebuild and Debug Gateway - removes
/distand rebuilds with debugging enabled, then starts the Gateway. - Debug Gateway - debugs an existing build without modifying
/dist.
Setup
- Open Run and Debug (from the Activity Bar, or press
Ctrl+Shift+D). - Choose Rebuild and Debug Gateway and click Start Debugging.
To handle the build and debug cycle manually instead:
- Enable source maps in a terminal:
- Linux/macOS:
export OUTPUT_SOURCE_MAPS=1 - Windows (PowerShell):
$env:OUTPUT_SOURCE_MAPS="1" - Windows (CMD):
set OUTPUT_SOURCE_MAPS=1
- Linux/macOS:
- Rebuild:
pnpm clean:dist && pnpm build - Select Debug Gateway and click Start Debugging.
Place breakpoints inside src/ TypeScript files; source maps let the debugger map them onto the compiled JavaScript.
Notes
- Rebuild and Debug Gateway removes
/distand performs a completepnpm buildwith source maps each time it launches. - Debug Gateway can be started and stopped without impacting
/dist, though you handle the build process in a separate terminal. - Modify
launch.jsonargsto debug other CLI subcommands. - To use the built CLI for other purposes (such as
dashboard --no-openwhen your debug session generates a new auth token), launch it from a different terminal:node ./openclaw.mjsor an alias likealias openclaw-build="node $(pwd)/openclaw.mjs".